System

Abstract

To provide a system that enables a user to understand the contents of a video only by listening to voice.SOLUTION: The system includes: means for receiving video data; means for dividing the received video data into frames; means for extracting a feature image from each of the divided frame; means for generating a prompt sentence for instructing generation of an explanation text about the feature image on the basis of the extracted feature image; means for generating an explanatory text about the feature image by using the generated prompt sentence and a generative AI model; means for transmitting the generated explanatory text to voice synthesis software; means for receiving voice data from the voice synthesis software; and means for outputting the received voice data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to understand the contents of video content, it is necessary to see the video visually. However, for people with visual impairments or those who cannot watch the video due to time constraints, it is difficult to understand the contents of video content. [Means for solving the problem]

[0005] The present invention analyzes video and extracts its characteristic images. It then generates image commentary text based on the characteristic images and converts the text into audio. This allows even people who cannot see the video to understand the content of the video content by simply listening to the audio. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16]FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The system of the present invention includes a means for analyzing a video, a means for extracting a characteristic image, a means for generating an image commentary text, and a means for converting the video into audio. The video analysis means uses artificial intelligence to extract a characteristic image from the video. The extracted characteristic image is converted into commentary text by the image commentary text generation means. The commentary text is converted into audio by the audio output means and output to the user.

[0029] "Example 2"

[0030] As a specific example, consider the analysis of a television program video. The video analysis means uses artificial intelligence to extract images of characteristic scenes and characters from the television program video. The extracted characteristic images are converted by the image commentary text generation means into commentary text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears." The commentary text is converted into audio by the audio output means, so that even people with visual impairments or those who cannot watch the video due to time constraints can understand the content of the television program just by listening to the audio.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The video analysis means analyzes the video. Artificial intelligence is used in this analysis, and characteristic images are extracted from the video.

[0034] Step 2: The extracted feature images are converted into explanatory text by an image explanatory text generating means.

[0035] Step 3: The generated commentary text is converted into speech by the speech output means.

[0036] Step 4: Finally, the converted audio is output to the user.

[0037] "Example 2"

[0038] Step 1: The television program footage is analyzed using a video analysis tool. Artificial intelligence is used in this analysis, and images of characteristic scenes and characters are extracted from the footage.

[0039] Step 2: The extracted feature images are converted into explanatory text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears," by an image explanatory text generating means.

[0040] Step 3: The generated commentary text is converted into speech by the speech output means.

[0041] Step 4: Finally, the converted audio is output to people with visual impairments or who are unable to watch the video due to time constraints, enabling them to understand the content of the TV program just by listening to the audio.

[0042] Example 1

[0043] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0044] Conventional video analysis systems have problems in that they do not efficiently extract characteristic images from a video, generate explanatory text based on the extracted images, and then convert the text into audio. In addition, they lack a means for users to quickly and accurately understand the content of the video.

[0045] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0046] In this invention, the server includes a means for analyzing the video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for transmitting the audio data to the user's terminal, thereby enabling the content of the video to be provided to the user quickly and accurately.

[0047] "Means for analyzing video" refers to technology for analyzing input video data and detecting characteristic elements within the video.

[0048] "Means for extracting characteristic images" refers to a technology that identifies important objects and scenes from the analyzed video data and extracts them as images.

[0049] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[0050] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data.

[0051] The "means for transmitting voice data to a user's terminal" refers to a technique for transmitting the generated voice data to a user's terminal so that the user can play back the voice.

[0052] A "machine learning model" is an algorithm or framework that learns from data and performs pattern recognition and predictions.

[0053] A "natural language processing model" is an algorithm or framework for analyzing, understanding, and generating text data.

[0054] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] The present invention is a system for analyzing a video, extracting characteristic images, generating image commentary text, converting the text into audio, and providing the text to a user. A specific embodiment of this system will be described below.

[0056] Video Analysis Methods

[0057] The server uses a machine learning model as a means of video analysis. Specifically, it uses deep learning frameworks such as TENSORFLOW (registered trademark) and PyTorch to analyze the input video data. The video data is uploaded to the server by the user via the terminal. The server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame. This identifies important objects in each frame.

[0058] Feature image extraction method

[0059] The server extracts characteristic images from the data obtained by the video analysis means. The extracted images are temporarily stored for use in subsequent processing. Specifically, frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) are selected from the output of the object detection model, and these are stored as characteristic images.

[0060] Image description text generation method

[0061] The server generates explanatory text using a generative AI model (e.g., GPT-3 (registered trademark)) based on the extracted feature images. The generated text is saved to be provided to the user. Specifically, a prompt sentence is sent to the generative AI model using the feature images as input. For example, using the prompt sentence "What is in this image?", the explanatory text "There is a red car in this image" is obtained from the model.

[0062] Audio conversion means

[0063] The server converts the generated commentary text into speech. Specifically, it uses a speech synthesis service such as Google (registered trademark) Text-to-Speech (TTS) or Amazon Polly. The server sends the commentary text to the speech synthesis service and obtains the speech data. For example, the server sends the text "This image shows a red car" and receives the corresponding speech data.

[0064] Audio data output means

[0065] The server sends the generated audio data to the user's device. The user can play the audio through the device and listen to the commentary. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button on the smartphone app to listen to the audio commentary.

[0066] Examples of concrete examples and prompts

[0067] Examples

[0068] A user uploads a video taken in a park to the system. The server analyzes the video and extracts images of playground equipment and people in the park. It then generates explanatory text for each extracted image, such as "Children are playing in the park," converts this into audio, and provides it to the user.

[0069] Examples of prompt statements

[0070] "Analyze the video footage taken in the park, extract characteristic images, and generate explanatory text. Then convert the generated text into audio and output it."

[0071] In this way, the user can easily understand the contents of the video through audio.

[0072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0073] Step 1:

[0074] A user uploads video data captured using a device to a server. Specifically, the user opens the application on the device and taps the "Video Upload" button. A file selection screen is displayed, and when the user selects a video file and presses the "Send" button, the video data is sent to the server via an HTTP request. The input is the user's video data, and the output is the video data saved on the server.

[0075] Step 2:

[0076] The server receives the uploaded video data and starts analyzing it using video analysis methods. Specifically, the server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame, which identifies important objects in each frame. The input is the video data stored on the server, and the output is the analyzed object data for each frame.

[0077] Step 3:

[0078] The server extracts feature images from the data obtained by the video analysis means. Specifically, it selects frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) from the output of the object detection model, and saves them as feature images. The input is the analyzed object data for each frame, and the output is the extracted feature images.

[0079] Step 4:

[0080] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. Specifically, it sends a prompt sentence to the generative AI model using the feature images as input. For example, it uses the prompt sentence "What is in this image?" to obtain explanatory text from the model, such as "There is a red car in this image." The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[0081] Step 5:

[0082] The server converts the generated commentary text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the commentary text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data. The input is the generated commentary text, and the output is the generated speech data.

[0083] Step 6:

[0084] The server sends the generated audio data to the user's device. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button in the application on the device to listen to the audio commentary. The input is the generated audio data, and the output is the audio played on the user's device.

[0085] (Application example 1)

[0086] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Autonomous vehicles are required to grasp the surrounding situation in real time and provide appropriate information to the driver and passengers. However, the current system does not have sufficient accuracy in image analysis and information provision, and improving safety and convenience is an issue. In particular, there is a lack of means to quickly and accurately recognize important information such as road signs and pedestrians and to convey that information to the driver and passengers in an easy-to-understand manner.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0089] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for analyzing a video around the autonomous vehicle in real time and extracting a characteristic image, and a means for generating commentary text for the characteristic image, converting it into audio, and providing it. This makes it possible to grasp the situation around the autonomous vehicle in real time and provide information to the driver and passengers quickly and accurately.

[0090] "Means for analyzing video" refers to a device or software for analyzing video data acquired from a video input device such as a camera and extracting specific information.

[0091] A "means for extracting characteristic images" is a device or software for selecting images having specific characteristics from the analyzed video data.

[0092] The "means for generating image explanatory text" refers to a device or software for generating explanatory text regarding the image based on the extracted characteristic image.

[0093] "Means for converting image description text into speech" refers to a device or software for converting the generated image description text into speech data and providing it to the user.

[0094] "Means for analyzing images around an autonomous vehicle in real time" refers to a device or software that analyzes images obtained from cameras or other devices mounted on an autonomous vehicle in real time and grasps the surrounding situation.

[0095] A "means for extracting characteristic images" is a device or software for selecting images having specific important information from video data analyzed in real time.

[0096] "Means for generating explanatory text and converting it into audio for providing" refers to a device or software for generating explanatory text based on the extracted characteristic images, converting the text into audio for providing to the user.

[0097] A system for implementing this invention is configured as follows: First, a server has a means for analyzing video, and analyzes video data acquired from a video input device such as a camera. Next, a means for extracting characteristic images from the analyzed video data is used to select images with specific characteristics. Based on the extracted characteristic images, a means for generating image commentary text generates commentary. Finally, a means for converting the generated image commentary text into audio is used to convert it into audio data, which is provided to the user.

[0098] Specifically, the server uses the following hardware and software: A camera is used for video capture, and an artificial intelligence model such as TensorFlow is used for video analysis. A generative AI model is used to extract feature images and generate explanatory text. Google Text-to-Speech API (gTTS) is used for voice conversion.

[0099] For example, in an autonomous vehicle, a camera captures images of the surroundings in real time. The server analyzes the images and extracts characteristic images such as road signs and pedestrians. For the extracted images, a generative AI model is used to generate explanatory text. The generated explanatory text is converted to speech using gTTS and provided to the driver and passengers through the speaker.

[0100] As a concrete example, the following prompt sentence is input to the generative AI model:

[0101] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[0102] By inputting this prompt into the generative AI model, the following explanatory text is generated:

[0103] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[0104] This explanatory text can be converted into audio and provided to drivers and passengers, improving the safety and convenience of self-driving vehicles.

[0105] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0106] Step 1:

[0107] The server captures video data from the camera in real time. The input is the video data from the camera and the output is the captured video frames. This video data is used for subsequent analysis processing.

[0108] Step 2:

[0109] The server analyzes the captured video frames using an artificial intelligence model such as TensorFlow. The input is the video frame, and the output is the analysis result, which is feature data. This analysis extracts important features in the video (e.g., road signs, pedestrians, etc.).

[0110] Step 3:

[0111] The server extracts a feature image from the analysis result. The input is the feature data as the analysis result, and the output is the feature image. This feature image is used for subsequent generation of explanatory text.

[0112] Step 4:

[0113] The server generates image description text using a generative AI model based on the feature image. The input is the feature image, and the output is the description text. This description text is an explanatory text about the feature image.

[0114] Step 5:

[0115] The server converts the generated commentary text into voice data using Google Text-to-Speech API (gTTS). The input is the commentary text, and the output is the voice data. This voice data is provided to the user.

[0116] Step 6:

[0117] The server provides the converted voice data to the driver or passengers through the speaker. The input is voice data and the output is voice output from the speaker. This allows the driver or passengers to obtain information about the surrounding situation in real time.

[0118] Example 2

[0119] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] Conventional video analysis systems have made it difficult for people with visual impairments or those who cannot watch the video due to time constraints to understand the content of television programs. In addition, there was a lack of efficient means to extract characteristic scenes and characters from the video, generate text to appropriately explain them, and then convert that text into speech.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0122] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into a voice, a means for dividing the video into frames, a means for transmitting the characteristic image to a generative AI model, a means for inputting a prompt sentence to the generative AI model, a means for receiving a commentary text returned from the generative AI model, a means for transmitting the commentary text to a voice synthesis software, a means for receiving voice data returned from the voice synthesis software, and a means for providing the voice data to a user. This allows even a person with a visual impairment or a person who cannot watch a video due to time constraints to understand the contents of a TV program just by listening to the sound.

[0123] "Means for analyzing video" refers to technology or devices for receiving video data and analyzing its contents.

[0124] A "means for extracting characteristic images" is a technology or device that selects important images, such as specific scenes or characters, from the analyzed video data.

[0125] The "means for generating image explanatory text" refers to a technique or device that generates text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0126] "Means for converting image description text into speech" refers to technology or devices that convert the generated text into speech data.

[0127] A "means for dividing an image into frames" is a technique or device for dividing continuous image data into individual still images (frames).

[0128] "Means for sending to a generative AI model" refers to technology or equipment for sending the extracted feature images to a generative AI model.

[0129] "Means for inputting a prompt sentence into a generative AI model" refers to technology or equipment for inputting an explanatory sentence (prompt sentence) regarding the content of an image into a generative AI model.

[0130] "Means for receiving explanatory text returned from a generative AI model" refers to technology or devices that receive explanatory text returned from a generative AI model.

[0131] The "means for transmitting to speech synthesis software" refers to a technique or device for transmitting the generated commentary text to speech synthesis software.

[0132] The "means for receiving voice data returned from the voice synthesis software" refers to a technique or device for receiving voice data returned from the voice synthesis software.

[0133] The "means for providing voice data to a user" refers to a technique or device for providing received voice data to a user.

[0134] The present invention relates to a system that analyzes video, extracts characteristic scenes and characters, generates text explaining them, and converts the text into speech. A specific embodiment of this system will be described below.

[0135] The server receives the video of a television program. The received video is first divided into frames. This division is performed using common video processing techniques. Next, the server uses a deep learning framework (e.g., TensorFlow, PyTorch) to analyze each frame. This analysis involves computer vision techniques such as object detection and face recognition. For example, the server detects specific characters or changes in the scene within a frame and extracts these images.

[0136] The extracted feature image is sent to a generative AI model (e.g., GPT-4 (registered trademark)). The server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is, "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." The generative AI model generates explanatory text based on this prompt sentence and returns it to the server.

[0137] The generated commentary text is sent to speech synthesis software (e.g. Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into audio data and returns it to the server. Finally, the server provides this audio data to the user. The user can understand the content of the TV program by listening to the audio.

[0138] As a concrete example, let's say there is a scene in a TV program where the protagonist confronts an enemy character. The server analyzes this scene and extracts characteristic images. Next, the server has a generative AI model (e.g. GPT-4) generate explanatory text such as "A scene where the protagonist confronts an enemy character." Finally, the server converts this explanatory text into audio using speech synthesis software (e.g. Google Text-to-Speech API) and provides it to the user. By listening to this audio, the user can understand the content of the scene without watching the video.

[0139] This system enables people with visual impairments or who are unable to watch the video due to time constraints to understand the content of television programs simply by listening to the audio.

[0140] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0141] Step 1:

[0142] The server receives the video of a television program. The received video is first divided into frames. Specifically, a process is performed to convert the video data into a series of still images (frames). This process divides the video data into individual frames. The input is the video data of a television program, and the output is the divided frame images.

[0143] Step 2:

[0144] The server uses a deep learning framework (e.g. TensorFlow, PyTorch) to analyze each frame. Computer vision techniques such as object detection and face recognition are used for the analysis. Specifically, it detects specific characters and scene changes in each frame. Through this process, images of characteristic scenes and characters are extracted. The input is the segmented frame images, and the output is the extracted feature images.

[0145] Step 3:

[0146] The server sends the extracted feature images to a generative AI model (e.g., GPT-4). Next, the server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is, "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." This process returns explanatory text from the generative AI model. The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[0147] Step 4:

[0148] The server sends the generated commentary text to speech synthesis software (e.g., Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into voice data and returns it to the server. Specifically, it performs a process to convert text data into voice data. This process generates voice data. The input is the commentary text, and the output is voice data.

[0149] Step 5:

[0150] The server provides the generated voice data to the user. Specifically, it transmits the voice data to the user's terminal so that the user can play it. This process allows the user to understand the contents of the TV program by listening to the voice. The input is the voice data, and the output is the voice provided to the user.

[0151] (Application example 2)

[0152] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] People with visual impairments and those who do not have time to watch videos have difficulty understanding the content of television programs and movies. In addition, there is a lack of technology that can analyze videos, extract characteristic scenes and characters, and provide them as audio, so there are limited ways to provide information to these users. Furthermore, there is a demand for a system that can perform video analysis and audio output in real time using devices such as smartphones and smart glasses.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0155] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image description text based on the characteristic image, a means for converting the image description text into audio, and a means for providing the audio to a user. This allows visually impaired people or people who do not have time to watch videos to understand the contents of a television program or movie through audio. In addition, real-time video analysis and audio output can be realized by using an application installed on a smartphone, smart glasses, a head-mounted display, or a robot. Furthermore, by including a means for generating a prompt sentence using a generative AI model, more advanced video analysis and information provision are possible.

[0156] A "means for analyzing video" refers to technology or equipment that receives video data as input and analyzes its contents.

[0157] "Means for extracting characteristic images" refers to technology or devices for extracting important images, such as specific scenes or characters, from the analyzed video data.

[0158] The "means for generating image explanatory text" refers to a technique or device for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0159] "Means for converting image description text into audio" refers to technology or devices for converting the generated image description text into audio data.

[0160] "Means for providing audio to a user" refers to technology or devices for transmitting converted audio data to a user.

[0161] "Artificial intelligence" is a technology that uses techniques such as machine learning and deep learning to analyze data and make judgments and predictions.

[0162] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[0163] "Smart glasses" are a device shaped like glasses that has a built-in display, camera, sensor, etc. and can display information and collect data.

[0164] A "head-mounted display" is a display device worn on the head, and refers to technology and devices for displaying information within the user's field of vision.

[0165] A "robot" is a mechanical device that operates autonomously or remotely to perform specific tasks.

[0166] A "generative AI model" is a model that uses artificial intelligence to generate new information and content from data.

[0167] A "prompt" is text that is entered into a generative AI model as instructions or questions that serve as guidelines for the model to generate appropriate output.

[0168] A system for implementing the present invention includes a means for analyzing a video, a means for extracting a feature image, a means for generating an image description text, a means for converting the image description text into speech, and a means for providing the speech to a user. The system further includes a means for generating a prompt sentence using a generative AI model.

[0169] System Configuration

[0170] Hardware:

[0171] server

[0172] Smartphone

[0173] Smart Glasses

[0174] Head-mounted display

[0175] robot

[0176] software:

[0177] OpenCV: A library for video analysis

[0178] pytesseract: A library for extracting text from images

[0179] gTTS: A text-to-speech library

[0180] mpg321: a tool for playing audio files

[0181] Processing flow

[0182] The server first analyzes the video. For video analysis, OpenCV is used to process the video data frame by frame. Next, pytesseract is used to extract feature images from the analyzed frames. Based on the extracted feature images, image description text is generated. A generative AI model is used to generate this text.

[0183] The generated image description text is converted to audio using gTTS, and the converted audio data is provided to the user using mpg321. This allows people with visual impairments or those who do not have time to watch the video to understand the content of TV programs and movies through audio.

[0184] Examples

[0185] For example, a user launches the "Audio Vision" app on a smartphone and inputs a video of a TV program. The app analyzes the video and generates text such as "a scene where the main character is running" or "a scene where an enemy character appears." The generated text is converted into audio and provided to the user.

[0186] Examples of prompt statements

[0187] An example of a prompt for a generative AI model is:

[0188] Analyze TV program footage and extract characteristic scenes and characters. I would like to develop an application that converts the extracted information into text and audio and provides it to users. Specifically, detect the following scenes:

[0189] Scene where the main character is running

[0190] Scenes in which enemy characters appear

[0191] In this way, a specific embodiment for carrying out the invention has been shown, which enables even those with visual impairments or those who do not have time to watch a video to understand the content of the video through audio.

[0192] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0193] Step 1:

[0194] The server receives video data input from the user. The input video data is video files of TV shows or movies. The server divides the video data into frames and prepares them for analysis.

[0195] Step 2:

[0196] The server uses OpenCV to analyze the video data. Specifically, it processes each frame sequentially and detects characteristic scenes and characters in the video. The input is the video frame, and the output is the detection result of characteristic scenes and characters.

[0197] Step 3:

[0198] The server uses pytesseract to extract text information from the detected characteristic scenes and characters. The input is the images of the characteristic scenes and characters, and the output is the extracted text information.

[0199] Step 4:

[0200] The server uses a generative AI model to generate image commentary text based on the extracted text information. The input is the extracted text information, and the output is the image commentary text. In concrete terms, the server inputs a prompt sentence into the generative AI model to generate appropriate commentary text.

[0201] Step 5:

[0202] The server converts the generated image description text into audio data using gTTS. The input is the image description text, and the output is audio data. Specifically, the server converts the text into an audio file and saves it.

[0203] Step 6:

[0204] The server uses mpg321 to provide the generated audio data to the user. The input is the audio data, and the output is the audio provided to the user. The specific operation is to play the audio file and let the user listen to it.

[0205] Step 7:

[0206] The user receives the provided audio using a smartphone, smart glasses, a head-mounted display, or a robot. This allows the user to understand the content of the video by audio. The input is the audio provided by the server, and the output is the user's understanding.

[0207] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0208] "Example 1"

[0209] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[0210] "Example 2"

[0211] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[0212] The process flow of each embodiment will be described below.

[0213] "Example 1"

[0214] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[0215] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[0216] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[0217] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[0218] "Example 2"

[0219] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[0220] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[0221] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[0222] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[0223] Example 1

[0224] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0225] Conventional video analysis systems were able to extract characteristic images from video and convert them into text or audio, but they were unable to provide appropriate information according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[0226] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0227] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for recognizing a user's emotion and adjusting the generation of the image commentary text and the conversion of the audio based on the emotion, thereby making it possible to provide appropriate information according to the user's emotion.

[0228] A "means for analyzing video" is a device or program for receiving video data as input and analyzing the content thereof.

[0229] The "means for extracting characteristic images" refers to a device or program for selecting images having specific characteristics from the analyzed video data.

[0230] The "means for generating image explanatory text" refers to a device or program for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0231] The "means for converting image description text into audio" refers to a device or program for converting the generated image description text into audio data.

[0232] The "means for recognizing the user's emotions" refers to a device or program for analyzing and recognizing the emotions from the user's voice, facial expressions, etc.

[0233] "Means for adjusting the generation of image description text or the conversion of audio based on emotions" refers to a device or program for adjusting the content of the generated image description text or audio in accordance with the recognized user's emotions.

[0234] "Machine learning" is a technology in which a computer learns patterns and rules from data and then uses that knowledge to analyze new data.

[0235] A "generative AI model" is a model that uses artificial intelligence to generate data such as text and images.

[0236] This invention is a system that analyzes video, extracts characteristic images, generates explanatory text based on the images, and converts the text into audio. It also includes a function to recognize the user's emotions and adjust the content of the explanatory text and audio based on the emotions.

[0237] The server uses machine learning to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data. For example, it acquires video data from a surveillance camera in real time and detects people and objects in the video.

[0238] The server then uses image processing libraries such as OpenCV to extract characteristic images from the analyzed video data, for example extracting the face of a particular person in the video and storing their facial features in a database.

[0239] The server uses a generative AI model to generate image description text based on the extracted feature images. Specifically, it uses generative AI models such as GPT-3 and BERT to generate text that describes the contents of the image. For example, it generates a description text such as "This image shows a man wearing a blue shirt."

[0240] The generated image description text is converted to speech using a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. For example, the text "This image shows a man wearing a blue shirt" is converted to speech and output to the user.

[0241] In addition, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's voice and facial expressions to recognize emotions. For example, it detects the emotion of joy from the user's voice.

[0242] Based on the recognized emotion, the server adjusts the image description text and audio content. For example, if the user expresses joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily" and outputs it as audio.

[0243] Examples

[0244] Suppose a user is viewing security camera footage. The server analyzes the footage and detects that a man wearing a blue shirt is in the image. The server then generates explanatory text based on this information, such as "This image shows a man wearing a blue shirt." The server then converts this text into speech and outputs it to the user. If the user expresses an emotion of joy, the server generates text such as "This image shows a man wearing a blue shirt smiling happily," converts it into speech, and outputs it.

[0245] Examples of prompt statements

[0246] "Analyze surveillance camera footage, extract characteristic images, generate explanatory text, and convert it into audio. Adjust the explanatory text according to the user's emotions."

[0247] In this way, the server performs a series of processes from video analysis to audio output, providing appropriate information to the user.

[0248] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0249] Step 1:

[0250] The server receives video data from security cameras and user devices. The input is a real-time video stream, and the output is the video data to be analyzed. For example, if a security camera is monitoring the entrance to an office, the server acquires that video.

[0251] Step 2:

[0252] The server analyzes the received video data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to detect people and objects in the video. The input is the video data acquired in step 1, and the output is information about the detected people and objects. For example, the server identifies the position and movement of people in the video.

[0253] Step 3:

[0254] The server extracts characteristic images from the analyzed video data. It uses image processing libraries such as OpenCV to perform face recognition and object detection. The input is the person and object information obtained in step 2, and the output is a characteristic image. For example, the server extracts the face of a specific person in the video and stores the facial features in a database.

[0255] Step 4:

[0256] The server generates image description text based on the extracted feature images. A generative AI model (e.g., GPT-3 or BERT) is used to generate text that describes the contents of the image. The input is the feature images extracted in step 3, and the output is the image description text. For example, the server generates description text such as "This image shows a man wearing a blue shirt."

[0257] Step 5:

[0258] The server converts the generated image description text into speech. It uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly to convert the text into speech. The input is the image description text generated in step 4, and the output is the audio data. For example, the server converts the text "This image shows a man wearing a blue shirt" into speech and outputs it to the user.

[0259] Step 6:

[0260] The server uses an emotion engine to recognize the user's emotion. The emotion engine analyzes the user's voice and facial expressions to recognize emotions. The input is the user's voice and facial expression data, and the output is the recognized emotion information. For example, the server detects the emotion of joy from the user's voice.

[0261] Step 7:

[0262] The server adjusts the content of the image description text and audio based on the recognized emotion. The input is the emotion information obtained in step 6 and the image description text generated in step 4, and the output is the adjusted image description text and audio data. For example, if the user indicates an emotion of joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily," converts it into audio, and outputs it.

[0263] (Application example 1)

[0264] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] Conventional video analysis systems have difficulty in providing appropriate explanations according to the user's emotions, and have not sufficiently improved the user experience. In addition, in physical stores, when a user shows interest in a product, there is a lack of a means to provide a real-time explanation according to that interest. This causes the user to be unable to obtain sufficient information about the product, which reduces their willingness to purchase.

[0266] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0267] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, a means for adjusting the image commentary text based on the emotion, and a means for converting the adjusted image commentary text into audio. This makes it possible to provide an appropriate commentary in real time according to the user's emotion, thereby improving the user experience in a brick-and-mortar store and increasing purchasing motivation.

[0268] "Means for analyzing video" refers to devices or software for analyzing video data obtained from a camera or sensor and extracting specific information.

[0269] A "means for extracting characteristic images" is a device or software for selecting images with specific characteristics from the analyzed video data.

[0270] The "means for generating image explanatory text" refers to a device or software for generating text explaining an image based on the extracted characteristic image.

[0271] The "means for converting image description text into audio" refers to a device or software for converting the generated text into audio data and providing it to the user.

[0272] "Means for recognizing user's emotions" refers to a device or software for analyzing the user's voice and facial expressions and identifying the user's emotional state.

[0273] A "means for adjusting image description text based on emotion" is a device or software for changing the content or expression of the generated image description text in response to the recognized emotion of the user.

[0274] The "means for converting the adjusted image description text into speech" refers to a device or software for converting the emotion-adjusted text into speech data and providing it to a user.

[0275] A system for implementing this invention includes means for analyzing video, means for extracting characteristic images, means for generating image description text, means for converting the image description text into speech, means for recognizing a user's emotions, means for adjusting the image description text based on the emotions, and means for converting the adjusted image description text into speech.

[0276] Hardware and Software Used

[0277] Hardware

[0278] Smart glasses: A device equipped with a camera, microphone, and speaker that is worn by the user and captures images of the inside of the store in real time.

[0279] Server: A high-performance computer used for processing video analysis, feature image extraction, text generation, emotion recognition, voice conversion, and other processes.

[0280] software

[0281] OpenCV: A library for video analysis and feature image extraction.

[0282] TensorFlow: A machine learning framework for running image analysis models.

[0283] Transformers (Hugging Face): A library for generating image description text using a generative AI model (GPT-3).

[0284] pyttsx3: A library for converting text to speech.

[0285] EmotionRecognizer: Software for recognizing user emotions.

[0286] Data processing and calculation

[0287] Video analysis and feature image extraction

[0288] The server uses OpenCV to analyze the video data acquired from the smart glasses' camera and TensorFlow to extract feature images, thereby identifying the products and areas that the user is paying attention to.

[0289] Image description text generation

[0290] The server inputs the extracted feature images into a generative AI model (GPT-3) using the Transformers library to generate image description text. This text is provided to users in a format that is easy to understand.

[0291] Emotion recognition and text adjustment

[0292] The server uses EmotionRecognizer to recognize emotions from the user's voice and facial expressions. Based on the recognized emotions, the generated image description text is adjusted. For example, if the user expresses an emotion of joy, the text is changed to a more positive expression.

[0293] Audio conversion

[0294] The server converts the adjusted image description text into audio using pyttsx3 and provides it to the user through the speakers of the smart glasses.

[0295] Examples

[0296] As a user walks through a store wearing the smart glasses, the camera in the glasses captures images of the shelves. The server analyzes the images and recognizes a particular product (such as a newly released smartphone). If the user shows interest in the product and smiles, the server will respond with a voice message saying, "This is the latest smartphone! It has lots of great features!"

[0297] Examples of prompt statements

[0298] As a user walks through a store wearing the smart glasses, the camera in the glasses captures images of the shelves. The server analyzes the images and recognizes a particular product (such as a newly released smartphone). If the user shows interest in the product and smiles, the server will respond with a voice message saying, "This is the latest smartphone! It has lots of great features!"

[0299] In this way, it becomes possible to provide appropriate commentary in real time according to the user's emotions, improving the user experience in physical stores and increasing purchasing motivation.

[0300] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0301] Step 1:

[0302] The server acquires video data from the smart glasses camera in real time. The input is video data from the camera, and the output is frame data for analysis. The server uses OpenCV to split the video data into frames and convert them into an analyzable format.

[0303] Step 2:

[0304] The server analyzes the acquired frame data and extracts feature images. The input is the frame data obtained in step 1, and the output is the feature image. The server uses TensorFlow to extract images with specific features from the frame data. These feature images are used to identify products or areas that users are paying attention to.

[0305] Step 3:

[0306] The server inputs the extracted feature images into a generative AI model (GPT-3) to generate image description text. The input is the feature images, and the output is the image description text. The server uses the Transformers library to generate description text based on the feature images. This text is provided in a format that is easy for users to understand.

[0307] Step 4:

[0308] The server analyzes the user's voice and facial expressions to recognize emotions. The input is the voice data and facial expression data acquired from the microphone and camera of the smart glasses, and the output is the recognized emotion. The server uses EmotionRecognizer to identify the user's emotional state.

[0309] Step 5:

[0310] The server adjusts the generated image description text based on the recognized emotion. The input is the image description text and the recognized emotion, and the output is the adjusted image description text. For example, if the user indicates an emotion of happiness, the server changes the text to a more positive expression.

[0311] Step 6:

[0312] The server converts the adjusted image description text into speech and provides it to the user. The input is the adjusted image description text, and the output is audio data. The server uses pyttsx3 to convert the text into speech and provides it to the user through the speaker of the smart glasses.

[0313] In this way, it becomes possible to provide appropriate commentary in real time according to the user's emotions, improving the user experience in physical stores and increasing purchasing motivation.

[0314] Example 2

[0315] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] Conventional video analysis systems have had difficulty effectively conveying the content of a video to people with visual impairments or those who cannot watch the video due to time constraints. In addition, they were unable to provide appropriate explanatory text or audio according to the user's emotions, making it difficult to improve the user experience.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0318] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, and a means for adjusting the generation of the image commentary text and the conversion of the audio based on the emotion. This allows even a person with a visual impairment or a person who cannot view the video due to time constraints to understand the content of the video through audio, and further allows the server to provide an appropriate commentary text and audio according to the user's emotion.

[0319] "Means for analyzing video" refers to technology for analyzing video data and recognizing specific scenes or objects.

[0320] "Means for extracting characteristic images" refers to technology for extracting images of specific scenes or characters from the analyzed video data.

[0321] The "means for generating image explanatory text" is a technology for generating appropriate explanatory text based on the extracted characteristic images.

[0322] "Means for converting image commentary text into audio" refers to technology for converting the generated commentary text into audio data.

[0323] "Means for recognizing a user's emotions" refers to technology for recognizing emotions from a user's voice and facial expressions.

[0324] The "means for adjusting the generation of image commentary text and the conversion of audio based on emotions" is a technology for adjusting the content of generated commentary text and audio based on the recognized emotions of a user.

[0325] "Machine learning" is a technology in which algorithms automatically learn patterns based on data and make predictions and classifications.

[0326] A "generative AI model" is an artificial intelligence model that generates natural language text based on input data.

[0327] This invention is a system that analyzes video, extracts characteristic images, generates image commentary text, and converts it into audio. It also includes a function to recognize the user's emotions and adjust the commentary text and audio based on those emotions.

[0328] First, the server receives the video data of the TV program. This process uses a streaming protocol (e.g. RTSP) or a file transfer protocol (e.g. FTP). Specifically, the server downloads the video data from the specified URL.

[0329] The server then analyzes the received video data using deep learning frameworks such as TensorFlow and PyTorch. The server splits the video data into frames and performs object detection and scene recognition on each frame.

[0330] From the analyzed video, the server extracts images of characteristic scenes and characters. This process uses image processing libraries such as OpenCV. The server extracts frames that contain specific scenes and characters based on the object detection results.

[0331] Based on the extracted feature images, the server generates image description text. A generative AI model such as GPT-3 is used for this process. The server inputs the feature images, gives a prompt to the generative AI model, and generates appropriate description text. For example, text such as "The main character is running in this scene" is generated.

[0332] The generated image description text is converted into speech by the server. For this process, speech synthesis services such as Google Text-to-Speech (TTS) and Amazon Polly are used. The server inputs the description text into the TTS engine and generates an audio file.

[0333] The device recognizes emotions from the user's voice and facial expressions. This process uses Microsoft Azure's Emotion API and Google Cloud's Vision API. The device analyzes the user's camera footage and microphone audio in real time and classifies emotions.

[0334] The server adjusts the generation of image description text and speech conversion based on information from the emotion engine. Specifically, if the user expresses the emotion of joy, the server generates positive description text with priority. For example, text such as "A scene in which the protagonist wins" is generated. On the other hand, if the user expresses the emotion of sadness, the server generates description text related to sadness with priority.

[0335] Examples

[0336] If the user is watching the first episode of a drama, the server receives the video using the RTSP protocol, divides it into frames, and analyzes it. As a result of the analysis, scenes in which the main character is running and scenes in which the enemy character appears are detected.

[0337] For the extracted scenes, the server generates explanatory text such as "In this scene, the main character is running." At this time, a prompt such as "Please generate an explanation for the scene where the main character is running" is input to the generative AI model.

[0338] When a user expresses joy, the device analyzes the camera footage and microphone audio, and the emotion engine recognizes this as "joy." Based on this information, the server generates positive explanatory text, such as "a scene in which the protagonist wins."

[0339] In this way, even if a user has a visual impairment, he or she can understand the contents of a television program through audio.

[0340] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0341] Step 1:

[0342] The server receives video data of a TV program. The input is the video data from a specified URL, and the output is the video data stored in the server. In concrete terms, the server downloads the video data using the RTSP protocol.

[0343] Step 2:

[0344] The server analyzes the received video data. The input is the stored video data, and the output is the analyzed feature information (e.g., scene and character detection results). Specifically, the server uses TensorFlow and PyTorch to divide the video data into frames, and performs object detection and scene recognition on each frame.

[0345] Step 3:

[0346] The server extracts feature images from the analyzed video. The input is the analyzed feature information, and the output is the extracted feature images. Specifically, the server uses OpenCV to extract frames that contain specific scenes or characters.

[0347] Step 4:

[0348] The server generates image description text based on the extracted feature images. The input is the extracted feature images, and the output is the generated image description text. In concrete terms, the server inputs a prompt sentence into a generative AI model such as GPT-3 to generate appropriate description text. For example, text such as "The main character is running in this scene" is generated.

[0349] Step 5:

[0350] The server converts the generated image description text into audio. The input is the generated image description text, and the output is the generated audio file. Specifically, the server converts the description text into audio data using Google Text-to-Speech (TTS) or Amazon Polly.

[0351] Step 6:

[0352] The device recognizes emotions from the user's voice and facial expressions. The input is the user's camera footage and microphone audio, and the output is the recognized emotion information. Specifically, the device uses Microsoft Azure's Emotion API and Google Cloud's Vision API to classify emotions in real time.

[0353] Step 7:

[0354] The server adjusts the generation of image commentary text and the conversion of voice based on information from the emotion engine. The input is the recognized emotion information, and the output is the adjusted commentary text and voice. In concrete terms, when the user shows an emotion of joy, the server preferentially generates a positive commentary text. For example, text such as "a scene in which the protagonist wins" is generated. When the user shows an emotion of sadness, the server preferentially generates a commentary text related to sadness.

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] There is a demand for systems that enable people with visual impairments or those who cannot watch videos due to time constraints to understand the content of video content. Another challenge is how to improve the user experience by providing appropriate explanatory text and audio according to the user's emotions.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0359] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, and a means for adjusting the generation of the image commentary text and the conversion of the audio based on the user's emotion. This allows even a person with a visual impairment or a person with time constraints to understand the contents of the video content through audio, and further allows the server to provide an appropriate commentary and audio according to the user's emotion.

[0360] A "means for analyzing video" is a technology that receives video data as input, analyzes its contents, and identifies characteristic scenes and characters.

[0361] "Means for extracting characteristic images" refers to technology for selecting images of specific scenes or characters from the analyzed video data.

[0362] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[0363] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data and providing it to the user.

[0364] "Means for recognizing a user's emotions" refers to technology for detecting emotions from the user's voice, facial expressions, etc., and identifying those emotions.

[0365] "Means for adjusting the generation of image commentary text and the conversion of audio based on user emotions" refers to technology for adjusting the content and tone of generated commentary text and audio in response to the recognized user emotions.

[0366] A system for implementing this invention includes means for analyzing video, means for extracting characteristic images, means for generating image description text, means for converting the image description text into speech, means for recognizing a user's emotions, and means for adjusting the generation of image description text and the speech conversion based on the user's emotions.

[0367] System Configuration

[0368] Hardware

[0369] Server: Performs video analysis, feature image extraction, image commentary text generation, voice conversion, and emotion recognition.

[0370] Device: The device used by the user, such as a smartphone, smart glasses, or a head-mounted display.

[0371] Input devices: Devices such as cameras and microphones for capturing the user's voice and facial expressions.

[0372] software

[0373] Video Analysis: Analyze video data using OpenCV.

[0374] Feature Image Extraction: Extracting characteristic scenes and characters using TensorFlow.

[0375] Image description text generation: Use the Transformers library to generate description text based on the extracted feature images.

[0376] Text-to-speech: gTTS is used to convert the generated commentary text into speech.

[0377] Emotion Recognition: Use the emotion_recognition library to recognize emotions from the user's voice and facial expressions.

[0378] Processing flow

[0379] 1. Video Analysis: The server uses OpenCV to analyze the video data frame by frame and identify characteristic scenes and characters.

[0380] 2. Feature Image Extraction: The server uses TensorFlow to extract feature scene and character images from the analyzed frames.

[0381] 3. Image description text generation: The server uses the Transformers library to generate description text based on the extracted feature images.

[0382] 4. Voice conversion: The server uses gTTS to convert the generated commentary text into voice data and sends it to the terminal.

[0383] 5. Emotion recognition: The server uses the emotion_recognition library to recognize emotions from the user's voice and facial expressions.

[0384] 6. Adjustment: The server adjusts the content and tone of the generated commentary text and audio based on the recognized user emotions.

[0385] Examples

[0386] For example, if a user feels "sad," the emotion recognition function detects that emotion, and the commentary text generation function preferentially generates "scenes in which the protagonist is sad." This allows the user to receive commentary that matches their emotions.

[0387] Examples of prompt statements

[0388] "Recognize the user's emotions and generate commentary text based on those emotions. For example, if the user is sad, generate commentary on scenes related to sadness with priority."

[0389] In this way, it is possible to provide more appropriate commentary and audio according to the user's emotions.

[0390] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0391] Step 1:

[0392] The server receives video data as input and uses OpenCV to analyze the video frame by frame. Specifically, it divides the video data into frames and analyzes the content of each frame to identify characteristic scenes and characters. The input is the video data, and the output is the analyzed frame data.

[0393] Step 2:

[0394] The server receives the analyzed frame data as input and uses TensorFlow to extract images of characteristic scenes and characters. Specifically, it selects images with specific characteristics from each frame and extracts them as feature images. The input is the analyzed frame data, and the output is the feature images.

[0395] Step 3:

[0396] The server receives the extracted feature images as input and generates image description text using the Transformers library. Specifically, it generates text that describes the contents of the feature images based on the feature images. The input is the feature images, and the output is the image description text.

[0397] Step 4:

[0398] The server receives the generated image description text as input and converts it into voice data using gTTS. Specifically, it converts the text data into voice data and generates an audio file to provide to the user. The input is the image description text and the output is the voice data.

[0399] Step 5:

[0400] The server receives the user's voice and facial expression as input and recognizes emotions using the emotion_recognition library. Specifically, it analyzes the user's voice and facial expression data and identifies the emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion data.

[0401] Step 6:

[0402] The server receives the recognized emotion data as input and adjusts the content and tone of the generated image commentary text and audio. Specifically, the server changes the content of the commentary text and the tone of the audio according to the user's emotion to provide more appropriate commentary and audio. The input is the recognized emotion data, and the output is the adjusted commentary text and audio data.

[0403] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0404] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0405] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0406] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0407] [Second embodiment]

[0408] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0409] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0410] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0411] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0412] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0413] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0414] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0415] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0416] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0417] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0418] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0419] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0420] "Example 1"

[0421] The system of the present invention includes a means for analyzing a video, a means for extracting a characteristic image, a means for generating an image commentary text, and a means for converting the video into audio. The video analysis means uses artificial intelligence to extract a characteristic image from the video. The extracted characteristic image is converted into commentary text by the image commentary text generation means. The commentary text is converted into audio by the audio output means and output to the user.

[0422] "Example 2"

[0423] As a specific example, consider the analysis of a television program video. The video analysis means uses artificial intelligence to extract images of characteristic scenes and characters from the television program video. The extracted characteristic images are converted by the image commentary text generation means into commentary text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears." The commentary text is converted into audio by the audio output means, so that even people with visual impairments or those who cannot watch the video due to time constraints can understand the content of the television program just by listening to the audio.

[0424] The process flow of each embodiment will be described below.

[0425] "Example 1"

[0426] Step 1: The video analysis means analyzes the video. Artificial intelligence is used in this analysis, and characteristic images are extracted from the video.

[0427] Step 2: The extracted feature images are converted into explanatory text by an image explanatory text generating means.

[0428] Step 3: The generated commentary text is converted into speech by the speech output means.

[0429] Step 4: Finally, the converted audio is output to the user.

[0430] "Example 2"

[0431] Step 1: The television program footage is analyzed using a video analysis tool. Artificial intelligence is used in this analysis, and images of characteristic scenes and characters are extracted from the footage.

[0432] Step 2: The extracted feature images are converted into explanatory text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears," by an image explanatory text generating means.

[0433] Step 3: The generated commentary text is converted into speech by the speech output means.

[0434] Step 4: Finally, the converted audio is output to people with visual impairments or who are unable to watch the video due to time constraints, enabling them to understand the content of the TV program just by listening to the audio.

[0435] Example 1

[0436] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0437] Conventional video analysis systems have problems in that they do not efficiently extract characteristic images from a video, generate explanatory text based on the extracted images, and then convert the text into audio. In addition, they lack a means for users to quickly and accurately understand the content of the video.

[0438] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0439] In this invention, the server includes a means for analyzing the video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for transmitting the audio data to the user's terminal, thereby enabling the content of the video to be provided to the user quickly and accurately.

[0440] "Means for analyzing video" refers to technology for analyzing input video data and detecting characteristic elements within the video.

[0441] "Means for extracting characteristic images" refers to a technology that identifies important objects and scenes from the analyzed video data and extracts them as images.

[0442] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[0443] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data.

[0444] The "means for transmitting voice data to a user's terminal" refers to a technique for transmitting the generated voice data to a user's terminal so that the user can play back the voice.

[0445] A "machine learning model" is an algorithm or framework that learns from data and performs pattern recognition and predictions.

[0446] A "natural language processing model" is an algorithm or framework for analyzing, understanding, and generating text data.

[0447] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0448] The present invention is a system for analyzing a video, extracting characteristic images, generating image commentary text, converting the text into audio, and providing the text to a user. A specific embodiment of this system will be described below.

[0449] Video Analysis Methods

[0450] The server uses machine learning models to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the input video data. The video data is uploaded to the server by the user via their device. The server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame. This identifies important objects in each frame.

[0451] Feature image extraction method

[0452] The server extracts characteristic images from the data obtained by the video analysis means. The extracted images are temporarily stored for use in subsequent processing. Specifically, frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) are selected from the output of the object detection model, and these are stored as characteristic images.

[0453] Image description text generation method

[0454] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. The generated text is saved to be provided to the user. Specifically, the feature images are used as input to send a prompt to the generative AI model. For example, using the prompt "What is in this image?", the explanatory text "There is a red car in this image" is obtained from the model.

[0455] Audio conversion means

[0456] The server converts the generated explanatory text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the explanatory text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data.

[0457] Audio data output means

[0458] The server sends the generated audio data to the user's device. The user can play the audio through the device and listen to the commentary. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button on the smartphone app to listen to the audio commentary.

[0459] Examples of concrete examples and prompts

[0460] Examples

[0461] A user uploads a video taken in a park to the system. The server analyzes the video and extracts images of playground equipment and people in the park. It then generates explanatory text for each extracted image, such as "Children are playing in the park," converts this into audio, and provides it to the user.

[0462] Examples of prompt statements

[0463] "Analyze the video footage taken in the park, extract characteristic images, and generate explanatory text. Then convert the generated text into audio and output it."

[0464] In this way, the user can easily understand the contents of the video through audio.

[0465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0466] Step 1:

[0467] A user uploads video data captured using a device to a server. Specifically, the user opens the application on the device and taps the "Video Upload" button. A file selection screen is displayed, and when the user selects a video file and presses the "Send" button, the video data is sent to the server via an HTTP request. The input is the user's video data, and the output is the video data saved on the server.

[0468] Step 2:

[0469] The server receives the uploaded video data and starts analyzing it using video analysis methods. Specifically, the server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame, which identifies important objects in each frame. The input is the video data stored on the server, and the output is the analyzed object data for each frame.

[0470] Step 3:

[0471] The server extracts feature images from the data obtained by the video analysis means. Specifically, it selects frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) from the output of the object detection model, and saves them as feature images. The input is the analyzed object data for each frame, and the output is the extracted feature images.

[0472] Step 4:

[0473] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. Specifically, it sends a prompt sentence to the generative AI model using the feature images as input. For example, it uses the prompt sentence "What is in this image?" to obtain explanatory text from the model, such as "There is a red car in this image." The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[0474] Step 5:

[0475] The server converts the generated commentary text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the commentary text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data. The input is the generated commentary text, and the output is the generated speech data.

[0476] Step 6:

[0477] The server sends the generated audio data to the user's device. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button in the application on the device to listen to the audio commentary. The input is the generated audio data, and the output is the audio played on the user's device.

[0478] (Application example 1)

[0479] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0480] Autonomous vehicles are required to grasp the surrounding situation in real time and provide appropriate information to the driver and passengers. However, the current system does not have sufficient accuracy in image analysis and information provision, and improving safety and convenience is an issue. In particular, there is a lack of means to quickly and accurately recognize important information such as road signs and pedestrians and to convey that information to the driver and passengers in an easy-to-understand manner.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0482] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for analyzing a video around the autonomous vehicle in real time and extracting a characteristic image, and a means for generating commentary text for the characteristic image, converting it into audio, and providing it. This makes it possible to grasp the situation around the autonomous vehicle in real time and provide information to the driver and passengers quickly and accurately.

[0483] "Means for analyzing video" refers to a device or software for analyzing video data acquired from a video input device such as a camera and extracting specific information.

[0484] A "means for extracting characteristic images" is a device or software for selecting images having specific characteristics from the analyzed video data.

[0485] The "means for generating image explanatory text" refers to a device or software for generating explanatory text regarding the image based on the extracted characteristic image.

[0486] "Means for converting image description text into speech" refers to a device or software for converting the generated image description text into speech data and providing it to the user.

[0487] "Means for analyzing images around an autonomous vehicle in real time" refers to a device or software that analyzes images obtained from cameras or other devices mounted on an autonomous vehicle in real time and grasps the surrounding situation.

[0488] A "means for extracting characteristic images" is a device or software for selecting images having specific important information from video data analyzed in real time.

[0489] "Means for generating explanatory text and converting it into audio for providing" refers to a device or software for generating explanatory text based on the extracted characteristic images, converting the text into audio for providing to the user.

[0490] A system for implementing this invention is configured as follows: First, a server has a means for analyzing video, and analyzes video data acquired from a video input device such as a camera. Next, a means for extracting characteristic images from the analyzed video data is used to select images with specific characteristics. Based on the extracted characteristic images, a means for generating image commentary text generates commentary. Finally, a means for converting the generated image commentary text into audio is used to convert it into audio data, which is provided to the user.

[0491] Specifically, the server uses the following hardware and software: A camera is used for video capture, and an artificial intelligence model such as TensorFlow is used for video analysis. A generative AI model is used to extract feature images and generate explanatory text. Google Text-to-Speech API (gTTS) is used for voice conversion.

[0492] For example, in an autonomous vehicle, a camera captures images of the surroundings in real time. The server analyzes the images and extracts characteristic images such as road signs and pedestrians. For the extracted images, a generative AI model is used to generate explanatory text. The generated explanatory text is converted to speech using gTTS and provided to the driver and passengers through the speaker.

[0493] As a concrete example, the following prompt sentence is input to the generative AI model:

[0494] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[0495] By inputting this prompt into the generative AI model, the following explanatory text is generated:

[0496] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[0497] This explanatory text can be converted into audio and provided to drivers and passengers, improving the safety and convenience of self-driving vehicles.

[0498] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0499] Step 1:

[0500] The server captures video data from the camera in real time. The input is the video data from the camera and the output is the captured video frames. This video data is used for subsequent analysis processing.

[0501] Step 2:

[0502] The server analyzes the captured video frames using an artificial intelligence model such as TensorFlow. The input is the video frame, and the output is the analysis result, which is feature data. This analysis extracts important features in the video (e.g., road signs, pedestrians, etc.).

[0503] Step 3:

[0504] The server extracts a feature image from the analysis result. The input is the feature data as the analysis result, and the output is the feature image. This feature image is used for subsequent generation of explanatory text.

[0505] Step 4:

[0506] The server generates image description text using a generative AI model based on the feature image. The input is the feature image, and the output is the description text. This description text is an explanatory text about the feature image.

[0507] Step 5:

[0508] The server converts the generated commentary text into voice data using Google Text-to-Speech API (gTTS). The input is the commentary text, and the output is the voice data. This voice data is provided to the user.

[0509] Step 6:

[0510] The server provides the converted voice data to the driver or passengers through the speaker. The input is voice data and the output is voice output from the speaker. This allows the driver or passengers to obtain information about the surrounding situation in real time.

[0511] Example 2

[0512] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0513] Conventional video analysis systems have made it difficult for people with visual impairments or those who cannot watch the video due to time constraints to understand the content of television programs. In addition, there was a lack of efficient means to extract characteristic scenes and characters from the video, generate text to appropriately explain them, and then convert that text into speech.

[0514] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0515] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into a voice, a means for dividing the video into frames, a means for transmitting the characteristic image to a generative AI model, a means for inputting a prompt sentence to the generative AI model, a means for receiving a commentary text returned from the generative AI model, a means for transmitting the commentary text to a voice synthesis software, a means for receiving voice data returned from the voice synthesis software, and a means for providing the voice data to a user. This allows even a person with a visual impairment or a person who cannot watch a video due to time constraints to understand the contents of a TV program just by listening to the sound.

[0516] "Means for analyzing video" refers to technology or devices for receiving video data and analyzing its contents.

[0517] A "means for extracting characteristic images" is a technology or device that selects important images, such as specific scenes or characters, from the analyzed video data.

[0518] The "means for generating image explanatory text" refers to a technique or device that generates text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0519] "Means for converting image description text into speech" refers to technology or devices that convert the generated text into speech data.

[0520] A "means for dividing an image into frames" is a technique or device for dividing continuous image data into individual still images (frames).

[0521] "Means for sending to a generative AI model" refers to technology or equipment for sending the extracted feature images to a generative AI model.

[0522] "Means for inputting a prompt sentence into a generative AI model" refers to technology or equipment for inputting an explanatory sentence (prompt sentence) regarding the content of an image into a generative AI model.

[0523] "Means for receiving explanatory text returned from a generative AI model" refers to technology or devices that receive explanatory text returned from a generative AI model.

[0524] The "means for transmitting to speech synthesis software" refers to a technique or device for transmitting the generated commentary text to speech synthesis software.

[0525] The "means for receiving voice data returned from the voice synthesis software" refers to a technique or device for receiving voice data returned from the voice synthesis software.

[0526] The "means for providing voice data to a user" refers to a technique or device for providing received voice data to a user.

[0527] The present invention relates to a system that analyzes video, extracts characteristic scenes and characters, generates text explaining them, and converts the text into speech. A specific embodiment of this system will be described below.

[0528] The server receives the video of a television program. The received video is first divided into frames. This division is performed using common video processing techniques. Next, the server uses a deep learning framework (e.g., TensorFlow, PyTorch) to analyze each frame. This analysis involves computer vision techniques such as object detection and face recognition. For example, the server detects specific characters or changes in the scene within a frame and extracts these images.

[0529] The extracted feature image is sent to a generative AI model (e.g. GPT-4). The server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." The generative AI model generates explanatory text based on this prompt sentence and returns it to the server.

[0530] The generated commentary text is sent to speech synthesis software (e.g. Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into audio data and returns it to the server. Finally, the server provides this audio data to the user. The user can understand the content of the TV program by listening to the audio.

[0531] As a concrete example, let's say there is a scene in a TV program where the protagonist confronts an enemy character. The server analyzes this scene and extracts characteristic images. Next, the server has a generative AI model (e.g. GPT-4) generate explanatory text such as "A scene where the protagonist confronts an enemy character." Finally, the server converts this explanatory text into audio using speech synthesis software (e.g. Google Text-to-Speech API) and provides it to the user. By listening to this audio, the user can understand the content of the scene without watching the video.

[0532] This system enables people with visual impairments or who are unable to watch the video due to time constraints to understand the content of television programs simply by listening to the audio.

[0533] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0534] Step 1:

[0535] The server receives the video of a television program. The received video is first divided into frames. Specifically, a process is performed to convert the video data into a series of still images (frames). This process divides the video data into individual frames. The input is the video data of a television program, and the output is the divided frame images.

[0536] Step 2:

[0537] The server uses a deep learning framework (e.g. TensorFlow, PyTorch) to analyze each frame. Computer vision techniques such as object detection and face recognition are used for the analysis. Specifically, it detects specific characters and scene changes in each frame. Through this process, images of characteristic scenes and characters are extracted. The input is the segmented frame images, and the output is the extracted feature images.

[0538] Step 3:

[0539] The server sends the extracted feature images to a generative AI model (e.g., GPT-4). Next, the server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is, "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." This process returns explanatory text from the generative AI model. The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[0540] Step 4:

[0541] The server sends the generated commentary text to speech synthesis software (e.g., Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into voice data and returns it to the server. Specifically, it performs a process to convert text data into voice data. This process generates voice data. The input is the commentary text, and the output is voice data.

[0542] Step 5:

[0543] The server provides the generated voice data to the user. Specifically, it transmits the voice data to the user's terminal so that the user can play it. This process allows the user to understand the contents of the TV program by listening to the voice. The input is the voice data, and the output is the voice provided to the user.

[0544] (Application example 2)

[0545] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0546] People with visual impairments and those who do not have time to watch videos have difficulty understanding the content of television programs and movies. In addition, there is a lack of technology that can analyze videos, extract characteristic scenes and characters, and provide them as audio, so there are limited ways to provide information to these users. Furthermore, there is a demand for a system that can perform video analysis and audio output in real time using devices such as smartphones and smart glasses.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0548] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image description text based on the characteristic image, a means for converting the image description text into audio, and a means for providing the audio to a user. This allows visually impaired people or people who do not have time to watch videos to understand the contents of a television program or movie through audio. In addition, real-time video analysis and audio output can be realized by using an application installed on a smartphone, smart glasses, a head-mounted display, or a robot. Furthermore, by including a means for generating a prompt sentence using a generative AI model, more advanced video analysis and information provision are possible.

[0549] A "means for analyzing video" refers to technology or equipment that receives video data as input and analyzes its contents.

[0550] "Means for extracting characteristic images" refers to technology or devices for extracting important images, such as specific scenes or characters, from the analyzed video data.

[0551] The "means for generating image explanatory text" refers to a technique or device for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0552] "Means for converting image description text into audio" refers to technology or devices for converting the generated image description text into audio data.

[0553] "Means for providing audio to a user" refers to technology or devices for transmitting converted audio data to a user.

[0554] "Artificial intelligence" is a technology that uses techniques such as machine learning and deep learning to analyze data and make judgments and predictions.

[0555] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[0556] "Smart glasses" are a device shaped like glasses that has a built-in display, camera, sensor, etc. and can display information and collect data.

[0557] A "head-mounted display" is a display device worn on the head, and refers to technology and devices for displaying information within the user's field of vision.

[0558] A "robot" is a mechanical device that operates autonomously or remotely to perform specific tasks.

[0559] A "generative AI model" is a model that uses artificial intelligence to generate new information and content from data.

[0560] A "prompt" is text that is entered into a generative AI model as instructions or questions that serve as guidelines for the model to generate appropriate output.

[0561] A system for implementing the present invention includes a means for analyzing a video, a means for extracting a feature image, a means for generating an image description text, a means for converting the image description text into speech, and a means for providing the speech to a user. The system further includes a means for generating a prompt sentence using a generative AI model.

[0562] System Configuration

[0563] Hardware:

[0564] server

[0565] Smartphone

[0566] Smart Glasses

[0567] Head-mounted display

[0568] robot

[0569] software:

[0570] OpenCV: A library for video analysis

[0571] pytesseract: A library for extracting text from images

[0572] gTTS: A text-to-speech library

[0573] mpg321: a tool for playing audio files

[0574] Processing flow

[0575] The server first analyzes the video. For video analysis, OpenCV is used to process the video data frame by frame. Next, pytesseract is used to extract feature images from the analyzed frames. Based on the extracted feature images, image description text is generated. A generative AI model is used to generate this text.

[0576] The generated image description text is converted to audio using gTTS, and the converted audio data is provided to the user using mpg321. This allows people with visual impairments or those who do not have time to watch the video to understand the content of TV programs and movies through audio.

[0577] Examples

[0578] For example, a user launches the "Audio Vision" app on a smartphone and inputs a video of a TV program. The app analyzes the video and generates text such as "a scene where the main character is running" or "a scene where an enemy character appears." The generated text is converted into audio and provided to the user.

[0579] Examples of prompt statements

[0580] An example of a prompt for a generative AI model is:

[0581] Analyze TV program footage and extract characteristic scenes and characters. I would like to develop an application that converts the extracted information into text and audio and provides it to users. Specifically, detect the following scenes:

[0582] Scene where the main character is running

[0583] Scenes in which enemy characters appear

[0584] In this way, a specific embodiment for carrying out the invention has been shown, which enables even those with visual impairments or those who do not have time to watch a video to understand the content of the video through audio.

[0585] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0586] Step 1:

[0587] The server receives video data input from the user. The input video data is video files of TV shows or movies. The server divides the video data into frames and prepares them for analysis.

[0588] Step 2:

[0589] The server uses OpenCV to analyze the video data. Specifically, it processes each frame sequentially and detects characteristic scenes and characters in the video. The input is the video frame, and the output is the detection result of characteristic scenes and characters.

[0590] Step 3:

[0591] The server uses pytesseract to extract text information from the detected characteristic scenes and characters. The input is the images of the characteristic scenes and characters, and the output is the extracted text information.

[0592] Step 4:

[0593] The server uses a generative AI model to generate image commentary text based on the extracted text information. The input is the extracted text information, and the output is the image commentary text. In concrete terms, the server inputs a prompt sentence into the generative AI model to generate appropriate commentary text.

[0594] Step 5:

[0595] The server converts the generated image description text into audio data using gTTS. The input is the image description text, and the output is audio data. Specifically, the server converts the text into an audio file and saves it.

[0596] Step 6:

[0597] The server uses mpg321 to provide the generated audio data to the user. The input is the audio data, and the output is the audio provided to the user. The specific operation is to play the audio file and let the user listen to it.

[0598] Step 7:

[0599] The user receives the provided audio using a smartphone, smart glasses, a head-mounted display, or a robot. This allows the user to understand the content of the video by audio. The input is the audio provided by the server, and the output is the user's understanding.

[0600] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] "Example 1"

[0602] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[0603] "Example 2"

[0604] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[0605] The process flow of each embodiment will be described below.

[0606] "Example 1"

[0607] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[0608] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[0609] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[0610] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[0611] "Example 2"

[0612] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[0613] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[0614] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[0615] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[0616] Example 1

[0617] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0618] Conventional video analysis systems were able to extract characteristic images from video and convert them into text or audio, but they were unable to provide appropriate information according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0620] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for recognizing a user's emotion and adjusting the generation of the image commentary text and the conversion of the audio based on the emotion, thereby making it possible to provide appropriate information according to the user's emotion.

[0621] A "means for analyzing video" is a device or program for receiving video data as input and analyzing the content thereof.

[0622] The "means for extracting characteristic images" refers to a device or program for selecting images having specific characteristics from the analyzed video data.

[0623] The "means for generating image explanatory text" refers to a device or program for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0624] The "means for converting image description text into audio" refers to a device or program for converting the generated image description text into audio data.

[0625] The "means for recognizing the user's emotions" refers to a device or program for analyzing and recognizing the emotions from the user's voice, facial expressions, etc.

[0626] "Means for adjusting the generation of image description text or the conversion of audio based on emotions" refers to a device or program for adjusting the content of the generated image description text or audio in accordance with the recognized user's emotions.

[0627] "Machine learning" is a technology in which a computer learns patterns and rules from data and then uses that knowledge to analyze new data.

[0628] A "generative AI model" is a model that uses artificial intelligence to generate data such as text and images.

[0629] This invention is a system that analyzes video, extracts characteristic images, generates explanatory text based on the images, and converts the text into audio. It also includes a function to recognize the user's emotions and adjust the content of the explanatory text and audio based on the emotions.

[0630] The server uses machine learning to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data. For example, it acquires video data from a surveillance camera in real time and detects people and objects in the video.

[0631] The server then uses image processing libraries such as OpenCV to extract characteristic images from the analyzed video data, for example extracting the face of a particular person in the video and storing their facial features in a database.

[0632] The server uses a generative AI model to generate image description text based on the extracted feature images. Specifically, it uses generative AI models such as GPT-3 and BERT to generate text that describes the contents of the image. For example, it generates a description text such as "This image shows a man wearing a blue shirt."

[0633] The generated image description text is converted to speech using a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. For example, the text "This image shows a man wearing a blue shirt" is converted to speech and output to the user.

[0634] In addition, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's voice and facial expressions to recognize emotions. For example, it detects the emotion of joy from the user's voice.

[0635] Based on the recognized emotion, the server adjusts the image description text and audio content. For example, if the user expresses joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily" and outputs it as audio.

[0636] Examples

[0637] Suppose a user is viewing security camera footage. The server analyzes the footage and detects that a man wearing a blue shirt is in the image. The server then generates explanatory text based on this information, such as "This image shows a man wearing a blue shirt." The server then converts this text into speech and outputs it to the user. If the user expresses an emotion of joy, the server generates text such as "This image shows a man wearing a blue shirt smiling happily," converts it into speech, and outputs it.

[0638] Examples of prompt statements

[0639] "Analyze surveillance camera footage, extract characteristic images, generate explanatory text, and convert it into audio. Adjust the explanatory text according to the user's emotions."

[0640] In this way, the server performs a series of processes from video analysis to audio output, providing appropriate information to the user.

[0641] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0642] Step 1:

[0643] The server receives video data from security cameras and user devices. The input is a real-time video stream, and the output is the video data to be analyzed. For example, if a security camera is monitoring the entrance to an office, the server acquires that video.

[0644] Step 2:

[0645] The server analyzes the received video data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to detect people and objects in the video. The input is the video data acquired in step 1, and the output is information about the detected people and objects. For example, the server identifies the position and movement of people in the video.

[0646] Step 3:

[0647] The server extracts characteristic images from the analyzed video data. It uses image processing libraries such as OpenCV to perform face recognition and object detection. The input is the person and object information obtained in step 2, and the output is a characteristic image. For example, the server extracts the face of a specific person in the video and stores the facial features in a database.

[0648] Step 4:

[0649] The server generates image description text based on the extracted feature images. A generative AI model (e.g., GPT-3 or BERT) is used to generate text that describes the contents of the image. The input is the feature images extracted in step 3, and the output is the image description text. For example, the server generates description text such as "This image shows a man wearing a blue shirt."

[0650] Step 5:

[0651] The server converts the generated image description text into speech. It uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly to convert the text into speech. The input is the image description text generated in step 4, and the output is the audio data. For example, the server converts the text "This image shows a man wearing a blue shirt" into speech and outputs it to the user.

[0652] Step 6:

[0653] The server uses an emotion engine to recognize the user's emotion. The emotion engine analyzes the user's voice and facial expression to recognize the emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion information. For example, the server detects the emotion of joy from the user's voice.

[0654] Step 7:

[0655] The server adjusts the content of the image description text and audio based on the recognized emotion. The input is the emotion information obtained in step 6 and the image description text generated in step 4, and the output is the adjusted image description text and audio data. For example, if the user indicates an emotion of joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily," converts it into audio, and outputs it.

[0656] (Application example 1)

[0657] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0658] Conventional video analysis systems have difficulty in providing appropriate explanations according to the user's emotions, and have not sufficiently improved the user experience. In addition, in physical stores, when a user shows interest in a product, there is a lack of means to provide a real-time explanation according to that interest. This causes the user to be unable to obtain sufficient information about the product, which reduces their willingness to purchase.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0660] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, a means for adjusting the image commentary text based on the emotion, and a means for converting the adjusted image commentary text into audio. This makes it possible to provide an appropriate commentary in real time according to the user's emotion, thereby improving the user experience in a brick-and-mortar store and increasing purchasing motivation.

[0661] "Means for analyzing video" refers to devices or software for analyzing video data obtained from a camera or sensor and extracting specific information.

[0662] A "means for extracting characteristic images" is a device or software for selecting images with specific characteristics from the analyzed video data.

[0663] The "means for generating image explanatory text" refers to a device or software for generating text explaining an image based on the extracted characteristic image.

[0664] The "means for converting image description text into audio" refers to a device or software for converting the generated text into audio data and providing it to the user.

[0665] "Means for recognizing user's emotions" refers to a device or software for analyzing the user's voice and facial expressions and identifying the user's emotional state.

[0666] A "means for adjusting image description text based on emotion" is a device or software for changing the content or expression of the generated image description text in response to the recognized emotion of the user.

[0667] The "means for converting the adjusted image description text into speech" refers to a device or software for converting the emotion-adjusted text into speech data and providing it to a user.

[0668] A system for implementing this invention includes means for analyzing video, means for extracting characteristic images, means for generating image description text, means for converting the image description text into speech, means for recognizing a user's emotions, means for adjusting the image description text based on the emotions, and means for converting the adjusted image description text into speech.

[0669] Hardware and Software Used

[0670] Hardware

[0671] Smart glasses: A device equipped with a camera, microphone, and speaker that is worn by the user and captures images of the inside of the store in real time.

[0672] Server: A high-performance computer used for processing video analysis, feature image extraction, text generation, emotion recognition, voice conversion, and other processes.

[0673] software

[0674] OpenCV: A library for video analysis and feature image extraction.

[0675] TensorFlow: A machine learning framework for running image analysis models.

[0676] Transformers (Hugging Face): A library for generating image description text using a generative AI model (GPT-3).

[0677] pyttsx3: A library for converting text to speech.

[0678] EmotionRecognizer: Software for recognizing user emotions.

[0679] Data processing and calculation

[0680] Video analysis and feature image extraction

[0681] The server uses OpenCV to analyze the video data acquired from the smart glasses' camera and TensorFlow to extract feature images, thereby identifying the products and areas that the user is paying attention to.

[0682] Image description text generation

[0683] The server inputs the extracted feature images into a generative AI model (GPT-3) using the Transformers library to generate image description text. This text is provided to users in a format that is easy to understand.

[0684] Emotion recognition and text adjustment

[0685] The server uses EmotionRecognizer to recognize emotions from the user's voice and facial expressions. Based on the recognized emotions, the generated image description text is adjusted. For example, if the user expresses an emotion of joy, the text is changed to a more positive expression.

[0686] Audio conversion

[0687] The server converts the adjusted image description text into audio using pyttsx3 and provides it to the user through the speakers of the smart glasses.

[0688] Examples

[0689] As a user walks through a store wearing the smart glasses, the camera in the glasses captures images of the shelves. The server analyzes the images and recognizes a particular product (such as a newly released smartphone). If the user shows interest in the product and smiles, the server will respond with a voice message saying, "This is the latest smartphone! It has lots of great features!"

[0690] Examples of prompt statements

[0691] As a user walks through a store wearing the smart glasses, the camera in the glasses captures images of the shelves. The server analyzes the images and recognizes a particular product (such as a newly released smartphone). If the user shows interest in the product and smiles, the server will respond with a voice message saying, "This is the latest smartphone! It has lots of great features!"

[0692] In this way, it becomes possible to provide appropriate commentary in real time according to the user's emotions, improving the user experience in physical stores and increasing purchasing motivation.

[0693] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0694] Step 1:

[0695] The server acquires video data from the smart glasses camera in real time. The input is video data from the camera, and the output is frame data for analysis. The server uses OpenCV to split the video data into frames and convert them into an analyzable format.

[0696] Step 2:

[0697] The server analyzes the acquired frame data and extracts feature images. The input is the frame data obtained in step 1, and the output is the feature image. The server uses TensorFlow to extract images with specific features from the frame data. These feature images are used to identify products or areas that users are paying attention to.

[0698] Step 3:

[0699] The server inputs the extracted feature images into a generative AI model (GPT-3) to generate image description text. The input is the feature images, and the output is the image description text. The server uses the Transformers library to generate description text based on the feature images. This text is provided in a format that is easy for users to understand.

[0700] Step 4:

[0701] The server analyzes the user's voice and facial expressions to recognize emotions. The input is the voice data and facial expression data acquired from the microphone and camera of the smart glasses, and the output is the recognized emotion. The server uses EmotionRecognizer to identify the user's emotional state.

[0702] Step 5:

[0703] The server adjusts the generated image description text based on the recognized emotion. The input is the image description text and the recognized emotion, and the output is the adjusted image description text. For example, if the user indicates an emotion of happiness, the server changes the text to a more positive expression.

[0704] Step 6:

[0705] The server converts the adjusted image description text into speech and provides it to the user. The input is the adjusted image description text, and the output is audio data. The server uses pyttsx3 to convert the text into speech and provides it to the user through the speaker of the smart glasses.

[0706] In this way, it becomes possible to provide appropriate commentary in real time according to the user's emotions, improving the user experience in physical stores and increasing purchasing motivation.

[0707] Example 2

[0708] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0709] Conventional video analysis systems have had difficulty effectively conveying the content of a video to people with visual impairments or those who cannot watch the video due to time constraints. In addition, they were unable to provide appropriate explanatory text or audio according to the user's emotions, making it difficult to improve the user experience.

[0710] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0711] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, and a means for adjusting the generation of the image commentary text and the conversion of the audio based on the emotion. This allows even a person with a visual impairment or a person who cannot view the video due to time constraints to understand the content of the video through audio, and further allows the server to provide an appropriate commentary text and audio according to the user's emotion.

[0712] "Means for analyzing video" refers to technology for analyzing video data and recognizing specific scenes or objects.

[0713] "Means for extracting characteristic images" refers to technology for extracting images of specific scenes or characters from the analyzed video data.

[0714] The "means for generating image explanatory text" is a technology for generating appropriate explanatory text based on the extracted characteristic images.

[0715] "Means for converting image commentary text into audio" refers to technology for converting the generated commentary text into audio data.

[0716] "Means for recognizing a user's emotions" refers to technology for recognizing emotions from a user's voice and facial expressions.

[0717] The "means for adjusting the generation of image commentary text and the conversion of audio based on emotions" is a technology for adjusting the content of generated commentary text and audio based on the recognized emotions of a user.

[0718] "Machine learning" is a technology in which algorithms automatically learn patterns based on data and make predictions and classifications.

[0719] A "generative AI model" is an artificial intelligence model that generates natural language text based on input data.

[0720] This invention is a system that analyzes video, extracts characteristic images, generates image commentary text, and converts it into audio. It also includes a function to recognize the user's emotions and adjust the commentary text and audio based on those emotions.

[0721] First, the server receives the video data of the TV program. This process uses a streaming protocol (e.g. RTSP) or a file transfer protocol (e.g. FTP). Specifically, the server downloads the video data from the specified URL.

[0722] The server then analyzes the received video data using deep learning frameworks such as TensorFlow and PyTorch. The server splits the video data into frames and performs object detection and scene recognition on each frame.

[0723] From the analyzed video, the server extracts images of characteristic scenes and characters. This process uses image processing libraries such as OpenCV. The server extracts frames that contain specific scenes and characters based on the object detection results.

[0724] Based on the extracted feature images, the server generates image description text. A generative AI model such as GPT-3 is used for this process. The server inputs the feature images, gives a prompt to the generative AI model, and generates appropriate description text. For example, text such as "The main character is running in this scene" is generated.

[0725] The generated image description text is converted into speech by the server. For this process, speech synthesis services such as Google Text-to-Speech (TTS) and Amazon Polly are used. The server inputs the description text into the TTS engine and generates an audio file.

[0726] The device recognizes emotions from the user's voice and facial expressions. This process uses Microsoft Azure's Emotion API and Google Cloud's Vision API. The device analyzes the user's camera footage and microphone audio in real time and classifies emotions.

[0727] The server adjusts the generation of image description text and speech conversion based on information from the emotion engine. Specifically, if the user expresses the emotion of joy, the server generates positive description text with priority. For example, text such as "A scene in which the protagonist wins" is generated. On the other hand, if the user expresses the emotion of sadness, the server generates description text related to sadness with priority.

[0728] Examples

[0729] If the user is watching the first episode of a drama, the server receives the video using the RTSP protocol, divides it into frames, and analyzes it. As a result of the analysis, scenes in which the main character is running and scenes in which the enemy character appears are detected.

[0730] For the extracted scenes, the server generates explanatory text such as "In this scene, the main character is running." At this time, a prompt such as "Please generate an explanation for the scene where the main character is running" is input to the generative AI model.

[0731] When a user expresses joy, the device analyzes the camera footage and microphone audio, and the emotion engine recognizes this as "joy." Based on this information, the server generates positive explanatory text, such as "a scene in which the protagonist wins."

[0732] In this way, even if a user has a visual impairment, he or she can understand the contents of a television program through audio.

[0733] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0734] Step 1:

[0735] The server receives video data of a TV program. The input is the video data from a specified URL, and the output is the video data stored in the server. In concrete terms, the server downloads the video data using the RTSP protocol.

[0736] Step 2:

[0737] The server analyzes the received video data. The input is the stored video data, and the output is the analyzed feature information (e.g., scene and character detection results). Specifically, the server uses TensorFlow and PyTorch to divide the video data into frames, and performs object detection and scene recognition on each frame.

[0738] Step 3:

[0739] The server extracts feature images from the analyzed video. The input is the analyzed feature information, and the output is the extracted feature images. Specifically, the server uses OpenCV to extract frames that contain specific scenes or characters.

[0740] Step 4:

[0741] The server generates image description text based on the extracted feature images. The input is the extracted feature images, and the output is the generated image description text. In concrete terms, the server inputs a prompt sentence into a generative AI model such as GPT-3 to generate appropriate description text. For example, text such as "The main character is running in this scene" is generated.

[0742] Step 5:

[0743] The server converts the generated image description text into audio. The input is the generated image description text, and the output is the generated audio file. Specifically, the server converts the description text into audio data using Google Text-to-Speech (TTS) or Amazon Polly.

[0744] Step 6:

[0745] The device recognizes emotions from the user's voice and facial expressions. The input is the user's camera footage and microphone audio, and the output is the recognized emotion information. Specifically, the device uses Microsoft Azure's Emotion API and Google Cloud's Vision API to classify emotions in real time.

[0746] Step 7:

[0747] The server adjusts the generation of image commentary text and the conversion of voice based on information from the emotion engine. The input is the recognized emotion information, and the output is the adjusted commentary text and voice. In concrete terms, when the user shows an emotion of joy, the server preferentially generates a positive commentary text. For example, text such as "a scene in which the protagonist wins" is generated. When the user shows an emotion of sadness, the server preferentially generates a commentary text related to sadness.

[0748] (Application example 2)

[0749] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0750] There is a demand for systems that enable people with visual impairments or those who cannot watch videos due to time constraints to understand the content of video content. Another challenge is how to improve the user experience by providing appropriate explanatory text and audio according to the user's emotions.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0752] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, and a means for adjusting the generation of the image commentary text and the conversion of the audio based on the user's emotion. This allows even a person with a visual impairment or a person with time constraints to understand the contents of the video content through audio, and further allows the server to provide an appropriate commentary and audio according to the user's emotion.

[0753] A "means for analyzing video" is a technology that receives video data as input, analyzes its contents, and identifies characteristic scenes and characters.

[0754] "Means for extracting characteristic images" refers to technology for selecting images of specific scenes or characters from the analyzed video data.

[0755] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[0756] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data and providing it to the user.

[0757] "Means for recognizing a user's emotions" refers to technology for detecting emotions from the user's voice, facial expressions, etc., and identifying those emotions.

[0758] "Means for adjusting the generation of image commentary text and the conversion of audio based on user emotions" refers to technology for adjusting the content and tone of generated commentary text and audio in response to the recognized user emotions.

[0759] A system for implementing this invention includes means for analyzing video, means for extracting characteristic images, means for generating image description text, means for converting the image description text into speech, means for recognizing a user's emotions, and means for adjusting the generation of image description text and the speech conversion based on the user's emotions.

[0760] System Configuration

[0761] Hardware

[0762] Server: Performs video analysis, feature image extraction, image commentary text generation, voice conversion, and emotion recognition.

[0763] Device: The device used by the user, such as a smartphone, smart glasses, or a head-mounted display.

[0764] Input devices: Devices such as cameras and microphones for capturing the user's voice and facial expressions.

[0765] software

[0766] Video Analysis: Analyze video data using OpenCV.

[0767] Feature Image Extraction: Extracting characteristic scenes and characters using TensorFlow.

[0768] Image description text generation: Use the Transformers library to generate description text based on the extracted feature images.

[0769] Text-to-speech: gTTS is used to convert the generated commentary text into speech.

[0770] Emotion Recognition: Use the emotion_recognition library to recognize emotions from the user's voice and facial expressions.

[0771] Processing flow

[0772] 1. Video Analysis: The server uses OpenCV to analyze the video data frame by frame and identify characteristic scenes and characters.

[0773] 2. Feature Image Extraction: The server uses TensorFlow to extract feature scene and character images from the analyzed frames.

[0774] 3. Image description text generation: The server uses the Transformers library to generate description text based on the extracted feature images.

[0775] 4. Voice conversion: The server uses gTTS to convert the generated commentary text into voice data and sends it to the terminal.

[0776] 5. Emotion recognition: The server uses the emotion_recognition library to recognize emotions from the user's voice and facial expressions.

[0777] 6. Adjustment: The server adjusts the content and tone of the generated commentary text and audio based on the recognized user emotions.

[0778] Examples

[0779] For example, if a user feels "sad," the emotion recognition function detects that emotion, and the commentary text generation function preferentially generates "scenes in which the protagonist is sad." This allows the user to receive commentary that matches their emotions.

[0780] Examples of prompt statements

[0781] "Recognize the user's emotions and generate commentary text based on those emotions. For example, if the user is sad, generate commentary on scenes related to sadness with priority."

[0782] In this way, it is possible to provide more appropriate commentary and audio according to the user's emotions.

[0783] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0784] Step 1:

[0785] The server receives video data as input and uses OpenCV to analyze the video frame by frame. Specifically, it divides the video data into frames and analyzes the content of each frame to identify characteristic scenes and characters. The input is the video data, and the output is the analyzed frame data.

[0786] Step 2:

[0787] The server receives the analyzed frame data as input and uses TensorFlow to extract images of characteristic scenes and characters. Specifically, it selects images with specific characteristics from each frame and extracts them as feature images. The input is the analyzed frame data, and the output is the feature images.

[0788] Step 3:

[0789] The server receives the extracted feature images as input and generates image description text using the Transformers library. Specifically, it generates text that describes the content of the feature images based on the feature images. The input is the feature images, and the output is the image description text.

[0790] Step 4:

[0791] The server receives the generated image description text as input and converts it into voice data using gTTS. Specifically, it converts the text data into voice data and generates an audio file to provide to the user. The input is the image description text and the output is the voice data.

[0792] Step 5:

[0793] The server receives the user's voice and facial expression as input and recognizes emotions using the emotion_recognition library. Specifically, it analyzes the user's voice and facial expression data and identifies the emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion data.

[0794] Step 6:

[0795] The server receives the recognized emotion data as input and adjusts the content and tone of the generated image commentary text and audio. Specifically, the server changes the content of the commentary text and the tone of the audio according to the user's emotion to provide more appropriate commentary and audio. The input is the recognized emotion data, and the output is the adjusted commentary text and audio data.

[0796] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0797] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0798] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0799] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0800] [Third embodiment]

[0801] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0802] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0803] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0804] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0805] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0806] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0807] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0808] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0809] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0810] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0811] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0812] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0813] "Example 1"

[0814] The system of the present invention includes a means for analyzing a video, a means for extracting a characteristic image, a means for generating an image commentary text, and a means for converting the video into audio. The video analysis means uses artificial intelligence to extract a characteristic image from the video. The extracted characteristic image is converted into commentary text by the image commentary text generation means. The commentary text is converted into audio by the audio output means and output to the user.

[0815] "Example 2"

[0816] As a specific example, consider the analysis of a television program video. The video analysis means uses artificial intelligence to extract images of characteristic scenes and characters from the television program video. The extracted characteristic images are converted by the image commentary text generation means into commentary text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears." The commentary text is converted into audio by the audio output means, so that even people with visual impairments or those who cannot watch the video due to time constraints can understand the content of the television program just by listening to the audio.

[0817] The process flow of each embodiment will be described below.

[0818] "Example 1"

[0819] Step 1: The video analysis means analyzes the video. Artificial intelligence is used in this analysis, and characteristic images are extracted from the video.

[0820] Step 2: The extracted feature images are converted into explanatory text by an image explanatory text generating means.

[0821] Step 3: The generated commentary text is converted into speech by the speech output means.

[0822] Step 4: Finally, the converted audio is output to the user.

[0823] "Example 2"

[0824] Step 1: The television program footage is analyzed using a video analysis tool. Artificial intelligence is used in this analysis, and images of characteristic scenes and characters are extracted from the footage.

[0825] Step 2: The extracted feature images are converted into explanatory text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears," by an image explanatory text generating means.

[0826] Step 3: The generated commentary text is converted into speech by the speech output means.

[0827] Step 4: Finally, the converted audio is output to people with visual impairments or who are unable to watch the video due to time constraints, enabling them to understand the content of the TV program just by listening to the audio.

[0828] Example 1

[0829] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0830] Conventional video analysis systems have problems in that they do not efficiently extract characteristic images from a video, generate explanatory text based on the extracted images, and then convert the text into audio. In addition, they lack a means for users to quickly and accurately understand the content of the video.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0832] In this invention, the server includes a means for analyzing the video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for transmitting the audio data to the user's terminal, thereby enabling the content of the video to be provided to the user quickly and accurately.

[0833] "Means for analyzing video" refers to technology for analyzing input video data and detecting characteristic elements within the video.

[0834] "Means for extracting characteristic images" refers to a technology that identifies important objects and scenes from the analyzed video data and extracts them as images.

[0835] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[0836] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data.

[0837] The "means for transmitting voice data to a user's terminal" refers to a technique for transmitting the generated voice data to a user's terminal so that the user can play back the voice.

[0838] A "machine learning model" is an algorithm or framework that learns from data and performs pattern recognition and predictions.

[0839] A "natural language processing model" is an algorithm or framework for analyzing, understanding, and generating text data.

[0840] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0841] The present invention is a system for analyzing a video, extracting characteristic images, generating image commentary text, converting the text into audio, and providing the text to a user. A specific embodiment of this system will be described below.

[0842] Video Analysis Methods

[0843] The server uses machine learning models to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the input video data. The video data is uploaded to the server by the user via their device. The server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame. This identifies important objects in each frame.

[0844] Feature image extraction method

[0845] The server extracts characteristic images from the data obtained by the video analysis means. The extracted images are temporarily stored for use in subsequent processing. Specifically, frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) are selected from the output of the object detection model, and these are stored as characteristic images.

[0846] Image description text generation method

[0847] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. The generated text is saved to be provided to the user. Specifically, the feature images are used as input to send a prompt to the generative AI model. For example, using the prompt "What is in this image?", the explanatory text "There is a red car in this image" is obtained from the model.

[0848] Audio conversion means

[0849] The server converts the generated explanatory text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the explanatory text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data.

[0850] Audio data output means

[0851] The server sends the generated audio data to the user's device. The user can play the audio through the device and listen to the commentary. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button on the smartphone app to listen to the audio commentary.

[0852] Examples of concrete examples and prompts

[0853] Examples

[0854] A user uploads a video taken in a park to the system. The server analyzes the video and extracts images of playground equipment and people in the park. It then generates explanatory text for each extracted image, such as "Children are playing in the park," converts this into audio, and provides it to the user.

[0855] Examples of prompt statements

[0856] "Analyze the video footage taken in the park, extract characteristic images, and generate explanatory text. Then convert the generated text into audio and output it."

[0857] In this way, the user can easily understand the contents of the video through audio.

[0858] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0859] Step 1:

[0860] A user uploads video data captured using a device to a server. Specifically, the user opens the application on the device and taps the "Video Upload" button. A file selection screen is displayed, and when the user selects a video file and presses the "Send" button, the video data is sent to the server via an HTTP request. The input is the user's video data, and the output is the video data saved on the server.

[0861] Step 2:

[0862] The server receives the uploaded video data and starts analyzing it using video analysis methods. Specifically, the server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame, which identifies important objects in each frame. The input is the video data stored on the server, and the output is the analyzed object data for each frame.

[0863] Step 3:

[0864] The server extracts feature images from the data obtained by the video analysis means. Specifically, it selects frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) from the output of the object detection model, and saves them as feature images. The input is the analyzed object data for each frame, and the output is the extracted feature images.

[0865] Step 4:

[0866] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. Specifically, it sends a prompt sentence to the generative AI model using the feature images as input. For example, it uses the prompt sentence "What is in this image?" to obtain explanatory text from the model, such as "There is a red car in this image." The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[0867] Step 5:

[0868] The server converts the generated commentary text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the commentary text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data. The input is the generated commentary text, and the output is the generated speech data.

[0869] Step 6:

[0870] The server sends the generated audio data to the user's device. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button in the application on the device to listen to the audio commentary. The input is the generated audio data, and the output is the audio played on the user's device.

[0871] (Application example 1)

[0872] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0873] Autonomous vehicles are required to grasp the surrounding situation in real time and provide appropriate information to the driver and passengers. However, the current system does not have sufficient accuracy in image analysis and information provision, and improving safety and convenience is an issue. In particular, there is a lack of means to quickly and accurately recognize important information such as road signs and pedestrians and to convey that information to the driver and passengers in an easy-to-understand manner.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0875] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for analyzing a video around the autonomous vehicle in real time and extracting a characteristic image, and a means for generating commentary text for the characteristic image, converting it into audio, and providing it. This makes it possible to grasp the situation around the autonomous vehicle in real time and provide information to the driver and passengers quickly and accurately.

[0876] "Means for analyzing video" refers to a device or software for analyzing video data acquired from a video input device such as a camera and extracting specific information.

[0877] A "means for extracting characteristic images" is a device or software for selecting images having specific characteristics from the analyzed video data.

[0878] The "means for generating image explanatory text" refers to a device or software for generating explanatory text regarding the image based on the extracted characteristic image.

[0879] "Means for converting image description text into speech" refers to a device or software for converting the generated image description text into speech data and providing it to the user.

[0880] "Means for analyzing images around an autonomous vehicle in real time" refers to a device or software that analyzes images obtained from cameras or other devices mounted on an autonomous vehicle in real time and grasps the surrounding situation.

[0881] A "means for extracting characteristic images" is a device or software for selecting images having specific important information from video data analyzed in real time.

[0882] "Means for generating explanatory text and converting it into audio for providing" refers to a device or software for generating explanatory text based on the extracted characteristic images, converting the text into audio for providing to the user.

[0883] A system for implementing this invention is configured as follows: First, a server has a means for analyzing video, and analyzes video data acquired from a video input device such as a camera. Next, a means for extracting characteristic images from the analyzed video data is used to select images with specific characteristics. Based on the extracted characteristic images, a means for generating image commentary text generates commentary. Finally, a means for converting the generated image commentary text into audio is used to convert it into audio data, which is provided to the user.

[0884] Specifically, the server uses the following hardware and software: A camera is used for video capture, and an artificial intelligence model such as TensorFlow is used for video analysis. A generative AI model is used to extract feature images and generate explanatory text. Google Text-to-Speech API (gTTS) is used for voice conversion.

[0885] For example, in an autonomous vehicle, a camera captures images of the surroundings in real time. The server analyzes the images and extracts characteristic images such as road signs and pedestrians. For the extracted images, a generative AI model is used to generate explanatory text. The generated explanatory text is converted to speech using gTTS and provided to the driver and passengers through the speaker.

[0886] As a concrete example, the following prompt sentence is input to the generative AI model:

[0887] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[0888] By inputting this prompt into the generative AI model, the following explanatory text is generated:

[0889] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[0890] This explanatory text can be converted into audio and provided to drivers and passengers, improving the safety and convenience of self-driving vehicles.

[0891] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0892] Step 1:

[0893] The server captures video data from the camera in real time. The input is the video data from the camera and the output is the captured video frames. This video data is used for subsequent analysis processing.

[0894] Step 2:

[0895] The server analyzes the captured video frames using an artificial intelligence model such as TensorFlow. The input is the video frame, and the output is the analysis result, which is feature data. This analysis extracts important features in the video (e.g., road signs, pedestrians, etc.).

[0896] Step 3:

[0897] The server extracts a feature image from the analysis result. The input is the feature data as the analysis result, and the output is the feature image. This feature image is used for subsequent generation of explanatory text.

[0898] Step 4:

[0899] The server generates image description text using a generative AI model based on the feature image. The input is the feature image, and the output is the description text. This description text is an explanatory text about the feature image.

[0900] Step 5:

[0901] The server converts the generated commentary text into voice data using Google Text-to-Speech API (gTTS). The input is the commentary text, and the output is the voice data. This voice data is provided to the user.

[0902] Step 6:

[0903] The server provides the converted voice data to the driver or passengers through the speaker. The input is voice data and the output is voice output from the speaker. This allows the driver or passengers to obtain information about the surrounding situation in real time.

[0904] Example 2

[0905] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0906] Conventional video analysis systems have made it difficult for people with visual impairments or those who cannot watch the video due to time constraints to understand the content of television programs. In addition, there was a lack of efficient means to extract characteristic scenes and characters from the video, generate text to appropriately explain them, and then convert that text into speech.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0908] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into a voice, a means for dividing the video into frames, a means for transmitting the characteristic image to a generative AI model, a means for inputting a prompt sentence to the generative AI model, a means for receiving a commentary text returned from the generative AI model, a means for transmitting the commentary text to a voice synthesis software, a means for receiving voice data returned from the voice synthesis software, and a means for providing the voice data to a user. This allows even a person with a visual impairment or a person who cannot watch a video due to time constraints to understand the contents of a TV program just by listening to the sound.

[0909] "Means for analyzing video" refers to technology or devices for receiving video data and analyzing its contents.

[0910] A "means for extracting characteristic images" is a technology or device that selects important images, such as specific scenes or characters, from the analyzed video data.

[0911] The "means for generating image explanatory text" refers to a technique or device that generates text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0912] "Means for converting image description text into speech" refers to technology or devices that convert the generated text into speech data.

[0913] A "means for dividing an image into frames" is a technique or device for dividing continuous image data into individual still images (frames).

[0914] "Means for sending to a generative AI model" refers to technology or equipment for sending the extracted feature images to a generative AI model.

[0915] "Means for inputting a prompt sentence into a generative AI model" refers to technology or equipment for inputting an explanatory sentence (prompt sentence) regarding the content of an image into a generative AI model.

[0916] "Means for receiving explanatory text returned from a generative AI model" refers to technology or devices that receive explanatory text returned from a generative AI model.

[0917] The "means for transmitting to speech synthesis software" refers to a technique or device for transmitting the generated commentary text to speech synthesis software.

[0918] The "means for receiving voice data returned from the voice synthesis software" refers to a technique or device for receiving voice data returned from the voice synthesis software.

[0919] The "means for providing voice data to a user" refers to a technique or device for providing received voice data to a user.

[0920] The present invention relates to a system that analyzes video, extracts characteristic scenes and characters, generates text explaining them, and converts the text into speech. A specific embodiment of this system will be described below.

[0921] The server receives the video of a television program. The received video is first divided into frames. This division is performed using common video processing techniques. Next, the server uses a deep learning framework (e.g., TensorFlow, PyTorch) to analyze each frame. This analysis involves computer vision techniques such as object detection and face recognition. For example, the server detects specific characters or changes in the scene within a frame and extracts these images.

[0922] The extracted feature image is sent to a generative AI model (e.g. GPT-4). The server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." The generative AI model generates explanatory text based on this prompt sentence and returns it to the server.

[0923] The generated commentary text is sent to speech synthesis software (e.g. Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into audio data and returns it to the server. Finally, the server provides this audio data to the user. The user can understand the content of the TV program by listening to the audio.

[0924] As a concrete example, let's say there is a scene in a TV program where the protagonist confronts an enemy character. The server analyzes this scene and extracts characteristic images. Next, the server has a generative AI model (e.g. GPT-4) generate explanatory text such as "A scene where the protagonist confronts an enemy character." Finally, the server converts this explanatory text into audio using speech synthesis software (e.g. Google Text-to-Speech API) and provides it to the user. By listening to this audio, the user can understand the content of the scene without watching the video.

[0925] This system enables people with visual impairments or who are unable to watch the video due to time constraints to understand the content of television programs simply by listening to the audio.

[0926] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0927] Step 1:

[0928] The server receives the video of a television program. The received video is first divided into frames. Specifically, a process is performed to convert the video data into a series of still images (frames). This process divides the video data into individual frames. The input is the video data of a television program, and the output is the divided frame images.

[0929] Step 2:

[0930] The server uses a deep learning framework (e.g. TensorFlow, PyTorch) to analyze each frame. Computer vision techniques such as object detection and face recognition are used for the analysis. Specifically, it detects specific characters and scene changes in each frame. Through this process, images of characteristic scenes and characters are extracted. The input is the segmented frame images, and the output is the extracted feature images.

[0931] Step 3:

[0932] The server sends the extracted feature images to a generative AI model (e.g., GPT-4). Next, the server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is, "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." This process returns explanatory text from the generative AI model. The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[0933] Step 4:

[0934] The server sends the generated commentary text to speech synthesis software (e.g., Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into voice data and returns it to the server. Specifically, it performs a process to convert text data into voice data. This process generates voice data. The input is the commentary text, and the output is voice data.

[0935] Step 5:

[0936] The server provides the generated voice data to the user. Specifically, it transmits the voice data to the user's terminal so that the user can play it. This process allows the user to understand the contents of the TV program by listening to the voice. The input is the voice data, and the output is the voice provided to the user.

[0937] (Application example 2)

[0938] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0939] People with visual impairments and those who do not have time to watch videos have difficulty understanding the content of television programs and movies. In addition, there is a lack of technology that can analyze videos, extract characteristic scenes and characters, and provide them as audio, so there are limited ways to provide information to these users. Furthermore, there is a demand for a system that can perform video analysis and audio output in real time using devices such as smartphones and smart glasses.

[0940] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0941] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image description text based on the characteristic image, a means for converting the image description text into audio, and a means for providing the audio to a user. This allows visually impaired people or people who do not have time to watch videos to understand the contents of a television program or movie through audio. In addition, real-time video analysis and audio output can be realized by using an application installed on a smartphone, smart glasses, a head-mounted display, or a robot. Furthermore, by including a means for generating a prompt sentence using a generative AI model, more advanced video analysis and information provision are possible.

[0942] A "means for analyzing video" refers to technology or equipment that receives video data as input and analyzes its contents.

[0943] "Means for extracting characteristic images" refers to technology or devices for extracting important images, such as specific scenes or characters, from the analyzed video data.

[0944] The "means for generating image explanatory text" refers to a technique or device for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[0945] "Means for converting image description text into audio" refers to technology or devices for converting the generated image description text into audio data.

[0946] "Means for providing audio to a user" refers to technology or devices for transmitting converted audio data to a user.

[0947] "Artificial intelligence" is a technology that uses techniques such as machine learning and deep learning to analyze data and make judgments and predictions.

[0948] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[0949] "Smart glasses" are a device shaped like glasses that has a built-in display, camera, sensor, etc. and can display information and collect data.

[0950] A "head-mounted display" is a display device worn on the head, and refers to technology and devices for displaying information within the user's field of vision.

[0951] A "robot" is a mechanical device that operates autonomously or remotely to perform specific tasks.

[0952] A "generative AI model" is a model that uses artificial intelligence to generate new information and content from data.

[0953] A "prompt" is text that is entered into a generative AI model as instructions or questions that serve as guidelines for the model to generate appropriate output.

[0954] A system for implementing the present invention includes a means for analyzing a video, a means for extracting a feature image, a means for generating an image description text, a means for converting the image description text into speech, and a means for providing the speech to a user. The system further includes a means for generating a prompt sentence using a generative AI model.

[0955] System Configuration

[0956] Hardware:

[0957] server

[0958] Smartphone

[0959] Smart Glasses

[0960] Head-mounted display

[0961] robot

[0962] software:

[0963] OpenCV: A library for video analysis

[0964] pytesseract: A library for extracting text from images

[0965] gTTS: A text-to-speech library

[0966] mpg321: a tool for playing audio files

[0967] Processing flow

[0968] The server first analyzes the video. For video analysis, OpenCV is used to process the video data frame by frame. Next, pytesseract is used to extract feature images from the analyzed frames. Based on the extracted feature images, image description text is generated. A generative AI model is used to generate this text.

[0969] The generated image description text is converted to audio using gTTS, and the converted audio data is provided to the user using mpg321. This allows people with visual impairments or those who do not have time to watch the video to understand the content of TV programs and movies through audio.

[0970] Examples

[0971] For example, a user launches the "Audio Vision" app on a smartphone and inputs a video of a TV program. The app analyzes the video and generates text such as "a scene where the main character is running" or "a scene where an enemy character appears." The generated text is converted into audio and provided to the user.

[0972] Examples of prompt statements

[0973] An example of a prompt for a generative AI model is:

[0974] Analyze TV program footage and extract characteristic scenes and characters. I would like to develop an application that converts the extracted information into text and audio and provides it to users. Specifically, detect the following scenes:

[0975] Scene where the main character is running

[0976] Scenes in which enemy characters appear

[0977] In this way, a specific embodiment for carrying out the invention has been shown, which enables even those with visual impairments or those who do not have time to watch a video to understand the content of the video through audio.

[0978] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0979] Step 1:

[0980] The server receives video data input from the user. The input video data is video files of TV shows or movies. The server divides the video data into frames and prepares them for analysis.

[0981] Step 2:

[0982] The server uses OpenCV to analyze the video data. Specifically, it processes each frame sequentially and detects characteristic scenes and characters in the video. The input is the video frame, and the output is the detection result of characteristic scenes and characters.

[0983] Step 3:

[0984] The server uses pytesseract to extract text information from the detected characteristic scenes and characters. The input is the images of the characteristic scenes and characters, and the output is the extracted text information.

[0985] Step 4:

[0986] The server uses a generative AI model to generate image commentary text based on the extracted text information. The input is the extracted text information, and the output is the image commentary text. In concrete terms, the server inputs a prompt sentence into the generative AI model to generate appropriate commentary text.

[0987] Step 5:

[0988] The server converts the generated image description text into audio data using gTTS. The input is the image description text, and the output is audio data. Specifically, the server converts the text into an audio file and saves it.

[0989] Step 6:

[0990] The server uses mpg321 to provide the generated audio data to the user. The input is the audio data, and the output is the audio provided to the user. The specific operation is to play the audio file and let the user listen to it.

[0991] Step 7:

[0992] The user receives the provided audio using a smartphone, smart glasses, a head-mounted display, or a robot. This allows the user to understand the content of the video by audio. The input is the audio provided by the server, and the output is the user's understanding.

[0993] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0994] "Example 1"

[0995] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[0996] "Example 2"

[0997] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[0998] The process flow of each embodiment will be described below.

[0999] "Example 1"

[1000] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[1001] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[1002] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[1003] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[1004] "Example 2"

[1005] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[1006] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[1007] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[1008] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[1009] Example 1

[1010] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1011] Conventional video analysis systems were able to extract characteristic images from video and convert them into text or audio, but they were unable to provide appropriate information according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1013] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for recognizing a user's emotion and adjusting the generation of the image commentary text and the conversion of the audio based on the emotion, thereby making it possible to provide appropriate information according to the user's emotion.

[1014] A "means for analyzing video" is a device or program for receiving video data as input and analyzing the content thereof.

[1015] The "means for extracting characteristic images" refers to a device or program for selecting images having specific characteristics from the analyzed video data.

[1016] The "means for generating image explanatory text" refers to a device or program for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[1017] The "means for converting image description text into audio" refers to a device or program for converting the generated image description text into audio data.

[1018] The "means for recognizing the user's emotions" refers to a device or program for analyzing and recognizing the emotions from the user's voice, facial expressions, etc.

[1019] "Means for adjusting the generation of image description text or the conversion of audio based on emotions" refers to a device or program for adjusting the content of the generated image description text or audio in accordance with the recognized user's emotions.

[1020] "Machine learning" is a technology in which a computer learns patterns and rules from data and then uses that knowledge to analyze new data.

[1021] A "generative AI model" is a model that uses artificial intelligence to generate data such as text and images.

[1022] This invention is a system that analyzes video, extracts characteristic images, generates explanatory text based on the images, and converts the text into audio. It also includes a function to recognize the user's emotions and adjust the content of the explanatory text and audio based on the emotions.

[1023] The server uses machine learning to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data. For example, it acquires video data from a surveillance camera in real time and detects people and objects in the video.

[1024] The server then uses image processing libraries such as OpenCV to extract characteristic images from the analyzed video data, for example extracting the face of a particular person in the video and storing their facial features in a database.

[1025] The server uses a generative AI model to generate image description text based on the extracted feature images. Specifically, it uses generative AI models such as GPT-3 and BERT to generate text that describes the contents of the image. For example, it generates a description text such as "This image shows a man wearing a blue shirt."

[1026] The generated image description text is converted to speech using a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. For example, the text "This image shows a man wearing a blue shirt" is converted to speech and output to the user.

[1027] In addition, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's voice and facial expressions to recognize emotions. For example, it detects the emotion of joy from the user's voice.

[1028] Based on the recognized emotion, the server adjusts the image description text and audio content. For example, if the user expresses joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily" and outputs it as audio.

[1029] Examples

[1030] Suppose a user is viewing security camera footage. The server analyzes the footage and detects that a man wearing a blue shirt is in the image. The server then generates explanatory text based on this information, such as "This image shows a man wearing a blue shirt." The server then converts this text into speech and outputs it to the user. If the user expresses an emotion of joy, the server generates text such as "This image shows a man wearing a blue shirt smiling happily," converts it into speech, and outputs it.

[1031] Examples of prompt statements

[1032] "Analyze surveillance camera footage, extract characteristic images, generate explanatory text, and convert it into audio. Adjust the explanatory text according to the user's emotions."

[1033] In this way, the server performs a series of processes from video analysis to audio output, providing appropriate information to the user.

[1034] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1035] Step 1:

[1036] The server receives video data from security cameras and user devices. The input is a real-time video stream, and the output is the video data to be analyzed. For example, if a security camera is monitoring the entrance to an office, the server acquires that video.

[1037] Step 2:

[1038] The server analyzes the received video data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to detect people and objects in the video. The input is the video data acquired in step 1, and the output is information about the detected people and objects. For example, the server identifies the position and movement of people in the video.

[1039] Step 3:

[1040] The server extracts characteristic images from the analyzed video data. It uses image processing libraries such as OpenCV to perform face recognition and object detection. The input is the person and object information obtained in step 2, and the output is a characteristic image. For example, the server extracts the face of a specific person in the video and stores the facial features in a database.

[1041] Step 4:

[1042] The server generates image description text based on the extracted feature images. A generative AI model (e.g., GPT-3 or BERT) is used to generate text that describes the contents of the image. The input is the feature images extracted in step 3, and the output is the image description text. For example, the server generates description text such as "This image shows a man wearing a blue shirt."

[1043] Step 5:

[1044] The server converts the generated image description text into speech. It uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly to convert the text into speech. The input is the image description text generated in step 4, and the output is the audio data. For example, the server converts the text "This image shows a man wearing a blue shirt" into speech and outputs it to the user.

[1045] Step 6:

[1046] The server uses an emotion engine to recognize the user's emotion. The emotion engine analyzes the user's voice and facial expressions to recognize emotions. The input is the user's voice and facial expression data, and the output is the recognized emotion information. For example, the server detects the emotion of joy from the user's voice.

[1047] Step 7:

[1048] The server adjusts the content of the image description text and audio based on the recognized emotion. The input is the emotion information obtained in step 6 and the image description text generated in step 4, and the output is the adjusted image description text and audio data. For example, if the user indicates an emotion of joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily," converts it into audio, and outputs it.

[1049] (Application example 1)

[1050] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1051] Conventional video analysis systems have difficulty in providing appropriate explanations according to the user's emotions, and have not sufficiently improved the user experience. In addition, in physical stores, when a user shows interest in a product, there is a lack of means to provide a real-time explanation according to that interest. This causes the user to be unable to obtain sufficient information about the product, which reduces their willingness to purchase.

[1052] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1053] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, a means for adjusting the image commentary text based on the emotion, and a means for converting the adjusted image commentary text into audio. This makes it possible to provide an appropriate commentary in real time according to the user's emotion, thereby improving the user experience in a brick-and-mortar store and increasing purchasing motivation.

[1054] "Means for analyzing video" refers to devices or software for analyzing video data obtained from a camera or sensor and extracting specific information.

[1055] A "means for extracting characteristic images" is a device or software for selecting images with specific characteristics from the analyzed video data.

[1056] The "means for generating image explanatory text" refers to a device or software for generating text explaining an image based on the extracted characteristic image.

[1057] The "means for converting image description text into audio" refers to a device or software for converting the generated text into audio data and providing it to the user.

[1058] "Means for recognizing user's emotions" refers to a device or software for analyzing the user's voice and facial expressions and identifying the user's emotional state.

[1059] A "means for adjusting image description text based on emotion" is a device or software for changing the content or expression of the generated image description text in response to the recognized emotion of the user.

[1060] The "means for converting the adjusted image description text into speech" refers to a device or software for converting the emotion-adjusted text into speech data and providing it to a user.

[1061] A system for implementing this invention includes means for analyzing video, means for extracting characteristic images, means for generating image description text, means for converting the image description text into speech, means for recognizing a user's emotions, means for adjusting the image description text based on the emotions, and means for converting the adjusted image description text into speech.

[1062] Hardware and Software Used

[1063] Hardware

[1064] Smart glasses: A device equipped with a camera, microphone, and speaker that is worn by the user and captures images of the inside of the store in real time.

[1065] Server: A high-performance computer used for processing video analysis, feature image extraction, text generation, emotion recognition, voice conversion, and other processes.

[1066] software

[1067] OpenCV: A library for video analysis and feature image extraction.

[1068] TensorFlow: A machine learning framework for running image analysis models.

[1069] Transformers (Hugging Face): A library for generating image description text using a generative AI model (GPT-3).

[1070] pyttsx3: A library for converting text to speech.

[1071] EmotionRecognizer: Software for recognizing user emotions.

[1072] Data processing and calculation

[1073] Video analysis and feature image extraction

[1074] The server uses OpenCV to analyze the video data acquired from the smart glasses' camera and TensorFlow to extract feature images, thereby identifying the products and areas that the user is paying attention to.

[1075] Image description text generation

[1076] The server inputs the extracted feature images into a generative AI model (GPT-3) using the Transformers library to generate image description text. This text is provided to users in a format that is easy to understand.

[1077] Emotion recognition and text adjustment

[1078] The server uses EmotionRecognizer to recognize emotions from the user's voice and facial expressions. Based on the recognized emotions, the generated image description text is adjusted. For example, if the user expresses an emotion of joy, the text is changed to a more positive expression.

[1079] Audio conversion

[1080] The server converts the adjusted image description text into audio using pyttsx3 and provides it to the user through the speakers of the smart glasses.

[1081] Examples

[1082] As a user walks through a store wearing the smart glasses, the camera in the glasses captures images of the shelves. The server analyzes the images and recognizes a particular product (such as a newly released smartphone). If the user shows interest in the product and smiles, the server will respond with a voice message saying, "This is the latest smartphone! It has lots of great features!"

[1083] Examples of prompt statements

[1084] As a user walks through a store wearing the smart glasses, the camera in the glasses captures images of the shelves. The server analyzes the images and recognizes a particular product (such as a newly released smartphone). If the user shows interest in the product and smiles, the server will respond with a voice message saying, "This is the latest smartphone! It has lots of great features!"

[1085] In this way, it becomes possible to provide appropriate commentary in real time according to the user's emotions, improving the user experience in physical stores and increasing purchasing motivation.

[1086] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1087] Step 1:

[1088] The server acquires video data from the smart glasses camera in real time. The input is video data from the camera, and the output is frame data for analysis. The server uses OpenCV to split the video data into frames and convert them into an analyzable format.

[1089] Step 2:

[1090] The server analyzes the acquired frame data and extracts feature images. The input is the frame data obtained in step 1, and the output is the feature image. The server uses TensorFlow to extract images with specific features from the frame data. These feature images are used to identify products or areas that users are paying attention to.

[1091] Step 3:

[1092] The server inputs the extracted feature images into a generative AI model (GPT-3) to generate image description text. The input is the feature images, and the output is the image description text. The server uses the Transformers library to generate description text based on the feature images. This text is provided in a format that is easy for users to understand.

[1093] Step 4:

[1094] The server analyzes the user's voice and facial expressions to recognize emotions. The input is the voice data and facial expression data acquired from the microphone and camera of the smart glasses, and the output is the recognized emotion. The server uses EmotionRecognizer to identify the user's emotional state.

[1095] Step 5:

[1096] The server adjusts the generated image description text based on the recognized emotion. The input is the image description text and the recognized emotion, and the output is the adjusted image description text. For example, if the user indicates an emotion of happiness, the server changes the text to a more positive expression.

[1097] Step 6:

[1098] The server converts the adjusted image description text into speech and provides it to the user. The input is the adjusted image description text, and the output is audio data. The server uses pyttsx3 to convert the text into speech and provides it to the user through the speaker of the smart glasses.

[1099] In this way, it becomes possible to provide appropriate commentary in real time according to the user's emotions, improving the user experience in physical stores and increasing purchasing motivation.

[1100] Example 2

[1101] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1102] Conventional video analysis systems have had difficulty effectively conveying the content of a video to people with visual impairments or those who cannot watch the video due to time constraints. In addition, they were unable to provide appropriate explanatory text or audio according to the user's emotions, making it difficult to improve the user experience.

[1103] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1104] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, and a means for adjusting the generation of the image commentary text and the conversion of the audio based on the emotion. This allows even a person with a visual impairment or a person who cannot view the video due to time constraints to understand the content of the video through audio, and further allows the server to provide an appropriate commentary text and audio according to the user's emotion.

[1105] "Means for analyzing video" refers to technology for analyzing video data and recognizing specific scenes or objects.

[1106] "Means for extracting characteristic images" refers to technology for extracting images of specific scenes or characters from the analyzed video data.

[1107] The "means for generating image explanatory text" is a technology for generating appropriate explanatory text based on the extracted characteristic images.

[1108] "Means for converting image commentary text into audio" refers to technology for converting the generated commentary text into audio data.

[1109] "Means for recognizing a user's emotions" refers to technology for recognizing emotions from a user's voice and facial expressions.

[1110] The "means for adjusting the generation of image commentary text and the conversion of audio based on emotions" is a technology for adjusting the content of generated commentary text and audio based on the recognized emotions of a user.

[1111] "Machine learning" is a technology in which algorithms automatically learn patterns based on data and make predictions and classifications.

[1112] A "generative AI model" is an artificial intelligence model that generates natural language text based on input data.

[1113] This invention is a system that analyzes video, extracts characteristic images, generates image commentary text, and converts it into audio. It also includes a function to recognize the user's emotions and adjust the commentary text and audio based on those emotions.

[1114] First, the server receives the video data of the TV program. This process uses a streaming protocol (e.g. RTSP) or a file transfer protocol (e.g. FTP). Specifically, the server downloads the video data from the specified URL.

[1115] The server then analyzes the received video data using deep learning frameworks such as TensorFlow and PyTorch. The server splits the video data into frames and performs object detection and scene recognition on each frame.

[1116] From the analyzed video, the server extracts images of characteristic scenes and characters. This process uses image processing libraries such as OpenCV. The server extracts frames that contain specific scenes and characters based on the object detection results.

[1117] Based on the extracted feature images, the server generates image description text. A generative AI model such as GPT-3 is used for this process. The server inputs the feature images, gives a prompt to the generative AI model, and generates appropriate description text. For example, text such as "The main character is running in this scene" is generated.

[1118] The generated image description text is converted into speech by the server. For this process, speech synthesis services such as Google Text-to-Speech (TTS) and Amazon Polly are used. The server inputs the description text into the TTS engine and generates an audio file.

[1119] The device recognizes emotions from the user's voice and facial expressions. This process uses Microsoft Azure's Emotion API and Google Cloud's Vision API. The device analyzes the user's camera footage and microphone audio in real time and classifies emotions.

[1120] The server adjusts the generation of image description text and speech conversion based on information from the emotion engine. Specifically, if the user expresses the emotion of joy, the server generates positive description text with priority. For example, text such as "A scene in which the protagonist wins" is generated. On the other hand, if the user expresses the emotion of sadness, the server generates description text related to sadness with priority.

[1121] Examples

[1122] If the user is watching the first episode of a drama, the server receives the video using the RTSP protocol, divides it into frames, and analyzes it. As a result of the analysis, scenes in which the main character is running and scenes in which the enemy character appears are detected.

[1123] For the extracted scenes, the server generates explanatory text such as "In this scene, the main character is running." At this time, a prompt such as "Please generate an explanation for the scene where the main character is running" is input to the generative AI model.

[1124] When a user expresses joy, the device analyzes the camera footage and microphone audio, and the emotion engine recognizes this as "joy." Based on this information, the server generates positive explanatory text, such as "a scene in which the protagonist wins."

[1125] In this way, even if a user has a visual impairment, he or she can understand the contents of a television program through audio.

[1126] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1127] Step 1:

[1128] The server receives video data of a TV program. The input is the video data from a specified URL, and the output is the video data stored in the server. In concrete terms, the server downloads the video data using the RTSP protocol.

[1129] Step 2:

[1130] The server analyzes the received video data. The input is the stored video data, and the output is the analyzed feature information (e.g., scene and character detection results). Specifically, the server uses TensorFlow and PyTorch to divide the video data into frames, and performs object detection and scene recognition on each frame.

[1131] Step 3:

[1132] The server extracts feature images from the analyzed video. The input is the analyzed feature information, and the output is the extracted feature images. Specifically, the server uses OpenCV to extract frames that contain specific scenes or characters.

[1133] Step 4:

[1134] The server generates image description text based on the extracted feature images. The input is the extracted feature images, and the output is the generated image description text. In concrete terms, the server inputs a prompt sentence into a generative AI model such as GPT-3 to generate appropriate description text. For example, text such as "The main character is running in this scene" is generated.

[1135] Step 5:

[1136] The server converts the generated image description text into audio. The input is the generated image description text, and the output is the generated audio file. Specifically, the server converts the description text into audio data using Google Text-to-Speech (TTS) or Amazon Polly.

[1137] Step 6:

[1138] The device recognizes emotions from the user's voice and facial expressions. The input is the user's camera footage and microphone audio, and the output is the recognized emotion information. Specifically, the device uses Microsoft Azure's Emotion API and Google Cloud's Vision API to classify emotions in real time.

[1139] Step 7:

[1140] The server adjusts the generation of image commentary text and the conversion of voice based on information from the emotion engine. The input is the recognized emotion information, and the output is the adjusted commentary text and voice. In concrete terms, when the user shows an emotion of joy, the server preferentially generates a positive commentary text. For example, text such as "a scene in which the protagonist wins" is generated. When the user shows an emotion of sadness, the server preferentially generates a commentary text related to sadness.

[1141] (Application example 2)

[1142] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1143] There is a demand for systems that enable people with visual impairments or those who cannot watch videos due to time constraints to understand the content of video content. Another challenge is how to improve the user experience by providing appropriate explanatory text and audio according to the user's emotions.

[1144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1145] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for recognizing a user's emotion, and a means for adjusting the generation of the image commentary text and the conversion of the audio based on the user's emotion. This allows even a person with a visual impairment or a person with time constraints to understand the contents of the video content through audio, and further allows the server to provide an appropriate commentary and audio according to the user's emotion.

[1146] A "means for analyzing video" is a technology that receives video data as input, analyzes its contents, and identifies characteristic scenes and characters.

[1147] "Means for extracting characteristic images" refers to technology for selecting images of specific scenes or characters from the analyzed video data.

[1148] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[1149] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data and providing it to the user.

[1150] "Means for recognizing a user's emotions" refers to technology for detecting emotions from the user's voice, facial expressions, etc., and identifying those emotions.

[1151] "Means for adjusting the generation of image commentary text and the conversion of audio based on user emotions" refers to technology for adjusting the content and tone of generated commentary text and audio in response to the recognized user emotions.

[1152] A system for implementing this invention includes means for analyzing video, means for extracting characteristic images, means for generating image description text, means for converting the image description text into speech, means for recognizing a user's emotions, and means for adjusting the generation of image description text and the speech conversion based on the user's emotions.

[1153] System Configuration

[1154] Hardware

[1155] Server: Performs video analysis, feature image extraction, image commentary text generation, voice conversion, and emotion recognition.

[1156] Device: The device used by the user, such as a smartphone, smart glasses, or a head-mounted display.

[1157] Input devices: Devices such as cameras and microphones for capturing the user's voice and facial expressions.

[1158] software

[1159] Video Analysis: Analyze video data using OpenCV.

[1160] Feature Image Extraction: Extracting characteristic scenes and characters using TensorFlow.

[1161] Image description text generation: Use the Transformers library to generate description text based on the extracted feature images.

[1162] Text-to-speech: gTTS is used to convert the generated commentary text into speech.

[1163] Emotion Recognition: Use the emotion_recognition library to recognize emotions from the user's voice and facial expressions.

[1164] Processing flow

[1165] 1. Video Analysis: The server uses OpenCV to analyze the video data frame by frame and identify characteristic scenes and characters.

[1166] 2. Feature Image Extraction: The server uses TensorFlow to extract feature scene and character images from the analyzed frames.

[1167] 3. Image description text generation: The server uses the Transformers library to generate description text based on the extracted feature images.

[1168] 4. Voice conversion: The server uses gTTS to convert the generated commentary text into voice data and sends it to the terminal.

[1169] 5. Emotion recognition: The server uses the emotion_recognition library to recognize emotions from the user's voice and facial expressions.

[1170] 6. Adjustment: The server adjusts the content and tone of the generated commentary text and audio based on the recognized user emotions.

[1171] Examples

[1172] For example, if a user feels "sad," the emotion recognition function detects that emotion, and the commentary text generation function preferentially generates "scenes in which the protagonist is sad." This allows the user to receive commentary that matches their emotions.

[1173] Examples of prompt statements

[1174] "Recognize the user's emotions and generate commentary text based on those emotions. For example, if the user is sad, generate commentary on scenes related to sadness with priority."

[1175] In this way, it is possible to provide more appropriate commentary and audio according to the user's emotions.

[1176] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1177] Step 1:

[1178] The server receives video data as input and uses OpenCV to analyze the video frame by frame. Specifically, it divides the video data into frames and analyzes the content of each frame to identify characteristic scenes and characters. The input is the video data, and the output is the analyzed frame data.

[1179] Step 2:

[1180] The server receives the analyzed frame data as input and uses TensorFlow to extract images of characteristic scenes and characters. Specifically, it selects images with specific characteristics from each frame and extracts them as feature images. The input is the analyzed frame data, and the output is the feature images.

[1181] Step 3:

[1182] The server receives the extracted feature images as input and generates image description text using the Transformers library. Specifically, it generates text that describes the contents of the feature images based on the feature images. The input is the feature images, and the output is the image description text.

[1183] Step 4:

[1184] The server receives the generated image description text as input and converts it into voice data using gTTS. Specifically, it converts the text data into voice data and generates an audio file to provide to the user. The input is the image description text and the output is the voice data.

[1185] Step 5:

[1186] The server receives the user's voice and facial expression as input and recognizes emotions using the emotion_recognition library. Specifically, it analyzes the user's voice and facial expression data and identifies the emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion data.

[1187] Step 6:

[1188] The server receives the recognized emotion data as input and adjusts the content and tone of the generated image commentary text and audio. Specifically, the server changes the content of the commentary text and the tone of the audio according to the user's emotion to provide more appropriate commentary and audio. The input is the recognized emotion data, and the output is the adjusted commentary text and audio data.

[1189] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1190] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1191] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1192] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1193] [Fourth embodiment]

[1194] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1195] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1196] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1197] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1198] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1199] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1200] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1201] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1202] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1203] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1204] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1205] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1206] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1207] "Example 1"

[1208] The system of the present invention includes a means for analyzing a video, a means for extracting a characteristic image, a means for generating an image commentary text, and a means for converting the video into audio. The video analysis means uses artificial intelligence to extract a characteristic image from the video. The extracted characteristic image is converted into commentary text by the image commentary text generation means. The commentary text is converted into audio by the audio output means and output to the user.

[1209] "Example 2"

[1210] As a specific example, consider the analysis of a television program video. The video analysis means uses artificial intelligence to extract images of characteristic scenes and characters from the television program video. The extracted characteristic images are converted by the image commentary text generation means into commentary text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears." The commentary text is converted into audio by the audio output means, so that even people with visual impairments or those who cannot watch the video due to time constraints can understand the content of the television program just by listening to the audio.

[1211] The process flow of each embodiment will be described below.

[1212] "Example 1"

[1213] Step 1: The video analysis means analyzes the video. Artificial intelligence is used in this analysis, and characteristic images are extracted from the video.

[1214] Step 2: The extracted feature images are converted into explanatory text by an image explanatory text generating means.

[1215] Step 3: The generated commentary text is converted into speech by the speech output means.

[1216] Step 4: Finally, the converted audio is output to the user.

[1217] "Example 2"

[1218] Step 1: The television program footage is analyzed using a video analysis tool. Artificial intelligence is used in this analysis, and images of characteristic scenes and characters are extracted from the footage.

[1219] Step 2: The extracted feature images are converted into explanatory text, such as "a scene in which the main character is running" or "a scene in which an enemy character appears," by an image explanatory text generating means.

[1220] Step 3: The generated commentary text is converted into speech by the speech output means.

[1221] Step 4: Finally, the converted audio is output to people with visual impairments or who are unable to watch the video due to time constraints, enabling them to understand the content of the TV program just by listening to the audio.

[1222] Example 1

[1223] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1224] Conventional video analysis systems have problems in that they do not efficiently extract characteristic images from a video, generate explanatory text based on the extracted images, and then convert the text into audio. In addition, they lack a means for users to quickly and accurately understand the content of the video.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1226] In this invention, the server includes a means for analyzing the video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for transmitting the audio data to the user's terminal, thereby enabling the content of the video to be provided to the user quickly and accurately.

[1227] "Means for analyzing video" refers to technology for analyzing input video data and detecting characteristic elements within the video.

[1228] "Means for extracting characteristic images" refers to a technology that identifies important objects and scenes from the analyzed video data and extracts them as images.

[1229] The "means for generating image explanatory text" is a technique for generating text explaining the contents of an extracted characteristic image based on the extracted characteristic image.

[1230] "Means for converting image description text into audio" refers to technology for converting the generated text into audio data.

[1231] The "means for transmitting voice data to a user's terminal" refers to a technique for transmitting the generated voice data to a user's terminal so that the user can play back the voice.

[1232] A "machine learning model" is an algorithm or framework that learns from data and performs pattern recognition and predictions.

[1233] A "natural language processing model" is an algorithm or framework for analyzing, understanding, and generating text data.

[1234] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1235] The present invention is a system for analyzing a video, extracting characteristic images, generating image commentary text, converting the text into audio, and providing the text to a user. A specific embodiment of this system will be described below.

[1236] Video Analysis Methods

[1237] The server uses machine learning models to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the input video data. The video data is uploaded to the server by the user via their device. The server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame. This identifies important objects in each frame.

[1238] Feature image extraction method

[1239] The server extracts characteristic images from the data obtained by the video analysis means. The extracted images are temporarily stored for use in subsequent processing. Specifically, frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) are selected from the output of the object detection model, and these are stored as characteristic images.

[1240] Image description text generation method

[1241] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. The generated text is saved to be provided to the user. Specifically, the feature images are used as input to send a prompt to the generative AI model. For example, using the prompt "What is in this image?", the explanatory text "There is a red car in this image" is obtained from the model.

[1242] Audio conversion means

[1243] The server converts the generated explanatory text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the explanatory text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data.

[1244] Audio data output means

[1245] The server sends the generated audio data to the user's device. The user can play the audio through the device and listen to the commentary. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button on the smartphone app to listen to the audio commentary.

[1246] Examples of concrete examples and prompts

[1247] Examples

[1248] A user uploads a video taken in a park to the system. The server analyzes the video and extracts images of playground equipment and people in the park. It then generates explanatory text for each extracted image, such as "Children are playing in the park," converts this into audio, and provides it to the user.

[1249] Examples of prompt statements

[1250] "Analyze the video footage taken in the park, extract characteristic images, and generate explanatory text. Then convert the generated text into audio and output it."

[1251] In this way, the user can easily understand the contents of the video through audio.

[1252] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1253] Step 1:

[1254] A user uploads video data captured using a device to a server. Specifically, the user opens the application on the device and taps the "Video Upload" button. A file selection screen is displayed, and when the user selects a video file and presses the "Send" button, the video data is sent to the server via an HTTP request. The input is the user's video data, and the output is the video data saved on the server.

[1255] Step 2:

[1256] The server receives the uploaded video data and starts analyzing it using video analysis methods. Specifically, the server splits the video data into frames and applies an object detection model (e.g., YOLO) to each frame, which identifies important objects in each frame. The input is the video data stored on the server, and the output is the analyzed object data for each frame.

[1257] Step 3:

[1258] The server extracts feature images from the data obtained by the video analysis means. Specifically, it selects frames containing objects that meet certain conditions (e.g., confidence scores above a certain level) from the output of the object detection model, and saves them as feature images. The input is the analyzed object data for each frame, and the output is the extracted feature images.

[1259] Step 4:

[1260] The server generates explanatory text using a generative AI model (e.g., GPT-3) based on the extracted feature images. Specifically, it sends a prompt sentence to the generative AI model using the feature images as input. For example, it uses the prompt sentence "What is in this image?" to obtain explanatory text from the model, such as "There is a red car in this image." The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[1261] Step 5:

[1262] The server converts the generated commentary text into speech. Specifically, it uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. The server sends the commentary text to the speech synthesis service and obtains the speech data. For example, it sends the text "This image shows a red car" and receives the corresponding speech data. The input is the generated commentary text, and the output is the generated speech data.

[1263] Step 6:

[1264] The server sends the generated audio data to the user's device. Specifically, the server sends the audio data to the user's device as an HTTP response. The user taps the "play" button in the application on the device to listen to the audio commentary. The input is the generated audio data, and the output is the audio played on the user's device.

[1265] (Application example 1)

[1266] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1267] Autonomous vehicles are required to grasp the surrounding situation in real time and provide appropriate information to the driver and passengers. However, the current system does not have sufficient accuracy in image analysis and information provision, and improving safety and convenience is an issue. In particular, there is a lack of means to quickly and accurately recognize important information such as road signs and pedestrians and to convey that information to the driver and passengers in an easy-to-understand manner.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1269] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, a means for analyzing a video around the autonomous vehicle in real time and extracting a characteristic image, and a means for generating commentary text for the characteristic image, converting it into audio, and providing it. This makes it possible to grasp the situation around the autonomous vehicle in real time and provide information to the driver and passengers quickly and accurately.

[1270] "Means for analyzing video" refers to a device or software for analyzing video data acquired from a video input device such as a camera and extracting specific information.

[1271] A "means for extracting characteristic images" is a device or software for selecting images having specific characteristics from the analyzed video data.

[1272] The "means for generating image explanatory text" refers to a device or software for generating explanatory text regarding the image based on the extracted characteristic image.

[1273] "Means for converting image description text into speech" refers to a device or software for converting the generated image description text into speech data and providing it to the user.

[1274] "Means for analyzing images around an autonomous vehicle in real time" refers to a device or software that analyzes images obtained from cameras or other devices mounted on an autonomous vehicle in real time and grasps the surrounding situation.

[1275] A "means for extracting characteristic images" is a device or software for selecting images having specific important information from video data analyzed in real time.

[1276] "Means for generating explanatory text and converting it into audio for providing" refers to a device or software for generating explanatory text based on the extracted characteristic images, converting the text into audio for providing to the user.

[1277] A system for implementing this invention is configured as follows: First, a server has a means for analyzing video, and analyzes video data acquired from a video input device such as a camera. Next, a means for extracting characteristic images from the analyzed video data is used to select images with specific characteristics. Based on the extracted characteristic images, a means for generating image commentary text generates commentary. Finally, a means for converting the generated image commentary text into audio is used to convert it into audio data, which is provided to the user.

[1278] Specifically, the server uses the following hardware and software: A camera is used for video capture, and an artificial intelligence model such as TensorFlow is used for video analysis. A generative AI model is used to extract feature images and generate explanatory text. Google Text-to-Speech API (gTTS) is used for voice conversion.

[1279] For example, in an autonomous vehicle, a camera captures images of the surroundings in real time. The server analyzes the images and extracts characteristic images such as road signs and pedestrians. For the extracted images, a generative AI model is used to generate explanatory text. The generated explanatory text is converted to speech using gTTS and provided to the driver and passengers through the speaker.

[1280] As a concrete example, the following prompt sentence is input to the generative AI model:

[1281] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[1282] By inputting this prompt into the generative AI model, the following explanatory text is generated:

[1283] "There is a pedestrian in this image. The pedestrian is about to cross the road."

[1284] This explanatory text can be converted into audio and provided to drivers and passengers, improving the safety and convenience of self-driving vehicles.

[1285] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1286] Step 1:

[1287] The server captures video data from the camera in real time. The input is the video data from the camera and the output is the captured video frames. This video data is used for subsequent analysis processing.

[1288] Step 2:

[1289] The server analyzes the captured video frames using an artificial intelligence model such as TensorFlow. The input is the video frame, and the output is the analysis result, which is feature data. This analysis extracts important features in the video (e.g., road signs, pedestrians, etc.).

[1290] Step 3:

[1291] The server extracts a feature image from the analysis result. The input is the feature data as the analysis result, and the output is the feature image. This feature image is used for subsequent generation of explanatory text.

[1292] Step 4:

[1293] The server generates image description text using a generative AI model based on the feature image. The input is the feature image, and the output is the description text. This description text is an explanatory text about the feature image.

[1294] Step 5:

[1295] The server converts the generated commentary text into voice data using Google Text-to-Speech API (gTTS). The input is the commentary text, and the output is the voice data. This voice data is provided to the user.

[1296] Step 6:

[1297] The server provides the converted voice data to the driver or passengers through the speaker. The input is voice data and the output is voice output from the speaker. This allows the driver or passengers to obtain information about the surrounding situation in real time.

[1298] Example 2

[1299] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1300] Conventional video analysis systems have made it difficult for people with visual impairments or those who cannot watch the video due to time constraints to understand the content of television programs. In addition, there was a lack of efficient means to extract characteristic scenes and characters from the video, generate text to appropriately explain them, and then convert that text into speech.

[1301] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1302] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into a voice, a means for dividing the video into frames, a means for transmitting the characteristic image to a generative AI model, a means for inputting a prompt sentence to the generative AI model, a means for receiving a commentary text returned from the generative AI model, a means for transmitting the commentary text to a voice synthesis software, a means for receiving voice data returned from the voice synthesis software, and a means for providing the voice data to a user. This allows even a person with a visual impairment or a person who cannot watch a video due to time constraints to understand the contents of a TV program just by listening to the sound.

[1303] "Means for analyzing video" refers to technology or devices for receiving video data and analyzing its contents.

[1304] A "means for extracting characteristic images" is a technology or device that selects important images, such as specific scenes or characters, from the analyzed video data.

[1305] The "means for generating image explanatory text" refers to a technique or device that generates text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[1306] "Means for converting image description text into speech" refers to technology or devices that convert the generated text into speech data.

[1307] A "means for dividing an image into frames" is a technique or device for dividing continuous image data into individual still images (frames).

[1308] "Means for sending to a generative AI model" refers to technology or equipment for sending the extracted feature images to a generative AI model.

[1309] "Means for inputting a prompt sentence into a generative AI model" refers to technology or equipment for inputting an explanatory sentence (prompt sentence) regarding the content of an image into a generative AI model.

[1310] "Means for receiving explanatory text returned from a generative AI model" refers to technology or devices that receive explanatory text returned from a generative AI model.

[1311] The "means for transmitting to speech synthesis software" refers to a technique or device for transmitting the generated commentary text to speech synthesis software.

[1312] The "means for receiving voice data returned from the voice synthesis software" refers to a technique or device for receiving voice data returned from the voice synthesis software.

[1313] The "means for providing voice data to a user" refers to a technique or device for providing received voice data to a user.

[1314] The present invention relates to a system that analyzes video, extracts characteristic scenes and characters, generates text explaining them, and converts the text into speech. A specific embodiment of this system will be described below.

[1315] The server receives the video of a television program. The received video is first divided into frames. This division is performed using common video processing techniques. Next, the server uses a deep learning framework (e.g., TensorFlow, PyTorch) to analyze each frame. This analysis involves computer vision techniques such as object detection and face recognition. For example, the server detects specific characters or changes in the scene within a frame and extracts these images.

[1316] The extracted feature image is sent to a generative AI model (e.g. GPT-4). The server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." The generative AI model generates explanatory text based on this prompt sentence and returns it to the server.

[1317] The generated commentary text is sent to speech synthesis software (e.g. Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into audio data and returns it to the server. Finally, the server provides this audio data to the user. The user can understand the content of the TV program by listening to the audio.

[1318] As a concrete example, let's say there is a scene in a TV program where the protagonist confronts an enemy character. The server analyzes this scene and extracts characteristic images. Next, the server has a generative AI model (e.g. GPT-4) generate explanatory text such as "A scene where the protagonist confronts an enemy character." Finally, the server converts this explanatory text into audio using speech synthesis software (e.g. Google Text-to-Speech API) and provides it to the user. By listening to this audio, the user can understand the content of the scene without watching the video.

[1319] This system enables people with visual impairments or who are unable to watch the video due to time constraints to understand the content of television programs simply by listening to the audio.

[1320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1321] Step 1:

[1322] The server receives the video of a television program. The received video is first divided into frames. Specifically, a process is performed to convert the video data into a series of still images (frames). This process divides the video data into individual frames. The input is the video data of a television program, and the output is the divided frame images.

[1323] Step 2:

[1324] The server uses a deep learning framework (e.g. TensorFlow, PyTorch) to analyze each frame. Computer vision techniques such as object detection and face recognition are used for the analysis. Specifically, it detects specific characters and scene changes in each frame. Through this process, images of characteristic scenes and characters are extracted. The input is the segmented frame images, and the output is the extracted feature images.

[1325] Step 3:

[1326] The server sends the extracted feature images to a generative AI model (e.g., GPT-4). Next, the server inputs a prompt sentence to the generative AI model. For example, the prompt sentence is, "Look at this image and generate explanatory text for the scene. The image contains a scene in which the protagonist is confronting an enemy character." This process returns explanatory text from the generative AI model. The input is the feature image and the prompt sentence, and the output is the generated explanatory text.

[1327] Step 4:

[1328] The server sends the generated commentary text to speech synthesis software (e.g., Google Text-to-Speech API, Amazon Polly). The speech synthesis software converts the text into voice data and returns it to the server. Specifically, it performs a process to convert text data into voice data. This process generates voice data. The input is the commentary text, and the output is voice data.

[1329] Step 5:

[1330] The server provides the generated voice data to the user. Specifically, it transmits the voice data to the user's terminal so that the user can play it. This process allows the user to understand the contents of the TV program by listening to the voice. The input is the voice data, and the output is the voice provided to the user.

[1331] (Application example 2)

[1332] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1333] People with visual impairments and those who do not have time to watch videos have difficulty understanding the content of television programs and movies. In addition, there is a lack of technology that can analyze videos, extract characteristic scenes and characters, and provide them as audio, so there are limited ways to provide information to these users. Furthermore, there is a demand for a system that can perform video analysis and audio output in real time using devices such as smartphones and smart glasses.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1335] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image description text based on the characteristic image, a means for converting the image description text into audio, and a means for providing the audio to a user. This allows visually impaired people or people who do not have time to watch videos to understand the contents of a television program or movie through audio. In addition, real-time video analysis and audio output can be realized by using an application installed on a smartphone, smart glasses, a head-mounted display, or a robot. Furthermore, by including a means for generating a prompt sentence using a generative AI model, more advanced video analysis and information provision are possible.

[1336] A "means for analyzing video" refers to technology or equipment that receives video data as input and analyzes its contents.

[1337] "Means for extracting characteristic images" refers to technology or devices for extracting important images, such as specific scenes or characters, from the analyzed video data.

[1338] The "means for generating image explanatory text" refers to a technique or device for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[1339] "Means for converting image description text into audio" refers to technology or devices for converting the generated image description text into audio data.

[1340] "Means for providing audio to a user" refers to technology or devices for transmitting converted audio data to a user.

[1341] "Artificial intelligence" is a technology that uses techniques such as machine learning and deep learning to analyze data and make judgments and predictions.

[1342] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[1343] "Smart glasses" are a device shaped like glasses that has a built-in display, camera, sensor, etc. and can display information and collect data.

[1344] A "head-mounted display" is a display device worn on the head, and refers to technology and devices for displaying information within the user's field of vision.

[1345] A "robot" is a mechanical device that operates autonomously or remotely to perform specific tasks.

[1346] A "generative AI model" is a model that uses artificial intelligence to generate new information and content from data.

[1347] A "prompt" is text that is entered into a generative AI model as instructions or questions that serve as guidelines for the model to generate appropriate output.

[1348] A system for implementing the present invention includes a means for analyzing a video, a means for extracting a feature image, a means for generating an image description text, a means for converting the image description text into speech, and a means for providing the speech to a user. The system further includes a means for generating a prompt sentence using a generative AI model.

[1349] System Configuration

[1350] Hardware:

[1351] server

[1352] Smartphone

[1353] Smart Glasses

[1354] Head-mounted display

[1355] robot

[1356] software:

[1357] OpenCV: A library for video analysis

[1358] pytesseract: A library for extracting text from images

[1359] gTTS: A text-to-speech library

[1360] mpg321: a tool for playing audio files

[1361] Processing flow

[1362] The server first analyzes the video. For video analysis, OpenCV is used to process the video data frame by frame. Next, pytesseract is used to extract feature images from the analyzed frames. Based on the extracted feature images, image description text is generated. A generative AI model is used to generate this text.

[1363] The generated image description text is converted to audio using gTTS, and the converted audio data is provided to the user using mpg321. This allows people with visual impairments or those who do not have time to watch the video to understand the content of TV programs and movies through audio.

[1364] Examples

[1365] For example, a user launches the "Audio Vision" app on a smartphone and inputs a video of a TV program. The app analyzes the video and generates text such as "a scene where the main character is running" or "a scene where an enemy character appears." The generated text is converted into audio and provided to the user.

[1366] Examples of prompt statements

[1367] An example of a prompt for a generative AI model is:

[1368] Analyze TV program footage and extract characteristic scenes and characters. I would like to develop an application that converts the extracted information into text and audio and provides it to users. Specifically, detect the following scenes:

[1369] Scene where the main character is running

[1370] Scenes in which enemy characters appear

[1371] In this way, a specific embodiment for carrying out the invention has been shown, which enables even those with visual impairments or those who do not have time to watch a video to understand the content of the video through audio.

[1372] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1373] Step 1:

[1374] The server receives video data input from the user. The input video data is video files of TV shows or movies. The server divides the video data into frames and prepares them for analysis.

[1375] Step 2:

[1376] The server uses OpenCV to analyze the video data. Specifically, it processes each frame sequentially and detects characteristic scenes and characters in the video. The input is the video frame, and the output is the detection result of characteristic scenes and characters.

[1377] Step 3:

[1378] The server uses pytesseract to extract text information from the detected characteristic scenes and characters. The input is the images of the characteristic scenes and characters, and the output is the extracted text information.

[1379] Step 4:

[1380] The server uses a generative AI model to generate image commentary text based on the extracted text information. The input is the extracted text information, and the output is the image commentary text. In concrete terms, the server inputs a prompt sentence into the generative AI model to generate appropriate commentary text.

[1381] Step 5:

[1382] The server converts the generated image description text into audio data using gTTS. The input is the image description text, and the output is audio data. Specifically, the server converts the text into an audio file and saves it.

[1383] Step 6:

[1384] The server uses mpg321 to provide the generated audio data to the user. The input is the audio data, and the output is the audio provided to the user. The specific operation is to play the audio file and let the user listen to it.

[1385] Step 7:

[1386] The user receives the provided audio using a smartphone, smart glasses, a head-mounted display, or a robot. This allows the user to understand the content of the video by audio. The input is the audio provided by the server, and the output is the user's understanding.

[1387] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1388] "Example 1"

[1389] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[1390] "Example 2"

[1391] In one embodiment of the present invention, in addition to the video analysis means, the feature image extraction means, the image commentary text generation means, and the voice conversion means, an emotion engine that recognizes the user's emotions is included. This emotion engine recognizes emotions from the user's voice and facial expressions, and adjusts the generation of image commentary text and the conversion of voice based on the emotions. For example, if the user shows an emotion of joy, the emotion engine preferentially generates image commentary text related to joy. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image commentary text related to sadness. This provides more appropriate image commentary text and voice according to the user's emotions, improving the user's experience.

[1392] The process flow of each embodiment will be described below.

[1393] "Example 1"

[1394] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[1395] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[1396] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[1397] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[1398] "Example 2"

[1399] Step 1: The video analysis means analyzes the video and extracts characteristic images.

[1400] Step 2: Generate image description text based on the image extracted by the feature image extraction means. Step 3: At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions.

[1401] Step 4: The emotion engine adjusts the generation of image description text and the conversion of voice based on the emotion recognized. For example, if the user shows an emotion of happiness, the emotion engine preferentially generates image description text related to happiness. Also, if the user shows an emotion of sadness, the emotion engine preferentially generates image description text related to sadness.

[1402] Step 5: The text generated by the image description text generating means is converted into speech by the speech converting means and provided to the user. This provides more appropriate image description text and speech in accordance with the user's emotions, improving the user's experience.

[1403] Example 1

[1404] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1405] Conventional video analysis systems were able to extract characteristic images from video and convert them into text or audio, but they were unable to provide appropriate information according to the user's emotions. As a result, the user experience was uniform, and flexible responses according to individual emotions were required.

[1406] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1407] In this invention, the server includes a means for analyzing a video, a means for extracting a characteristic image from the video, a means for generating an image commentary text based on the characteristic image, a means for converting the image commentary text into audio, and a means for recognizing a user's emotion and adjusting the generation of the image commentary text and the conversion of the audio based on the emotion, thereby making it possible to provide appropriate information according to the user's emotion.

[1408] A "means for analyzing video" is a device or program for receiving video data as input and analyzing the content thereof.

[1409] The "means for extracting characteristic images" refers to a device or program for selecting images having specific characteristics from the analyzed video data.

[1410] The "means for generating image explanatory text" refers to a device or program for generating text explaining the content of an extracted characteristic image based on the extracted characteristic image.

[1411] The "means for converting image description text into audio" refers to a device or program for converting the generated image description text into audio data.

[1412] The "means for recognizing the user's emotions" refers to a device or program for analyzing and recognizing the emotions from the user's voice, facial expressions, etc.

[1413] "Means for adjusting the generation of image description text or the conversion of audio based on emotions" refers to a device or program for adjusting the content of the generated image description text or audio in accordance with the recognized user's emotions.

[1414] "Machine learning" is a technology in which a computer learns patterns and rules from data and then uses that knowledge to analyze new data.

[1415] A "generative AI model" is a model that uses artificial intelligence to generate data such as text and images.

[1416] This invention is a system that analyzes video, extracts characteristic images, generates explanatory text based on the images, and converts the text into audio. It also includes a function to recognize the user's emotions and adjust the content of the explanatory text and audio based on the emotions.

[1417] The server uses machine learning to analyze the video. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to analyze the video data. For example, it acquires video data from a surveillance camera in real time and detects people and objects in the video.

[1418] The server then uses image processing libraries such as OpenCV to extract characteristic images from the analyzed video data, for example extracting the face of a particular person in the video and storing their facial features in a database.

[1419] The server uses a generative AI model to generate image description text based on the extracted feature images. Specifically, it uses generative AI models such as GPT-3 and BERT to generate text that describes the contents of the image. For example, it generates a description text such as "This image shows a man wearing a blue shirt."

[1420] The generated image description text is converted to speech using a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly. For example, the text "This image shows a man wearing a blue shirt" is converted to speech and output to the user.

[1421] In addition, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's voice and facial expressions to recognize emotions. For example, it detects the emotion of joy from the user's voice.

[1422] Based on the recognized emotion, the server adjusts the image description text and audio content. For example, if the user expresses joy, the server generates a description text such as "This image shows a man in a blue shirt smiling happily" and outputs it as audio.

[1423] Examples

[1424] Suppose a user is viewing security camera footage. The server analyzes the footage and detects that a man wearing a blue shirt is in the image. The server then generates explanatory text based on this information, such as "This image shows a man wearing a blue shirt." The server then converts this text into speech and outputs it to the user. If the user expresses an emotion of joy, the server generates text such as "This image shows a man wearing a blue shirt smiling happily," converts it into speech, and outputs it.

[1425] Examples of prompt statements

[1426] "Analyze surveillance camera footage, extract characteristic images, generate explanatory text, and convert it into audio. Adjust the explanatory text according to the user's emotions."

[1427] In this way, the server performs a series of processes from video analysis to audio output, providing appropriate information to the user.

[1428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1429] Step 1:

[1430] The server receives video data from security cameras and user devices. The input is a real-time video stream, and the output is the video data to be analyzed. For example, if a security camera is monitoring the entrance to an office, the server acquires that video.

[1431] Step 2:

[1432] The server analyzes the received video data. Specifically, it uses deep learning frameworks such as TensorFlow and PyTorch to detect people and objects in the video. The input is the video data acquired in step 1, and the output is information about the detected people and objects. For example, the server identifies the position and movement of people in the video.

[1433] Step 3:

[1434] The server extracts characteristic images from the analyzed video data. It uses image processing libraries such as OpenCV to perform face recognition and object detection. The input is the person and object information obtained in step 2, and the output is a characteristic image. For example, the server extracts the face of a specific person in the video and stores the facial features in a database.

[1435] Step 4:

[1436] The server generates image description text based on the extracted feature images. A generative AI model (e.g., GPT-3 or BERT) is used to generate text that describes the contents of the image. The input is the feature images extracted in step 3, and the output is the image description text. For example, the server generates description text such as "This image shows a man wearing a blue shirt."

[1437] Step 5:

[1438] The server converts the generated image description text into speech. It uses a speech synthesis service such as Google Text-to-Speech (TTS) or Amazon Polly to convert the text into speech. The input is the image description text generated in step 4, and the output is the audio data. For example, the server converts the text "This image shows a man wearing a blue shirt" into speech and outputs it to the user.

[1439] Step 6:

[1440] The server uses an emotion engine to recognize the user's emotion. The emotion engine analyzes the user's voice and facial expression to recognize the emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion information. For example, the server detects the emotion of joy ...

Claims

1. A means for receiving video data; A means for dividing the received video data into frames; A means for extracting a feature image from the divided frames; a means for generating a prompt sentence for instructing generation of an explanatory text for the extracted characteristic image based on the characteristic image; A means for generating explanatory text for the characteristic image using the generated prompt sentence and a generative AI model; means for transmitting the generated commentary text to speech synthesis software; means for receiving voice data from said voice synthesis software; and means for outputting the received audio data.

2. means for identifying an emotion of the user using an emotion engine; The system of claim 1 , wherein the means for generating a prompt sentence generates a prompt sentence that instructs the generation of an explanatory text for the characteristic image based on the identified user's emotion and the extracted characteristic image.

3. The system of claim 2, wherein the means for generating commentary text preferentially generates image commentary text related to joy when the user's emotion is an emotion of joy, and preferentially generates image commentary text related to sadness when the user's emotion is an emotion of sadness.

Images