Multimodal user interface technology

Abstract

The present invention provides a multimodal information processing device provided with a user interface. The device comprises: a text accepting unit that accepts text data; an emotion parameter calculating unit that calculates a prescribed emotion parameter indicating an emotion of a person on the basis of the text data; a presentation information determining unit that determines presentation information to be presented to a user on the basis of the text data and the prescribed emotion parameter; and an output control unit that performs control such that the presentation information is output via the user interface. The text data may be generated from various images.

Description

Multimodal User Interface Technology

[0001] The present invention relates to a multimodal user interface technology, and more particularly to a multimodal information processing device, a multimodal information processing method using the device, and a computer program for executing the method.

[0002] "Multimodal information" is information that is perceived through various modalities (sensory channels such as vision, hearing, touch, taste, and smell) that a person can sense or experience. Multimodal information processing recognizes information that combines multiple modalities, enabling a richer cognitive experience.

[0003] For example, we can recognize text by simply reading it, but we can also simultaneously listen to it being read aloud by a text-to-speech function, which allows us to use both our senses of sight and hearing to facilitate our understanding of the information contained in the text.

[0004] Furthermore, multimodal information processing enables richer interpretation of information by integrating information from different modalities. For example, when a person watches a movie, they combine information from multiple sources, such as music, sound effects, and the actors' facial expressions and movements, in order to understand the story and emotions of the movie. This allows for information and a deeper understanding that cannot be obtained from a single source alone.

[0005] For example, Patent Document 1 listed below discloses a technology for providing accurate and personalized answers that accurately match user needs during intelligent chat between humans and machines, thereby enabling natural intelligent chat between humans and machines, thereby satisfying the user's chat needs and improving the user experience. More specifically, Patent Document 1 discloses a method for intelligent chat between humans and machines using artificial intelligence, including the steps of receiving multimodal input signals including audio signals, image signals, sensor signals, and / or event-driven signals, processing the multimodal input signals to obtain text data, obtaining a user's intention from the text data, obtaining an answer corresponding to the user's intention, converting the answer into a multimodal output signal, and outputting the multimodal output signal.

[0006] Japanese Patent Application Laid-Open No. 2017-010517

[0007] However, the technology described in Patent Document 1 processes multimodal input signals to obtain text data, and obtains the user's intentions from the obtained text data, but does not obtain the emotions of specific related people from the text data, and therefore cannot present information to the user that takes such emotions into consideration.

[0008] Furthermore, the pleasant or tingly sensations that people experience when stimulated by their hearing or vision are known as the Autonomous Sensory Meridian Response (ASMR). Therefore, stimulating sensory channels that people find pleasant is thought to be effective in stimulating and activating people's thinking.

[0009] However, in the technology described in Patent Document 1 above, although audio signals and / or image signals are shown as multimodal output signals, no consideration is given to the manner in which they should be presented to the user, and it cannot be said that this is sufficient to stimulate and activate human thinking.

[0010] Furthermore, although the above-mentioned Patent Document 1 describes acquiring text data by processing multimodal input signals such as image signals, no consideration is given to acquiring text data that will attract people's attention or interest.

[0011] Therefore, the present invention aims to provide a multimodal user interface technology that can attract people's attention and interest through multimodal information provided through multiple modalities, while stimulating, activating, and organizing their thoughts.

[0012] The present invention, which aims to solve the above problems, comprises the following invention-specifying matters and technical features.

[0013] According to one aspect, the present invention provides a multimodal information processing device having a user interface, the multimodal information processing device including: a text accepting unit that accepts text data; an emotion parameter calculation unit that calculates predetermined emotion parameters indicative of human emotions based on the text data; a presentation information determination unit that determines presentation information to be presented to a user based on the text data and the predetermined emotion parameters; and an output control unit that controls output of the presentation information via the user interface.

[0014] Here, the emotion parameter calculation section can calculate the predetermined emotion parameter based on voice data generated based on the text data.

[0015] The multimodal information processing device may further include an emotion AI model that has been subjected to machine learning, with text data as an explanatory variable and voice data that reproduces the voice of a specific person according to emotional expressions based on the personality of the specific person as a target variable.

[0016] Further, the emotion parameter calculation unit may calculate the predetermined emotion parameter based on the voice data that reproduces the voice of the specific person in accordance with emotional expression based on the personality of the specific person, which is generated by the emotion AI model.

[0017] The emotion parameter calculation section can also calculate the predetermined emotion parameter based on a predetermined emotion element reproduced by the voice data.

[0018] Furthermore, the presentation information determination unit may determine, as the presentation information, keywords extracted from the text data and / or information related to the keywords. Here, the keyword is used in a broad sense to include not only a single word but also a phrase or phrase consisting of multiple words.

[0019] The presentation information determination unit may determine a presentation format for the presentation information.

[0020] The presentation information determination unit may generate an animation sequence for an object that is the visual presentation information, and the output control unit may control the object to be animated and displayed on a screen that represents a virtual space on the user interface in accordance with the animation sequence.

[0021] The presentation information determination unit may generate the animation sequence for displaying the objects in animation in order with a time difference that allows the objects to be visually recognized.

[0022] The animation display may also include at least one of a manner in which the object is visually gradually completed, a manner in which the object is visually represented in perspective, and a manner in which the object is visually emphasized.

[0023] Additionally, the visual presentation information may be displayed without being geometrically aligned with one another within the screen.

[0024] The presentation information determination unit may also generate the animation sequence for gradually disappearing the displayed object after a predetermined time has elapsed.

[0025] The presentation information determination unit may further include an audio data generation unit that generates audio data related to the presentation information, and the output control unit may control, based on the audio data, to output audio from a speaker in accordance with the animation display of the object.

[0026] The multimodal information processing device may further include an image data receiving unit that receives input image data, and a text data generating unit that generates the text data from the received image data. The text data generating unit may generate the text data from the received image data using a text AI model that has been subjected to machine learning, with the image data as an explanatory variable and text data consisting of sentences that explain the image data as a target variable.

[0027] Another aspect of the present invention is a method for processing multimodal information, the method including: receiving text data; generating, based on the text data, voice data that reproduces the voice of a specific person in accordance with emotional expressions that are based on the personality of the specific person; calculating, based on the voice data, predetermined emotional parameters that indicate the person's emotions; a presentation information determination unit that determines presentation information to be presented to a user based on the text data and the predetermined emotional parameters; and controlling the presentation information to be output via a predetermined user interface.

[0028] Furthermore, the present invention according to another aspect may be a computer program that causes a computing device to execute the above method under the control of a processor, or a recording medium on which the computer program is non-transitoryly recorded.

[0029] In this disclosure, "means" or "unit" does not simply mean physical means, but also includes cases where all or part of the functions of the means or unit are realized by software. Furthermore, the functions of one means or unit may be realized by two or more physical means, and the functions of two or more means or units may be realized by one physical means.

[0030] Furthermore, in this disclosure, a "system" includes a logical collection of multiple devices (or functional modules that realize specific functions), regardless of whether each device or functional module is physically configured as a single entity or as separate entities.

[0031] According to the present invention, it is possible to attract people's attention and interest while stimulating, activating and organizing their thoughts.

[0032] Other technical features, objects, effects, and advantages of the present invention will become apparent from the following embodiments described with reference to the accompanying drawings.

[0033] FIG. 1 is a diagram illustrating an example of a schematic configuration of a multimodal information processing system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of a multimodal information processing device adapted to a multimodal information processing system according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of the contents of a memory module of a multimodal information processing device according to an embodiment of the present invention. FIG. 4 is a block diagram illustrating an example of a functional configuration model of a multimodal information processing device according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating an example of processing by a multimodal information processing device according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating an example of animation display processing by a multimodal information processing device according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of text data input to a multimodal information processing device according to an embodiment of the present invention. FIG. 8A is a diagram illustrating an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 8B is a diagram illustrating an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 8C is a diagram illustrating an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 8D is a diagram illustrating an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 9A is a diagram illustrating an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. Fig. 9B is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. Fig. 9C is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. Fig. 9D is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. Fig. 10 is a block diagram showing an example of a functional configuration model of a multimodal information processing device according to another embodiment of the present invention. Fig. 11 is a flowchart showing an example of processing by a multimodal information processing device according to an embodiment of the present invention.FIG. 12A is a diagram showing an example of image data input to a multimodal information processing device according to an embodiment of the present invention. FIG. 12B is a diagram showing an example of image data input to a multimodal information processing device according to an embodiment of the present invention. FIG. 12C is a diagram showing an example of image data input to a multimodal information processing device according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of text data generated in a multimodal information processing device according to an embodiment of the present invention. FIG. 14A is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 14B is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 14C is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention. FIG. 14D is a diagram showing an example of a screen displayed on a user interface of a multimodal information processing device according to an embodiment of the present invention.

[0034] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and are not intended to exclude various modifications or applications of techniques not explicitly described below. The present invention can be implemented in various modifications (e.g., by combining the various embodiments) without departing from the spirit of the present invention. Furthermore, in the following description of the drawings, identical or similar parts are denoted by identical or similar reference numerals. The drawings are schematic and do not necessarily correspond to actual dimensions, ratios, etc. Parts in the drawings may have different dimensional relationships or ratios.

[0035] <First Embodiment> This embodiment is characterized in that, in a multimodal information processing system, predetermined emotional parameters are calculated based on given text data, multimodal presentation information to be presented to a user is determined based on the calculated predetermined emotional parameters, and the determined presentation information is output via a user interface.

[0036] (Overall Configuration) Fig. 1 is a diagram illustrating an example of a schematic configuration of a multimodal information processing system according to an embodiment of the present invention. As shown in the figure, the multimodal information processing system 1 of this embodiment may be configured to include, for example, a multimodal information processing device 10 and a cloud computer 20 connected via a communication network. Note that while the figure illustrates a cloud-based multimodal information processing system 1, the system is not limited to this, and the multimodal information processing device 10 may also be configured to function as a standalone device.

[0037] The multimodal information processing device 10 is a computing device operated by a user, and may be, for example, a personal computer such as a notebook computer or a desktop computer, or a tablet computer, but is not limited to these, and may be any device configured to realize the present invention. The multimodal information processing device 10 implements a multimodal information processing program for presenting information including keywords and the like generated based on text data including text sentences to a user through multiple modalities (sensory channels), as will be described later.

[0038] The cloud computer 20 is a computing device that responds to requests provided by the multimodal information processing device 10. For example, the cloud computer 20 generates predetermined "information" using AI in response to a prompt request provided by the multimodal information processing device 10 and returns the generated information to the multimodal information processing device 10. The cloud computer 20 may be configured to include, for example, various databases 21 and an artificial intelligence (AI) model 22 that has undergone predetermined machine learning. Note that the cloud computer 20 may be any computer accessible to the multimodal information processing device 10, and may utilize a known AI cloud service on the Internet. In the present disclosure, the artificial intelligence model 22 may be configured to include an emotion AI model and a presentation information determination model, as described below.

[0039] (Hardware Configuration of Multimodal Information Processing Device) Fig. 2 is a diagram showing an example of the hardware configuration of a multimodal information processing device adapted to a multimodal information processing system according to an embodiment of the present invention. Of the various hardware resources (components) that make up the multimodal information processing device 10, those particularly relevant to the present invention are shown in the diagram.

[0040] That is, as shown in the figure, the multimodal information processing device 10 can typically be configured to include one or more processor modules 11, a chipset 12, a memory module 13, an I / O controller 14, various peripheral interfaces 15, and various input / output devices 16 (such as a display 16a).

[0041] The processor module 11 includes, but is not limited to, for example, a processor (processor core), a microcontroller, a digital signal processor, and / or a combination thereof. The chipset 12 is composed of a circuit that integrates a bridge for a bus that connects the processor module 11, the memory module 13, the I / O controller 14, etc., as well as other components necessary to configure the multimodal information processing device 10. The chipset 12 is controlled, for example, by the processor module 11. In this disclosure, the processor module 11 may be simply referred to as a "processor."

[0042] The memory module 13 is a primary storage device made up of volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, flash memory, etc.), and / or a combination thereof, available for use by the processor module 11. The memory module 13 may also include video memory (VRAM) for the display 16a. In this disclosure, the memory module 13 may be simply referred to as "memory."

[0043] As shown in FIG. 3 , the memory module 13 stores various software resources, such as device drivers, an operating system (OS) program, one or more application programs, and various data. In the present disclosure, the application program includes a multimodal information processing program executed on the OS of the multimodal information processing device 10 under the control of the processor module 11. The multimodal information processing program, under the control of the processor module 11, enables the multimodal information processing device 10 to present information including keywords generated based on text sentences to a user in multiple modalities. Some processes in the multimodal information processing program may be executed as different threads or processes on the processor module 11. The various data include, for example, text data and image data (still image data and video image data). The various data are stored as data files in folders logically configured on the memory module 13.

[0044] The I / O controller 14 is a circuit that controls data transfer between various peripheral interfaces 15 (e.g., an I / O interface 15a and a communication interface 15b). The I / O interface 15a controls the operation of the input / output devices 16 that make up the user interface unit 130. The communication interface 15b is a circuit that enables computer communication with the outside via the IP network 12.

[0045] The input / output device 16 includes, but is not limited to, a display 16 a, a speaker 16 b, a microphone 16 c, etc. For example, the input / output device 16 may include a keyboard, a mouse, a camera, a scanner, a vibrator, an actuator, a lighting device, etc.

[0046] The display 16a is a display device for displaying displayable multimedia data such as text data, image data (e.g., still image data and moving image data), graphic data, etc. The display 16a may be configured with a projector and an external large screen such as those used for presentations.

[0047] The speaker 16b outputs audio or sound based on an audio signal generated by an audio driver (not shown). Under the control of the processor module 11, the speaker 16b reads out text data, outputs music, sound effects, etc.

[0048] The microphone 36c collects, for example, the user's voice, generates voice data, and outputs it. The voice data generated by the microphone 36c is stored in the memory module 13 as a voice file, and is also converted into text data by voice analysis under the control of the processor module 11.

[0049] (Functional Configuration Model of a Multimodal Information Processing Device) Figure 4 is a block diagram showing an example of a functional configuration model of a multimodal information processing device according to an embodiment of the present invention. In the figure, the multimodal information processing device 10 is shown as a functional configuration model including, for example, a control unit 110, a storage unit 120, a user interface unit 130, and a communication interface unit 140. Such a functional configuration model is realized on the multimodal information processing device 10 by executing a multimodal information processing program under the control of the processor module 11 and cooperating with the various software and / or hardware resources described above.

[0050] The control unit 110 comprehensively controls and executes various processes (tasks) in the multimodal information processing device 10. In this example, the control unit 110 is configured to include a text data receiving unit 111, an emotion parameter calculation unit 112, a presentation information determination unit 113, and an output control unit 114. Details of these functional configurations of the control unit 110 will be described later.

[0051] The storage unit 120 stores the various programs and data described above. For example, in connection with the execution of the multimodal information processing program, the storage unit 120 stores text data, audio data (voice data, sound effects, background music), image data (still image data and video data), screen data, and the like, for example, in a predetermined file format. The text data is data that has been converted into text based on acquired voice data. The audio data includes voice data converted from text data, as described below. The screen data is data for representing the screen displayed on the user interface unit 130 as a virtual space under the control of the control unit 110. The screen data may include objects (keywords, etc.) and their coordinate information, etc.

[0052] The user interface unit 130 is a user interface for providing a multimodal environment to the user. Under the control of the control unit 110, the user interface unit 130 accepts various input operations from the user and collects the user's voice using the microphone 36c. In response to the input, the user interface unit 130 displays a screen representing a virtual space on the display 16a and outputs audio from the speaker 16b.

[0053] The communication interface unit 140 controls communication so that the multimodal information processing device 10 can access the search server 20 via the communication network 10. For example, the communication interface unit 330, under the control of the control unit 110, controls the transmission of a request based on the generated prompt to the cloud computer 20. Also, for example, the communication interface unit 140 receives a generation result (product) transmitted from the cloud computer 20 in response to the request, and hands it over to the control unit 110.

[0054] Next, the functional configuration of the control unit 110 will be described in detail.

[0055] The text data receiving unit 111 receives text data input from an external source. Alternatively, the text data receiving unit 111 may read the text data from the storage unit 120. The text data typically includes sentences with a certain meaning. A script for a presentation is an example of text data. The text data may be a document file that includes character codes and is configured in a predetermined format, or may be a document file that includes character codes obtained by reading a document printed on a paper medium using an OCR and is configured in a predetermined format. The text data receiving unit 111 passes the received text data to the emotion parameter calculation unit 112 and the presentation information determination unit 113.

[0056] The emotion parameter calculation unit 112 calculates emotion parameters based on the provided text data. The emotion parameters are multidimensional parameters that indicate human emotions that may be expressed based on the context of the text data. The emotion parameters include, for example, feature quantities for each parameter. The emotion parameter calculation unit 112 is configured to include, for example, a trained emotion artificial intelligence (AI) model. Alternatively, the trained emotion AI model may be built on the cloud computer 20. For example, the emotion parameter calculation unit 112 performs text mining processing on the text data as pre-processing for the trained emotion AI model. Text mining divides sentences based on the text data into "smallest linguistically meaningful units" such as words, phrases, or parts of speech such as morphemes, and extracts feature quantities related to, for example, frequency of occurrence, correlation between words and phrases, trends and / or patterns of change over time, etc. The emotion AI model generates voice data that sounds as if a specific person (whether real or virtual) is reading the text data with emotional expressions based on their personality. In other words, the emotion AI model is a machine learning model that has undergone machine learning using text data as an explanatory variable and voice data that reproduces the voice of a specific person according to emotional expressions based on the personality of the specific person as a target variable. For example, an emotion AI model that has undergone machine learning to be able to reproduce Steve Jobs' reading will generate voice data that reproduces Steve Jobs' speaking style based on given text data. Note that the voice data referred to here does not have to be configured to be recognizable by humans, as long as it is recognizable by the emotion parameter calculation unit 112.

[0057] The emotion parameter calculation unit 112 calculates predetermined emotion parameters based on various emotion elements reproduced by the voice data generated by the emotion AI model. The emotion elements include, but are not limited to, voice timbre (frequency), pitch, pauses, tone, intonation, accent, and accent. The emotion parameter calculation unit 112 passes the calculated predetermined emotion parameters to the presentation information determination unit 113.

[0058] The presentation information determination unit 113 determines information to be presented to a user (hereinafter referred to as "presentation information") based on given text data and predetermined emotional parameters. The presentation information includes, but is not limited to, visual presentation information and auditory presentation information. That is, the presentation information is multimodal information. As an example, the presentation information may be a keyword extracted from the text data. Note that, in this disclosure, the term "keyword" is used in a broad sense to include not only a single word but also a phrase consisting of multiple words. As another example, the presentation information may be information related to the keyword (e.g., related keywords, phrases, images, and / or audio, etc.). If the presentation information determination unit 113 determines, for example, that a specific keyword is important in the context based on the given text data and predetermined emotional parameters, the presentation information determination unit 113 may determine the specific keyword as information to be presented. Furthermore, depending on the emotion, context, etc., the presentation information determination unit 113 may determine image data or audio data, which are other modalities, as information to be presented. The image data or audio may be obtained, for example, by an Internet search.

[0059] The presentation information determination unit 113 may also determine the presentation information in accordance with a request and response from the cloud computer 20. That is, the presentation information determination unit 113 creates a prompt request for requesting presentation information based on given text data and predetermined emotion parameters, and transmits the created prompt request to the cloud computer 20. The cloud computer 20 determines keywords and the like as described above using a presentation information determination model based on the transmitted prompt request, and transmits the determined keywords to the presentation information determination unit 113. The presentation information determination unit 113 determines the response result transmitted from the cloud computer 20 as the presentation information.

[0060] The presentation information determination unit 113 also determines an optimal presentation format for the presentation information. For example, the presentation information determination unit 113 includes an animation generation unit 1131 that generates an animation sequence for visual presentation information (e.g., objects such as keywords and images). The animation generation unit 1131 generates an animation sequence for displaying objects such as extracted keywords with a predetermined animation and associates the animation sequence with the object. The predetermined animation may be, for example, randomly selected from an animation library.

[0061] One example of animation is a presentation mode in which objects representing visual presentation information appear in sequence at arbitrary positions on the screen with a visually perceptible time difference. The arbitrary positions may be determined, for example, randomly. Therefore, the appearance positions of the objects do not need to be geometrically aligned with each other. Another example of an animation display in which an object appears on the screen is a mode in which the object gradually completes from a visually empty or blurred state. Another example is a presentation mode in which an object appears to be moving in a virtual space. Presentation modes that appear to be moving in a virtual space include, for example, a presentation mode in which the object appears to be visually approaching using perspective. Yet another example is a presentation mode in which the object is visually emphasized. Animation presentation modes are not limited to those listed here and may be any mode that provides some visual display effect. Presentation modes may include various display effects such as display timing and display time. These various presentation modes may stimulate the user more than when objects are simply displayed in a vague, inorganic manner.

[0062] The animation generation unit 1131 may also include a physics calculation engine (not shown). The physics calculation engine simulates the behavior of objects (e.g., search result images) in the virtual space by performing physical calculations according to predetermined physical parameters. Examples of predetermined physical parameters include mass, velocity, external force, and fluid velocity. As an example, the physics calculation engine simulates the behavior of presented information in the virtual space by likening it to the behavior of sparks floating in the air from a bonfire. As another example, the physics calculation engine simulates the behavior of keywords in the virtual space by likening it to the behavior of bubbles rising from water. As a result, instead of or in addition to the above-described animation display, the animation generation unit 1131 enables the presented information on the screen to be animated, as if floating in the virtual space. Such animated keyword display makes the user feel comfortable, mentally stabilized, and focused, while stimulating and activating their thinking.

[0063] The physics engine can also simulate the fluctuation of presented information floating in the virtual space to be a 1 / f fluctuation. 1 / f fluctuation is a fluctuation in which the power spectral density is inversely proportional to the frequency f (where f > 0). By displaying an animation that expresses the behavior of keywords with such 1 / f fluctuation, users can feel comfortable, stabilize their mind, increase their concentration, and stimulate and activate their thinking by resonating with their own biological rhythms.

[0064] Furthermore, when a predetermined time (e.g., 10 seconds) has elapsed since the animation display of the presented information, the animation generation unit 1131 generates an animation sequence that recreates a scene in which the keyword appears to gradually or naturally disappear (fade out). This prevents the keyword on the screen from appearing and disappearing, thereby preventing it from disrupting the user's thought process.

[0065] The presentation information determination unit 113 also includes an audio generation unit 1132 that generates audio data for auditory presentation information. The audio generation unit 1132 generates predetermined audio data (e.g., sound effect data) according to, for example, the display time (animation display time) indicated by an animation sequence associated with visual presentation information (object) and associates the predetermined audio data with the presentation information. For example, the predetermined audio data may be randomly selected from an audio library. Alternatively, the predetermined audio data may be voice data of a keyword being read aloud, which is the presentation information. The audio generation unit 1132 may also generate a prompt based on the presentation information, provide the generated prompt to the AI ​​model of the cloud computer 20 via a network, and obtain audio data returned from the AI ​​model in response to the prompt.

[0066] The output control unit 114 controls the output of the multimodal presentation information determined by the presentation information determination unit 113 via the user interface unit 130. The output control unit 114 may include, for example, a display control unit 1141 that controls the display 16a to display a screen including a keyword, and an audio control unit 1142 that controls the output of sound effects from the speaker 16b. That is, the display control unit 1141 controls the display 16a to display visual presentation information (objects) in a predetermined animation sequence on a screen representing a virtual space, and the audio control unit 1142 controls the output of auditory presentation information (audio) from the speaker 16b in synchronization with the animation sequence. As a result, the visual presentation information is visually emphasized and animatedly displayed on the screen in synchronization with the auditory presentation information. Although not shown, the output control unit 114 may also control lighting equipment to change the lighting (brightness, color, blinking, directionality, type, etc.) in accordance with the animation displayed on the display 16a.

[0067] 5 is a flowchart showing an example of processing by the multimodal information processing device according to one embodiment of the present invention. This processing is realized by the multimodal information processing device 10 executing a multimodal information processing program under the control of the processor module 11 and cooperating with various software and / or hardware resources.

[0068] That is, when the control unit 110 of the multimodal information processing device 10 starts executing the multimodal information processing program, it monitors whether or not text data has been input from the outside (S501).

[0069] If the control unit 110 determines that text data has been input (Yes in S501), the control unit 110 then calculates a predetermined emotion parameter based on the input text data (S502). The emotion parameter is a parameter that indicates a human emotion that may be expressed based on the context of the text data. The control unit 110 generates voice data using, for example, a trained emotion AI model, and further calculates the predetermined emotion parameter based on the generated voice data.

[0070] Next, the control unit 110 determines presentation information to be presented to the user based on the input text data and the calculated predetermined emotion parameters (S503). The control unit 110 determines, for example, keywords extracted from the text data and other information related thereto (for example, related keywords, images, audio, and / or sound effects) as presentation information.

[0071] Next, the control unit 110 generates an animation for the visual presentation information (S504). For example, the control unit 110 selects or generates an animation sequence for displaying the extracted keyword as presentation information or an object such as an image related thereto with a predetermined animation, and associates the animation sequence with the object. One example is an animation sequence in which the keyword gradually fades in from a randomly determined position on the screen and then gradually fades out.

[0072] Next, the control unit 110 generates audio for the auditory presentation information (S505). For example, the control unit 110 selects or generates predetermined audio data according to the display time indicated by the animation sequence assigned to the object, and associates the audio data with the object.

[0073] Next, the control unit 110 performs display control so that a screen including visual presentation information is displayed on the display 16a, and also performs audio control so that auditory presentation information is output from the speaker 16b (S506).

[0074] 6 is a flowchart showing an example of animation display processing by a multimodal information processing device according to an embodiment of the present invention. This processing shows details of the animation display processing (S504) shown in FIG.

[0075] As shown in the figure, the presentation information determination unit 113 simulates the behavior of the keyword in the virtual space by performing physical calculations based on predetermined physical parameters (e.g., mass, speed, external force, air current, etc.) using a physics calculation engine (S601). For example, the physics calculation engine simulates the keyword as if it were floating in the virtual space. In this case, the physics calculation engine may perform physical calculations so that the fluctuation of the search result image floating in the virtual space is 1 / f fluctuation.

[0076] Next, the presentation information determination unit 113 performs control so that the search result image is displayed as an animation on the screen representing the virtual space according to the result of the physical calculation (S601). That is, the presentation information determination unit 113 generates screen data according to the result of the physical calculation and outputs this to the output control unit 116. As a result, for example, keywords are displayed as if they are floating on the screen representing the virtual space on the user interface unit 130.

[0077] Next, the presentation information determination unit 113 determines whether the animation display of the keyword satisfies a predetermined end condition (S603). The predetermined end condition is, for example, whether a predetermined time (e.g., 10 seconds) has elapsed since the start of the animation display of the keyword, but is not limited to this.

[0078] If the presentation information determination unit 113 determines that the animation display of the keyword does not satisfy the predetermined termination condition (No in S603), it continues to output screen data for the animation display by the above simulation. On the other hand, if the presentation information determination unit 113 determines that the animation display of the keyword satisfies the predetermined termination condition (Yes in S603), the presentation information determination unit 113 generates screen data for an animation display in which the keyword appears to fade out naturally (S604).

[0079] In this way, the multimodal information processing device 10 calculates predetermined emotion parameters based on the received text data, and further determines the multimodal information to be presented based on the text data and the predetermined emotion parameters. The multimodal information processing device 10 then controls the presentation information to be output from the user interface. This allows the multimodal information processing device 10 to attract a person's attention and interest while stimulating, activating, and organizing their thoughts.

[0080] (Specific Example) Next, an example of an operation of the multimodal information processing device 10 by a user will be described. Here, it is assumed that the multimodal information processing device 10 receives text data such as that shown in Fig. 7. Such text data may be created by a person, or may be generated by a so-called generation AI in accordance with a prompt request.

[0081] First, the multimodal information processing device 10 calculates emotion parameters based on the received text data. That is, the multimodal information processing device 10 uses an emotion AI model to generate voice data including emotion expressions based on the text data. In this example, the emotion AI model has been subjected to machine learning so as to be able to reproduce the speech of Steve Jobs. Next, the multimodal information processing device 10 calculates emotion parameters based on the generated voice data.

[0082] Next, the multimodal information processing device 10 determines presentation information based on the text data and audio data. In this example, the multimodal information processing device 10 extracts keywords, such as "Macintosh computer" and "developing," from audio data reproducing Steve Jobs' speech while taking into account the context. The multimodal information processing device 10 then acquires image data related to these keywords from an image library or the Internet. The multimodal information processing device 10 then displays these objects, such as keywords and images, as animations on the screen, as shown in, for example, FIGS. 8A to 8D . That is, as shown in FIGS. 8A to 8D , the keywords "Macintosh computer" and "developing" are each animated, with each character quickly flowing in from the right side of the screen and disappearing to the left, while images related to each keyword are animated and gradually appear and disappear. At this time, the multimodal information processing device 10 plays sound effects, such as background music, to accompany the animation.

[0083] The multimodal information processing device 10 also extracts keywords such as "Steve Jobs," "Steve Wozniak," and "Ronald Wayne" from the audio data while taking into account the context, and then acquires image data of these people from an image library or the Internet. Next, the multimodal information processing device 10 displays these objects, such as keywords and images, as animations on the screen, and plays sound effects such as background music in sync with the animation, as shown in Figures 9A to 9D.

[0084] The multimodal information processing device 10 may display animation and play sound effects in synchronization with the playback of audio data that reproduces Steve Jobs' reading.

[0085] (Effects, advantages, etc.) As described above, according to this embodiment, the multimodal information processing device 10 calculates emotional parameters from given text data, determines multimodal presentation information for the user based on the text data and the emotional parameters, and presents this to the user. This enables the user to interpret information more richly and deeply, while attracting the user's attention and interest and encouraging the user to stimulate, activate, and organize their thoughts.

[0086] In particular, according to the present embodiment, the multimodal information processing device 10 uses an emotion AI model to generate voice data from given text data and calculates emotion parameters based on the generated voice data, making it possible to calculate emotion parameters based on voice data that includes the emotional expressions of a specific person, which cannot be fully expressed by text data alone.

[0087] Furthermore, according to this embodiment, the multimodal information processing device 10 determines visual and / or auditory information to be presented to the user based on the emotion parameters, and further displays animation and outputs audio, thereby enabling the user to interpret information more richly and deeply, while attracting the user's attention and interest and encouraging the user to stimulate, activate, and organize their thoughts.

[0088] Furthermore, in animation display, the multimodal information processing device 10 displays each of the objects representing visually presented information in various ways, such as displaying them in sequence with a visually recognizable time difference, or randomly at any position on the screen, thereby enabling the user to interpret information more richly and deeply, while attracting the user's attention and interest and encouraging the user to stimulate, activate, and organize their thoughts.

[0089] Second Embodiment This embodiment is a modification of the first embodiment, and is characterized in that in a multimodal information processing system, text data is generated from given image data, predetermined emotional parameters are calculated based on the generated text data, multimodal presentation information to be presented to a user is determined based on the calculated predetermined emotional parameters, and the determined presentation information is output via a user interface.

[0090] (Functional Configuration Model of a Multimodal Information Processing Device) Figure 10 is a block diagram showing an example of a functional configuration model of a multimodal information processing device according to one embodiment of the present invention. The functional configuration model of a multimodal information processing device 10' shown in this figure differs from that shown in Figure 4 in that the control unit 110 further includes an image data receiving unit 115 and a text data generating unit 116. The multimodal information processing device 10' of this embodiment will be described below, focusing on the differences from the first embodiment and omitting other differences as appropriate.

[0091] The image data receiving unit 115 receives image data input from an external device. Alternatively, the image data receiving unit 115 may read image data from the storage unit 120. The image data is a data file configured in a predetermined image format (e.g., JPEG, PNG, etc.). The image data is typically, but is not limited to, color data. Furthermore, the image data may be not only still image data but also moving image data with audio. The image data receiving unit 115 passes the received image data to the text data generating unit 116.

[0092] The text data generation unit 116 generates text data based on image data. Specifically, the text data generation unit 116 creates a prompt request that instructs the generation of text data related to the image based on the image represented by the given image data, and provides this to the text generation AI to acquire the text data. In other words, the text AI model is a machine learning model that has undergone machine learning using image data as an explanatory variable and text data consisting of sentences describing the image data as a target variable. The text generation AI may be configured as part of the text data generation unit 116 or may be built on the cloud computer 20. The text data generation unit 116 passes the generated text data to the text data generation unit 111. Note that in this example, the text data generation unit 116 and the text data reception unit 111 are configured as separate functional components, but this is not limited thereto; the text data generation unit 116 may be configured as a part of the text data 111 as a functional component.

[0093] (Processing by Multimodal Information Processing Device) Fig. 11 is a flowchart showing an example of processing by a multimodal information processing device according to an embodiment of the present invention. The flowchart shown in Fig. 11 is the same as that shown in Fig. 5, except for processing steps S1101 and S1102.

[0094] As shown in the figure, when the control unit 110 of the multimodal information processing device 10' starts executing the multimodal information processing program, it monitors whether or not image data has been input from the outside (S1101).

[0095] If the control unit 110 determines that image data has been input (Yes in S1101), the control unit 110 then generates text data based on the input image data (S1102). As described above, the text data is generated by providing the generation AI with a prompt request created based on the image data.

[0096] Next, the control unit 110 calculates predetermined emotion parameters based on the generated text data (S1103), and determines presentation information to be presented to the user based on the generated text data and the calculated predetermined emotion parameters (S1104). Subsequently, the control unit 110 generates animation for the visual presentation information (S1105), and further generates audio for the auditory presentation information (S1106). The control unit 110 then controls display so that a screen including the visual presentation information is displayed on the display 16a, and controls audio so that the auditory presentation information is output from the speaker 16b (S1107). (Specific Example 2) Next, an example will be described in which the multimodal information processing device 10′ receives image data, generates text data from the received image data, and provides multimodal presentation information. Here, it is assumed that the multimodal information processing device 10′ receives image data representing each image shown in FIGS. 12A to 12C.

[0097] The multimodal information processing device 10' issues a prompt request to, for example, a generation AI instructing it to generate text data related to the image from the image, and as a result, the multimodal information processing device 10' generates text data such as that shown in Figure 13, generated by the generation AI.

[0098] Next, the multimodal information processing device 10′ calculates emotion parameters based on the received text data, as described above. In this example, the emotion AI model is assumed to have undergone machine learning to reproduce the reading of a famous reciter. Subsequently, the multimodal information processing device 10 calculates emotion parameters based on the generated voice data.

[0099] Next, the multimodal information processing device 10' determines the information to be presented based on the text data and audio data. For example, the multimodal information processing device 10 extracts several keywords, such as "technology," "drone," "imagination," "exploration," and "development," based on the text data and audio data, and further acquires image data related to these keywords from an image library or the Internet. The multimodal information processing device 10' then displays these objects, such as keywords and images, in an animated manner on the screen, while playing audio data based on the generated text data, as shown in, for example, Figures 14A to 14D. Specifically, in Figure 14A, an image is introduced as an introduction, and then various keywords and images, such as "technology," "drone," and "imagination," appear sequentially in Figures 14B and 14C. Finally, in Figure 14D, the keywords and images previously described are introduced. At this time, the multimodal information processing device 10 plays sound effects, such as background music, to accompany the animated display.

[0100] (Effects, Advantages, etc.) As described above, according to this embodiment, the same effects and advantages as those of the first embodiment can be achieved. In particular, according to this embodiment, the multimodal information processing device 10′ generates text data related to images based on several images, so that even if there is no text data such as a script, as long as there are images, it becomes possible to determine presentation information according to emotion parameters and output it via a user interface.

[0101] The above-described embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to these embodiments. The present invention can be embodied in various forms without departing from the spirit of the present invention.

[0102] For example, the multimodal information processing device 10 may be configured to acquire specific keywords based on a presentation script or materials prepared in advance by a user. Alternatively, the multimodal information processing device 10 may be configured to convert mechanically read speech data into text data, convert this into speech data including emotional expressions according to an emotion AI model, and then acquire specific keywords.

[0103] Furthermore, in the methods disclosed herein, steps, operations, or functions may be performed in parallel or in a different order, provided that the results do not contradict. The steps, operations, and functions described are provided merely as examples, and some of the steps, operations, and functions may be omitted or combined into one, or other steps, operations, or functions may be added, without departing from the spirit of the invention.

[0104] Furthermore, although various embodiments are disclosed in this specification, specific features (technical matters) in one embodiment can be added to or substituted for specific features in other embodiments, with appropriate modifications, and such forms are also included in the spirit of the present invention.

[0105] 1...Multimodal information processing system 10, 10'...Multimodal information processing device 11...Processor module 12...Chipset 13...Memory module 14...I / O controller 15...Peripheral interface 15a...I / O interface 15b...Communication interface 16...Input / output device 16a...Touch panel 16b...Speaker 16c...Microphone 110...Control unit 111...Text data reception unit 112...Emotion parameter calculation unit 113...Presentation information determination unit 1131...Animation generation unit 1132...Audio generation unit 114...Output control unit 1141...Display control unit 1142...Audio control unit 115...Image data reception unit 116...Text data generation unit 120...Memory unit 130...User interface unit 140...Communication interface unit 20...Cloud computer 21...Database 22...Artificial intelligence model (machine learning model)

Claims

1. A multimodal information processing device with a user interface, comprising: a text accepting unit that accepts text data; an emotion parameter calculation unit that calculates predetermined emotion parameters indicating human emotions based on the text data; a presentation information determination unit that determines presentation information to be presented to a user based on the text data and the predetermined emotion parameters; and an output control unit that controls the presentation information to be output via the user interface.

2. The multimodal information processing device according to claim 1, wherein the emotion parameter calculation unit calculates the predetermined emotion parameter based on voice data generated based on the text data.

3. The multimodal information processing device according to claim 2, further comprising an emotion AI model that has been subjected to machine learning using text data as an explanatory variable and voice data that reproduces the voice of a specific person according to emotional expressions based on the personality of the specific person as a target variable.

4. The multimodal information processing device according to claim 3, wherein the emotion parameter calculation unit calculates the predetermined emotion parameter based on the voice data that reproduces the voice of the specific person in accordance with the emotion expression based on the personality of the specific person, which is generated by the emotion AI model.

5. The multimodal information processing device according to claim 4, wherein the emotion parameter calculation unit calculates the predetermined emotion parameter based on emotion elements reproduced by the voice data.

6. The multimodal information processing device according to claim 1, wherein the presentation information determination unit determines keywords extracted from the text data and / or information related to the keywords as the presentation information.

7. The multimodal information processing device according to claim 6, wherein the presentation information determination unit determines a presentation form for the presentation information.

8. A multimodal information processing device as described in claim 7, wherein the presentation information determination unit generates an animation sequence for an object which is the visual presentation information, and the output control unit controls the object to be animatedly displayed on a screen which represents a virtual space on the user interface in accordance with the animation sequence.

9. The multimodal information processing device according to claim 8, wherein the presentation information determination unit generates the animation sequence for displaying the objects in animation in sequence with a time difference that allows the objects to be visually recognized.

10. The multimodal information processing device according to claim 9, wherein the animation display includes at least one of a manner in which the object is visually gradually completed, a manner in which the object is visually represented in perspective, and a manner in which the object is visually emphasized.

11. The multimodal information processing device according to claim 10, wherein the visual presentation information is displayed without being geometrically aligned with one another on the screen.

12. The multimodal information processing device according to claim 10, wherein the presentation information determination unit generates the animation sequence for causing the displayed object to gradually disappear after a predetermined time has elapsed.

13. A multimodal information processing device as described in claim 8, wherein the presentation information determination unit further includes an audio data generation unit that generates audio data related to the presentation information, and the output control unit controls the output of audio from a speaker in accordance with the animation display of the object based on the audio data.

14. A multimodal information processing device as described in claim 1, further comprising: an image data accepting unit that accepts input image data; and a text data generating unit that generates the text data from the accepted image data, wherein the text data generating unit generates the text data from the accepted image data using a text AI model that has been subjected to machine learning, with image data as an explanatory variable and text data consisting of sentences describing the image data as a target variable.

15. A method for processing multimodal information, comprising: accepting input text data; generating voice data based on the text data, which reproduces the voice of a specific person in accordance with emotional expression based on the personality of the specific person; calculating predetermined emotional parameters indicative of human emotions based on the voice data; a presentation information determination unit that determines presentation information to be presented to a user based on the text data and the predetermined emotional parameters; and controlling so that the presentation information is output via a predetermined user interface.

16. A computer program causing a computing device to execute a method for processing multimodal information under the control of a processor, the method comprising: accepting input text data; generating voice data based on the text data, the voice data reproducing the voice of a specific person according to emotional expression based on the personality of the specific person; calculating predetermined emotional parameters indicating human emotions based on the voice data; a presentation information determination unit that determines presentation information to be presented to a user based on the text data and the predetermined emotional parameters; and controlling the presentation information to be output via a predetermined user interface.

Images