Processing device, processing program, and processing method

Abstract

To provide a user-friendly processing apparatus for a user, such as a receiver, a processing program, and a processing method.SOLUTION: A processing apparatus is equipped with at least one processor, the at least one processor being configured to: receive, from a sender terminal device, speech information input in association with each of a plurality of objects included in content generated in the sender terminal device, via a communication interface; select at least one object of the multiple objects via an input interface; and execute a process for outputting the speech information associated with the selected at least one object when outputting the speech information via an output interface.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure relates to a processing device, a processing program, and a processing method for outputting speech information associated with a selected object. [Background technology]

[0002] Conventionally, video distribution systems via the Internet have been known. For example, Patent Document 1 describes a video distribution system including a recruitment management unit that notifies a user terminal of recruitment requirements including video distribution conditions and acquires posted videos from the user terminal, a video analysis unit that analyzes whether the posted videos are available for distribution and sets posted videos that are available for distribution as videos to be distributed, and a video distribution management unit that distributes the videos to be distributed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180967 Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, in consideration of the above-described techniques, an object of the present disclosure is to provide, through various embodiments, a processing device, a processing program, and a processing method that are more user-friendly for users such as recipients. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a processing device having at least one processor, wherein the at least one processor is configured to receive, from the sender terminal device via a communication interface, speech information input in association with each of a plurality of objects included in content generated in the sender terminal device, select at least one object from the plurality of objects via the input interface, and, when outputting the speech information via an output interface, execute processing to output the speech information associated with the selected at least one object.

[0006] According to one aspect of the present disclosure, there is provided a processing program that causes a computer having at least one processor to receive, via a communication interface, speech information input from a sender terminal device in association with each of a plurality of objects included in content generated in the sender terminal device, select at least one of the plurality of objects via the input interface, and, when outputting the speech information via an output interface, execute a process for outputting the speech information associated with the selected at least one object, in the computer having at least one processor.

[0007] According to one aspect of the present disclosure, there is provided a processing method executed by at least one processor in a computer having the at least one processor, the processing method including the steps of receiving, from the sender terminal device via a communication interface, speech information input in association with each of a plurality of objects included in content generated in the sender terminal device, selecting at least one object from the plurality of objects via the input interface, and outputting the speech information associated with the selected at least one object when outputting the speech information via an output interface. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a processing device, a processing program, and a processing method that are more user-friendly for users such as recipients.

[0009] It should be noted that the above effects are merely illustrative for the sake of convenience and are not limiting. In addition to or instead of the above effects, any effect described in this disclosure or an effect obvious to a person skilled in the art may be achieved. [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1A is a diagram showing an overview of processing related to a processing system 1 according to an embodiment of the present disclosure. [Figure 1B] FIG. 1B is a block diagram showing the configuration of a processing system 1 according to an embodiment of the present disclosure. [Figure 2A] FIG. 2A is a block diagram showing a configuration of a server device 100 according to an embodiment of the present disclosure. [Figure 2B] FIG. 2B is a block diagram showing a configuration of a terminal device 200 according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram schematically illustrating information transmitted from the sender terminal device 200-1 as transmission information according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing a processing sequence executed by the processing system 1 according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a diagram showing a processing flow executed in the receiver terminal device 200-2 according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram showing an example of a screen output on the sender terminal device 200-1 according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram showing an example of a screen output on the receiver terminal device 200-2 according to an embodiment of the present disclosure. [Figure 8A]FIG. 8A is a diagram showing a processing sequence executed by the processing system 1 according to an embodiment of the present disclosure. [Figure 8B] FIG. 8B is a diagram showing a processing sequence executed by the processing system 1 according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram showing an overview of processing related to the processing system 1 according to the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1. Overview of Processing System 1 The processing system 1 according to the present disclosure is used for outputting speech information associated with a desired object of a recipient for content transmitted from a sender. As an example, the processing system 1 is used for video content transmitted from a sender so that a recipient can select one of the character objects appearing in the video content and output only the sound associated with the character object.

[0012] Here, FIG. 1A is a diagram illustrating an overview of processing related to a processing system 1 according to an embodiment of the present disclosure. Specifically, FIG. 1A illustrates an example of processing in video content distribution performed using the processing system 1. According to FIG. 1A, a user who is a sender uses an available sender terminal device to transmit video content featuring objects, character A and character B, to a receiver terminal device of a user who is a receiver, via a server device. For example, the sender himself / herself inputs both audio information, which is voice A of character A, and audio information, which is voice B of character B, into the video content (typically, it is assumed that the sender plays both character A and character B in the video content).

[0013] Then, a user (recipient) receives and plays video content from the sender terminal device via the server device using an available receiver terminal device. However, depending on, for example, one's preferences or circumstances, the receiver may need to output only one of the voices of character A and character B from the video content being played, or conversely, mute the other. In this case, if the receiver simply changes the audio volume or audio settings of the playback application, the receiver can only mute all of the audio transmitted from the sender terminal device, i.e., both character A's voice A and character B's voice B, or continue to output both. However, with the processing system 1, identification information for identifying each audio information is assigned in advance to the audio information of character A's voice A and character B's voice B, making it possible for the receiver to output only the audio desired by the receiver and limit, or mute, the output of the other audio. In the example of FIG. 1A, only audio A is output, and the output of audio B is limited, or muted.

[0014] Such a processing system 1 is typically used in video content in which character A and character B appear, but can also be used in other video content such as video conferences and telephone conferences. In such cases, similar to the above, it is possible to restrict the output of any of the audio by specifying the character or user identification information of the user participating in the video conference or telephone conference.

[0015] In this way, in the processing system 1, speech information (e.g., the speech information of audio A and the speech information of audio B) associated with each of a plurality of objects (e.g., character A and character B) included in content (e.g., video content) is input to the sender terminal device. Meanwhile, at least one object (e.g., character A) from among the plurality of objects is selected in the receiver terminal device. Then, the receiver terminal device allows output of speech information (e.g., the speech information of audio A) associated with the selected at least one object (e.g., character A), while restricting output of speech information (e.g., the speech information of audio B) associated with objects other than the selected object (e.g., character B).

[0016] In this disclosure, the terms "sender" and "receiver" are merely used to distinguish between those who send content and those who receive content. In other words, even if a person is described as a "sender," they may also be a "receiver" if they receive content from another person, and even if a person is described as a "receiver," they may also be a "sender" if they send content to another person. Furthermore, senders and receivers are not limited to individuals, but may also be organizations such as companies or groups. While the description mainly focuses on cases where the sender generates the content themselves, the sender and the person who generates the content may be different parties. In this case, even if a person who generates content only generates the content but does not actually generate it, if the generated content is sent by someone, they are included in the "sender."

[0017] Furthermore, in this disclosure, the terms "sender terminal device" and "recipient terminal device" are merely names used to distinguish between a terminal device that transmits content and a terminal device that receives content. In other words, even if a device is described as a sender terminal device, it may become a recipient terminal device if it receives content from another terminal device, and even if a device is described as a recipient terminal device, it may become a sender terminal device if it transmits content to another terminal device.

[0018] In this disclosure, "content" refers to a collection of electronic information transmitted and received via a communication network. Examples of such content include video content, music content, game content, publication content, chat content, social networking content, web content, and combinations thereof. Among these, the processing system 1 is preferably used for video content that includes at least image information containing multiple character objects as characters and audio information associated with each character object. Note that in this disclosure, video content includes not only content distributed through video content distribution sites, but also, for example, video conferencing content (including content transmitted and received only via audio with the camera function turned off), telephone conference content, and digital advertising content such as digital signage. Furthermore, unless otherwise specified, the following description will use video content as an example of content, but the content is not limited to video content.

[0019] In this disclosure, "object" refers to data included in content or a means for operating and inputting the data. Examples of such objects include character objects, structure objects, decorative objects, text objects, image objects, GUI objects, and combinations thereof. Among these, the processing system 1 is preferably used for character objects that appear in video content (e.g., Character A and Character B in FIG. 1A). Note that, unless otherwise specified, the following description will be given using character objects as an example of an object, but the object is not limited to a character object.

[0020] In this disclosure, a "processing device" refers to any of the devices constituting the processing system 1, and may be any of a server device, a sender terminal device, and a receiver terminal device. Furthermore, the processing device is not limited to any of these devices alone, but may be a combination of multiple devices that can distribute and process the processing performed in the processing device. Furthermore, a "processing program" and a "processing method" refer to a program and a method executed in the processing device.

[0021] 2. Configuration of Processing System 1 1B is a block diagram showing a configuration of a processing system 1 according to an embodiment of the present disclosure. According to FIG. 1B, the processing system 1 includes a server device 100 for processing content (e.g., video content), a sender terminal device 200-1 for transmitting the content, and a receiver terminal device 200-2 for receiving the content, which are communicatively connected via a communication network.

[0022] In FIG. 1B, sender terminal device 200-1 and receiver terminal device 200-2 are each shown as a single device, but naturally, each may include multiple devices.

[0023] 1B shows a single server device 100, multiple server devices or other devices may be combined to distribute processing and storage. In this case, the server device 100 may also include a combination of multiple server devices or other devices.

[0024] 3. Configuration of Server Device 100 Fig. 2A is a block diagram showing the configuration of a server device 100 according to an embodiment of the present disclosure. The server device 100 does not need to include all of the components shown in Fig. 2A, and may have a configuration in which some components are omitted, or may include other components. The server device 100 does not need to include the components shown in Fig. 2A in a single housing, and the components and processes of the server device 100 may be distributed across multiple devices.

[0025] 2A, the server device 100 includes a processor 111 configured with a CPU or the like, a memory 112 including RAM, ROM, non-volatile memory, an HDD or the like, and a communication interface 113. These components are electrically connected to each other via control lines and data lines.

[0026] The processor 111 is configured with a CPU (microcomputer) and functions as a control unit for controlling other connected components based on various programs stored in the memory 112. The processor 111 reads out from the memory 112 a program for executing an application according to the present disclosure and a program for executing an OS, and executes these programs. Specifically, the processor 111 executes, based on the programs stored in the memory 112, "a process of receiving, from the sender terminal device 200-1 via the communication interface 113, utterance information that is input in association with each of a plurality of objects included in content generated in the sender terminal device 200-1" and "a process of transmitting the received content to the receiver terminal device 200-2 via the communication interface 113." The processor 111 is mainly configured with one or more CPUs, but may also be combined with a GPU, an FPGA, or the like as appropriate.

[0027] The memory 112 includes RAM, ROM, nonvolatile memory, and HDD, and functions as a storage unit. The ROM stores instructions and commands for executing the application and OS according to the present disclosure as a program. Such a program is loaded and executed by the processor 111. The RAM is used to write and read data while the program stored in the ROM is being processed by the processor 111. The nonvolatile memory is a memory into which data is written and read as the program is executed, and the data written therein is saved even after the execution of the program has ended. Specifically, the memory 112 stores a program for the processor 111 to execute the above-mentioned processes, etc.

[0028] The communication interface 113 functions as a communication unit that transmits and receives information to and from other devices, such as the sender terminal device 200-1 and the receiver terminal device 200-2, which are installed remotely, via a communication processing circuit and an antenna. The communication processing circuit processes programs used in the processing system 1, various types of information, and the like, for transmitting and receiving information as the processing progresses. The communication processing circuit processes based on a wideband wireless communication method such as the LTE method, but it is also possible to process based on a method related to narrowband wireless communication or a method related to contactless wireless communication, such as a wireless LAN such as IEEE802.11 or Bluetooth (registered trademark). Wired communication can also be used instead of or in addition to wireless communication.

[0029] 4. Configuration of Terminal Device 200 2B is a block diagram showing the configuration of a terminal device 200 according to an embodiment of the present disclosure. The terminal device 200 does not need to have all of the components shown in FIG. 2B, and it is possible to adopt a configuration in which some components are omitted, or to add other components. Furthermore, the terminal device 200 is used as a sender terminal device 200-1 or a receiver terminal device 200-2, but both do not need to have the same configuration, and each terminal device may have a different configuration.

[0030] 2B, the terminal device 200 includes a processor 211 configured with a CPU or the like, a memory 212 including RAM, ROM, non-volatile memory, an HDD, and the like, a communication interface 213, an input interface 214, and an output interface 215. These components are electrically connected to each other via control lines and data lines.

[0031] The processor 211 is configured with a CPU (microcomputer) and functions as a control unit for controlling other connected components based on various programs stored in the memory 212. The processor 211 reads out and executes a program for executing an application according to the present disclosure and a program for executing an OS from the memory 212. The processor 211 is mainly configured with one or more CPUs, but may also be appropriately combined with a GPU, FPGA, or the like.

[0032] When functioning as sender terminal device 200-1, processor 111 executes the following processes based on programs stored in memory 212: "a process of accepting operational input from the sender via input interface 214 and launching an application program for generating content," "a process of accepting operational input from the sender via input interface 214 and selecting one of multiple objects included in the content based on identification information entered to identify that one object," "a process of inputting speech information associated with the selected one of the objects via input interface 214," and "a process of transmitting content including image information and speech information to server device 100 via communication interface 213."

[0033] Furthermore, when functioning as the receiver terminal device 200-2, the processor 111 executes the following processes based on the programs stored in the memory 212: "a process of accepting operational input from the receiver via the input interface 214, launching an application program for outputting content, and selecting the desired content," "a process of receiving content from the sender terminal device 200-1 via the communication interface 213, including speech information input in association with each of a plurality of objects included in the content generated in the sender terminal device 200-1," "a process of outputting the selected content via the output interface 215," "a process of selecting at least one object from a plurality of objects included in the content via the input interface 214," and "a process of outputting speech information associated with at least one selected object when outputting speech information included in the content via the output interface 215."

[0034] The memory 212 includes RAM, ROM, or nonvolatile memory and functions as a storage unit. The ROM stores instructions and commands for executing the application and OS according to the present disclosure as a program. Such a program is loaded and executed by the processor 211. The RAM is used to write and read data while the program stored in the ROM is being processed by the processor 211. The nonvolatile memory is a memory into which data is written and read by the execution of the program, and the data written therein is saved even after the execution of the program has ended. Specifically, the memory 212 stores a program for the processor 211 to execute the above-mentioned processes, etc.

[0035] The communication interface 213 functions as a communication unit that transmits and receives information to and from the electrically connected server device 100 and other terminal devices 200 via a communication processing circuit. The communication processing circuit processes programs used in the processing system 1, various types of information, and the like, to transmit and receive information according to the progress of processing. The communication processing circuit processes based on a wideband wireless communication method such as the LTE method, but can also process based on a method related to narrowband wireless communication or a method related to contactless wireless communication such as a wireless LAN such as IEEE802.11 or Bluetooth (registered trademark). Furthermore, wired communication can be used instead of or in addition to wireless communication.

[0036] The input interface 214 functions as an input unit that accepts input of operational inputs from the sender or receiver to the terminal device 200 and various information inputs by the sender or receiver. Examples of the input interface 214 include various hard keys such as a keyboard and a mouse, a touch panel that is superimposed on the display of the display device and has an input coordinate system corresponding to the display coordinate system of the display, a microphone for inputting voice information, which is one type of speech information, and a sensor for sensing the external environment, such as a camera for capturing images. In the case of a touch panel, icons corresponding to commands to be input are displayed on the display, and each icon is selected by the user or operator performing operational input via the touch panel. The touch panel may detect operational inputs using any method, such as a capacitive or resistive type. The input interface 214 does not always need to be physically provided on the terminal device 200 and may be connected as needed via a wired or wireless network.

[0037] The output interface 215 functions as an output unit for outputting various types of information. An example of the output interface 215 is an interface for connecting to an external device or equipment, such as a display device configured with a liquid crystal panel, an organic EL display, or a plasma display. However, if the terminal device 200 itself has a display, the display can function as the output interface. Furthermore, if the terminal device 200 is connected to a display device or the like via a communication interface 213, the communication interface 213 can also function as the output interface 215.

[0038] 6. Content Examples In this embodiment, as described above, content is generated in the sender terminal device 200-1 and output to the receiver terminal device 200-2 via the server device 100. Such content includes multiple objects, with speech information associated with each object. This content refers to a collection of electronic information transmitted and received via a communication network. Examples include video content, music content, game content, publication content, chat content, SNS content, web content, and combinations thereof. Among these, the processing system 1 is preferably used for video content that includes at least image information containing multiple character objects as characters and audio information associated with each character object. Note that, unless otherwise specified, the following description will be given assuming that the content is video content. However, the content is not limited to video content, and the processing according to this embodiment can be similarly performed for other content.

[0039] 3 is a diagram schematically illustrating information transmitted from the sender terminal device 200-1 as transmission information according to an embodiment of the present disclosure. Specifically, FIG. 3 is a diagram illustrating an example of video content that is generated in the sender terminal device 200-1, stored in the memory 212, and then transmitted to the server device 100 and stored in the content management table.

[0040] 3, video content includes image information and audio information associated with content ID information of the video content. The "content ID information" is information unique to each video content and is used to identify each video content. The content ID information is generated each time new video content is generated in sender terminal device 200-1 or each time new video content is received in server device 100.

[0041] "Image information" refers to image data constituting video content. The image information may be a still image, a video image, or a combination thereof. Such image information may be captured in real space by a camera provided as one of the input interfaces 214 in the sender terminal device 200-1, or may be virtually generated by processing by the processor 211. The image information includes at least a plurality of mutually identifiable objects, each of which is assigned object ID information. For example, in the example of FIG. 3, the image information includes a character object of character A with object ID information "B1" and a character object of character B with object ID information "B2." Examples of such objects include character objects, structure objects, decorative objects, text objects, image objects, GUI objects, and combinations thereof. Among these, the processing system 1 is preferably used for character objects that appear in video content (e.g., characters A and B in FIG. 1A). Hereinafter, unless otherwise specified, character objects will be described as an example of an object, but the objects are not limited to character objects.

[0042] "Audio information" is a type of speech information and is audio data constituting video content. One example of the audio information is audio data obtained by inputting the sender's voice or the like via a microphone provided as one of the input interfaces 214 in the sender terminal device 200-1. However, the audio information may also be, for example, audio data reproducing the voice of a character object based on text information input via the input interface 214, text data obtained by converting the sender's voice input via the microphone, text data generated based on text information input via the input interface 214, or other data obtained by converting at least one of these. Such audio information is typically stored in association with the object ID information of each object included in the video content. For example, in the example of FIG. 3, audio information A is stored in association with "B1," which is the object ID information of character A, audio information B is stored in association with "B2," which is the object ID information of character B, and background music audio information is stored as audio information not associated with any object ID information.

[0043] That is, FIG. 3 shows an example of video content with content ID information "A1." The video content includes image information, which is a video consisting of multiple frames F1 to Fn and spanning a time period from time t0 to time tn. At least one frame of the image information includes, as objects, character A with object ID information "B1" and character B with object ID information "B2." Here, as shown in FIG. 1A, for example, it is assumed that the sender uses his / her own sender terminal device 200-1 to play the roles of character A and character B. Therefore, the video content includes voice information A of character A, whose input begins at time t0 and ends at time t2. The video content also includes voice information B of character B, whose input begins at time t1 and ends at time t4, as a result of receiving an operational input from the sender at time t1. The video content also includes voice information A of character A, whose input started at time t3 and ended at time t6 as a result of receiving an operational input from the sender at time t3. The video content also includes voice information B of character B, whose input started at time t4 and ended at time tn as a result of receiving an operational input from the sender at time t5. Furthermore, during the period from time t1 to time t6, BGM voice information is included as voice information not associated with any object. That is, in the example of Fig. 3, for example, voice information A of character A and voice information B of character B are played simultaneously from time t1 to time t2, from time t3 to time t4, and from time t5 to time t6.

[0044] Thus, taking video content as an example, content includes image information and audio information associated with content ID information. Furthermore, the image information includes each frame (image data) constituting the video, associated with time (e.g., t0 to tn), frame ID information (e.g., F1 to Fn) identifying each frame, and object ID information identifying each object included in at least one frame of the image information, all of which are synchronized with time (e.g., t0 to tn). Furthermore, the audio information includes each piece of audio data associated with time (e.g., t0 to tn), and object ID information associated with each piece of audio data (although there may be no associated object ID information).

[0045] As mentioned above, the video content shown in Fig. 3 is merely an example of the content. Therefore, even when video content is used as the content, it is not necessary to include all of the various types of information exemplified above, and other information may also be included.

[0046] Furthermore, as described above, voice information is one type of speech information, and the speech information can be any information that is associated with the object ID information of the object and can reproduce the information entered by the sender, and can be, in addition to voice information, for example, text information or image information.

[0047] 7. Processing sequence performed by processing system 1 Fig. 4 is a diagram showing a processing sequence executed in the processing system 1 according to an embodiment of the present disclosure. Specifically, Fig. 4 is a diagram showing a series of processing sequences from when content is generated in the sender terminal device 200-1 to when the generated content is output in the receiver terminal device 200-2 via the server device 100. The processing in each device is executed by a processor running a program stored in the memory of each device.

[0048] (A) Processing related to content generation 4, first, processing related to content generation is executed mainly in sender terminal device 200-1. Processor 211 of sender terminal device 200-1 accepts operational input from the sender via input interface 214, reads an application program for generating content from memory 212, and starts the application program (S11). When the application program is started, processor 211 begins storing image data of frame F1 as image information of the content, as shown in FIG. 3. At this time, processor 211 accepts operational input from the sender via input interface 214 to request input of audio information in association with an object included in the image information of the content (S12), and outputs an audio input screen via output interface 215.

[0049] Here, Fig. 6 is a diagram illustrating an example of a screen output on the sender terminal device 200-1 according to an embodiment of the present disclosure. Specifically, Fig. 6 is a diagram illustrating an example of a voice input screen 10 output on the sender terminal device 200-1 when an operational input from the sender for requesting voice information input is received in S12 of Fig. 4. According to Fig. 6, the voice input screen 10 includes an image information display area 11 and an object selection area 12, along with the name of an application program called "Application A." In the image information display area 11, image data 13 of frame F1, for example, is output as currently recorded image information. The image data 13 includes an image 14 of character A and an image 15 of character B as character objects.

[0050] Here, character A shown in image 14 is assigned object ID information "B1," and character B shown in image 15 is assigned object ID information "B2." The object ID information is assigned, for example, when a video is being captured by a camera, by having processor 211 execute an object detection process to detect objects contained in each frame, and assign object ID information to each project when each object is first detected. Furthermore, when image information is virtually generated in a virtual space by sender terminal device 200-1, the object ID information is assigned by processor 211 assigning object ID information to objects drawn at the time of generation. Therefore, in the example of FIG. 6, only image 14 of character A and image 15 of character B happen to be included, but if an image of another character is newly included, character ID information for that other character will be generated.

[0051] Object selection area 12 includes icons for selecting each object corresponding to an object to which object ID information is assigned among the objects included in the image information. In the example of FIG. 6, object selection area 12 includes a character A icon 16 corresponding to character A and a character B icon 17 corresponding to character B. When processor 211 receives an operation input from the sender via input interface 214 for any of the icons included in object selection area 12 (for example, character A icon 16 or character B icon 17), processor 211 selects the character corresponding to the icon for which the operation input was made. In the example of FIG. 6, character A icon 16 is displayed so as to be distinguishable from the other icons, which indicates that the object ID information of character A (i.e., "B1") has been selected as the object ID information of the character to which the voice information to be input will be associated.

[0052] 4 again, as shown in Fig. 6, when the object ID information of a character for which voice information input is desired is selected, the processor 211 of the sender terminal device 200-1 accepts the input of the voice information in association with the object ID information via the input interface 214 (S13). Specifically, the processor 211 stores voice data uttered by the sender from a microphone, which is one of the input interfaces 214, as voice information in the memory 212 in synchronization with each frame of the image information associated with the time (for example, T0) at which the voice information is being input.

[0053] The processor 211 of the sender terminal device 200-1 repeats the recording of image information from S11 to S13, the selection of object ID information to associate with audio information, and the input of audio information, to generate content with content ID information "A1", for example, as shown in Fig. 3. When the processor 211 has finished generating the content, it stores it in the memory 212 in association with the content ID information, and transmits the generated content (T11) to the server device 100 via the communication interface 213.

[0054] When processor 111 of server device 100 receives the content from sender terminal device 200-1, it stores the received content in a content management table (not shown) in memory 112 in association with the content ID information (S14). Specifically, processor 111 of server device 100 stores each piece of information (image information, object ID information, audio information, etc.) included in the content shown in Fig. 3, for example, in association with the content ID information in the content management table (not shown) in memory 112. This completes the processing related to content generation.

[0055] In FIG. 4, the processor 211 of the sender terminal device 200-1 transmits the content to the server device 100 when the generation of the content is completed, but the content may be divided and transmitted each time a predetermined number of frames or amount of data is generated.

[0056] 4 has been explained on the assumption that audio information is input while image information is being recorded, but the processor 211 of the sender terminal device 200-1 may first generate image information and then input the audio information later in synchronization with each frame of the image information. For example, in the example shown in Fig. 3, audio information A and audio information B are input in overlapping fashion from time t1 to time t2, but by inputting each piece of audio information in synchronization with each frame after the image information has been generated, the same sender can play both character A and character B.

[0057] Although not specifically shown in FIG. 4, as shown in FIG. 3, it is also possible to input audio information (for example, BGM audio information) that is not associated with the object ID information of a specific object.

[0058] (B) Processing related to content output Next, referring to FIG. 4, processing related to content output is mainly executed in the receiver terminal device 200-2. For example, if the content is video content, this processing involves selecting desired video content in the receiver terminal device 200-2 and playing the video content. The processor 211 of the receiver terminal device 200-2 accepts an operational input from the receiver via the input interface 214, reads an application program for content output from the memory 212, and starts the application program (S21). When the application program is started, the processor 211 outputs, via the output interface 215, a content selection screen displaying a list of thumbnail images for selecting one or more video content items that can be output to the receiver terminal device 200-2. The processor 211 then accepts an operational input from the receiver via the input interface 214 to select a thumbnail image of the desired content item from a list on the content selection screen, and selects the content item to be output (S22). The processor 211 transmits, via the communication interface 213, a content request (T21) to the server device 100 requesting transmission of the selected content, together with content ID information associated with the selected content (for example, content ID information "A1").

[0059] When the processor 111 of the server device 100 receives a content request from the receiver terminal device 200-2 via the communication interface 113, it refers to the content management table based on the content ID information (e.g., A1) received together with the content request, and reads out the content (e.g., the information exemplified in FIG. 3) (S23). The processor 111 transmits the read out content (T22) via the communication interface 113 to the receiver terminal device 200-2 that has sent the content request.

[0060] When the processor 211 of the receiver terminal device 200-2 receives content via the communication interface 213, it outputs the received content via the output interface 215 (S24). Here, when object ID information is associated with a frame constituting the image information currently being output in the content received via the input interface 214, the processor 211 of the receiver terminal device 200-2 can select audio information to be output via an operational input from the receiver. That is, when the processor 211 receives an operational input from the sender requesting input of audio information in association with an object (S25), it selects audio information to be output via the output interface 215 and executes processing to change the audio information to be output (S26). Note that the series of processes relating to S24 to S26 will be described in detail later with reference to FIG. 5.

[0061] The receiver terminal device 200-2 then repeats the steps of S24 to S26, the output of the content, the selection of the object, and the change of the audio information to be output in accordance with the selection, and ends the output of the content when the time tn is reached. This completes the process related to the output of the content.

[0062] In FIG. 4, the processor 211 of the receiver terminal device 200-2 receives the content as a single block of data from the server device 100, but it may also be configured to receive the content in units of a predetermined number of frames or data amount and output the data sequentially.

[0063] 8. Processing flow of the receiver terminal device 200-2 Fig. 5 is a diagram showing a processing flow executed in the server device 100 according to an embodiment of the present disclosure. Specifically, Fig. 5 is a diagram showing a processing flow related to content output performed by the receiver terminal device 200-2 in S24 to S26 of Fig. 4. This processing flow is mainly performed by the receiver terminal device 200-2 by reading and executing a program stored in the memory 212.

[0064] 5, processor 211 receives desired content (for example, content having content ID information A1 shown in FIG. 3) from server device 100 via communication interface 213 (S111). Then, upon receiving the content, processor 211 outputs the received content via output interface 215. Specifically, processor 211 sequentially outputs image information included in the received content starting from frame F1 via a display, which is one of output interfaces 215. Processor 211 also outputs audio information included in the received content in synchronization with the frame of image information to be output via a speaker, which is one of output interfaces 215. In the example of FIG. 3, audio information A of character A is output from time t0, and at time t1, audio information B of character B and BGM audio information are each output in addition to audio information A.

[0065] When object ID information is associated with a frame constituting image information currently being output in the content received via input interface 214, processor 211 can select audio information to be output by accepting an operational input from the receiver via input interface 214. Therefore, processor 211 determines whether an object has been selected by accepting the operational input (S113).

[0066] Here, Fig. 7 is a diagram illustrating an example of a screen output on the receiver terminal device 200-2 according to an embodiment of the present disclosure. Specifically, Fig. 7 is a diagram illustrating an example of a content output screen 20 when the receiver terminal device 200-2 receives an operation input from the receiver to select audio information to be output in S112 to S113 of Fig. 5. According to Fig. 7, the content output screen 20 includes an image information display area 21 and an object selection area 22 along with the name of an application program "Application B." In the image information display area 21, image data 23 of frame F3, for example, is output as the image information currently being output. The image data 23 includes an image 24 of character A and an image 25 of character B as character objects. Character A shown in image 24 is assigned object ID information "B1," and character B shown in image 25 is assigned object ID information "B2."

[0067] The object selection area 22 includes icons for selecting each object, i.e., character, corresponding to object ID information associated with audio information synchronized with the frame of image information currently being output from among the audio information. For example, taking the content output screen 20 at any timing between time t1 and time t2 in FIG. 3 as an example, both audio information A for character A and audio information B for character B are output at that time. Therefore, the object selection area 22 includes a character A icon 26 corresponding to character A and a character B icon 27 corresponding to character B. When the processor 211 receives an operation input from the receiver via the input interface 214 for any of the icons included in the object selection area 22 (e.g., character A icon 26 or character B icon 27), the processor 211 selects the character corresponding to the icon for which the operation input was made. In the example of FIG. 7, the character A icon 26 is displayed so as to be distinguishable from the other icons, indicating that the object ID information of character A (i.e., "B1") has been selected as the object ID information of the character to which the audio information to be input will be associated.

[0068] 7, the object selection area 22 includes an icon for selecting audio information to be output corresponding to the character included in the currently output frame. However, this is not limiting, and an icon for selecting audio information may always be included for a character that appears in at least one frame throughout the entire content, so that audio information can be selected even when no audio information is being output.

[0069] 5 and 7, the case where an object is selected by receiving an operation input by the receiver via the input interface 214 has been described. However, instead of or in addition to this, it is also possible to select an object using a sensor such as a microphone or a camera as the input interface 214. For example, the processor 211 uses a camera to recognize the attributes (e.g., age, gender, etc.) of the receiver using the receiver terminal device 200-2. Then, the processor 211 selects an object for which audio is to be output based on the recognition results. For example, a terminal device for digital signage is prepared as the receiver terminal device 200-2, and a camera mounted on the terminal device recognizes the attributes of the user (receiver) viewing the display of the terminal device. Then, if the user (receiver) is recognized as a “child,” audio other than that of objects for children (e.g., animal characters) is muted; and if the user (receiver) is recognized as an “adult,” audio other than that of objects for adults (e.g., human characters) is muted. In this way, by using a sensor such as a camera as the input interface 214, it is possible to realize a wider variety of selection methods.

[0070] 5 again, as shown in Fig. 7, when object ID information of a character for which output of voice information is desired is selected, processor 211 outputs only the voice information of the selected character and restricts (e.g., mutes) the output of voice information of other characters (S114). That is, when object ID information of character A is selected from time t1 to time t2 in Fig. 3, processor 211 restricts the output of voice information B of character B from output interface 215 (e.g., speaker) so that only voice information A of character A is output. On the other hand, if no object is selected in S113, the processing related to S114 is skipped.

[0071] Processor 211 constantly repeats the processes of S112 to S114 until the output of a series of contents from time t0 to tn is completed. This completes this processing flow.

[0072] In this case, since the content only includes the voice information A of character A and the voice information B of character B, when character A is selected, the output of voice information B is restricted and only voice information A is output. However, it is also possible to restrict the output of the voice information of the character selected in Fig. 7 and output the voice information of the character not selected without restriction.

[0073] In addition, if the content contains three or more audio pieces of information, (1) Output only the voice information of the selected character and restrict the output of voice information of all remaining characters. (2) Restrict the output of the voice information of the selected character and output the voice information of all the remaining characters. (3) Output the voice information of the selected characters and restrict the output of the voice information of all the remaining characters. (4) Restrict the output of voice information for selected characters and output voice information for all remaining characters. It is possible to output audio information in various combinations such as:

[0074] Furthermore, although the above example shows the case where the output of audio information is restricted by "muting," various other restriction methods may be adopted, such as changing the volume at the time of output (for example, lowering it), or outputting subtitle text information simultaneously with audio information that is normally output but not outputting subtitles for restricted audio information.

[0075] As described above, in this embodiment, it is possible to provide a processing device, processing program, and processing method that are more user-friendly for users such as recipients. In particular, when output content includes multiple pieces of speech information (e.g., audio information), the recipient can select the speech information (e.g., audio information) to be output. For example, in the past, if a recipient did not want to output audio information associated with some objects, output was restricted by controlling the volume button on the recipient terminal device 200-2, etc. Therefore, output of all audio information was restricted. However, in this embodiment, it is possible to selectively output only audio information associated with an object desired by the recipient at a timing desired by the recipient, or selectively restrict output.

[0076] 9. Variations Below, modifications of the above-described embodiment shown in Figures 1 to 7 are shown. The following modifications and the embodiment shown in Figures 1 to 7 can also be implemented in combination with each other. Furthermore, except for the points specifically mentioned below, the same processing as that explained in the embodiment shown in Figures 1 to 7 can be carried out.

[0077] (A) Modification 1 related to selection of audio information In the above, as shown in Fig. 4 etc., a case has been described in which the voice information associated with the character selected in the receiver terminal device 200-2 is selected by the processor 211 of the receiver terminal device 200-2, and the output of the voice information of the other characters is restricted. However, instead of this, the voice information associated with the character selected in the receiver terminal device 200-2 may be reorganized by the processor 111 of the server device 100, and the output of the voice information of the other characters may be restricted.

[0078] Fig. 8A is a diagram showing a processing sequence executed in the processing system 1 according to an embodiment of the present disclosure. Specifically, Fig. 8A is a diagram showing a processing sequence when processing related to selection of voice information, which is one type of utterance information, is performed by the processor 111 of the server device 100. The processing in each device is executed by the processor processing a program stored in the memory of each device.

[0079] The processing from S31 to S34 relating to content generation is the same as the processing from S11 to S14 relating to content generation shown in FIG. 4, and therefore a description thereof will be omitted.

[0080] Furthermore, among the processes related to content output, the processes of S41 to S45, from accepting the content output and starting the application program, to outputting the desired content via the output interface, to selecting the audio information associated with the desired object, are the same as the processes of S21 to S25 among the processes related to content output shown in Figure 4, and therefore their explanation will be omitted.

[0081] When the object ID information of the character for which audio information is desired to be output is selected using the method shown in Figure 7 or the like, the processor 211 of the receiver terminal device 200-2 transmits object selection information (T43) including the content ID information of the content currently being output and the selected object ID information to the server device 100 via the communication interface 213.

[0082] When the processor 111 of the server device 100 receives the object selection information, it reads out the content associated with the content ID information from the memory 112 and executes a process to reorganize the content (S46). Specifically, the processor 111 leaves the audio information associated with the selected object ID information (audio information A in the example of FIG. 7) of the audio information A and audio information B included in the read content as is, and deletes the other audio information that was not selected (audio information B in the example of FIG. 7) from the content. Then, when the processor 111 has reorganized the content through the above process, it newly stores the reorganized content in the memory 112 and transmits the content (T44) via the communication interface 113 to the receiver terminal device 200-2 that has transmitted the object selection information.

[0083] When the receiver terminal device 200-2 receives the reorganized content via the communication interface 213, it outputs the received content via the output interface 215, as in S44. At this time, the voice information of the content does not include voice information B of character B. Therefore, the processor 111 of the receiver terminal device 200-2 outputs only voice information A of character A via the output interface 215, without outputting voice information B of character B.

[0084] It should be noted that, for example, icons corresponding to objects (characters) included in at least some frames of the image information are always displayed in the object selection area 22 of the content output screen 20 output in Fig. 7. This allows the receiver to select audio information B if he or she desires to output the audio information B again, even if the output of audio information B is restricted.

[0085] As described above, the example shown in FIG. 8A also makes it possible to selectively output audio information, similar to the embodiments shown in FIGS.

[0086] (B) Modification 2 related to selection of audio information In the above, as shown in Fig. 4 etc., a case has been described in which the processor 211 of the receiver terminal device 200-2 selects the voice information associated with the character selected in the receiver terminal device 200-2, and the output of the voice information of the other characters is restricted. However, instead of this, the processor 211 of the sender terminal device 200-1 may select the voice information associated with the character selected in the receiver terminal device 200-2, and the output of the voice information of the other characters may be restricted.

[0087] Fig. 8B is a diagram showing a processing sequence executed in the processing system 1 according to an embodiment of the present disclosure. Specifically, Fig. 8B is a diagram showing a processing sequence when processing related to selection of voice information, which is one type of utterance information, is performed by the processor 211 of the sender terminal device 200-1. The processing in each device is executed by the processor processing a program stored in the memory of each device.

[0088] Note that, except for the fact that the content is streamed in fixed data amounts, the processes of S61 to S64 related to content generation are the same as the processes of S11 to S14 related to content generation shown in Figure 4, and therefore a description thereof will be omitted.

[0089] Furthermore, except for the fact that the content is streamed in fixed data amounts, the processes related to content output, from S71 to S75, which involve accepting the content output, starting the application program, outputting the desired content via the output interface, and selecting audio information associated with the desired object, are the same as the processes related to content output, from S21 to S25, shown in Figure 4, and therefore will not be described again.

[0090] When the object ID information of the character for which audio information is desired to be output is selected using the method shown in Figure 7 or the like, the processor 211 of the receiver terminal device 200-2 transmits object selection information (T43) including the content ID information of the content currently being output and the selected object ID information to the server device 100 via the communication interface 213.

[0091] When the processor 111 of the server device 100 receives the object selection information (T73), it identifies the sender terminal device 200-1 that is the sender of the content based on the content ID information (S76). Then, the processor 111 transmits the object selection information (T74) to the identified sender terminal device 200-1 via the communication interface 113.

[0092] When the processor 211 of the sender terminal device 200-1 receives the object selection information via the communication interface 213, it selectively executes input of audio information (S77). Specifically, while image information and audio information are being input and distributed in real time in the sender terminal device 200-1, the processor 211 accepts input of audio information associated with object ID information through the processes shown in S62 and S63 (i.e., the processes shown in S22 and S23 of FIG. 4). The processor 211 then refers to the object ID information received through the object selection information, and if audio information associated with the same object ID information as the object ID information has been input, it stores the audio information in synchronization with the image information. On the other hand, the processor 211 accepts input of audio information associated with object ID information different from the received object ID information, but does not include it in the content to be transmitted. In other words, the processor 211 generates content that includes only audio information associated with the object ID information of the character selected by the receiver, and does not include audio information associated with the object ID information of other characters.

[0093] The processor 211 of the sender terminal device 200-1 transmits the content (T75) generated as described above together with the content ID information to the server device 100 via the communication interface 213. When the processor 111 of the server device 100 receives the content via the communication interface 113, it stores the content in the content management table in association with the content ID information, and also transmits the received content (T76) via the communication interface 113 to the receiver terminal device 200-2 that has transmitted the object selection information.

[0094] When the processor 211 of the receiver terminal device 200-2 receives the content via the communication interface 213, it outputs the received content via the output interface 215. At this time, the received content contains only the audio information associated with the object ID information of the selected character, as described above, and does not contain audio information associated with the object ID information of other characters. In other words, the transmission of audio information associated with object ID information of characters other than the character selected by the receiver is restricted, and as a result, output from the receiver terminal device 200-2 is restricted.

[0095] As described above, the example shown in FIG. 8B also makes it possible to selectively output audio information, similar to the embodiments shown in FIGS.

[0096] (C) Modifications related to restricted audio information 1 to 8B illustrate a case where only the voice information A of character A and the voice information B of character B are included in the content, so when character A is selected, the output of voice information B is restricted and only voice information A is output. However, it is also possible to restrict the output of the voice information of the selected character and output the voice information of the unselected character without any restriction.

[0097] In addition, if the content contains three or more audio pieces of information, (1) Output only the voice information of the selected character and restrict the output of voice information of all remaining characters. (2) Restrict the output of the voice information of the selected character and output the voice information of all the remaining characters. (3) Output the voice information of the selected characters and restrict the output of the voice information of all the remaining characters. (4) Restrict the output of voice information for selected characters and output voice information for all remaining characters. It is possible to output audio information in various combinations such as:

[0098] (C) A variant in which multiple senders exist 1 to 8B, a case has been described in which one sender plays multiple characters by inputting voice information of multiple characters in association with object ID information into one sender terminal device 200-1. However, instead of or in addition to this, it is also possible to have multiple senders play the same character, or multiple senders play multiple characters, by inputting voice information of multiple characters in association with object ID information into multiple sender terminal devices 200-1.

[0099] FIG. 9 is a diagram illustrating an overview of processing related to the processing system 1 according to an embodiment of the present disclosure. Specifically, FIG. 9 illustrates an example of processing in video content distribution performed using the processing system 1. According to FIG. 9, for the same video content, voice information A of character A and voice information B of character B are input into a sender terminal device of sender A, and transmitted to a receiver terminal device of a receiver via a server device. Furthermore, voice information C of character C and voice information D of character D are input into a sender terminal device of sender B, and transmitted to a receiver terminal device of a receiver via a server device. At this time, sender ID information for identifying sender A or the sender terminal device of sender A is associated with the voice information A and the voice information B. Furthermore, sender ID information for identifying sender B or the sender terminal device of sender B is associated with the voice information C and the voice information D. Therefore, when selecting voice information to be output in a receiver terminal device, it is also possible to select sender ID information instead of selecting object ID information. For example, when sender ID information of sender A is selected in the receiver terminal device, only audio information A and audio information B are output, and the output of audio information C and audio information D is restricted. Also, when sender ID information of sender B is selected in the receiver terminal device, only audio information C and audio information D are output, and the output of audio information A and audio information B is restricted.

[0100] As described above, the example shown in FIG. 9 also makes it possible to selectively output audio information, similar to the embodiments shown in FIGS. 1 to 8B.

[0101] (D) Modifications related to content, objects, and speech information In the examples of FIGS. 1 to 8B, video content is used as an example of content, and therefore, the description has been given with reference to a case where the object is a character object and the speech information is audio information. However, regardless of whether the content is video content or other content, similar processing is possible for other objects and other speech information. For example, content may include, in addition to video content, music content, game content, publication content, chat content, SNS content, web content, and combinations thereof. Furthermore, objects may include, in addition to character objects, structure objects, decorative objects, text objects, image objects, GUI objects, and combinations thereof. Furthermore, speech information may include, in addition to audio information, text information, image information, and combinations thereof.

[0102] For example, when chat content is applied to an embodiment of the present disclosure, the objects include speech bubble-shaped GUI objects associated with each sender, and the speech information includes text information entered as chat by each user. Even in such a case, a receiver can select the GUI object of a desired sender to limit the output (display) of chat (text information) associated with the GUI objects of other senders. This makes it possible to selectively output only specific senders.

[0103] The processes and procedures described herein can be realized not only by those explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described herein can be realized by implementing logic corresponding to the processes in media such as integrated circuits, volatile memory, nonvolatile memory, magnetic disks, and optical storage. Furthermore, the processes and procedures described herein can be implemented as computer programs and executed by various computers, including processing devices and server devices.

[0104] Although processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple components, and / or multiple modules. Furthermore, although various information described herein is described as being stored in a single memory or storage unit, such information may be stored in multiple memories within a single device or multiple memories distributed across multiple devices. Furthermore, software and hardware elements described herein may be realized by integrating them into fewer components or by decomposing them into more components. [Explanation of symbols]

[0105] 1 Processing System 100 Server device 200 Terminal Device 200-1 Sender terminal device 200-2 Receiver terminal device

Claims

1. A processing device comprising at least one processor, The at least one processor: receiving, via a communication interface, from a sender terminal device, utterance information input in association with each of a plurality of objects included in content generated in the sender terminal device; selecting at least one object from the plurality of objects via an input interface; outputting the speech information via an output interface, outputting speech information associated with the at least one selected object; a processing unit configured to perform processing for:

2. the content is video content, the object is a character object included in the video content; The processing device of claim 1 .

3. The processing device according to claim 2 , wherein the speech information is voice information input in association with the character object.

4. The at least one processor: receiving each piece of utterance information input in association with each of the plurality of objects; When outputting the speech information through the output interface, output of the speech information associated with the at least one selected object is limited, thereby outputting the speech information associated with the at least one selected object. The processing device of claim 1 configured to perform processing for:

5. The at least one processor: receiving only utterance information of the at least one selected object from among utterance information input in association with the plurality of objects; When outputting the speech information through the output interface, only the speech information associated with the at least one received object is output. The processing device of claim 1 configured to perform processing for:

6. The processing device according to claim 5 , wherein utterance information associated with an object other than the at least one selected object is restricted from being transmitted from the sender terminal device to the processing device.

7. the speech information is received via a server device installed remotely from the sender terminal device, The processing device according to claim 5 , wherein speech information associated with objects other than the at least one selected object is restricted from being transmitted from the server device to the processing device.

8. 2. The processing device according to claim 1, wherein the utterance information is input in association with each of the plurality of objects by a sender selecting one of the plurality of objects in advance on a sender terminal device.

9. In a computer having at least one processor, the at least one processor: receiving, via a communication interface, from a sender terminal device, utterance information input in association with each of a plurality of objects included in content generated in the sender terminal device; selecting at least one object from the plurality of objects via an input interface; outputting the speech information via an output interface, outputting speech information associated with the at least one selected object; A processing program that functions to perform processing for the purpose.

10. A processing method executed by at least one processor in a computer having the at least one processor, comprising: receiving, via a communication interface, from a sender terminal device, utterance information input in association with each of a plurality of objects included in content generated in the sender terminal device; selecting at least one object from the plurality of objects via an input interface; outputting speech information associated with the at least one selected object when outputting the speech information via an output interface; A processing method comprising:

Images