Caption display control system and caption display control method

Abstract

To provide a caption display control system and a caption display control method which are capable of displaying captions that match the atmosphere of a content better.SOLUTION: The caption display control system comprises: a display unit that displays a content; an image feature extraction unit that extracts a feature of the content; a display mode determination unit that determines a caption display mode based on the feature of the content; and a display control unit that displays the caption on the display unit in the display mode determined by the display mode determination unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a subtitle display control system and a subtitle display control method. [Background technology]

[0002] Conventionally, a subtitle display control system has been proposed that changes the display of subtitles based on the audio signal of a content. For example, Patent Document 1 describes a subtitle display control system that determines a subtitle display method based on audio features extracted from an audio signal, and superimposes the subtitles on a video image using the determined display method. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-300323 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when changing the display of subtitles based on an audio signal, the display mode of the subtitles may not necessarily match the atmosphere of the image of the content.

[0005] The present disclosure has been made in view of the above problems. It is an object of the present disclosure to provide a subtitle display control system and a subtitle display control method that are capable of displaying subtitles that are in harmony with the atmosphere of content. [Means for solving the problem]

[0006] A subtitle display control system according to one embodiment of the present disclosure includes a display unit that displays content, an image feature extraction unit that extracts features of the content, a display mode determination unit that determines a display mode of subtitles based on the features of the content, and a display control unit that displays the subtitles on the display unit in the display mode determined by the display mode determination unit.

[0007] A subtitle display control method of one embodiment of the present disclosure is a subtitle display control method executed by a subtitle display control system, and includes the steps of extracting features of content, determining a display mode of subtitles based on the features of the content, and displaying the subtitles on the display unit in the determined display mode. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a basic configuration of a subtitle display control system according to a first embodiment. [Diagram 2] 2 is a diagram showing an example of a table stored in a storage unit in FIG. 1; [Diagram 3] 1. FIG. 4 is a diagram showing an example of another table stored in the storage unit of FIG. [Figure 4] 2 is a flowchart showing an example of a process executed by the subtitle display control system of FIG. [Diagram 5] FIG. 11 is a diagram illustrating an example of a table stored in a storage unit in another processing example 1. [Figure 6A] FIG. 13 is a schematic diagram showing an example of a subtitle on-screen display image. [Figure 6B] FIG. 13 is a schematic diagram showing an example of a display image of a subtitle outscreen. [Figure 7] FIG. 11 is a diagram illustrating an example of a table stored in a storage unit in another processing example 2. [Figure 8] FIG. 13 is a diagram illustrating an example of a table stored in a storage unit in another processing example 3. [Figure 9] FIG. 11 is a block diagram showing the configuration of a subtitle display control system according to a second embodiment. [Figure 10] FIG. 11 is a diagram showing another method for providing content in the second embodiment. [Figure 11] FIG. 13 is a block diagram showing the configuration of a subtitle display control system according to a third embodiment. [Figure 12] FIG. 4 is a diagram showing an example of an image represented by an image signal. [Figure 13]FIG. 13 is a block diagram showing a configuration of a subtitle display control system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent components are denoted by the same reference numerals, and redundant descriptions of the same or equivalent components will be omitted as appropriate.

[0010] [First embodiment / Configuration of subtitle display control system] FIG. 1 is a functional block diagram showing a basic configuration of a subtitle display control system 100 according to a first embodiment. The subtitle display control system 100 can be realized by one or more devices including some or all of the functional blocks shown in FIG. 1. In other words, the subtitle display control system 100 may be configured by one device including all of the functional blocks shown in FIG. 1. Alternatively, the subtitle display control system 100 may be realized by multiple devices each including some of the functional blocks shown in FIG. 1, and the entire system configured by the multiple devices including all of the functional blocks shown in FIG. 1. In this case, the multiple devices are connected to each other so that they can communicate information with each other.

[0011] The subtitle display control system 100 includes, for example, any display device that displays content on a display unit. Examples of the display device include, but are not limited to, a television receiver, a monitor, a display, a computer, a tablet terminal, a smartphone, and a projector.

[0012] One or more devices constituting the subtitle display control system 100 each include, for example, a control unit, a storage unit, and a communication unit. Depending on the function of each device, one or more devices may further include functional units other than the control unit, the storage unit, and the communication unit.

[0013] The control unit controls and manages the entire device, including each functional unit of the device. The control unit performs various controls, for example, by running a control program stored in the storage unit. For example, the control unit can be configured with a control device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).

[0014] The storage unit is a storage medium capable of storing programs and data. The storage unit may be configured, for example, with a semiconductor memory or a magnetic memory. Specifically, the storage unit may be configured, for example, with an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage unit may store, for example, a program for operating the control unit.

[0015] The communication unit executes information communication with an external device. The communication unit is configured to include an appropriate interface depending on the method of information communication. The device can transmit and receive various types of information to and from the external device via the communication unit.

[0016] As shown in FIG. 1, the subtitle display control system 100 includes, as functional units, a content signal receiving unit 1, a signal separating unit 2, a subtitle signal decoding unit 3, an image signal decoding unit 4, an audio signal decoding unit 5, a subtitle feature extracting unit 6, an image feature extracting unit 7, an audio feature extracting unit 8, a memory unit 9, a display mode determining unit 10, a subtitle signal converting unit 11, a subtitle signal processing unit 12, an image signal processing unit 13, an audio signal processing unit 14, a display control unit 15, a display unit 16, and an audio output unit 17. These functional units are realized by each functional unit included in one or more devices. For example, the content signal receiving unit 1 is realized by a communication unit of one or more devices. For example, the memory unit 9 is realized by a memory unit of one or more devices. For example, the signal separation unit 2, the subtitle signal decoding unit 3, the image signal decoding unit 4, the audio signal decoding unit 5, the subtitle feature extraction unit 6, the image feature extraction unit 7, the audio feature extraction unit 8, the display mode determination unit 10, the subtitle signal conversion unit 11, the subtitle signal processing unit 12, the image signal processing unit 13, the audio signal processing unit 14 and the display control unit 15 are realized by a control unit of one or more devices. The display unit 16 and the audio output unit 17 are provided in a display device which is one of the one or more devices.

[0017] The content signal receiving unit 1 receives a content signal transmitted by an external device. The content signal is a signal related to information of the content to be reproduced by the subtitle display control system 100.

[0018] In this embodiment, the content includes images. For example, the content includes video, video, or still images. Specifically, the content is a movie, a drama, a play, an anime, a computer game, or the like. However, the content is not limited to the examples given here. The content may further include audio. In this embodiment, the content will be described as including both images and audio.

[0019] The content signal is generated, for example, in an external device that is a signal transmission source. The external device generates the content signal, for example, by multiplexing a subtitle signal, an image signal, and an audio signal. Here, the subtitle signal, the image signal, and the audio signal are signals related to the subtitle, image, and audio information of the content, respectively. The external device transmits the content signal generated by multiplexing to the subtitle display control system 100. The subtitle display control system 100 receives the content signal thus transmitted by the external device at a content signal receiving unit 1.

[0020] The signal separation unit 2 separates the multiplexed content signal into the original signals. In this embodiment, the signal separation unit 2 separates the content signal into a subtitle signal, an image signal, and an audio signal. The separated subtitle signal, image signal, and audio signal are input to a subtitle signal decoding unit 3, an image signal decoding unit 4, and an audio signal decoding unit 5, respectively.

[0021] The subtitle signal decoding unit 3, the image signal decoding unit 4, and the audio signal decoding unit 5 decode the subtitle signal, the image signal, and the audio signal, respectively. The decoded subtitle signal, the image signal, and the audio signal are input to the subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8, respectively.

[0022] The feature extraction unit, which includes the subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8, extracts features of the content. Specifically, the subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8 extract the features of the subtitles, the images, and the audio, respectively. The subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8 extract the features based on, for example, a predetermined algorithm.

[0023] The subtitle feature extraction unit 6 extracts the features of the subtitles. For example, the subtitle feature extraction unit 6 extracts text data of the subtitles as the features of the subtitles.

[0024] Also, for example, the subtitle feature extraction unit 6 extracts a specific character string included in the text data as a subtitle feature. The specific character string extracted by the subtitle feature extraction unit 6 is determined in advance and stored, for example, in a storage unit of the device. The subtitle feature extraction unit 6 can extract the specific character string by searching the text data for the specific character string stored in the storage unit.

[0025] The specific character string can be determined appropriately. The specific character string can be, for example, a character string representing a specific voice or sound. As an example, the specific character string can be a character string representing a scream or laughter. Character strings representing a scream include, for example, "Kyaa" or "Woah." Character strings representing laughter include, for example, "Wahaha" or "Kusukusu." However, the character strings representing a scream or laughter are not limited to the examples shown here. Furthermore, the specific character string is not limited to a character string representing a scream or laughter. For example, the specific character string can be an onomatopoeia including onomatopoeic words and mimetic words.

[0026] Also, for example, the subtitle feature extraction unit 6 extracts an explanatory sentence related to sound contained in the text data as a feature of the subtitle. The explanatory sentence related to sound is a sentence explaining the content of the sound, such as, for example, the sound of a door opening and closing, the sound of rain, the sound of a telephone ringing, the sound of thunder, or the sound of a siren. The explanatory sentence related to sound is indicated in a specific manner in the text data of the subtitle. As an example, the explanatory sentence related to sound is indicated using parentheses as the specific manner. Specifically, the sound of a door opening and closing is indicated as (sound of a door opening and closing) in the text data of the subtitle. In this case, the subtitle feature extraction unit 6 can extract the explanatory sentence related to sound by searching the portion indicated in the specific manner from the text data.

[0027] Note that the subtitle feature extraction unit 6 is not limited to explanatory text related to a specific character string and audio, and may extract other features that can be extracted from a subtitle signal as subtitle features.

[0028] The image feature extraction unit 7 extracts features of an image. For example, the image feature extraction unit 7 extracts color information of an image, people or objects appearing in the image, and the like as image features.

[0029] The color information of an image is information about the colors contained in the image, for example, information about the colors contained in the entire image or a part of the image. The image feature extraction unit 7 can extract the color information, for example, as RGB values. For example, the image feature extraction unit 7 can extract the RGB values ​​as the image features by converting the colors of the entire image into RGB values ​​from the color signals included in the image signal. The image feature extraction unit 7 can convert the colors of the image into RGB values ​​by a known method.

[0030] The color information may be represented by a plurality of colors that are divided in advance. For example, the color information is divided into 11 colors, namely, red, orange, yellow, green, blue, purple, pink, brown, white, gray, and black. The image feature extraction unit 7 determines which of the 11 colors the color of the image is similar to, for example, based on the RGB value, and sets the determined color as the color information of the image. Details of the method of determining the color information of the image based on the RGB value will be described later with reference to FIG. 4. Note that the color information does not necessarily have to be extracted as RGB values ​​or a plurality of colors, and may be extracted, for example, by hue or other indexes related to color.

[0031] The person or object shown in the image is information about the person or object shown in the image, and may be the person or object itself, or may include the movement of the person or object. The image feature extraction unit 7 can extract the person or object shown in the image using a known image recognition technology. For example, the image feature extraction unit 7 identifies the person shown in the image. The image feature extraction unit 7 may extract the mouth movement of the person shown in the image. By extracting the mouth movement, the person (speaker) who is speaking in the image can be identified. The image feature extraction unit 7 may extract the facial expression of the person shown in the image (e.g., the speaker). The image feature extraction unit 7 may extract information that classifies the speaker's facial expression into a bright expression and a dark expression, etc., as the image feature. In addition, for example, the image feature extraction unit 7 extracts a specific object shown in the image. The specific object to be extracted is, for example, predetermined and stored in the storage unit of the device. The specific object can be, for example, a door, rain, a telephone, thunder, or a specific vehicle (e.g., a patrol car, an ambulance, or a fire engine), but is not limited thereto. Furthermore, the image feature extraction unit 7 may extract the position of a person or object appearing in the image as an image feature.

[0032] The image feature extraction unit 7 may extract other features that can be extracted from the image signal as image features, in addition to color information of the image and people or objects appearing in the image. The image feature extraction unit 7 also transmits the image signal to the image signal processing unit 13.

[0033] The voice feature extraction unit 8 extracts voice features. For example, the voice feature extraction unit 8 extracts volume, pitch, a specific type of sound, and the like as voice features.

[0034] The audio feature extraction unit 8 can extract the volume and pitch of a sound based on, for example, the waveform of the sound in the audio signal. The audio feature extraction unit 8 may extract, as the audio feature, whether the sound included in the audio signal exceeds or falls below a predetermined threshold value related to the volume or pitch of the sound. The predetermined threshold value is, for example, determined in advance and stored in the storage unit of the device.

[0035] The specific type of sound is, for example, a person's voice or a sound emitted by a specific object. The sound emitted by a specific object may be, for example, but is not limited to, a door opening / closing sound, rain, a telephone ringing, thunder, or a siren. The voice feature extraction unit 8 can extract the specific type of sound by using, for example, a known voice recognition technique. The type of sound extracted by the voice feature extraction unit 8 is, for example, predetermined and stored in the storage unit of the device.

[0036] Furthermore, the audio feature extraction unit 8 may extract the length of a silent period as an audio feature. The audio feature extraction unit 8 can extract the length of a silent period based on, for example, the waveform of audio in an audio signal.

[0037] The voice feature extraction unit 8 may extract, as voice features, features other than the volume, pitch, specific types of sounds, and the length of silent periods that can be extracted from a voice signal. The voice feature extraction unit 8 also transmits the voice signal to the voice signal processing unit 14.

[0038] The subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8 provide information on the extracted features to the display mode determination unit .

[0039] The storage unit 9 stores information required to determine the display mode based on the features. For example, the storage unit 9 stores a table showing the correspondence between the features and the display mode. Alternatively, the storage unit 9 stores an algorithm for determining the display mode based on the features.

[0040] Here, the features are features extracted by the subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8. The display mode is the display mode of the subtitles. The display mode of the subtitles includes at least any element of the character size, font, color, and display position of the subtitles, for example. In the present embodiment, the display mode of the subtitles will be described below as including all elements of the character size, font, color, and display position of the subtitles.

[0041] Each element related to the display mode of subtitles is associated with a specific feature. For example, the character size is associated with the volume, which is a feature of the audio. The font is associated with the color information, which is a feature of the image. The color of the characters is associated with the expression of the speaker, which is a feature of the image. The display position is associated with the position of the person or object shown in the image, which is a feature of the image. Note that the association between each element related to the display mode of subtitles and the feature is not limited to this.

[0042] The storage unit 9 stores a table showing the correspondence relationship between the feature and the display mode for each element related to the display mode. For example, in the above example, the character size is associated with the volume, which is a feature of the audio. In this case, the storage unit 9 stores a table showing the correspondence relationship between the volume and the character size.

[0043] FIG. 2 is a diagram showing an example of the table stored in the storage unit 9. Specifically, FIG. 2 is a table showing the correspondence relationship between the volume and the character size. In FIG. 2, the volume is classified into three levels: less than V1, greater than or equal to V1 and less than V2, and greater than or equal to V2. Here, V1 and V2 are thresholds related to the volume, and V1 < V2 is satisfied. The values of V1 and V2 are, for example, predetermined. For the three-level classification of the volume, the first size, the second size, and the third size are associated as the character sizes. The first size, the second size, and the third size indicate the size of the characters and are, for example, the number of points of the characters. Here, it is assumed that the second size is larger than the first size and the third size is larger than the second size. As understood from the table in FIG. 2, the first size is associated with a volume less than V1, the second size is associated with a volume greater than or equal to V1 and less than V2, and the third size is associated with a volume greater than or equal to V2. That is, the larger the volume, the larger the associated character size.

[0044] For example, in the above example, the font is associated with the color information, which is a feature of the image. In this case, the storage unit 9 stores a table showing the correspondence relationship between the color information and the font.

[0045] FIG. 3 is a diagram showing an example of another table stored in the storage unit 9. Specifically, FIG. 3 is a table showing the correspondence between color information and fonts. In FIG. 3, color information is divided into 11 colors, namely, red, orange, yellow, green, blue, purple, pink, brown, white, gray, and black. Fonts are associated with the 11 color information divisions. In the table shown in FIG. 3, cursive script is associated with red and gray, round gothic is associated with orange, Meiryo is associated with yellow, Ming font is associated with green and blue, textbook font is associated with purple, thick round gothic is associated with pink, textbook font is associated with brown, regular script is associated with white, and square gothic is associated with black.

[0046] The association between colors and fonts is performed in advance, for example, based on the objects imagined by each color and / or the image of each color itself. For example, the color red evokes images of objects such as appearance, blood, lipstick, and apples. Also, for example, the color red itself is associated with images of hotness, flashiness, passion, and danger. Fonts matching such objects and images are associated with red. The same applies to other colors. The association between colors and fonts is performed in advance using such a method, for example, by an administrator of the subtitle display control system 100 or a content provider.

[0047] Here, the table showing the correspondence between the volume and the character size and the table showing the correspondence between the color information and the font have been specifically explained using Figures 2 and 3, but a similar table showing the correspondence between the facial expression of the speaker and the color of the characters is also stored in the storage unit 9. For example, cool colors are associated with a dark facial expression, and warm colors are associated with a bright facial expression.

[0048] Concerning the display position of the characters, for example, an algorithm that can determine the display position of the characters so that it has a predetermined positional relationship with the speaker is stored in the storage unit 9. The predetermined positional relationship is, for example, a relationship in which the distance from the speaker is within a certain range. It is preferable that the predetermined positional relationship is a close distance from the speaker. By closing the distance from the speaker, when the spoken content is displayed as subtitles, the viewer can easily understand the speaker of the subtitles.

[0049] 1 again, the display mode determination unit 10 determines the display mode of the subtitles based on the features of the content. In this embodiment, the display mode determination unit 10 determines the display mode of the subtitles based on the features extracted by the subtitle feature extraction unit 6, the image feature extraction unit 7, and the audio feature extraction unit 8. Specifically, the display mode determination unit 10 can determine the display mode of the subtitles by referring to the storage unit 9.

[0050] For example, the display mode determination unit 10 determines the display mode of the subtitles based on the characteristics of the image. As an example, the display mode determination unit 10 determines a font as the display mode by referring to a table stored in the storage unit 9 based on color information as the characteristics of the image. Specifically, the display mode determination unit 10 determines a font corresponding to the color information as the display mode of the subtitles by referring to the table shown in FIG. 3, for example. In a similar manner, the display mode determination unit 10 can determine the color and display position of the characters as the display mode of the subtitles.

[0051] The display mode determination unit 10 may also determine the display mode of the subtitles based on other characteristics. For example, the display mode determination unit 10 can determine the character size as the display mode based on the audio characteristic. That is, the display mode determination unit 10 refers to the table shown in FIG. 2 to determine the character size corresponding to the volume, which is an audio characteristic, as the display mode of the subtitles.

[0052] The display mode determination unit 10 may determine the display mode of the subtitles based on a combination of a plurality of features. For example, the display mode determination unit 10 can determine the character size, font, color, and display position of the subtitles as the display mode of the subtitles based on the image features and the audio features.

[0053] The subtitle signal conversion unit 11 converts the text data of the subtitles into the format determined by the display format determination unit 10. For example, the storage unit 9 stores in advance data on characters of all character sizes, fonts, and colors used in the subtitle display control system 100. The subtitle signal conversion unit 11 refers to the storage unit 9 and converts the text data of the subtitles into the character size, font, and color determined by the display format determination unit 10.

[0054] Subtitle signal processing unit 12 processes the subtitle data converted by subtitle signal conversion unit 11 into a format that can be displayed together with an image. For example, when subtitles are displayed superimposed on an image, subtitle signal processing unit 12 processes the subtitle data converted by subtitle signal conversion unit 11 into a format that can be superimposed on an image.

[0055] Based on the image signal, the image signal processing unit 13 generates an image to be displayed on the display unit 16. For example, the image signal processing unit 13 processes the image signal into a format that can be displayed on the display unit 16.

[0056] The audio signal processing unit 14 processes the audio signal into a format that can be output by the audio output unit 17. For example, the audio signal processing unit 14 converts the audio signal, which is a digital signal, into an analog signal.

[0057] The display control unit 15 causes the display unit 16 to display the image and subtitles. Specifically, the display control unit 15 causes the display unit 16 to display the image and subtitles based on the image signal processed by the image signal processing unit 13 and the subtitle data processed by the subtitle signal processing unit 12. At this time, the display control unit 15 causes the display unit 16 to display the subtitles in the display mode determined by the display mode determination unit 10. The display control unit 15 causes the display unit 16 to display the subtitles, for example, by superimposing them on the image.

[0058] The display unit 16 is a device that displays an image. The display unit 16 may be configured with a known display such as a liquid crystal display (LCD), an organic electro-luminescence display (OELD), or an inorganic electro-luminescence display (IELD). The display unit 16 displays various information based on the control of the display control unit 15. For example, the display unit 16 displays content including images and subtitles based on the control of the display control unit 15.

[0059] The audio output unit 17 is a device that outputs sound. The audio output unit 17 is, for example, a speaker. The audio output unit 17 outputs, for example, the sound of the content under the control of the audio signal processing unit 14.

[0060] [Example of processing by the subtitle display control system] Next, an example of the process executed by the subtitle display control system 100 will be described. Fig. 4 is a flow chart showing an example of the process of displaying subtitles in a display mode determined based on the characteristics of an image. Specifically, Fig. 4 shows an example of the process of determining the display mode of subtitles based on color information, which is a characteristic of an image, and displaying the subtitles in the determined display mode.

[0061] 4, the subtitle display control system 100 first determines color information of an image based on RGB values. Specifically, the subtitle display control system 100 determines the color information of an image in steps S11 to S13.

[0062] That is, the image feature extraction unit 7 extracts color signals for one frame of the image based on the decoded image signal (step S11). The image feature extraction unit 7 converts the extracted color signals for one frame into RGB values ​​(step S12). As a result, the RGB values ​​for one frame of the image are extracted.

[0063] The image feature extraction unit 7 determines the approximate color information for the image of one frame based on the extracted RGB values ​​of the one frame (step S13). For example, the image feature extraction unit 7 determines the color information based on which color information section the extracted RGB values ​​of the one frame are approximated to among the pre-classified color information. Since the color information is classified into 11 colors in the above example, the image feature extraction unit 7 determines which section of the 11 color sections the extracted RGB values ​​of the one frame are approximated to. For example, if the extracted RGB values ​​of the one frame are most approximate to the RGB values ​​of red among the 11 color sections, red is determined as the approximate color information.

[0064] Specifically, in step S13, the image feature extraction unit 7 can refer to the storage unit 9 to determine the approximate color information. For example, the storage unit 9 stores predetermined RGB values ​​for each of the 11 color sections. The image feature extraction unit 7 refers to the storage unit 9 to compare the RGB values ​​of the extracted one frame with the RGB values ​​of each of the 11 color sections, thereby determining color information of the section having the RGB values ​​that are most approximate to the RGB values ​​of the extracted one frame. The color information of the section having the RGB values ​​that are most approximate to the RGB values ​​of the extracted one frame is determined as the color information that is most approximate to the RGB values ​​of the extracted one frame. The comparison of the RGB values ​​may be performed by comparing the numerical values ​​of red (R), green (G), and blue (B), or may be performed based on values ​​calculated by performing a predetermined numerical processing or weighting using the numerical values ​​of red (R), green (G), and blue (B). Therefore, the comparison of the RGB values ​​can be performed using various comparison methods.

[0065] The image feature extraction unit 7 accumulates the color information for one frame determined in step S13 (step S14). For example, the image feature extraction unit 7 stores the color information for one frame determined in step S13 in the storage unit 9. In this case, the color information only needs to be temporarily stored at least until the font is determined in step S16.

[0066] Then, the image feature extraction unit 7 judges whether or not color information has been accumulated for frames for a predetermined time (step S15). The predetermined time can be set to an appropriate value in advance, for example, within a range of 1 second to several seconds. However, the predetermined time is not limited to the range shown here. Here, the predetermined time is assumed to be 1 second. That is, the image feature extraction unit 7 judges whether or not color information for frames for 1 second has been accumulated in step S15. For example, a video for 1 second is composed of images of multiple frames. Therefore, the image feature extraction unit 7 judges whether or not color information has been accumulated for images of the number of frames that constitute a video for 1 second by the processes of steps S11 to S14.

[0067] When it is determined that color information for a predetermined time (here, 1 second) has not been accumulated (No in step S15), the image feature extraction unit 7 executes steps S11 to S14 for the frame following the frame for which color information was accumulated in step S14. This accumulates color information for the next frame. By repeating this process, the image feature extraction unit 7 accumulates color information for the number of frames for the predetermined time.

[0068] When it is determined that color information for a predetermined time has been accumulated (Yes in step S15), the image feature extraction unit 7 transmits the accumulated color information as an image feature to the display mode determination unit 10. The display mode determination unit 10 determines the display mode of the subtitles based on the accumulated color information. In this embodiment, since the image feature of color information is associated with the subtitle display mode of font, the display mode determination unit 10 determines the font as the display mode of the subtitles based on the accumulated color information (step S16).

[0069] At this time, the display mode determination unit 10 determines the display mode of the subtitles, for example, by referring to a table stored in the storage unit 9. In this example, the display mode determination unit 10 determines the font as the display mode of the subtitles, for example, by referring to a table showing the correspondence between color information and fonts as shown in Fig. 3.

[0070] The display mode determination unit 10 can determine the font using various determination methods based on the accumulated color information. For example, the accumulated color information includes color information of the number of frames for one second. The display mode determination unit 10 can determine the font associated with the most color information among the accumulated color information of the number of frames for one second as the display mode of the subtitles. For example, assume that the number of frames determined as red is X, the number of frames determined as orange is Y, and the number of frames determined as yellow is Z in the accumulated color information. The display mode determination unit 10 determines the font associated with the color information having the most number of frames among these X, Y, and Z as the display mode of the subtitles. For example, when X is greater than Y and Z, the display mode determination unit 10 determines the font associated with red having X frames (cursive font in the example shown in FIG. 3) as the display mode of the subtitles. For example, when Y is greater than X and Z, the display mode determination unit 10 determines the font associated with orange having Y frames (rounded gothic font in the example shown in FIG. 3) as the display mode of the subtitles. For example, if Z is greater than X and Y, a font associated with yellow having Z frames (Meiryo in the example shown in FIG. 3) is determined as the display mode for the subtitles.

[0071] The display mode determination unit 10 may determine the font using other methods. For example, a predetermined calculation may be performed based on color information of the number of frames for one second, and the font may be determined from the information obtained as a result of the calculation. As an example, the display mode determination unit 10 weights the RGB values ​​of red, orange, and yellow based on X, Y, and Z, which are the number of frames, and determines the font based on the RGB values ​​obtained as a result. In this case, for example, color information that is close to the RGB values ​​obtained as a result of the weighting may be determined again, and the font associated with the determined color information may be determined as the display mode of the subtitles.

[0072] When the display mode determination unit 10 determines the font, the subtitle signal conversion unit 11 and the subtitle signal processing unit 12 perform processing, and the display control unit 15 causes the display unit 16 to display the subtitles in the determined font (step S17).

[0073] The display control unit 15 determines whether or not the subtitle display for one phrase has been completed (step S18). The subtitle display for one phrase means the subtitle display for one screen. In other words, the subtitle display for one phrase refers to the subtitle display range that does not switch the screen display. For example, when the subtitle screen switches once and subtitles are displayed before and after the switch, it means that one phrase of subtitles has been displayed before and after the switch.

[0074] If the subtitle display for one phrase is not complete (No in step S18), the display control unit 15 continues to display the same subtitle until the subtitle display for one phrase is complete. On the other hand, if the subtitle display for one phrase is complete (Yes in step S18), the process proceeds to step S11, where the image feature extraction unit 7 extracts the color signal for the next frame.

[0075] By repeating steps S11 to S18 in this manner, the subtitle display control system 100 can determine a font based on color information that is a feature of the image, and display the subtitles in the determined font. In particular, in the example shown in the flow of FIG. 4, the font of the subtitles is determined based on color information for a predetermined period (e.g., one second) at the beginning of the video in which the subtitles are displayed. The subtitles are displayed in the determined font, and in step S18, the font is not changed and the same font continues to be displayed until the subtitle display for one phrase is completed. This makes it easier to prevent the subtitles from becoming difficult to read due to frequent font changes.

[0076] However, step S18 in the flow of Fig. 4 does not necessarily have to be executed. In this case, the font of the subtitles is determined based on the color information of the image at predetermined time intervals, and the subtitles are displayed in the determined font. Therefore, the font can be changed in real time in response to changes in the characteristics (color information) of the image.

[0077] In FIG. 4, a processing example in which the font of the subtitle is determined based on color information and the subtitle is displayed with the determined font has been described, but the processing in which the display mode of other subtitles is determined based on other characteristics and displayed can also be executed in a manner conforming to FIG. 4. For example, the subtitle display control system 100 can determine the character size based on the volume, which is a feature of the voice, determine the color of the subtitle characters based on the facial expression of the speaker, which is a feature of the image, and determine the display position of the subtitles based on the position of a person or object appearing in the image, which is a feature of the image, in a manner conforming to FIG. 4, and display the subtitles in the determined manner. In addition, the subtitle display control system 100 can determine multiple elements of the display mode of the subtitles based on multiple features, and display the subtitles by appropriately combining these. For example, the subtitle display control system 100 can display the subtitles using the character size determined based on the volume and the font determined based on the color information. The same applies to other combinations.

[0078] As described above, in the subtitle display control system 100 according to this embodiment, the display mode determination unit 10 determines the display mode of the subtitles based on the characteristics of the image, and the subtitles are displayed on the display unit 16 in the display mode determined by the display mode determination unit. Therefore, it is possible to display subtitles that harmonize with the atmosphere of the content based on the characteristics of the image.

[0079] The subtitle display control system 100 can display subtitles based on features in a manner other than the manner described in the above embodiment, or in addition to the manner described in the above embodiment. Some other processing examples executed by the subtitle display control system 100 will be described below.

[0080] [Other processing example 1] In the subtitle display control system 100, the display mode determination unit 10 can determine the display mode of the subtitles based on the image feature and the subtitle feature. Here, a case will be described in which the background color of the image is used as the image feature and a specific character string is used as the subtitle feature.

[0081] The background color of an image is the color of the background of the image. For example, the color of the background part of the image excluding the person is used as the background color of the image. Alternatively, the color information in the above embodiment may be used as the background color of the image. Therefore, the background color of the image can be determined based on the RGB value of the image or its background. The background color of the image is divided into a plurality of colors, for example. The background color of the image may be divided into 11 colors, as in the above embodiment. The background color of the image may be divided by a different standard from the 11 colors in the above embodiment. Here, the background color of the image is divided into two, a light background color and a dark background color. For example, a classification standard RGB value stored in advance in a storage unit is used as a classification standard for the light background color and the dark background color.

[0082] The specific character string may be, for example, a character string representing a specific voice or sound. In this example, the specific character string is assumed to be a character string representing a scream and a laugh. The character strings representing a scream and a laugh are, for example, determined in advance and stored in the storage unit 9. For example, "Kyaa" and "Woah" are stored as character strings representing a scream. Also, "Wahaha" and "Kusukusu" are stored as character strings representing laughter.

[0083] The storage unit 9 stores, for example, a table in which image features and subtitle features are associated with subtitle display modes. FIG. 5 is a diagram showing an example of a table stored in the storage unit 9 in another processing example 1. In the example shown in FIG. 5, a total of four display modes are determined by combinations of shouting and laughter, which are subtitle features, and light and dark background colors, which are image features. Specifically, a first display mode is associated with a shouting voice and a light background color. Similarly, a second display mode is associated with a shouting voice and a dark background color, a third display mode is associated with a laughter voice and a light background color, and a fourth display mode is associated with a laughter voice and a dark background color.

[0084] The first to fourth display modes are specific display modes related to subtitles, and are defined by, for example, one or more combinations of character size, font, and color of the subtitles. The first to fourth display modes are predefined as display modes that respectively represent the characteristics of the corresponding images and the characteristics of the subtitles. The first display mode associated with a scream and a bright background color is, for example, a light white or yellow font that evokes a happy cheer. The second display mode associated with a scream and a dark background color is, for example, a blurred red or gray font that evokes a fear or a sad feeling. The third display mode associated with laughter and a bright background color is, for example, a pop font in light blue or pink that evokes an excited or lively state. The fourth display mode associated with laughter and a dark background color is, for example, a dull brown or black font that evokes a chuckle. However, the display modes described here are merely examples, and the first to fourth display modes can be other appropriate display modes.

[0085] In another processing example 1, the display mode determination unit 10 determines the display mode of the subtitles based on the image features and the subtitle features. For example, the image feature extraction unit 7 determines whether the image has a light background color or a dark background color, as an image feature, using RGB values ​​for classification criteria stored in the storage unit 9. Furthermore, the subtitle feature extraction unit 6 determines whether the subtitles contain a character string representing a shout or laughter, as a subtitle feature. The display mode determination unit 10 determines the display mode of the subtitles based on whether the image has a light background color or a dark background color, and whether the image contains a character string representing a shout or laughter.

[0086] For example, when an image has a light background color and includes a character string representing a scream, the display mode determination unit 10 refers to the table of FIG. 5, for example, and determines the display mode of the subtitles including the character string representing the scream as the first display mode. For example, when an image has a dark background color and includes a character string representing a scream, the display mode determination unit 10 refers to the table of FIG. 5, for example, and determines the display mode of the subtitles including the character string representing the scream as the second display mode. Similarly, when the image includes a character string representing a laugh, the display mode determination unit 10 determines the display mode of the subtitles to be the third display mode or the fourth display mode depending on whether the background color is light or dark. When the display mode determination unit 10 determines the display mode, the display control unit 15 causes the display unit 16 to display the subtitles in the determined display mode through processing by the subtitle signal conversion unit 11 and the subtitle signal processing unit 12.

[0087] The first to fourth display modes are determined based on a specific character string that is a characteristic of the subtitles and a background color that is a characteristic of the image. Therefore, by determining the display mode of the subtitles based on the characteristics of the subtitles and the characteristics of the image and displaying the subtitles, it is possible to display subtitles that better harmonize with the atmosphere of the content.

[0088] [Other processing example 2] In the subtitle display control system 100, the display mode determination unit 10 can determine the display mode based on the genre of the content. The genre of the content is information that is assigned to the content in advance and indicates the classification of the content. The genre of the content may be appropriately classified, for example, into "news / reporting," "sports," "information / discussion show," "drama," "music," "variety," "movie," "anime / special effects," "documentary / educational," "theater / performance," and "other." However, the classification shown here is merely an example. The information on the genre of the content is included in, for example, a multiplexed content signal. The display mode determination unit 10 can recognize the genre of the content based on the genre information included in the content signal.

[0089] The storage unit 9 stores in advance a table in which, for example, each content genre is associated with a display mode that matches the image of the genre. The display mode determination unit 10 can determine the display mode of the subtitles by referring to the table, for example, from genre information. Note that the display mode of the subtitles can be, for example, the font of the subtitles, but is not limited to this.

[0090] Furthermore, the subtitle display control system 100 may be configured so that subtitle on-screen and subtitle out-screen are alternatively selectable as a subtitle display method. Here, the subtitle on-screen is a display method in which an image of a content is displayed on the entire display unit 16 (display screen) and subtitles are displayed superimposed on the image of the content, as shown, for example, in FIG. 6A. On the other hand, the subtitle out-screen is a display method in which a dedicated area for displaying subtitles is provided on the display unit 16 (display screen), and the image of the content and the subtitles are displayed without superimposing each other, as shown, for example, in FIG. 6B. Note that, in FIG. 6A and FIG. 6B, the subtitles are displayed at the center of the lower part of the display unit 16, but the display position of the subtitles does not necessarily have to be the center of the lower part of the display unit 16, and may be at the upper part, left or right. The subtitle on-screen and the subtitle out-screen are alternatively selectable, for example, based on a predetermined operation input by a user who watches the content.

[0091] When subtitle on-screen is selected according to the genre of the content, the display mode determination unit 10 can add animation (text animation) to the subtitles as a display mode of the subtitles. The storage unit 9 stores in advance, for example, a table that associates the genre of the content with whether or not animation is to be added to the subtitles. The display mode determination unit 10 can refer to the table to determine in which genre animation is to be added to the subtitles.

[0092] FIG. 7 is a diagram showing an example of a table stored by the storage unit 9 in another processing example 2. As shown in FIG. 7, the table stores genres, fonts, and subtitle display methods in association with each other. The subtitle display method is a part of the display manner of the subtitles, and in the example shown in FIG. 7, for the cases of subtitle on-screen and subtitle out-screen, whether to display in the form of subtitles or with animation added is stored. That is, when "subtitles" is written in FIG. 7, the text (characters) of the subtitles is displayed as a normal subtitle at a fixed position, for example, at the top, bottom, left, or right of the screen. On the other hand, when "animation" is written in FIG. 7, the text of the subtitles is displayed on the display screen with an animation effect, for example, the display position is appropriately changed, or the subtitles move or expand / shrink. Note that the type of animation to be added may be determined in advance as one, for example, or may be determined by a predetermined determination method according to the text of the subtitles.

[0093] For example, if the genre of the content is drama, the subtitles are displayed as animation near the person who is speaking the words in the subtitles. This makes it possible to clarify the speaker and enhance the sense of realism. Also, if the genre of the content is variety, the subtitles can be moved or expanded / contracted as animation to make the program more entertaining. In addition to the examples shown here, the atmosphere of the content can be more easily conveyed to the user, the viewer, by adding appropriate animation to the subtitles depending on the genre of the content.

[0094] 7, when "Subtitles / Animation" is written, the subtitle text is displayed either in a fixed position as a normal subtitle or with an animation effect added, depending on the content. For example, subtitle text with explanatory content is displayed as a normal subtitle, and subtitle text showing dialogue by a person is displayed with an animation effect added.

[0095] In another processing example 2, the display mode determination unit 10 determines the font as the display mode of the subtitles, for example, by referring to the table shown in FIG. 7 based on the information of the genre of the content. The display mode determination unit 10 also determines the display mode of the subtitles, based on the information of the genre of the content and the display method (i.e., subtitles on-screen or subtitles out-screen) determined based on the user's input. For example, when the genre is sports and the display method is subtitles on-screen, the display mode determination unit 10 determines the font to be "Mincho" and the display method to be "subtitles" by referring to the table in FIG. 7. For example, when the genre is variety and the display method is subtitles out-screen, the display mode determination unit 10 determines the font to be "sans serif" and the display method to be "subtitles" by referring to the table in FIG. 7. For example, when the genre is anime / special effects and the display method is subtitles on-screen, the display mode determination unit 10 determines the font to be "Gothic" and the display method to be "animation" by referring to the table in FIG. 7. When the display mode determination unit 10 determines the display mode (font and display method), after processing by the subtitle signal conversion unit 11 and the subtitle signal processing unit 12, the display control unit 15 displays the subtitles on the display unit 16 in the determined display mode.

[0096] This allows subtitles to be displayed in a font that matches the image of the content genre, making it possible to display subtitles that blend in better with the atmosphere of the content. Also, in the case of on-screen subtitles, adding animation to the subtitles according to the content genre makes it easier to convey the atmosphere of the content to the viewer, the user.

[0097] [Other processing example 3] In the subtitle display control system 100, when a description regarding sound is included as a subtitle feature, the display mode determination unit 10 can replace the description regarding sound with characters representing the sound represented by the description regarding sound. The description regarding sound is extracted as a subtitle feature by, for example, the subtitle feature extraction unit 6, as described in the above embodiment.

[0098] The storage unit 9 stores in advance a table in which, for example, explanatory text about a sound is associated with characters that represent the sound represented by the explanatory text about the sound. The display mode determination unit 10 replaces, for example, the extracted explanatory text about the sound with characters that represent the sound written in the table.

[0099] The display mode determination unit 10 can also change the display mode of the characters representing the replaced sound based on whether or not a sound source that emits the sound represented by the description regarding the sound is included in the image as a feature of the image. For example, the display mode determination unit 10 changes the display method as the display mode based on whether or not a sound source is included in the image. That is, the display mode determination unit 10 changes whether the characters representing the sound are displayed as normal subtitles or with animation added. The display mode determination unit 10 can change the display mode by referring to a predetermined table stored in the storage unit 9, for example.

[0100] Fig. 8 is a diagram showing an example of a table stored in the storage unit 9 in another processing example 3. As shown in Fig. 8, the table stores explanatory text relating to audio, characters representing audio, and subtitle display methods in association with each other. The subtitle display method is a part of the display manner of subtitles, and in the example shown in Fig. 8, for the cases of "with audio source" indicating that the audio source is included in the image and "without audio source" indicating that the audio source is not included in the image, it is stored whether to display in the form of subtitles or with animation added.

[0101] In another processing example 3, the display mode determination unit 10 replaces the explanatory text about the sound with characters representing the sound, based on the explanatory text about the sound, which is a feature of the subtitle, and the presence or absence of a sound source, which is a feature of the image, and determines the display mode of the subtitle. For example, the subtitle feature extraction unit 6 extracts a subtitle with an explanatory text of "sound of rain" as a feature of the subtitle. In this case, the display mode determination unit 10 replaces the subtitle "sound of rain" with a subtitle of characters representing the sound "buzzing" by referring to the table shown in FIG. 8, for example. Furthermore, the image feature extraction unit 7 extracts whether or not "rain", which is the sound source of the sound of rain, is included in the image in which the subtitle is displayed, as a feature of the image. If "rain" is included in the image, that is, if there is a sound source, the display mode determination unit 10 determines that the replaced subtitle "buzzing" is displayed with animation added as the display mode of the subtitle. On the other hand, if "rain" is not included in the image, that is, if there is no sound source, the display mode determination unit 10 determines that the replaced subtitle "buzzing" is displayed as a normal subtitle as the display mode of the subtitle. When animation is added to the subtitle "zar", for example, the subtitle "zar" is displayed by moving on the display screen.

[0102] For example, the subtitle feature extraction unit 6 extracts a subtitle with the explanatory text "sound of telephone bell" as a subtitle feature. In this case, the display mode determination unit 10 replaces the subtitle "sound of telephone bell" with a subtitle of characters representing the sound "pururu-ruru" by referring to the table shown in FIG. 8, for example. Furthermore, the image feature extraction unit 7 extracts whether or not the image in which the subtitle is displayed includes a "telephone" that is the sound source of the sound of the telephone bell, as an image feature. If the image includes a "telephone", that is, if there is a sound source, the display mode determination unit 10 determines that the replaced subtitle "pururu-ruru" is displayed with animation added as a subtitle display mode. In this case, the subtitle "pururu-ruru" is displayed, for example, by being positioned near the displayed telephone on the display screen. On the other hand, if the image does not include a "telephone", that is, if there is no sound source, the display mode determination unit 10 determines that the replaced subtitle "pururu-ruru" is displayed as a normal subtitle as a subtitle display mode. In this case, the subtitle "pururururu" is displayed, for example, at the bottom center on the display screen.

[0103] This allows the user to feel more immersive because the explanation about the sound is replaced with characters that represent the sound. Also, since the display mode can be changed depending on the presence or absence of a sound source, the display mode can be changed based on the characteristics of the image as well as the characteristics of the subtitles.

[0104] In the table shown in Fig. 8, each description about sound does not necessarily have to be associated with only one character representing the sound, and may be associated with multiple characters representing the sound. In this case, the character representing the sound may be determined based on not only the description about the sound, which is a feature of the subtitles, but also the feature of the video.

[0105] For example, in the table shown in FIG. 8, the explanatory text about the sound "sound of a siren" is associated with characters expressing the sound "beep beep beep". However, the explanatory text about the sound "sound of a siren" may be associated with characters expressing three sounds, "beep beep beep", "woo woo" and "woo ding ding ding". In this case, when the display mode determination unit 10 extracts the caption of the explanatory text "sound of a siren" as a feature of the caption, it further specifies the type of vehicle from the vehicle shown in the image as a feature of the image. When the identified type of vehicle is an ambulance, the display mode determination unit 10 determines "beep beep beep" as the characters expressing the sound, and converts the caption of the explanatory text "sound of a siren" to a caption "beep beep beep". When the identified type of vehicle is a patrol car, the display mode determination unit 10 determines "woo woo" as the characters expressing the sound, and converts the caption of the explanatory text "sound of a siren" to a caption "woo woo". When the identified type of vehicle is a fire engine, the display mode determination unit 10 determines "Woo-kan-kan-kan" as the characters representing the sound, and converts the caption of the explanation "The sound of a siren" into the caption "Woo-kan-kan-kan."

[0106] As another example, when the display mode determination unit 10 extracts a caption with a description "sound of rain" as a feature of the caption, the display mode determination unit 10 acquires the manner in which the rain falls in the image as a feature of the image. Depending on the manner in which the rain falls, the display mode determination unit 10 can replace the caption with the description "sound of rain" with, for example, "drops," "pitter patter," "buzz," "thud," or the like.

[0107] [Second embodiment] 9 is a block diagram showing the configuration of a subtitle display control system 200 according to the second embodiment. The second embodiment will be described while assigning the same reference numerals to components common to the first embodiment and omitting explanations as appropriate.

[0108] As shown in Fig. 9, the subtitle display control system 200 according to the second embodiment differs from the subtitle display control system 100 according to the first embodiment in that it does not include a content signal receiving unit 1, a signal separating unit 2, a subtitle signal decoding unit 3, an image signal decoding unit 4, and an audio signal decoding unit 5. The subtitle display control system 200 according to the second embodiment includes a plurality of signal receiving units that respectively receive the subtitle signal, the image signal, and the audio signal of the content. Specifically, as shown in Fig. 9, the subtitle display control system 200 includes a subtitle signal receiving unit 23, an image signal receiving unit 24, and an audio signal receiving unit 25.

[0109] In the second embodiment, the server 30 stores subtitle data, image data, and audio data of the content. The server 30 transmits a subtitle signal, an image signal, and an audio signal, which respectively indicate the subtitle data, the image data, and the audio data, to the subtitle display control system 200. The subtitle signal receiving unit 23 receives the subtitle signal transmitted from the server 30. The image signal receiving unit 24 receives the image signal transmitted from the server 30. The audio signal receiving unit 25 receives the audio signal transmitted from the server 30. That is, unlike the first embodiment, the subtitle display control system 200 according to the second embodiment receives a subtitle signal, an image signal, and an audio signal, instead of a multiplexed content signal. The subtitle signal receiving unit 23 outputs the received subtitle signal to the subtitle feature extraction unit 6. The image signal receiving unit 24 outputs the received image signal to the image feature extraction unit 7. The audio signal receiving unit 25 outputs the received audio signal to the audio feature extraction unit 8. The subsequent processes are the same as those in the first embodiment, and therefore a detailed description thereof will be omitted.

[0110] In the second embodiment, each signal of the content does not necessarily have to be provided to the subtitle display control system 200 from one server 30. Each signal of the content may be provided from two or more different servers. For example, as shown in FIG. 10, the subtitle signal may be provided from the first server 31, and the image signal and the audio signal may be provided from the second server 32. In this case, the first server 31 stores the subtitle data of the content, and the second server 32 stores the image data and the audio data of the content. The combination of each data stored in the multiple data is not limited to those shown here, and can be any combination. Also, each signal of the content does not necessarily have to be provided from two servers, and may be provided from three or more servers.

[0111] [Third embodiment] 11 is a block diagram showing the configuration of a subtitle display control system 300 according to the third embodiment. The third embodiment will be described while assigning the same reference numerals to components common to the first embodiment and omitting explanations as appropriate.

[0112] As shown in FIG. 11, the subtitle display control system 300 according to the third embodiment further includes an image data cut-out processing unit 41 and a conversion processing unit 42 in addition to the respective functional units of the subtitle display control system 100 according to the first embodiment.

[0113] The image data cutout processing unit 41 obtains the decoded image signal from the image signal decoding unit 4. The image data cutout processing unit 41 cuts out characters included in the image represented by the image signal. For example, the image data cutout processing unit 41 cuts out the characters by extracting character information embedded in the image using a known image recognition technique. Since the image data cutout processing unit 41 extracts the characters embedded in the image, the characters are not included in the subtitles represented by the subtitle signal. In other words, the image data cutout processing unit 41 cuts out the characters included as an image.

[0114] As an example, assume that the image represented by the image signal is the image shown in Fig. 12. The characters "Autumn leaves in Kyoto" are embedded in this image. The image data cutout processing unit 41 cuts out the characters "Autumn leaves in Kyoto" embedded in the image shown in Fig. 12 using image recognition technology. The image data cutout processing unit 41 inputs data of the cut out characters (here, the characters "Autumn leaves in Kyoto") to the conversion processing unit 42.

[0115] The conversion processing unit 42 converts the characters cut out by the image data cut-out processing unit 41 into a subtitle signal. In this example, the conversion processing unit 42 converts the character data of "Autumn leaves scenery in Kyoto" acquired from the image data cut-out processing unit 41 into a subtitle signal. The conversion processing unit 42 inputs the converted subtitle signal to the subtitle feature extraction unit 6.

[0116] The processes executed after the subtitle feature extraction unit 6 are the same as those in the first embodiment. That is, the subtitle feature extraction unit 6 extracts the features of the subtitles, and the display mode determination unit 10 determines the display mode of the subtitles based on the features of the image. In the third embodiment, the subtitle signal transmitted from the conversion processing unit 42 is treated in the same manner as the subtitle signal input from the subtitle signal decoding unit 3. Therefore, the display mode determination unit 10 determines the display mode of the subtitles shown by the subtitle signal converted by the conversion processing unit 42 based on, for example, the features of the image. The subtitles are displayed on the display unit 16 in the display mode determined by the display mode determination unit 10.

[0117] In the third embodiment, the image signal processing unit 13 may erase, by image processing, the characters cut out by the image data cutout processing unit 41 in the image represented by the image signal. That is, in the example shown in Fig. 12, the characters "Autumn leaves in Kyoto" in the image represented by the image signal are erased by the image signal processing unit 13, and instead, the characters "Autumn leaves in Kyoto" converted into a subtitle signal by the conversion processing unit 42 are displayed in the display mode determined by the display mode determination unit 10.

[0118] [Fourth embodiment] 13 is a block diagram showing the configuration of a subtitle display control system 400 according to the fourth embodiment. The fourth embodiment will be described while assigning the same reference numerals to configurations common to the first embodiment and omitting explanations as appropriate.

[0119] As shown in FIG. 13, the subtitle display control system 400 according to the fourth embodiment further includes an audio data cut-out processing unit 43 and a conversion processing unit 44 in addition to the respective functional units of the subtitle display control system 100 according to the first embodiment.

[0120] The audio data cutout processing unit 43 obtains the decoded audio signal from the audio signal decoding unit 5. The audio data cutout processing unit 43 cuts out the audio indicated by the audio signal. For example, the audio data cutout processing unit 43 cuts out the audio using a known voice recognition technique.

[0121] As an example, assume that the voice represented by the audio signal includes a narration (utterance) of "What beautiful autumn leaves." In this case, the audio data cutout processing unit 43 cuts out the voice of "What beautiful autumn leaves." The audio data cutout processing unit 43 inputs the cut-out audio data to the conversion processing unit 44.

[0122] The conversion processing unit 44 converts the audio clipped by the audio data clipping processing unit 43 into a subtitle signal. In this example, the conversion processing unit 44 converts the audio data of "What beautiful autumn leaves" acquired from the audio data clipping processing unit 43 into a subtitle signal. The conversion processing unit 44 inputs the converted subtitle signal to the subtitle feature extraction unit 6.

[0123] The processes executed after the subtitle feature extraction unit 6 are the same as those in the first embodiment. That is, the subtitle feature extraction unit 6 extracts the features of the subtitles, and the display mode determination unit 10 determines the display mode of the subtitles based on the features of the image. In the fourth embodiment, the subtitle signal transmitted from the conversion processing unit 44 is treated in the same manner as the subtitle signal input from the subtitle signal decoding unit 3. Therefore, the display mode determination unit 10 determines the display mode of the subtitles shown by the subtitle signal converted by the conversion processing unit 44 based on, for example, the features of the image. The subtitles are displayed on the display unit 16 in the display mode determined by the display mode determination unit 10.

[0124] In the above-described embodiment and other processing examples, the contents of each table stored in the storage unit 9 may be preset and stored in the storage unit 9, or may be set by the user himself by performing a predetermined operation input. For example, in the table shown in FIG. 2, the user may be able to freely set the relationship between the volume and the character size. Alternatively, in the table shown in FIG. 3, the user may be able to freely set the relationship between the color information and the font. Furthermore, in the table shown in FIG. 8, the user may be able to freely set the characters representing the sound associated with each of the descriptions relating to the sound. For example, in the example shown in FIG. 8, the characters representing the sound "pururururu" are associated with the description relating to the sound "the sound of the telephone bell", but the user can associate the characters "ring ring" instead of the characters "pururururu" as the characters representing the sound. The same applies to other tables.

[0125] When the user sets the contents of each table stored in the storage unit 9, the user can set the table with contents that match the image while referring to, for example, actual television broadcasts, or set the table with contents according to the user's preference. Alternatively, the user can set the table by searching or downloading preferred font data, character colors, and onomatopoeias (characters that represent sounds) via the network.

[0126] In the above embodiment, the content signal has been described as a signal obtained by multiplexing a subtitle signal, an image signal, and an audio signal. However, the content signal may be a signal obtained by multiplexing any combination of the subtitle signal, the image signal, and the audio signal. In other words, the content signal does not necessarily have to be a signal obtained by multiplexing all of the subtitle signal, the image signal, and the audio signal. In this case, the signal separator 2 may simply separate the multiplexed content signal into the original signals.

[0127] In the above embodiment, it has been described that, with regard to the association between the elements related to the display mode of the subtitles and the features, the character size is associated with the volume, the font is associated with the color information, the character color is associated with the facial expression of the speaker, and the display position is associated with the position of a person or object shown in the image. However, the association between each element related to the display mode of the subtitles and the features is not limited to this, and any association can be made. Therefore, the display mode determination unit 10 can determine the display mode associated with the feature from the feature of the subtitles, image, or sound based on the predetermined association.

[0128] Although the present disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present disclosure. For example, functions included in each functional unit or each step can be rearranged so as not to be logically inconsistent, and multiple functional units or steps can be combined or divided into one. [Explanation of symbols]

[0129] 1 Content signal receiving section 2 Signal separation section 3. Subtitle signal decoding section 4. Image signal decoding section 5 Audio signal decoding section 6. Subtitle feature extraction 7 Image feature extraction section 8. Audio feature extraction unit 9 Memory section 10 Display mode determination unit 11 Subtitle signal conversion unit 12 Subtitle signal processing unit 13 Image signal processing section 14 Audio signal processing section 15 Display control section 16 Display 17 Audio output section 23 Subtitle signal receiving unit 24 Image signal receiving section 25 Audio signal receiving section 30 Servers 31 First Server 32 Second Server 41 Image data cutout processing unit 42,44 Conversion processing section 43 Audio data cutout processing unit 100,200,300,400 Subtitle display control system

Claims

1. A display unit for displaying content; A feature extraction unit that extracts features of the content; a display mode determination unit that determines a display mode of a subtitle based on a feature of the content; a display control unit that causes the subtitle to be displayed on the display unit in the display mode determined by the display mode determination unit; A subtitle display control system comprising:

2. a signal separation unit that separates a subtitle signal, an image signal, and an audio signal from the multiplexed received signal related to the content, The feature extraction unit extracts image features as features of the content from the image signal, the display mode determination unit determines a display mode of the subtitles included in the subtitle signal based on a feature of the image. The subtitle display control system according to claim 1 .

3. The subtitle display control system according to claim 2 , wherein the display mode determination unit determines the display mode based on the characteristics of the image by referring to a table indicating a correspondence relationship between the characteristics of the image and the display mode that is set in advance.

4. The signal separation unit further separates an audio signal from the received signal, An audio feature extraction unit that extracts audio features of the content from the audio signal, The display mode determination unit determines the display mode based on a feature of the image and a feature of the sound. The subtitle display control system according to claim 2 .

5. The subtitle display control system according to claim 1 , wherein the display manner includes at least any one of a character size, a font, a color, and a display position of the subtitles.

6. The subtitle display control system according to claim 2 , wherein the display mode determination unit determines a display position as a display mode of the subtitles based on a position of a sound source as a feature of the image.

7. a subtitle feature extraction unit that extracts features of subtitles of the content from the subtitle signal, The display mode determination unit determines the display mode based on a feature of the image and a feature of the subtitle. The subtitle display control system according to claim 2 .

8. The subtitle display control system according to claim 7 , wherein when the subtitle includes a specific character string as a feature of the subtitle, the display mode determination unit determines the display mode based on color information of the image as a feature of the image.

9. The subtitle display control system according to claim 1 , wherein the display mode determination unit determines a font as the display mode based on a genre of the content.

10. As the subtitle display method, a subtitle on-screen display in which the subtitle is displayed superimposed on the image and a subtitle out-screen display in which the subtitle is displayed without being superimposed on the image can be alternatively selected, The subtitle display control system according to claim 1 , wherein the display mode determination unit adds an animation to the subtitle as the display mode when the subtitle on-screen is selected according to a genre of the content.

11. a subtitle feature extraction unit that extracts features of subtitles of the content from the subtitle signal, When a description regarding an audio is included as a feature of the subtitle, the display mode determination unit replaces the description regarding the audio with characters representing the audio represented by the description regarding the audio. The subtitle display control system according to claim 2 .

12. The subtitle display control system according to claim 11, wherein the display mode determination unit changes the display mode of the replaced characters representing the sound based on whether or not a sound source emitting the sound represented by the description regarding the sound is included in the image as a feature of the image.

13. The subtitle display control system according to claim 1 , further comprising a plurality of signal receiving units each receiving a subtitle signal, an image signal, and an audio signal of the content.

14. an image data cutout processing unit that cuts out characters included in the image represented by the image signal; a conversion processing unit that converts the characters cut out by the image data cut-out processing unit into a subtitle signal; Further equipped with the display mode determination unit determines a display mode of the subtitles represented by the subtitle signal converted by the conversion processing unit. The subtitle display control system according to claim 2 or 13.

15. The subtitle display control system according to claim 14 , further comprising an image signal processing unit that erases, in the image represented by the image signal, the characters cut out by the image data cut-out processing unit through image processing.

16. an audio data clipping processor that clips out the audio represented by the audio signal; a conversion processing unit that converts the audio cut out by the audio data cut-out processing unit into the subtitle signal indicating characters expressing the audio; Further equipped with the display mode determination unit determines a display mode of the subtitles represented by the subtitle signal converted by the conversion processing unit. The subtitle display control system according to claim 2 or 13.

17. A subtitle display control method executed by a subtitle display control system, comprising: Extracting features of the content; determining a display mode of the subtitles based on the characteristics of the content; displaying the subtitle on the display unit in the determined display mode; A subtitle display control method comprising:

Images