Information processing apparatus, control method, and program

Abstract

To provide an information processing apparatus, a control method, and a program capable of making ruby easier to read.SOLUTION: A CPU displays, for characters with ruby in a lyric caption to be displayed on a touch panel monitor, the characters at a character display position P1 and the ruby at a ruby display position P2 adjacent to the character display position P1. The CPU accepts ON / OFF for a main text / ruby switching process via a caption setting screen 52 displayed on the touch panel monitor. If the CPU accepts ON for the main text / ruby switching process, the CPU displays the ruby at the character display position P1 and the characters at the ruby display position P2 at displaying on the touch panel monitor. If the CPU accepts OFF for the main text / ruby switching process, the CPU does not execute the main text / ruby switching process at displaying on the touch panel monitor.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method, and a program. [Background technology]

[0002] As part of universal design, karaoke systems that can display kanji in lyrics with ruby ​​characters are known. For example, when a user logs into the karaoke system, the system identifies and acquires the user ID from a user ID recording medium, and when the user selects a song for which ruby ​​characters are registered, the karaoke system determines whether or not ruby ​​characters should be displayed according to the registered information, and displays the corresponding lyrics caption when the song is played (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-271816 A Summary of the Invention [Problem to be solved by the invention]

[0004] Because ruby ​​is displayed above the lyrics subtitle text, there was a problem that it was difficult to read for users who primarily wanted to see the ruby.

[0005] An object of the present invention is to provide an information processing device, a control method, and a program that can make ruby ​​characters easier to read. [Means for solving the problem]

[0006] An information processing device according to a first aspect of the present invention includes a display control unit that, for characters with ruby ​​annotations in text data to be displayed on a display unit, displays the characters at a character display position and also displays the ruby ​​annotation at a ruby ​​display position adjacent to the character display position; a replacement control unit that executes a replacement process in which, in the display by the display control unit, the ruby ​​annotation is displayed at the character display position and the character is displayed at the ruby ​​display position; and a replacement acceptance unit that accepts the enablement or disablement of the replacement process, wherein the replacement control unit executes the replacement process when the replacement acceptance unit accepts the enablement of the replacement process, and does not execute the replacement process when the replacement acceptance unit accepts the disablement of the replacement process.

[0007] According to the first aspect, by executing a swapping process for the display of characters with ruby ​​annotations, ruby ​​annotations are displayed at the character display position, and characters are displayed at the ruby ​​annotation display position. As a result, when a user mainly wants to look at ruby ​​annotations, ruby ​​annotations are displayed at the same position as other characters without ruby ​​annotations, making it easier to read the text displayed on the display unit. Furthermore, since both characters and ruby ​​annotations are displayed even after the swapping process is executed, even a user who mainly wants to look at characters can read the text displayed on the display unit smoothly. Furthermore, by displaying both characters and ruby ​​annotations, the user can understand the correspondence between characters and ruby ​​annotations.

[0008] A control method according to a second aspect of the present invention is a control method performed by an information processing device, and includes a display control step of, for characters with ruby ​​annotations in text data to be displayed on a display unit, displaying the characters at a character display position and displaying the ruby ​​annotation at a ruby ​​display position adjacent to the character display position, a replacement control step of executing a replacement process of displaying the ruby ​​annotation at the character display position and displaying the character at the ruby ​​display position in the display by the display control step, and a replacement acceptance step of accepting whether the replacement process is enabled or disabled, wherein the replacement control step executes the replacement process if the replacement acceptance step accepts that the replacement process is enabled, and does not execute the replacement process if the replacement acceptance step accepts that the replacement process is disabled.

[0009] According to the second aspect, the effect recited in claim 1 can be obtained by the information processing device performing the above steps.

[0010] A program according to a third aspect of the present invention causes a computer of an information processing device to execute a display control step for displaying characters with ruby ​​annotations in text data to be displayed on a display unit at a character display position and displaying the ruby ​​annotation at a ruby ​​display position adjacent to the character display position, a replacement control step for executing a replacement process for displaying the ruby ​​annotation at the character display position and displaying the character at the ruby ​​display position in the display performed by the display control step, and a replacement acceptance step for accepting whether the replacement process is enabled or disabled, wherein the replacement control step executes the replacement process if the replacement acceptance step accepts that the replacement process is enabled, and does not execute the replacement process if the replacement acceptance step accepts that the replacement process is disabled.

[0011] According to the third aspect, the effect recited in claim 1 can be obtained by causing a computer of an information processing device to carry out the above steps. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the electrical configuration of a karaoke system 10. FIG. [Diagram 2] FIG. 13 is a diagram for explaining the formation of texture data. [Diagram 3] FIG. 4 is a diagram for explaining vertex data. [Figure 4] FIG. 2 is a diagram for explaining index data. [Diagram 5] FIG. 11 is a diagram for explaining index data of the 0th phrase and the 1st phrase. [Figure 6] FIG. 13 is a diagram for explaining index data of "teacher." [Figure 7] FIG. 13 is a diagram for explaining index data of "on" (favor). [Figure 8]FIG. 2 is a diagram showing a configuration of reference data. [Figure 9] FIG. 13 is a diagram showing an example of how lyrics captions are displayed (normally). [Figure 10] FIG. 10 is a diagram showing an example of how lyrics captions are displayed (with the main text and ruby ​​characters swapped from the state shown in FIG. 9). [Figure 11] FIG. 11 is a diagram showing an example of how lyrics captions are displayed (with ruby ​​characters removed from the state shown in FIG. 10). [Figure 12] FIG. 13 is a diagram showing an example of a lyric caption display (with romanized ruby ​​characters added). [Figure 13] FIG. 13 is a diagram showing an example of how lyrics captions are displayed (with the main text and ruby ​​swapped from the state shown in FIG. 12). [Figure 14] FIG. 14 is a diagram showing an example of how lyrics captions are displayed (with ruby ​​characters removed from the state shown in FIG. 13). [Figure 15] 13 is a flowchart of a playback process. [Figure 16] 13 is a flowchart of a telop data generating process. [Figure 17] 13 is a flowchart of a section loop playback process. [Figure 18] FIG. 13 is a diagram showing a section loop playback setting screen (with a starting phrase 611 selected). [Figure 19] FIG. 13 is a diagram showing a section loop playback setting screen (in a state where an end phrase 612 has been selected). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An embodiment of the present invention will be described. The drawings to be referred to are used to explain technical features that the present invention can employ. In other words, the configurations, controls, and the like described in the drawings are merely illustrative examples and are not intended to be limiting.

[0014] The configuration of a karaoke system 10 will be described with reference to Fig. 1. The karaoke system 10 includes a karaoke device 2 (sometimes called a commander) and a remote control device 1. The karaoke device 2 and the remote control device 1 are communicatively connected to a LAN 100 by wire or wirelessly to form a network.

[0015] The karaoke device 2 is installed in a store such as a karaoke booth. The karaoke device 2 includes an audio control unit 25. The audio control unit 25 plays music (accompaniment). The audio control unit 25 adds effects such as echo to the audio signals input from the microphones 43a, 43b, and adds a playback signal based on music information and outputs the result to the speaker 42. For convenience, the speaker 42 is described as one, but various configurations can be adopted, such as multiple speakers 42 for stereo output or surround output. The karaoke device 2 is connected to various devices via the interfaces 22a to 22d. The four interfaces 22a to 22d can be in various configurations, such as one interface, or one interface for each device.

[0016] The karaoke machine 2 comprises an operation unit 21. The operation unit 21 accepts various inputs from the user. The karaoke machine 2 comprises a hard disk 32. The hard disk 32 is a storage unit that stores various information. The karaoke machine 2 comprises a LAN communication unit 24a. The LAN communication unit 24a is a communication means that connects to the LAN 100 and joins the network. The karaoke machine 2 further comprises a wireless LAN communication unit 24b. The karaoke machine 2 can also make a wireless network connection using the wireless LAN communication unit 24b, instead of a wired network connection using the LAN communication unit 24a.

[0017] The karaoke device 2 includes a video playback unit. The video playback unit displays lyrics video, background video, and the like on a monitor 41. The video playback unit includes a GPU 29 (Graphics Processing Unit), a RAM 28, and a video output unit 31. The GPU 29 forms video based on various information. The RAM 28 stores information required for video formation. The video output unit 31 outputs the played video to both the monitor 41 and the touch panel monitor 33.

[0018] The karaoke device 2 can display various information on the touch panel monitor 33 in addition to the externally connected monitor 41. The touch panel monitor 33 includes a display unit 35 and a touch panel 34. The display unit 35 displays the video input from the video output unit 31. The touch panel 34 outputs the touch input position to the interface 22b. The touch panel monitor 33 is disposed on the front of the housing of the karaoke device 2, and functions as an input unit that accepts various operations from the user. By selecting a song on the touch panel monitor 33, the user can perform various operations on the karaoke device 2, such as directly reserving the song on the karaoke device 2.

[0019] The control unit of the karaoke device 2 includes a CPU 30 (Central Processing Unit) and a memory 27. The CPU 30 controls each component of the karaoke device 2. The memory 27 temporarily stores information required for executing various programs. The video playback unit may be a video card configured as a single device. A cache for temporarily storing data may be provided between the GPU 33 and the RAM 28.

[0020] The karaoke device 2 having the above configuration can execute song reservation processing and playback processing, etc. The song reservation processing is processing for specifying and reserving a song based on a user's instruction, and is executed in cooperation with the remote control device 1. Reservation information formed by the song selection processing of the remote control device 1 is transmitted to the karaoke device 2. The karaoke device 2 registers the received reservation information in a reservation table in the memory 27. The playback processing is processing for playing a reserved song, and is processing in which a performance processing, a lyric display processing, and a video playback processing are executed synchronously.

[0021] The performance process is a process for causing the audio control unit 25 to perform a performance based on the performance data included in the music data. The music performed by the audio control unit 25 is mixed with audio signals input from the microphones 43a and 43b and is output from the speaker 42. The lyrics display process is a process for providing singing assistance by displaying a lyrics video (telop) based on the lyrics data included in the music data on the monitor 41, the touch panel monitor 33, etc. Note that a background video display process may be executed in which a background video is superimposed on the lyrics video displayed in the lyrics display process.

[0022] The remote control device 1 can execute a song selection process. In the song selection process, a song is searched for based on an instruction from the user, and reservation information for the song instructed to be played is sent to the karaoke device 2. The remote control device 1 can receive various information from the karaoke device 2 or a server device 5 connected to the Internet, and execute various processes. The operation unit 17 and the touch panel monitor 11 are a user interface that receives various instructions from the user. The touch panel monitor 11 includes a display unit 11a and a touch panel 11b, and can display various information on the display unit 11a and receive touch input from the user.

[0023] The remote control device 1 comprises a memory 14 and a remote control control section. The memory 14 stores a database required for music selection processing, various programs, and various information generated with program execution. The remote control control section controls the overall configuration of the remote control device 1. The remote control control section comprises a CPU 15, an image control section 13, a video RAM 12, and an operation processing section 18. The image control section 13 forms an image to be displayed on the touch panel monitor 11. The video RAM 12 temporarily stores image data of the image to be displayed. The operation processing section 18 interprets input from the touch panel monitor 11 or the operation section 17 and transmits it to the CPU 15.

[0024] The remote control device 1 is connected to a network formed by a LAN 100 by being wirelessly connected to an access point 130 via a wireless LAN communication unit 16. Each remote control device 1 is pre-associated with a specific karaoke device 2. Various commands output from the remote control device 1 are received by the associated karaoke device 2.

[0025] The remote control device 1 having the above configuration accepts various inputs from a user via the touch panel monitor 11 or the operation unit 17. The remote control device 1 can perform various processes such as song selection by displaying various information on the touch panel monitor 11. The song selection process is a process of transmitting reservation information to be output to the karaoke device 2.

[0026] A method for creating telop data will now be described. The telop data is data for displaying lyric telops on the monitor 41 and touch panel monitor 33 using the GPU 29. In this embodiment, the telop data is created from lyric data, but instead of this, the telop data may be included in the music data in advance.

[0027] The lyric data is composed of text data, and includes spaces, line breaks, and blank lines. The lyric data also includes timing data that indicates the display start time, erasure time, and singing position for each appropriate unit such as each character. In this embodiment, the lyric data is converted into telop data that is easy to handle by the GPU 29, and the lyrics are displayed using the GPU 29. The CPU 30 forms telop data based on the lyric data. The telop data in this embodiment includes (1) texture data, (2) vertex data, and (3) reference data.

[0028] (1) Texture data FIG. 2 is a diagram for explaining the formation of texture data. This example uses the lyrics of the Japanese song "Aogeba Totoshi" shown in FIG. 3 as an example. First, overlapping characters are removed from the text data forming the lyrics information. Next, using font data or the like, the characters from which overlapping has been removed are rasterized (bitmapped) to form a single piece of image data (texture data). Note that ruby ​​characters are added to the kanji characters in the lyrics of "Aogeba Totoshi." Characters with ruby ​​characters added are rasterized as characters including the ruby ​​characters.

[0029] (2) Vertex data The vertex data is an array of data related to each vertex on a three-dimensional model. In other words, it is an array of data related to each vertex that forms a plurality of polygons. In the embodiment, one polygon is assigned to one character. Note that one polygon may be assigned to a portion of a character or to multiple characters. The data related to each vertex includes three-dimensional coordinates, texture coordinates, and index data.

[0030] (2-1) Three-dimensional coordinates Three-dimensional coordinates are data that define the positions of polygons that are placed in three-dimensional space. FIG. 3 shows an example of the lyrics of the Japanese song "Aogeba Totoshi" (only the first verse is shown) placed in three-dimensional space. The three-dimensional coordinates indicate the positions, i.e., coordinates, of each vertex of each polygon that corresponds to each character that makes up the lyrics when the lyrics are placed in three-dimensional space. Note that the characters shown within the polygons in FIG. 3 are added for ease of understanding, and in reality, the characters are visualized by pasting image data cut out from texture data into the polygons.

[0031] (2-2) Texture coordinates Texture coordinates are data that define the position of the portion of the texture data to be applied to each polygon. In other words, texture coordinates indicate the positions, or coordinates, that correspond to the vertices of each polygon in the texture data. By referencing the texture coordinates, it is possible to extract the necessary character image data from the texture data. In other words, it is possible to extract the necessary character image data from the texture data and apply it to each polygon.

[0032] (2-3) Index Data Index data is information assigned to each vertex forming a polygon. FIG. 4 is a diagram for explaining index data. In this embodiment, lyrics are assigned information on a single character unit "character", a "phrase" formed of one or more characters, a "line" formed of one or more phrases, and a "chapter" formed of one or more lines. As described above, lyrics data is composed of text data including spaces, line breaks, and blank lines. When index data is created from lyrics data, for example, a group up to a space is determined as a "phrase", a group up to a line break is determined as a "line", and a group up to a blank line is determined as a "chapter". In addition to this form, various forms such as tagging lyrics data with "phrases", "lines", and "chapters" can be adopted.

[0033] Fig. 5 is a diagram for explaining index data assigned to each vertex of a polygon. Fig. 5 explains the index data assigned to each vertex of a polygon by taking the first two phrases in Fig. 4, "Aogeba toushi waga shi no on" (Aogeba toushi waga shi no on) as an example. In this embodiment, the initial value for characters, phrases, lines, and chapters is "0".

[0034] For the first phrase (phrase 0), for the first character (character 0), the character 'あ', index data (0, 0, 0, 0) is assigned to each of the upper left and lower left vertices. This index data indicates that it is the 0th character, 0th phrase, 0th line, and 0th chapter. Also, index data (0.99, 0, 0, 0, 0) is assigned to each of the upper right and lower right vertices of the first character 'あ'. The value '0.99' is assigned to the character because a value smaller than 1 is used in order to assign 1 to the next character (character 1).

[0035] For the next character (character 1), the character 'お', index data (1, 0, 0, 0) is assigned to each of the upper left and lower left vertices. And index data (1.99, 0, 0, 0, 0) is assigned to each of the upper right and lower right vertices of 'お'. For the 2nd character 'げ', 3rd character 'ば', 4th character 'と', 5th character 'う', 6th character 'と', 7th character 'し', index data is assigned to each vertex according to the same rule.

[0036] For the next phrase (phrase 1), for the 8th character 'わ', index data (8, 1, 0, 0) is assigned to each of the upper left and lower left vertices. This index data indicates that it is the 8th character, 1st phrase, 0th line, and 0th chapter. And index data (8.99, 1, 0, 0, 0) is assigned to each of the upper right and lower right vertices of 'わ'. For the 9th character 'が', 10th character '師', 11th character 'の', 12th character '恩', index data is assigned to each vertex according to the same rule.

[0037] Here, as shown in FIG. 6, the 10th character '師' has a furigana of one character 'し'. For the furigana 'し', index data (10, 1, 0, 0) is assigned to each of the upper left and lower left vertices, similar to the index data of '師'. Index data (10.99, 1, 0, 0) is assigned to each of the upper right and lower right vertices.

[0038] As shown in FIG. 7, for the 12th character "恩", two furigana characters "お" and "ん" are arranged horizontally. For the furigana character "お", index data (12, 1, 0, 0) is assigned to each of the upper left and lower left vertices. For each of the upper right and lower right vertices, index data (12.49, 1, 0, 0) is assigned. Similarly, for the furigana character "ん", index data (12.50, 1, 0, 0) is assigned to each of the upper left and lower left vertices. For each of the upper right and lower right vertices, index data (12.99, 1, 0, 0) is assigned. That is, in order to distinguish "お" and "ん" from each other, index data (12.49, 1, 0, 0) is assigned to each of the upper right and lower right vertices of "お", and index data (12.50, 1, 0, 0) is assigned to each of the upper left and lower left vertices of "ん".

[0039] By assigning index data to each vertex of the polygon in this way, it is possible to easily perform telop control in units of characters, phrases, lines, and chapters. Also, since the index data has a data structure of a four-dimensional array (characters, phrases, lines, chapters), similar to the data handled by the GPU 29, efficient processing can be performed in the parallel processing of the GPU 29.

[0040] (3) Reference Data The reference data is data that defines various information about the characters, phrases, lines, and chapters that form the lyrics. FIG. 8 is a diagram for explaining the reference data of the present embodiment. As shown in FIG. 8, the reference data has information for each character T, phrase P, line L, and chapter C.

[0041] For the 0th character (the character "あ" in the example of the lyrics in FIG. 4), it includes index data, rectangular information indicating the position and size of the character, and the time range of the character. The index data is the information described in FIG. 5, and for this 0th character, it is (0, 0, 0, 0).

[0042] The rectangular information indicating the position and size of the character is the same as the vertex data explained in Fig. 3, and in the case of the 0th character, it is information indicating the position and size of the polygon that constitutes "a". When processing each piece of vertex data, the GPU 29 cannot refer to information on other vertex data. Therefore, by including rectangular information indicating the position and size of the character in the reference data, when processing each piece of vertex data, the control position of the character to which the vertex data belongs or other characters can be referenced by referring to the reference data.

[0043] The time range of a character is information indicating timing. In this embodiment, (Sn-1_end, Sn_start, Sn_end, Sn+1_start) is assigned as the time range of a character. Sn-1_end indicates the end timing of the previous character Sn-1, Sn_start indicates the start timing of the character Sn, Sn_end indicates the end timing of the character Sn, and Sn+1_start indicates the start timing of the next character Sn+1.

[0044] Here, when displaying the character Sn, two pieces of information, the start timing Sn_start of the character Sn and the end timing Sn_end of the character Sn, are sufficient. In this embodiment, like other data handled by the GPU 29, a four-dimensional array data structure is used, and Sn-1_end and Sn+1_start are included. By including these, processing using the timing of the previous character and the following character can be performed. As a comparative example, when Sn-1_end and Sn+1_start are not included, when processing using the timing of the previous character and the following character is performed, it is necessary to refer to the reference data of the character Sn, the reference data of the character Sn-1, and the reference data of the character Sn+1, which results in a large number of references and a high processing load. In this embodiment, it is possible to reduce the number of references compared to the comparative example, thereby reducing the processing load.

[0045] Regarding Sn-1_end and Sn+1_start, they may not be included in the time range. In that case, since the time range will have a two-dimensional array data structure, it may be made into a four-dimensional array data structure by including two empty data items to facilitate handling by GPU29. This also applies to the time ranges of phrases, lines, and chapters, which will be explained later. And in the same way as for the 0th item, the reference data will define the information from the 1st to the mXth (last character).

[0046] In this embodiment, the index data of the 10th character (the character "師" in the example lyrics of FIG. 4) includes the ruby index data (the single character "し" in the example lyrics of FIG. 4). The rectangular information indicating the position and size of the character is the rectangular information of the size including the ruby. The time range of the ruby is the same as the time range of the character corresponding to that ruby. Also, the index data of the 12th character (the character "恩" in the example lyrics of FIG. 4) includes the ruby index data (the characters "おん" in the example lyrics of FIG. 4). And similar to the 10th character, the rectangular information and the time range are defined.

[0047] For the 0th phrase (the characters "あおげばとうとし" in the example lyrics of FIG. 4), it includes the index data of the first character of the phrase, the rectangular information indicating the position and size of the phrase, and the time range of the phrase. The index data of the first character of the phrase stores the index data of the 0th character and (0,0,0,0).

[0048] The rectangular information indicating the position and size of the phrase, in the case of the 0th phrase, is the information indicating the position and size of the area surrounding "あおげばとうとし". When GPU29 processes each vertex data item, it cannot refer to the information of other vertex data items. Therefore, by including the rectangular information indicating the position and size of the phrase in the reference data, the control position of the phrase can be referred to when referring to the reference data.

[0049] The time range of a phrase is information indicating timing. In this embodiment, (Tn-1_end, Tn_start, Tn_end, Tn+1_start) is assigned as the time range of a phrase. Tn1_end indicates the end timing of the previous phrase Tn-1, Tn_start indicates the start timing of the phrase Tn, Tn_end indicates the end timing of the phrase Tn, and Tn+1_start indicates the start timing of the subsequent phrase Tn+1.

[0050] As in the case of characters, when displaying the phrase Tn, two pieces of information are sufficient: the start timing Tn_start of the phrase Tn and the end timing Tn_end of the phrase Tn, but in this embodiment, by including Sn-1_end and Sn+1_start, processing can be performed using the timing of the previous and following phrases. And, like the 0th phrase, the reference data specifies information on the 1st to mYth phrases (the last phrase).

[0051] As with phrases, the data for a line (or chapter) also includes index data for the first character of the line (or chapter), rectangular information indicating the position and size of the line (or chapter), and the time range of the line (or chapter).

[0052] In this way, the reference data specifies index data, rectangular information indicating the position and size, and time range (timing) for characters, phrases, lines, and chapters, making it easy to control the display timing of characters, phrases, lines, and chapters. Note that it is sufficient for the reference data to include information indicating the time range (timing) for characters, phrases, lines, and chapters.

[0053] In this embodiment, the above-mentioned telop data (texture data, vertex data, reference data) is used to cause the GPU 29 to display lyrics telop on the touch panel monitor 33. At this time, a variety of telop expressions can be achieved by changing the shader program for forming the telop in the GPU 29. As will be described later, the CPU 30 transmits a shader program corresponding to a piece of music to the RAM 28 for each playback process. For example, by using a shader program corresponding to a piece of music, a telop expression corresponding to the piece of music can be achieved.

[0054] The shader program is a program that specifies how the subtitles are displayed on the screen, and can control the position, color, etc., at which characters, chapters, lines, and phrases are displayed. The shader program of this embodiment includes a vertex shader program and a pixel shader program.

[0055] The vertex shader program is a program that specifies the display position, and can control display, for example, by character, phrase, line, or chapter. Display control includes layouts such as right-justified, left-justified, stepped, and line breaks, as well as animations such as scrolling and pop-ups. Display control is not limited to the layouts and animations exemplified above, but includes various other layouts and animations. In addition, the vertex shader program can express subtitles as a stereoscopic image captured from a specified camera viewpoint in a three-dimensional space.

[0056] The pixel shader program is capable of controlling color information for each pixel that forms the caption, and can control various color expressions for the caption to be displayed, such as using a single color or a gradation.

[0057] In this embodiment, a shader program is formed by combining a vertex shader program and a pixel shader program. By preparing a plurality of vertex shader programs and a plurality of pixel shader programs in advance and changing the combination, it is possible to form shader programs that express a variety of subtitles.

[0058] 9 to 14, the display settings of the playback screen 51 and ruby ​​displayed on the touch panel monitor 33 will be described. When a music playback operation is performed, the playback screen 51 (see FIG. 9) is displayed in the entire area of ​​the touch panel monitor 33. The lyrics of the music being played are displayed as a telop on the playback screen 51. The telop display is, for example, broken into lines for each phrase, and the multiple phrases displayed on the playback screen 51 are displayed in a staircase pattern. In this case, the vertex shader program performs coordinate conversion on the vertex data described in FIG. 3, and arranges multiple polygons on the screen so that a predetermined number of phrases (two phrases in the example of FIG. 9) are arranged in a staircase pattern. The pixel shader program displays characters on each polygon based on an image extracted from the texture data. The pixel shader program determines the current singing position based on the current position transmitted from the CPU 30, and displays and outputs a telop by changing the color before and after the current singing position.

[0059] When the caption display settings are operated while music is being played, the playback screen 51 slides to the right and shrinks, and the caption setting screen 52 appears to the left of it (see Fig. 9). On the caption setting screen 52, it is possible to select the caption font, select the text layout, turn the ruby ​​display function on / off, turn the romaji ruby ​​function on / off, turn the text / ruby swap function on / off, etc.

[0060] The ruby ​​display function is a function for displaying ruby ​​in a telop, and is turned ON by sliding slider 511 to the right and turned OFF by sliding slider 511 to the left. When the ruby ​​display function is turned ON, ruby ​​is displayed, and when it is turned OFF, ruby ​​is not displayed. The romaji ruby ​​function is a function for displaying romaji ruby, and is turned ON by sliding slider 512 to the right and turned OFF by sliding slider 512 to the left. When the romaji ruby ​​function is turned ON, ruby ​​in romaji is displayed, and when it is turned OFF, normal hiragana ruby ​​is displayed. The text / ruby swap function is a function for swapping the display positions of characters and ruby ​​for characters in a telop to which ruby ​​is added, and is turned ON by sliding slider 513 to the right and turned OFF by sliding slider 513 to the left. When the text / ruby swap function is turned ON, the display positions of the characters and ruby ​​are swapped, and when it is turned OFF, the characters and ruby ​​are displayed in their designated display positions. Regarding ON / OFF switching of the above three functions, it is advisable to highlight the sliders 511 to 513 or change their color when the sliders 511 to 513 are slid to the right to turn them ON. This allows the user to clearly distinguish the ON / OFF status of each function.

[0061] An example of switching between the above three functions will be described below. The telop display on the playback screen 51 shown in Fig. 9 is in the initialized normal state. At this time, the ruby ​​display function is ON, the romaji ruby ​​function is OFF, and the text-ruby swap function is OFF. The telop characters are displayed at character display position P1, and the ruby ​​is displayed at ruby ​​display position P2 in a size smaller than the characters. The ruby ​​display position P2 is the position directly above the character display position P1. In the example shown in Fig. 9, the ruby ​​"shi" is displayed above the character "shi" and the ruby ​​"on" is displayed above the character "on."

[0062] When the text and ruby replacement function is turned on from the normal state, as shown in FIG. 10, for the characters with ruby in the telop, the display positions of the character and the ruby are swapped. The ruby is displayed at the character display position P1 in the same size as other characters, and the character is displayed at the ruby display position P2 in a size smaller than the ruby. In the example shown in FIG. 10, the display positions of "師" and "し", and "恩" and "おん" are swapped with each other. "し" is displayed at the character display position P1 in the same size as other characters, and "師" is displayed at the ruby display position P2 in a size smaller than the character and in the normal ruby size. "おん" is also displayed at the character display position P1 in the same size as other characters, and "恩" is displayed at the ruby display position P2 in a size smaller than the character and in the normal ruby size.

[0063] In this way, by swapping the display positions of the character and the ruby with each other, the telop displayed on the playback screen 51 can be made easier to read. For example, it can be made even easier to read for users who mainly want to see the ruby, such as children who are not good at Chinese characters or foreigners. Also, in this embodiment, since both the character and the ruby are displayed even when the text and ruby replacement function is turned on, users who mainly want to see the character can also read the telop displayed on the playback screen 51 smoothly. Also, by displaying both the character and the ruby, the correspondence between the character and the ruby can be grasped.

[0064] Next, when the ruby display function is turned off from the state of FIG. 10, as shown in FIG. 11, the character displayed at the ruby display position P2 is erased in the state where the display positions of the character and the ruby are swapped with each other. Thereby, a telop with only hiragana reflecting the reading of the ruby can be displayed, so that the telop can be made even easier to read for users who mainly want to see the ruby.

[0065] Next, if, from the state shown in Figure 11, the ruby ​​display function is turned ON, the romaji ruby ​​function is turned ON, and the text / ruby swap function is turned OFF, then, as shown in Figure 12, all characters in the caption will be displayed with romaji ruby ​​added, as shown in Figure 12. The characters are displayed at character display position P1, and the romaji ruby ​​is displayed at ruby ​​display position P2. In the example shown in Figure 12, the phrases "aogebatoutoshi" and "wagashi no on" are displayed at character display position P1, and the romaji ruby ​​for "aogebatoutoshi" and "wagashinoon" are displayed above them at ruby ​​display position P2 for each character.

[0066] Next, when the main text / ruby swap function is turned ON from the state shown in Fig. 12, the display positions of the text and ruby ​​are swapped as shown in Fig. 13, with the romaji ruby ​​being displayed in the same size as the normal text at text display position P1, and the text being displayed in the same size as the normal ruby ​​at ruby ​​display position P2. In the example shown in Fig. 13, the romaji ruby ​​"aogebatoutoshi" and "wagashinoon" are displayed in the same size as the normal text at text display position P1 for each phrase, and above that "aogebatoutoshi" and "wagashinoon" are displayed in the same size as the normal ruby ​​for each phrase at ruby ​​display position P2.

[0067] In this way, by switching the display positions of the text and ruby, the telop display mainly uses romaji, making it easier for foreigners who are not fluent in Japanese to read the telop displayed on the playback screen 51. Even when the text / ruby swap function is ON, both hiragana characters and romaji ruby ​​are displayed, so for example, if a foreigner and a Japanese person are enjoying karaoke together, not only the foreigner but also the Japanese person can read the telop displayed on the playback screen 51 without any problems. Even when the text / ruby swap function is ON, by displaying both the text and ruby, the correspondence between the text and the romaji ruby ​​can be understood.

[0068] Then, when the ruby ​​display function is turned off in the state of Fig. 13, the display positions of the hiragana characters and the romaji ruby ​​are switched, and the characters displayed at ruby ​​display position P2 are erased, as shown in Fig. 14. This allows captions to be displayed in romaji only, making the captions easier to read for users who primarily want to see the romaji ruby.

[0069] The playback process executed by the CPU 30 will be described with reference to Fig. 15. When the power supply of the karaoke device 2 is turned on and the music playback mode is set, the CPU 30 of the karaoke device 2 reads out the playback program from the hard disk 32 and executes this process. When a music piece is reserved using the remote control device 1 or the operation unit 21 of the karaoke device 2, the music ID indicating the reserved music piece is registered in a reservation table managed in the memory 27.

[0070] The CPU 30 checks the reservation table (S11) and determines whether there is a song to be played next (S12). If there is no song to be played next (S12: NO), the CPU 30 returns to S11 and waits until a song to be played is reserved. If there is a song to be played next (S12: NO), the CPU 30 initializes various flag variables D, R, and S (S13). The variable D is a flag variable that indicates whether the ruby ​​display function is ON / OFF, and is set to "true" when it is ON and "false" when it is NO. The variable R is a flag variable that indicates whether the romaji ruby ​​function is ON / OFF, and is set to "true" when it is ON and "false" when it is NO. The variable S is a flag variable that indicates whether the main text / ruby swap function is ON / OFF, and is set to "true" when it is ON and "false" when it is NO. When the various flag variables are initialized, the variable D="true", the variable R="false", and the variable S="false". That is, when the various functions are initialized, the ruby ​​display function is ON, the romaji ruby ​​function is OFF, and the main text / ruby swap function is OFF. The various flag variables may be stored in the memory 27, for example.

[0071] Next, the CPU 30 transmits the shader program to the RAM 28 (S14). By transmitting the shader program to the RAM 28, it becomes possible for the GPU 29 to display a telop. The CPU 30 determines whether a new piece of music has been designated, or whether the variable R or the variable S has been changed (S15). If a new piece of music has not been designated, and neither the variable R nor the variable S has been changed (S15: NO), the CPU 30 advances the process to S17, which will be described later, without generating new telop data. On the other hand, if a new piece of music has been designated, or whether the variable R or the variable S has been changed (S15: YES), the CPU 30 executes a telop data generation process (S16).

[0072] The subtitle data generation process will be described with reference to FIG. 16. The CPU 30 reads out the specified music data (S31). The CPU 30 judges whether the variable R is "true" (S32). If the variable R is "true" (S32: YES), the romaji ruby ​​function is ON and the CPU 30 executes the romaji ruby ​​adding process (S33). In the romaji ruby ​​adding process, the CPU 30 performs a process of adding romaji ruby ​​to the lyric characters in the lyric data of the specified music. If the variable R is "false" (S32: NO), the romaji ruby ​​function is OFF and the CPU 30 does nothing and proceeds to S34 described below.

[0073] Next, CPU 30 determines whether variable S is "true" (S34). If variable S is "true" (S34: YES), the main text / ruby swap function is ON, so CPU 30 executes main text / ruby swap processing (S35). In main text / ruby swap processing, CPU 30 performs processing to mutually swap the positions of the characters and ruby ​​in the lyric data of the specified song. If variable S is "false" (S34: NO), the main text / ruby swap function is OFF, so CPU 30 does nothing and proceeds to S36 described below.

[0074] Next, the CPU 30 creates telop data from the lyrics data (S36) and transmits the created telop data to the RAM 28 (S37). In this manner, the CPU 30 ends the telop data generation process, and returns to the playback process shown in Figure 15, proceeding to S17.

[0075] Next, the CPU 30 causes the audio control unit 25 to start playing the performance data (S17). At this time, the CPU 30 synchronously executes the performance process, lyric display process, and video playback process described above. The CPU 30 determines whether the variable D has been initialized or changed (S18). If the variable D has been initialized or changed (S18: YES), the ruby ​​display function has been switched ON / OFF, so the CPU 30 sends the variable D to the shader program stored in the RAM 28 (S19). This allows the GPU 29 to display or not display ruby ​​in the telop based on the shader program reflecting the variable D. In this case, the GPU 29 simply needs to make the ruby ​​transparent.

[0076] Next, the CPU 30 transmits the current performance position (playback position) to the shader program stored in the RAM 28 in synchronization with the playback of the performance data (S20). The CPU 30 transmits the current performance position to the shader program periodically (for example, for each frame to be drawn). The GPU 29 performs lyrics display processing based on the telop data stored in the RAM 28, the shader program, and the current performance position received from the CPU 30, and displays the lyrics (telop display) on the touch panel monitor 33.

[0077] The vertex shader program executed by the GPU 29 refers to the time range (timing) in the reference data, determines the characters (or phrases, lines, or chapters) to be displayed, and determines the display position based on the vertex data. At this time, depending on the vertex shader program, coordinate conversion is performed. The pixel shader program can also control the color of the character image to be displayed, and can display characters on polygons based on images extracted from texture data. The pixel shader program can also change the color of characters based on the current playing position, or can display characters in a variety of colors.

[0078] The CPU 30 determines whether or not the playback of the performance data has ended (S21). If the playback has not ended (S21: NO), the CPU 30 returns to S15 and repeats the above process until the playback has ended. If the playback has ended (S21: YES), the CPU 30 determines whether or not the execution of the music playback mode of the playback mode has ended (S22). If the music playback mode has not ended (S22: NO), the CPU 30 returns to S11 and repeats the above process. If the execution of the music playback mode has ended (S22: YES), the CPU 30 ends the playback process.

[0079] The section loop playback process will be described with reference to Figs. 17 to 19. Section loop playback is a function for repeatedly playing back a section of a specified song. Note that this function may be so-called non-loop section playback, in which a section of a specified song is played back. When the user performs an operation to execute section loop playback on the operation unit 21, the CPU 30 reads out a section loop playback program from the hard disk 32 and executes this process.

[0080] As shown in Fig. 17, the CPU 30 displays a setting screen 61 on the touch panel monitor 33 (S41). As shown in Fig. 18, a phrase list button 621, a performance stop button 622, a decision button 623, etc. are displayed in the lower right of the setting screen 61. When the user specifies a piece of music and presses the phrase list button 621, the CPU 30 displays a list of phrases for the specified piece of music.

[0081] For example, when the song "Aogeba Totoshi" is specified and the phrase list button 621 is pressed, the lyrics of "Aogeba Totoshi" are displayed in a list on the setting screen 61, phrase by phrase. Each phrase is enclosed in a roughly square frame, and two phrases are displayed side by side vertically, with a new line every two phrases. A space is provided between the frames of adjacent phrases. This makes it easier for the user to intuitively identify the lyrics on a phrase by phrase basis.

[0082] To specify the section to be played back in a loop, the user selects the start phrase of the section to be played back in a loop from the list of phrases displayed on the setting screen 61. Although not shown, it is advisable to display a message such as "Loop playback: Select start position" at the top of the setting screen 61. This allows the user to clearly see the next operational step to be taken. The CPU 30 determines whether the start phrase has been selected (S42). Until the start phrase is selected (S42: NO), the CPU 30 returns to S42 and waits. For example, to set the four phrases from "Now is the time to say goodbye" to "Daily favors" as the section to be played back in a loop, the user selects the phrase "Now is the time to say goodbye" as the start phrase.

[0083] When a start phrase is selected (S42: YES), the CPU 30 highlights the selected start phrase (S43) and grays out the phrases before the selected start phrase (S44). For example, as shown in FIG. 18, the phrase "Now is the time to say goodbye" selected as the start phrase 611 is highlighted. This allows the user to clearly identify the specified start phrase and its position in the music. Furthermore, phrases before the start phrase 611 are grayed out, and these grayed out phrases cannot be selected as the end position. This makes it easy to recognize the range that can be selected as the end phrase. Note that the message "Loop playback: Select end position" is displayed at the top of the setting screen 61. This allows the user to clearly recognize that the next operation step is to select the end phrase.

[0084] The CPU 30 judges whether an end phrase has been selected (S45). Until an end phrase is selected (S45: NO), the CPU 30 returns to S45 and waits. If the user wants to reselect the start phrase 611, the user can cancel the selection of the start phrase 611. In this case, the CPU 30 returns to the process of S42.

[0085] When the end phrase 612 is selected (S45: YES), the CPU 30 highlights the start phrase 611 to the end phrase 612 (S46). For example, as shown in FIG. 19, when "Daigoro no On" is selected as the end phrase 612, the four phrases from the start phrase 611 to the end phrase 612 are highlighted as the loop playback section. This makes it easy to identify the section in the song that will be looped. At the top of the setting screen 61, a message is displayed saying "Loop playback: Press 'OK' when decided." This allows the user to clearly recognize that the next operational step is the step of deciding the playback section.

[0086] The CPU 30 judges whether the decision button 623 has been pressed (S47). Until the decision button 623 is pressed (S47: NO), the CPU 30 returns to S45 and repeats the above process. If the user reselects another phrase as the end phrase 612 (S45: YES), the CPU 30 highlights the section from the start phrase 611 to the reselected end phrase 612 as the loop playback section (S46). This allows the user to reselect the end phrase 612 without canceling the selection. Then, if the decision button 623 is pressed (S47: YES), the CPU 30 executes loop playback of the specified section (S48). When section loop playback is executed, the specified section is repeatedly played back.

[0087] To end the section loop playback, the user need only press the performance stop button 622. The CPU 30 judges whether the performance stop button 622 has been pressed (S49). Until the performance stop button 622 is pressed (S49: NO), the CPU 30 returns to S48 and continues the section loop playback. If the performance stop button 622 has been pressed (S49: YES), the CPU 30 ends this process.

[0088] In the above description, the touch panel monitor 33 is an example of a "display unit" of the present invention. The CPU 30 that executes the process of S15 is an example of an "exchange receiving unit" and an "addition receiving unit" of the present invention. The CPU 30 that executes the process of S18 is an example of a "display receiving unit" of the present invention. The CPU 30 that executes the process of S35 is an example of an "exchange control unit". The CPU 30 that executes the process of S33 is an example of a "Roman alphabet ruby ​​addition unit". The text / ruby exchange process is an example of the "exchange process" of the present invention. The Roman alphabet ruby ​​addition process is an example of the "addition process" of the present invention.

[0089] As described above, the karaoke system 10 of the present embodiment includes the karaoke device 2. The CPU 30 of the karaoke device 2 displays lyric telop on the touch panel monitor 33. For characters with ruby ​​added in the lyric telop, the CPU 30 displays the characters at the character display position P1 and also displays ruby ​​at the ruby ​​display position P2 adjacent to the character display position P1. The CPU 30 accepts ON / OFF of the text / ruby swap process on the telop setting screen 52 displayed on the touch panel monitor 33. When the CPU 30 accepts ON of the text / ruby swap process, the CPU 30 displays ruby ​​at the character display position P1 and also displays characters at the ruby ​​display position P2 in the display on the touch panel monitor 33. When the CPU 30 accepts OFF of the text / ruby swap process, the CPU 30 does not execute the text / ruby swap process in the display on the touch panel monitor 33. As a result, when the user mainly wants to see ruby, the ruby ​​is displayed at the same character display position P1 as other characters without ruby ​​added, so that the telop displayed on the touch panel monitor 33 can be easily read. For example, in the case of lyrics captions in which hiragana ruby ​​characters are added to kanji characters, turning on the text / ruby swap process has the effect of making it easier for people who are not good at kanji to sing. Even when the text / ruby swap process is executed, both the text and ruby ​​characters are displayed, so even users who mainly want to see the text can smoothly read the caption displayed on the touch panel monitor 33. Also, by displaying both the text and ruby ​​characters even when the text / ruby swap process is executed, the user can understand the correspondence between the text and ruby ​​characters.

[0090] According to the above embodiment, the CPU 30 accepts ON / OFF of the ruby ​​display function on the telop setting screen 52 displayed on the touch panel monitor 33. When ON of the ruby ​​display function is accepted, the CPU 30 displays characters or ruby ​​at the ruby ​​display position P2 depending on whether the main text / ruby exchange process is ON / OFF. When OFF of the ruby ​​display function is accepted, the CPU 30 does not display either characters or ruby ​​at the ruby ​​display position P2 regardless of whether the main text / ruby exchange process is ON / OFF. For example, by executing the main text / ruby exchange process and turning OFF the ruby ​​display function, it is possible to display a lyric telop consisting of only characters that reflect the reading of the ruby. This allows the karaoke system 10 to make the lyric telop easier to read for users who want to mainly see the ruby.

[0091] According to the above embodiment, the CPU 30 accepts ON / OFF of the romaji ruby ​​function on the telop setting screen 52 displayed on the touch panel monitor 33. When ON of the romaji ruby ​​function is accepted, the CPU 30 generates romaji ruby ​​based on the lyrics data of the music piece, and adds the generated romaji ruby ​​to the lyrics telop displayed on the touch panel monitor 33. When OFF of the romaji ruby ​​function is accepted, the CPU 30 does not generate romaji ruby. For example, by adding romaji ruby ​​to the characters of the lyrics telop displayed on the touch panel monitor 33 and executing a text / ruby exchange process, the romaji lyrics telop can be displayed in an easy-to-read manner. This allows the karaoke system 10 to make the text even easier to read for foreigners, such as foreign students studying abroad.

[0092] Also, according to the above embodiment, when the CPU 30 receives a command to switch the main text / ruby swap process OFF, it displays characters at a preset character size at character display position P1, and displays ruby ​​at a size smaller than the character size at ruby ​​display position P2. On the other hand, when the CPU 30 receives a command to switch the main text / ruby swap process ON, it displays ruby ​​at the character size at character display position P1, and displays characters at a size smaller than the character size at ruby ​​display position P2. In other words, the characters and ruby ​​displayed at character display position P1 after the main text / ruby swap process are displayed at the same size, making it easier to read the entire lyrics subtitle.

[0093] The present invention is not limited to the above embodiment, and various modifications are possible. The karaoke system 10 is an example of an information processing device of the present invention, but it can be applied to any device that can display text on a display unit. For example, it can be used as various information processing devices such as a personal computer, a game device, a smartphone, etc. In addition, the subtitle display is not limited to the display of lyrics, and may be, for example, subtitles for movies and videos. In addition, the display unit may be integrated with the information processing device or may be separate. In addition, the playback program of the above embodiment may be downloaded to a mobile terminal such as a smartphone, so that the playback process of the above embodiment is executed on the mobile terminal. The remote control device 1 of the karaoke system 10 may be omitted.

[0094] In the above embodiment, the CPU 30 causes the GPU 29 to control the display of the monitor 41 and the touch panel monitor 33, but the CPU 30 may control them. Also, in the above embodiment, the screens of Fig. 9 to Fig. 14, Fig. 18, Fig. 19, etc. are displayed on the touch panel monitor 33 and various inputs are accepted, but the above screens may also be displayed on the touch panel monitor 11 of the remote control device 1 and various inputs may also be accepted.

[0095] In the above embodiment, the CPU 30 transmits the shader program to the RAM 28 for each song, but if the display method of the lyrics subtitle is the same, it does not have to be transmitted every time. [Explanation of symbols]

[0096] 10 Karaoke equipment 30 CPU 33 Touch panel monitor

Claims

1. a display control unit that displays characters to which ruby ​​notation is added in the text data to be displayed on the display unit at a character display position and also displays the ruby ​​notation at a ruby ​​display position adjacent to the character display position; a replacement control unit that executes a replacement process in which, in a display by the display control unit, the ruby ​​is displayed at the character display position and the character is displayed at the ruby ​​display position; an exchange reception unit that receives validity / invalidity of the exchange process; Equipped with The replacement control unit is When the replacement reception unit receives a signal that the replacement process is valid, the replacement process is executed; When the replacement reception unit receives a request to invalidate the replacement process, the replacement process is not executed. An information processing device comprising:

2. a display reception unit that receives a display enable / disable command for the ruby ​​display position, The display control unit is When the display receiving unit receives the enablement of the display, the character or the ruby ​​is displayed at the ruby ​​display position according to the enablement or disablement of the replacement process; When the display reception unit receives the invalidation of the display, regardless of whether the replacement process is valid or invalid, neither the character nor the ruby ​​is displayed at the ruby ​​display position.

2. The information processing apparatus according to claim 1,

3. a romaji ruby ​​adding unit that executes an adding process of adding romaji ruby ​​to characters of the text data to be displayed on the display unit; an assignment acceptance unit that accepts the validity or invalidity of the assignment process; Equipped with The romaji ruby ​​providing unit is When the grant acceptance unit accepts that the grant process is valid, the grant process is executed, When the grant acceptance unit accepts an invalidation of the grant process, the grant process is not executed.

3. The information processing device according to claim 1 or 2.

4. when the replacement process is not executed, the display control unit displays the character in a preset first size at the character display position and displays the ruby ​​in a preset second size smaller than the first size at the ruby ​​display position; When the replacement process is executed, the ruby ​​is displayed in the first size at the character display position, and the character is displayed in the second size at the ruby ​​display position.

2. The information processing apparatus according to claim 1,

5. The text data is lyric and subtitle data, The information processing device is a karaoke device which performs playback based on performance data and the lyrics and subtitle data.

2. The information processing apparatus according to claim 1,

6. A control method performed by an information processing device, comprising: a display control step of displaying characters to which ruby ​​notation is added in the text data to be displayed on the display unit at a character display position and displaying the ruby ​​notation at a ruby ​​display position adjacent to the character display position; a replacement control step of executing a replacement process of displaying the ruby ​​at the character display position and displaying the character at the ruby ​​display position in the display control step; a replacement reception step for receiving validity or invalidity of the replacement process; Equipped with In the replacement control step, When the replacement process is confirmed as valid in the replacement reception step, the replacement process is executed; When the replacement process is invalidated in the replacement reception step, the replacement process is not executed. A control method comprising:

7. The computer of the information processing device a display control step of displaying characters to which ruby ​​notation is added in the text data to be displayed on the display unit at a character display position and displaying the ruby ​​notation at a ruby ​​display position adjacent to the character display position; a replacement control step of executing a replacement process of displaying the ruby ​​at the character display position and displaying the character at the ruby ​​display position in the display control step; a replacement reception step for receiving validity or invalidity of the replacement process; Run the command, In the replacement control step, When the replacement process is confirmed as valid in the replacement reception step, the replacement process is executed; When the replacement process is invalidated in the replacement reception step, the replacement process is not executed. A program characterized by.

Images