Image processing device, image processing program, and image processing method

Abstract

To achieve telop reproduction using a Graphics Processing Unit (GPU).SOLUTION: An image processing device is configured to reproduce telop data, and comprises a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a storage unit accessed by the GPU. The telop data includes: apex data indicative of display positions of respective fragments of the telop; and texture data stuck to the respective fragments of the telop. The image processing device causes the CPU to perform: first transmission processing for transmitting a telop formation program to the storage unit; second transmission processing for transmitting the telop data to the storage unit; and content reproduction processing for reproducing content data. The image processing device further causes the GPU to perform telop reproduction processing of performing display based on the telop data in synchronization with reproduction of content data on the basis of the telop formation program.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing program, and an image processing method. [Background technology]

[0002] In recent years, development methods that actively use GPUs (Graphics Processing Units) for image formation in various information processing devices have become common. GPUs are calculation devices, or processors, specialized for real-time image processing, and excel at parallel calculation processing, achieving calculation speeds several times to over 100 times faster than CPUs (Central Processing Units).

[0003] Patent Document 1 describes how one screen is divided into a video display area (TV or other images), an information display area (waiting times and call numbers often used in hospitals), and a subtitle display area (advertisements). The subtitle display area is an area for displaying static information such as text advertisements. It describes how by displaying images converted into a raster format, processing by the GPU is easy, allowing the subtitles to be displayed smoothly without any jerks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-174734 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the invention described in Patent Document 1, the caption can only be displayed in the caption display area, and the caption cannot be displayed in a free layout or with a free expression.

[0006] In addition, in various information processing devices, the processing load on the CPU is increasing as functions are added and improved. In a processing device that handles telops, such as a karaoke device, if the telop-related processing is performed by the CPU, it is conceivable that the processing resources of the CPU will be reduced.

[0007] The present invention has been made in consideration of these circumstances, and has as its object to make it possible to express a variety of telops using GPU processing by executing image processing related to telops on a GPU. Another object of the present invention is to efficiently perform image processing related to telops, thereby reducing the amount of processing by the CPU. [Means for solving the problem]

[0008] For this reason, the image processing device according to the present invention employs the following configuration. An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that plays back telop data, The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, Reference data consisting of a data dimension array handled by the GPU, with at least the timing of each fragment specified Including, causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; Based on the subtitle generation program, the GPU executes subtitle playback processing for displaying subtitle data in synchronization with playback of the content data. 、 The CPU executes a third transmission process of transmitting a playback position of the content data to the GPU; The GPU executes the subtitle playback process based on the playback position sent from the CPU and the timing of each fragment defined in the reference data.

[0009] Further, in the image processing device according to the present invention, The subtitle formation program includes a vertex shader program and a pixel shader program.

[0010] Further, in the image processing device according to the present invention, The first transmission process enables a different telop formation program to be transmitted.

[0011] Further, in the image processing device according to the present invention, The first transmission process transmits a telop formation program corresponding to the content data to be reproduced in the content reproduction process.

[0012] Further, in the image processing device according to the present invention, The first transmission process transmits a telop formation program selected by the user.

[0015] Further, in the image processing device according to the present invention, The timings defined in the reference data include the start and end times of each fragment.

[0016] Further, in the image processing device according to the present invention, An image processing program executed on an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and at least reference data that specifies the timing of each fragment, Including, causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; Based on the subtitle generation program, the GPU executes subtitle playback processing for displaying subtitle data in synchronization with playback of the content data. 、 a third transmission process for transmitting a playback position of the content data to the GPU; The GPU executes the subtitle playback process based on the playback position sent from the CPU and the timing of each fragment specified in the reference data.

[0017] The image processing method according to the present invention further comprises the steps of: An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, comprising: The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and at least reference data that specifies the timing of each fragment, Including, causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; Based on the subtitle generation program, the GPU executes subtitle playback processing for displaying subtitle data in synchronization with playback of the content data. 、 a third transmission process for transmitting a playback position of the content data to the GPU; The GPU executes the subtitle playback process based on the playback position sent from the CPU and the timing of each fragment specified in the reference data.

[0018] The present invention may also adopt other forms that will be described below. According to another embodiment, an image forming apparatus includes: An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that plays back telop data, The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and contains at least reference data that specifies the timing of each fragment, The CPU is a transmission process for transmitting the telop data to a storage unit; A content playback process for playing back the content data is executed; GPUs are execute a telop reproduction process for reproducing a telop based on the telop data; The telop playback process is performed in synchronization with the content playback process at a timing defined by the reference data.

[0019] In still another embodiment of the image processing device, The CPU generates at least one of the vertex data, the texture data, and the reference data in the texture data from the telop source data corresponding to the content data.

[0020] In still another embodiment of the image processing device, The timing of each segment defined in the reference data includes the start and end times of each segment.

[0021] In still another embodiment of the image processing device, The timing of each segment defined in the reference data further includes the end time of the previous segment and the start time of the next segment.

[0022] In still another embodiment of the image processing device, The reference data defines the timing of a fragment collection that includes multiple fragments.

[0023] In still another embodiment of the image processing device, The timing of a fragment collection defined in the reference data includes a start time and an end time of the fragment collection.

[0024] In still another embodiment of the image processing device, The reference data includes position information for each fragment, The telop playback process plays back the telop based on the position information defined by the reference data.

[0025] In still another embodiment of the image processing device, The position information of each fragment defined by the reference data includes rectangular information indicating the planar extent of each fragment.

[0026] In still another embodiment of the image processing device, The reference data defines position information of a fragment collection that includes a plurality of fragments.

[0027] In still another embodiment of the image processing device, The position information of the fragment collection defined by the reference data includes rectangular information indicating the planar extent of the fragment collection.

[0028] Further, an image processing program according to another embodiment includes: An image processing program executed on an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and contains at least reference data that specifies the timing of each fragment, causing a CPU to execute a transmission process for transmitting telop data to a storage unit and a content reproduction process for reproducing content data; A telop playback process is executed on the GPU to play back telops based on the telop data; The telop playback process is performed in synchronization with the content playback process, based on the timing defined by the reference data.

[0029] In addition, an image processing method according to another embodiment includes the steps of: An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, comprising: The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and contains at least reference data that specifies the timing of each fragment, causing a CPU to execute a transmission process for transmitting telop data to a storage unit and a content reproduction process for reproducing content data; A telop playback process is executed on the GPU to play back telops based on the telop data; The telop playback process is performed in synchronization with the content playback process at a timing defined by the reference data. Effect of the Invention

[0030] According to the present invention, it is possible to easily express various types of telops using GPU processing. Also, it is possible to have the GPU perform image processing related to telops, thereby reducing the amount of processing by the CPU. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 shows a configuration of a karaoke system according to an embodiment of the present invention. [Diagram 2] A flow diagram showing the regeneration process of this embodiment. [Diagram 3] FIG. 1 is a diagram for explaining the formation of texture data according to the present embodiment; [Figure 4] FIG. 1 is a diagram for explaining vertex data according to an embodiment of the present invention; [Diagram 5] FIG. 1 is a diagram for explaining the concept of index data according to the present embodiment; [Figure 6] FIG. 1 is a diagram showing a configuration of reference data according to an embodiment of the present invention; [Figure 7] FIG. 13 is a diagram showing an example of caption generation on a screen according to the present embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] 1 is a diagram showing the configuration of a karaoke system according to this embodiment. The karaoke system according to this embodiment includes a karaoke device 2 (sometimes called a commander) and a remote control device 1. In this embodiment, the karaoke device 2 and the remote control device 1 are communicatively connected to a LAN 100 by wire or wirelessly to form a network.

[0033] The karaoke device 2 installed in a store such as a karaoke booth includes an audio control unit 25 that reproduces music (accompaniment). The audio control unit 25 of this embodiment applies effects such as echo to the audio signals input from the microphones 43a and 43b, adds a reproduction signal based on music information, and outputs the result to a speaker 42. Although FIG. 1 shows one speaker 42 for convenience, it is possible to adopt various configurations such as a plurality of speakers 42 and stereo or surround output.

[0034] The karaoke device 2 of this embodiment is configured by connecting various devices via the interfaces 22a to 22d. Although four interfaces 22a to 22d are used in Fig. 1, the interfaces 22a to 22d may take various forms, such as a single interface or an interface for each device.

[0035] The karaoke machine 2 includes an operation unit 21 that accepts various inputs from a user. The karaoke machine 2 also includes a hard disk 32 that serves as a storage unit that stores various information. The karaoke machine 2 also includes a LAN communication unit 24a that serves as communication means for connecting to the LAN 100 and joining the network. The karaoke machine 2 of this embodiment also includes a wireless LAN communication unit 24b, and instead of a wired network connection using the LAN communication unit 24a, it is also possible to establish a wireless network connection using the wireless LAN communication unit 24b.

[0036] The karaoke device 2 also includes a video playback unit that displays lyrics video, background video, and the like on the monitor 41. The video playback unit includes a GPU 29 (Graphics Processing Unit) that forms video based on various information, a RAM 28 (corresponding to the "storage unit accessed by the GPU" according to the present invention) that stores information required for video formation, and a video output unit 31 that outputs the played video to the monitor 41 and the touch panel monitor 33. These video playback units may be provided as a video card configured as a single device. Note that a cache that temporarily stores data may be provided between the GPU 33 and the RAM 28. This cache also corresponds to the "storage unit accessed by the GPU" according to the present invention.

[0037] Furthermore, in this karaoke device 2, various information can be displayed on the touch panel monitor 33 in addition to the externally connected monitor 41. The touch panel monitor 33 is composed of a display unit 35 that displays the video input from the video output unit 31, and a touch panel 34 that outputs the touch input position to the interface 22b. The touch panel monitor 33 of this embodiment is disposed on the front of the housing of the karaoke device 2, and functions as an input unit that accepts various operations from the user. By selecting a song on the touch panel monitor 33, the user can perform various operations on the karaoke device 2, such as directly reserving the song on the karaoke device 2.

[0038] Furthermore, the karaoke device 2 is configured with a control section including a CPU 30 (Central Processing Unit) for controlling each component in an integrated manner, and a memory 27 for temporarily storing information required for executing various programs.

[0039] With this configuration, the karaoke device 2 executes various processes, but the main functions of the karaoke device 2 are to execute song reservation process, playback process, etc. The song reservation process is a process for specifying and reserving a song based on a user's specification, and is executed in cooperation with the remote control device 1. Reservation information formed by the song selection process of the remote control device 1 is transmitted to the karaoke device 2. The karaoke device 2 registers the received reservation information in a reservation table in the memory 27. The playback process is a process for playing back a reserved song, and is a process in which a performance process, a lyric display process, and a video playback process are executed synchronously.

[0040] The performance process is a process for causing the audio control unit 25 to perform a performance based on the performance data included in the music data. The music performed by the audio control unit 25 is mixed with audio signals input from the microphones 43a and 43b and is output from the speaker 42. The lyrics display process is a process for providing singing assistance by displaying a lyrics video (telop) based on the lyrics data included in the music data on the monitor 41. A background video display process may be executed to superimpose a background video on the lyrics video displayed in the lyrics display process.

[0041] Meanwhile, the remote control device 1 is capable of executing a song selection process in which it searches for songs based on instructions from the user and transmits reservation information for the song instructed to be played to the karaoke device 2. The remote control device 1 is also capable of receiving various information from the karaoke device 2 or a server device 5 connected to the Internet and executing various processes. In this embodiment, the remote control device 1 is provided with an operation unit 17 and a touch panel monitor 11 as a user interface for receiving various instructions from the user. The touch panel monitor 11 is configured with a display unit 11a and a touch panel 11b, and is capable of displaying various information on the display unit 11a and receiving touch input from the user.

[0042] Furthermore, the remote control device 1 is equipped with a memory 14 as a storage unit for storing databases required for music selection processing, various programs, and various information generated with program execution, and a remote control side control unit for controlling these components in an integrated manner. The remote control side control unit includes a CPU 15, an image control unit 13 that forms an image to be displayed on the touch panel monitor 11, a video RAM 12 that temporarily stores image data of the image to be displayed, and an operation processing unit 18 that interprets input from the touch panel monitor 11 or the operation unit 17 and transmits it to the CPU 15.

[0043] The remote control device 1 is connected to the network formed by the LAN 100 by being wirelessly connected to the access point 130 by the wireless LAN communication unit 16. Each remote control device 1 is pre-associated with a specific karaoke device 2. Various commands output from the remote control device 1 are received by the associated karaoke device 2.

[0044] With such a configuration of the remote control device 1, various inputs from the user can be received from the touch panel monitor 11 or the operation unit 17, and various information can be provided by displaying on the touch panel monitor 11, making it possible to perform various processes such as song selection processing which transmits reservation information to be output to the karaoke device 2.

[0045] 2 is a flow diagram showing the playback process according to this embodiment. The playback process is executed by the CPU 30. When a song is reserved using the remote control device 1 or the operation unit 21 of the karaoke device 2, a song ID indicating the reserved song is registered in a reservation table managed in the memory 27. In the playback process, this reservation table is checked (S101) to confirm whether or not there is a song to be played next.

[0046] If there is a song to be played next, song data corresponding to that song is read out (S103). The song data may be pre-stored in the karaoke device 2, or may be obtained by being distributed from the server device 5 or the like. The song data in this embodiment is composed of performance data and lyric data (corresponding to the "subtitle source data" of the present invention). The performance data is data that is played (performed) by the audio control unit 25. Moreover, the lyric data is data that is displayed on the monitor 41 in synchronization with the performance to assist singing.

[0047] In this embodiment, before a performance based on the performance data is executed, caption data is created based on the lyric data (S104). The caption data is data for displaying captions using the GPU 29. Note that, although the caption data is created from the lyric data in this embodiment, caption data may alternatively be included in the music data in advance.

[0048] The lyrics data in this embodiment is composed of text data, and includes spaces, line breaks, and blank lines. In addition, timing data indicating the display start time, erasure time, and singing position are specified for each character or other appropriate unit. Conventionally, the CPU 30 mounted on the karaoke device 2 has used such lyrics data to display lyrics on the monitor 41.

[0049] However, with the karaoke device 2, the processing load on the CPU 30 is increasing with an increase in functions, etc. In this situation, processing related to displaying lyric data is one of the factors that increase the processing load on the CPU 30. Also, having the CPU 30 express new lyrics is likely to result in a significant processing load on the CPU 30.

[0050] Therefore, in this embodiment, the lyrics data that has conventionally been used for displaying lyrics is converted into telop data that can be easily handled by the GPU 29, and the lyrics are displayed using the GPU 29. The CPU 30 forms the telop data based on the lyrics data.

[0051] The telop data in this embodiment is composed of (1) texture data, (2) vertex data, and (3) reference data. These data will be explained below.

[0052] In the embodiment, a character is used as the smallest unit, but the smallest unit can take various forms, such as multiple characters or a portion of a character. In the present invention, a smallest unit such as a "character" used in the above embodiment is called a "fragment," and a collection of "fragments" such as a "phrase," "line," or "chapter" is called a "fragment collection."

[0053] (1) Texture data Fig. 3 is a diagram for explaining the formation of texture data. This example takes the lyrics of the nursery rhyme "Elephant" shown in Fig. 4 as an example. First, overlapping characters are removed from the text data forming the lyrics information. Next, using font data or the like, the characters from which overlapping has been removed are rasterized (bitmapped) to form one piece of image data.

[0054] (2) Vertex data The vertex data is an array of data related to each vertex on a three-dimensional model. In other words, it is an array of data related to each vertex that forms a plurality of polygons. In the embodiment, one polygon is assigned to one character. Note that one polygon may be assigned to a portion of a character or to multiple characters. The data related to each vertex includes three-dimensional coordinates, texture coordinates, and index data.

[0055] (2-1) Three-dimensional coordinates Three-dimensional coordinates are data that define the positions of polygons placed in three-dimensional space. Figure 4 shows an example of the lyrics of the nursery rhyme "Elephant" placed in three-dimensional space. The three-dimensional coordinates indicate the positions, i.e., coordinates, of each vertex of each polygon that corresponds to each character that makes up the lyrics when the lyrics are placed in three-dimensional space. Note that the characters shown within the polygons in the figure are added for ease of understanding, and in reality, the characters are visualized by pasting image data cut out from texture data into the polygons.

[0056] (2-2) Texture coordinates Texture coordinates are data that define the position of the portion of the texture data that is to be applied to each polygon. In other words, texture coordinates indicate the positions, or coordinates, that correspond to the vertices of each polygon in the texture data. By referencing the texture coordinates, it is possible to extract the necessary character image data from the texture data. In other words, it is possible to extract the necessary character image data from the texture data and apply it to each polygon.

[0057] (2-3) Index Data Index data is information assigned to each vertex forming a polygon. Fig. 5(A) is a diagram for explaining index data. In this embodiment, lyrics are assigned information on "characters" each consisting of one character, "phrases" consisting of one or more characters, "lines" consisting of one or more phrases, and "chapters" consisting of one or more lines.

[0058] As described above, the lyrics data in this embodiment is composed of text data including spaces, line breaks, and blank lines. When creating index data from lyrics data, for example, a block up to a space is determined as a "phrase," a block up to a line break as a "line," and a block up to a blank line as a "chapter." In addition to this format, various other formats can be adopted, such as tagging lyrics data with "phrases," "lines," and "chapters."

[0059] Fig. 5(B) is a diagram for explaining index data assigned to each vertex of a polygon. Fig. 5(B) explains the index data by taking the first two phrases "Zou-san Zou-san" in Fig. 5(A) as an example. In this embodiment, the initial value for characters, phrases, lines, and chapters is "0".

[0060] For the first character (0th character), "zo", index data (0,0,0,0) is assigned to the top left and bottom left vertices. This index data indicates that it is the 0th character, 0th phrase, 0th line, and 0th chapter. Additionally, index data (0.99,0,0,0,0) is assigned to the top right and bottom right vertices of the first character (0th character), "zo". The value "0.99" is assigned to the character because a smaller value is used in order to assign 1 to the next character (1st character).

[0061] For the next character (the first character), "u", index data (1,0,0,0) is assigned to the top left and bottom left vertices. Index data (1.99,0,0,0,0) is assigned to the top right and bottom right vertices of "u". Index data is assigned to the vertices of the second character, "sa", and the third character, "n", using the same rule.

[0062] In the next phrase (first phrase), the fourth character "zo" has index data (4,1,0,0) assigned to its upper left and lower left vertices. This index data indicates that it is the fourth character, first phrase, 0th line, and 0th chapter. Furthermore, index data (4.99,1,0,0,0) is assigned to its upper right and lower right vertices. The same rule is used to assign index data to the vertices of the fifth character "u", the sixth character "sa", and the seventh character "n".

[0063] By assigning index data to each vertex of a polygon in this way, it is possible to easily control subtitles by character, phrase, line, or chapter.In addition, because the index data is a four-dimensional array (character, phrase, line, chapter) like the data handled by the GPU, it can be processed efficiently in parallel on the GPU.

[0064] (3) Reference data The reference data is data that specifies various information about the characters, phrases, lines, and chapters that form lyrics. Fig. 6 is a diagram for explaining the reference data of this embodiment. As shown in Fig. 6, the reference data has information for each character T, phrase P, line L, and chapter C.

[0065] For the 0th character (the character "zo" in the lyrics example in Fig. 5(A)), index data, rectangular information indicating the position and size of the character (corresponding to "position information"), and the time range of the character are included. The index data is the information explained in Fig. 5(B), and for this 0th character, it is (0,0,0,0).

[0066] The rectangular information indicating the position and size of the character is the same as the vertex data explained in FIG. 4, and in the case of the 0th character, it is information indicating the position and size of the polygon that constitutes "zo". In the GPU, when processing each piece of vertex data, it is not possible to refer to information on other vertex data. Therefore, by including rectangular information indicating the position and size of the character in the reference data, it is possible to refer to the control position of the character to which the vertex data belongs or other characters by referring to the reference data when processing each piece of vertex data.

[0067] The time range of a character is information indicating timing. In this embodiment, (Sn-1_end, Sn_start, Sn_end, Sn+1_start) is assigned as the time range of a character. Sn-1_end indicates the end timing of the previous character Sn-1, Sn_start indicates the start timing of the character Sn, Sn_end indicates the end timing of the character Sn, and Sn+1_start indicates the start timing of the next character Sn+1.

[0068] Here, when displaying the character Sn, two pieces of information, the start timing Sn_start of the character Sn and the end timing Sn_end of the character Sn, are sufficient. However, in this embodiment, like other data handled by the GPU, a four-dimensional array data structure is used, and Sn-1_end and Sn+1_start are included. By including these, it is possible to perform processing using the timing of the previous character and the following character. As a comparative example, when Sn-1_end and Sn+1_start are not included, when performing processing using the timing of the previous character and the following character, it is necessary to refer to the reference data of the character Sn, the reference data of the character Sn-1, and the reference data of the character Sn+1, which results in a large number of references and a high processing load. On the other hand, in this embodiment, it is possible to reduce the number of references compared to the comparative example, thereby reducing the processing load.

[0069] Note that Sn-1_end and Sn+1_start may not be included in the time range. In that case, the time range will have a two-dimensional data array, so by including two null data, it may have a four-dimensional data array, making it easier to handle on the GPU. This also applies to the time ranges of phrases, lines, and chapters, which will be explained later.

[0070] Similarly to the 0th character, the 1st to mXth characters (the last character) of information are defined in the reference data.

[0071] For the 0th phrase (the character "Elephant" in the example lyrics of FIG. 5(A)), index data for the first character of the phrase, rectangular information indicating the position and size of the phrase, and the time range of the phrase are included. The index data for the first character of the phrase is stored as the index data for the 0th character, (0,0,0,0).

[0072] In the case of the 0th phrase, the rectangular information indicating the position and size of the phrase is information indicating the position and size of the area surrounding the "elephant." When processing each piece of vertex data, the GPU cannot refer to information on other vertex data. Therefore, by including rectangular information indicating the position and size of the phrase in the reference data, it is possible to refer to the control position of the phrase when referencing the reference data.

[0073] The time range of a phrase is information indicating timing. In this embodiment, (Tn-1_end, Tn_start, Tn_end, Tn+1_start) is assigned as the time range of a phrase. Tn-1_end indicates the end timing of the previous phrase Tn-1, Tn_start indicates the start timing of the phrase Tn, Tn_end indicates the end timing of the phrase Tn, and Tn+1_start indicates the start timing of the subsequent phrase Tn+1.

[0074] As with text, when displaying the phrase Tn, two pieces of information are sufficient: the start timing Tn_start of the phrase Tn and the end timing Tn_end of the phrase Tn. However, in this embodiment, by including Sn-1_end and Sn+1_start, it is possible to perform processing using the timing of the previous and following phrases.

[0075] Similarly to the 0th phrase, the reference data defines information on the 1st to mYth phrases (the last phrase).

[0076] As with phrases, lines (or chapters) also include index data for the first character of the line (or chapter), rectangular information indicating the position and size of the line (or chapter), and the time range of the line (or chapter).

[0077] In this way, the reference data of this embodiment specifies index data, rectangular information indicating the position and size, and time range (timing) for characters, phrases, lines, and chapters, making it possible to easily control the display timing of characters, phrases, lines, and chapters. Note that it is sufficient for the reference data to include information indicating the time range (timing) for characters, phrases, lines, and chapters.

[0078] In this embodiment, the above-mentioned telop data (texture data, vertex data, reference data) is used to cause the GPU 29 to display telop on the monitor 41. At this time, a shader program for forming the telop is changed for the GPU 29, thereby making it possible to perform a variety of telop expressions. In this embodiment, the CPU 30 transmits a shader program corresponding to a piece of music to the RAM 28 for each playback process (S106). For example, by using a shader program corresponding to a piece of music, it is possible to realize a telop expression corresponding to the piece of music.

[0079] The shader program of this embodiment (corresponding to the "telop formation program" of the present invention) is a program that specifies how the telop is to be displayed on the screen, and is capable of controlling the display position, color tone, etc. of characters, chapters, lines, and phrases. The shader program of this embodiment is composed of a vertex shader program and a pixel shader program.

[0080] The vertex shader program is a program that specifies the display position, and can control display, for example, by character, phrase, line, or chapter. Display control includes layouts such as right-justified, left-justified, stepped, and line breaks, as well as animations such as scrolling and pop-ups. Display control is not limited to the layouts and animations exemplified above, but includes various other layouts and animations. In addition, the vertex shader program can express subtitles as a stereoscopic image captured from a specified camera viewpoint in a three-dimensional space.

[0081] On the other hand, the pixel shader program is capable of controlling the color information for each pixel that forms the caption, and can control various color expressions for the caption to be displayed, such as using a single color or a gradation.

[0082] In this embodiment, a shader program (corresponding to the "telop formation program" of the present invention) is formed by combining a vertex shader program and a pixel shader program. By preparing a plurality of vertex shader programs and a plurality of pixel shader programs in advance and changing the combination, it is possible to form shader programs that express a variety of telops.

[0083] After transmitting the telop data and shader program to RAM 28, enabling the GPU 29 to display the telop, the CPU 30 causes the audio control unit 25 to start playing the performance data (S107). At that time, the current performance position (playback position) is transmitted to the GPU 29 in synchronization with the playback of the performance data. The current performance position is transmitted to the GPU 29 periodically (for example, for each frame to be drawn). The GPU 29 displays lyrics (telop display) on the monitor 41 based on the telop data and shader program stored in RAM 28 and the current performance position received from the CPU 30.

[0084] The vertex shader program executed by the GPU 29 refers to the time range (timing) in the reference data, determines the characters (or phrases, lines, or chapters) to be displayed, and determines the display position based on the vertex data. At this time, depending on the vertex shader program, coordinate conversion is performed. The pixel shader program can also control the color of the character image to be displayed, and can display characters on polygons based on images extracted from texture data. The pixel shader program can also change the color of characters based on the current playing position, or can display characters in a variety of colors.

[0085] FIG. 7 shows an example of the caption displayed on the screen in this embodiment. FIG. 7(A) shows a display form in which captions are displayed horizontally in the lower half of the screen, similar to conventional karaoke devices. The vertex program performs coordinate conversion on the vertex data described in FIG. 4, and arranges a number of polygons for a given number of lines (two lines in the example of FIG. 7) on the screen. Next, the pixel shader program displays characters on each polygon based on an image extracted from the texture data. The pixel shader program also determines the current singing position based on the current position transmitted from the CPU, and displays and outputs the color change before and after the current singing position, thereby displaying the caption.

[0086] FIG. 7(B) shows a form in which telops are displayed over the entire screen. FIG. 7(B) shows a display form in which a line break is performed for each phrase, and furthermore, multiple phrases displayed on the screen are displayed in a staircase pattern. In this case, the vertex shader program performs coordinate conversion on the vertex data described in FIG. 4, and arranges multiple polygons on the screen so that a predetermined amount of phrases (four phrases in the example of FIG. 7) are arranged in a staircase pattern. Next, the pixel shader program displays characters on each polygon based on an image extracted from the texture data. In addition, the pixel shader program determines the current singing position based on the current position transmitted from the CPU, and displays and outputs a display with a different color before and after the current singing position, thereby displaying the telop.

[0087] Although two-dimensional display has been described in Figures 7(A) and 7(B), by changing the shader program it is possible to freely express layouts, such as displaying subtitles viewed from a certain viewpoint in three-dimensional space.

[0088] In this manner, in this embodiment, the processing related to telop playback can be realized by the GPU 29, and the CPU 30 only needs to transmit the current performance position to the GPU 29, thereby reducing the processing load on the CPU 30. Also, by changing the shader program, it is possible to realize a variety of telop expressions.

[0089] The transmission of the current playing position by the CPU 30 (S108) is continued until the playing of the performance data is completed. When the playing of the performance data is completed (S109: Yes), the reservation table is checked for playing the next song (S101).

[0090] In the above, the display of lyrics (telop) has been described using a karaoke system as an example in this embodiment, but in this embodiment, by using a shader program, and telop data and shader program optimized for the GPU 29, it is possible to display telops using the GPU 29. Therefore, it is possible to display telops in a variety of ways and to reduce the processing load on the CPU 30.

[0091] [First Modification] Next, a modified example of the image processing device according to the present invention will be described. In the above embodiment, the display of telops has been described by taking the example of displaying lyrics using the karaoke device 2. The display of telops is not limited to displaying lyrics, and may be used for subtitles of movies and moving images. In this case, the information processing device that reproduces the telops corresponds to the image processing device.

[0092] [Second modified example] In the above embodiment, a shader program (corresponding to the "telop formation program" of the present invention) is transmitted to the RAM 28 for each playback process. In addition to this form, various forms can be adopted for changing the shader program. For example, the shader program may be changed by a user operation. In this case, the shader program may be one prepared in advance, or may be changeable to a shader program created by the user himself.

[0093] Although the above embodiment does not refer in detail to the correspondence between music and shader programs, shader programs may be associated with at least one of music songs, music genres, singers, composers, and lyricists. Alternatively, shader programs may be associated with timekeeping information such as seasons, days of the week, and times. Alternatively, shader programs may be selected randomly.

[0094] Although various embodiments have been described above using the karaoke machine 2 as an example, the image processing device according to the present invention can be used not only as the karaoke machine 2 but also as various information processing devices such as a personal computer, a game device, a smartphone, etc. Furthermore, image processing programs or image processing methods executed in various information processing devices also fall within the scope of the invention. [Explanation of symbols]

[0095] 1: Remote control device 24a: LAN communication unit 2: Karaoke device 24b: Wireless LAN communication unit 5: Server device 25: Sound control unit 11: Touch panel monitor 27: Memory 11a:Display section 28:RAM 11b: Touch panel 30: CPU 12: Video RAM 31: Video output section 13: Video control unit 32: Hard disk 14: Memory 33: Touch panel monitor 15: CPU 34: Touch panel 16:Wireless LAN communication section 35:Display section 17: Operation unit 41: Monitor 18: Operation processing unit 42: Speaker 21: Operation unit 43a, 43b: Microphone 22a~22d: Interface 130: Access point

Claims

1. An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that reproduces telop data, The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, The data is composed of a data dimensional array that is handled by the GPU, and includes at least reference data that defines the timing of each fragment; causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; causing the GPU to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data based on the telop generation program; the CPU executes a third transmission process of transmitting a playback position of the content data to the GPU; The GPU executes the subtitle playback process based on the playback position sent from the CPU and the timing of each fragment defined by the reference data. Image processing device.

2. The subtitle generation program includes a vertex shader program and a pixel shader program. The image processing device according to claim 1 .

3. The first transmission process enables transmission of different telop formation programs. The image processing device according to claim 1 .

4. The first transmission process transmits a telop formation program corresponding to the content data to be reproduced in the content reproduction process. The image processing device according to claim 1 .

5. The first transmission process transmits a telop generation program selected by a user. The image processing device according to claim 1 .

6. The timings specified in the reference data include the start and end times of each segment. The image processing device according to claim 1 .

7. An image processing program executed on an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, and reference data that is configured as a data dimensional array that is handled by the GPU and that defines at least the timing of each fragment; causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; causing the GPU to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data based on the telop generation program; causing the CPU to execute a third transmission process of transmitting a playback position of the content data to the GPU; The GPU executes subtitle playback processing based on the playback position sent from the CPU and the timing of each fragment defined by the reference data. Image Processing Program

8. An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, comprising: The telop data is vertex data indicating the display position of each fragment of the telop; The texture data to be attached to each subtitle fragment, and reference data that is configured as a data dimensional array that is handled by the GPU and that defines at least the timing of each fragment; causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; causing the GPU to execute a telop playback process for performing a display based on the telop data in synchronization with the playback of the content data based on the telop generation program; causing the CPU to execute a third transmission process of transmitting a playback position of the content data to the GPU; The GPU executes subtitle playback processing based on the playback position sent from the CPU and the timing of each fragment defined by the reference data. Image processing methods.

Images