Image processing device, image processing program, and image processing method

Abstract

To realize a telop reproduction using a GPU (Graphics Processing Unit).SOLUTION: An image processing device that is constructed by a top data indicating a display position of each piece of a telop, a texture data attached to each piece of the telop, and a data dimension arrangement dealed with a GPU (Graphics Processing Unit), and reproduces a telop data containing a reference data in which a timing of at least each piece is regulated. A CPU (Central Processing Unit) executes a transmission processing for transmitting the telop data to a storage part; and a content reproduction processing for reproducing a content data, and the GPU executes the telop reproduction processing for reproducing the telop on the basis of the telop data, and the telop reproduction processing is performed so as to be synchronized to the content reproduction processing on the basis of the timing regulated by the reference data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing program, and an image processing method. [Background technology]

[0002] In recent years, development methods that actively use GPUs (Graphics Processing Units) for image formation in various information processing devices have become common. GPUs are computing devices, or processors, specialized for real-time image processing, and excel at parallel computing, achieving computing speeds several times to over 100 times faster than CPUs (Central Processing Units).

[0003] Patent Document 1 describes how a screen is divided into a video display area (TV or other video), an information display area (waiting times and call numbers commonly used in hospitals), and a caption display area (advertisements). The caption display area is an area where static information such as text advertisements is displayed. The document describes how converting text data into a raster format makes it easy for the GPU to process the images, allowing captions to be displayed smoothly without any jerks. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-174734 Summary of the Invention [Problem to be solved by the invention]

[0005] In the invention described in Patent Document 1, the caption can only be displayed in the caption display area, and the caption cannot be displayed in a free layout or with free display timing.

[0006] In various information processing devices, the processing load on the CPU is increasing as functions are added and improved. In a processing device that handles telops, such as a karaoke device, if the telop-related processing is performed by the CPU, it is conceivable that the processing resources of the CPU will be reduced.

[0007] The present invention has been made in consideration of these circumstances, and one of its objects is to efficiently perform image processing related to telops and reduce the amount of processing required by the CPU. Another object is to enable a variety of telop expressions using GPU processing by having the image processing related to telops executed by the GPU. [Means for solving the problem]

[0008] For this reason, the image processing device according to the present invention employs the following configuration. An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that plays back telop data, The caption data is vertex data indicating the display position of each fragment of the telop; Texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and includes at least reference data that specifies the timing of each fragment, the reference data defines timing of each of the plurality of fragments and timing of a fragment collection including a portion of the plurality of fragments; The CPU is a transmission process for transmitting the telop data to a storage unit; a content playback process for playing back the content data; GPUs are executes a telop playback process for playing back telops based on the telop data; The telop playback process is specified by the reference data. Fragment units and fragment aggregate units The processing is performed in synchronization with the content playback processing based on the timing.

[0009] Furthermore, in the image processing device according to the present invention, The CPU generates at least one of vertex data, texture data, and reference data in the texture data from the telop source data corresponding to the content data.

[0010] Furthermore, in the image processing device according to the present invention, The timing of each segment defined in the reference data includes the start and end times of each segment.

[0011] Furthermore, in the image processing device according to the present invention, The timing of each segment defined in the reference data further includes the end time of the previous segment and the start time of the next segment.

[0012] Furthermore, in the image processing device according to the present invention, The reference data defines the timing of a fragment collection that includes multiple fragments.

[0013] Furthermore, in the image processing device according to the present invention, The timing of a fragment collection defined in the reference data includes the start and end times of the fragment collection.

[0014] Furthermore, in the image processing device according to the present invention, the reference data includes position information of each fragment, The telop playback process plays back the telop based on the position information defined in the reference data.

[0015] Furthermore, in the image processing device according to the present invention, The position information of each fragment defined by the reference data includes rectangular information indicating the plane range of each fragment.

[0016] Furthermore, in the image processing device according to the present invention, The reference data defines the position information of a fragment collection that includes a plurality of fragments.

[0017] Furthermore, in the image processing device according to the present invention, The position information of the fragment collection defined by the reference data includes rectangular information indicating the planar range of the fragment collection.

[0018] The image processing program according to the present invention also includes: An image processing program executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The caption data is vertex data indicating the display position of each fragment of the telop; Texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and includes at least reference data that specifies the timing of each fragment, the reference data defines timing of each of the plurality of fragments and timing of a fragment collection including a portion of the plurality of fragments; causing a CPU to execute a transmission process for transmitting telop data to a storage unit and a content playback process for playing back content data; The GPU executes a caption playback process for playing captions based on the caption data, The telop playback process is specified by the reference data. Fragment units and fragment aggregate units The processing is performed in synchronization with the content playback processing based on the timing.

[0019] The image processing method according to the present invention further comprises: An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The caption data is vertex data indicating the display position of each fragment of the telop; Texture data to be attached to each subtitle fragment, It consists of a data dimensional array that is handled by the GPU, and includes at least reference data that specifies the timing of each fragment, the reference data defines timing of each of the plurality of fragments and timing of a fragment collection including a portion of the plurality of fragments; causing a CPU to execute a transmission process for transmitting telop data to a storage unit and a content playback process for playing back content data; The GPU executes a caption playback process for playing captions based on the caption data, The telop playback process is specified by the reference data. Fragment units and fragment aggregate units timing Based on , is performed in synchronization with the content playback process.

[0020] The present invention may also adopt other forms described below. An image forming apparatus according to another embodiment includes: An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that plays back telop data, The caption data is vertex data indicating the display position of each fragment of the telop; texture data to be attached to each fragment of the telop; causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; Based on the caption generation program, the GPU is caused to execute a caption playback process that performs a display based on caption data in synchronization with the playback of the content data.

[0021] In still another embodiment of the image processing device, The telop formation program includes a vertex shader program and a pixel shader program.

[0022] In still another embodiment of the image processing device, The first transmission process enables different telop formation programs to be transmitted.

[0023] In still another embodiment of the image processing device, The first transmission process transmits a telop formation program corresponding to the content data to be played back in the content playback process.

[0024] In still another embodiment of the image processing device, The first transmission process transmits a telop formation program selected by the user.

[0025] In still another embodiment of the image processing device, The telop data is composed of a data dimensional array that is handled by the GPU and includes at least reference data that defines the timing of each fragment.

[0026] In still another embodiment of the image processing device, the CPU executes a third transmission process of transmitting the playback position of the content data to the GPU; The GPU performs subtitle playback processing based on the playback position sent from the CPU and the timing of each fragment specified in the reference data.

[0027] In still another embodiment of the image processing device, The timings specified in the reference data include the start and end times of each segment.

[0028] Furthermore, an image processing program according to another embodiment includes: An image processing program executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The caption data is vertex data indicating the display position of each fragment of the telop; texture data to be attached to each fragment of the telop; causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; Based on the caption generation program, the GPU is caused to execute a caption playback process that performs a display based on caption data in synchronization with the playback of the content data.

[0029] Further, an image processing method according to another embodiment includes: An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The caption data is vertex data indicating the display position of each fragment of the telop; texture data to be attached to each fragment of the telop; causing a CPU to execute a first transmission process of transmitting a telop formation program to a storage unit, a second transmission process of transmitting telop data to a storage unit, and a content reproduction process of reproducing content data; Based on the caption generation program, the GPU is caused to execute a caption playback process that performs a display based on caption data in synchronization with the playback of the content data. [Effects of the Invention]

[0030] According to the present invention, it is possible to perform image processing related to telops on a GPU, thereby reducing the amount of processing required on the CPU. It is also possible to easily create a variety of telops using GPU processing. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 shows the configuration of a karaoke system according to an embodiment of the present invention. [Figure 2] Flow diagram showing the regeneration process of this embodiment [Figure 3] FIG. 1 is a diagram for explaining the formation of texture data according to the present embodiment; [Figure 4] FIG. 1 is a diagram illustrating vertex data according to an embodiment of the present invention. [Figure 5] FIG. 1 is a diagram for explaining the concept of index data according to the present embodiment. [Figure 6] FIG. 1 is a diagram showing the configuration of reference data according to the present embodiment; [Figure 7] FIG. 10 shows an example of caption generation on the screen according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0032] 1 is a diagram showing the configuration of a karaoke system according to this embodiment. The karaoke system according to this embodiment includes a karaoke device 2 (sometimes called a commander) and a remote control device 1. In this embodiment, the karaoke device 2 and the remote control device 1 are communicatively connected to a LAN 100 via wired or wireless communication to form a network.

[0033] A karaoke device 2 installed in a store such as a karaoke booth includes an acoustic control unit 25 that plays back music (accompaniment). The acoustic control unit 25 of this embodiment applies effects such as echo to the audio signals input from the microphones 43a and 43b, adds a playback signal based on music information, and outputs the result to a speaker 42. Although FIG. 1 shows only one speaker 42 for convenience, various configurations are possible, such as multiple speakers 42, stereo output, or surround output.

[0034] The karaoke device 2 of this embodiment is configured by connecting various devices via interfaces 22a to 22d. Although four interfaces 22a to 22d are used in Fig. 1, the interfaces 22a to 22d can take various forms, such as a single interface or one for each device.

[0035] The karaoke machine 2 includes an operation unit 21 that accepts various inputs from the user. The karaoke machine 2 also includes a hard disk 32 that serves as a storage unit for storing various types of information. The karaoke machine 2 also includes a LAN communication unit 24a that serves as communication means for connecting to the LAN 100 and joining the network. The karaoke machine 2 of this embodiment also includes a wireless LAN communication unit 24b, and instead of a wired network connection using the LAN communication unit 24a, it is also possible to establish a wireless network connection using the wireless LAN communication unit 24b.

[0036] The karaoke device 2 also includes a video playback unit that displays lyric videos, background videos, etc. on the monitor 41. The video playback unit includes a GPU 29 (Graphics Processing Unit) that creates videos based on various information, a RAM 28 that stores information necessary for creating the videos (corresponding to the "storage unit accessed by the GPU" according to the present invention), and a video output unit 31 that outputs the played videos to the monitor 41 and the touch panel monitor 33. These video playback units may be provided as a video card configured as a single device. A cache that temporarily stores data may be provided between the GPU 33 and the RAM 28. This cache also corresponds to the "storage unit accessed by the GPU" according to the present invention.

[0037] Furthermore, in this karaoke device 2, various information can be displayed on the touch panel monitor 33 in addition to the externally connected monitor 41. The touch panel monitor 33 is composed of a display unit 35 that displays the video input from the video output unit 31, and a touch panel 34 that outputs the touch input position to the interface 22b. The touch panel monitor 33 of this embodiment is disposed on the front of the housing of the karaoke device 2, and functions as an input unit that accepts various operations from the user. By selecting a song on the touch panel monitor 33, the user can perform various operations on the karaoke device 2, such as directly reserving a song on the karaoke device 2.

[0038] Furthermore, the karaoke device 2 is configured with a control unit including a CPU 30 (Central Processing Unit) for controlling each component in an integrated manner, and a memory 27 for temporarily storing information required for executing various programs.

[0039] With this configuration, the karaoke device 2 executes various processes, but its main functions include song reservation and playback. The song reservation process is a process for specifying and reserving a song based on a user's instruction, and is executed in cooperation with the remote control device 1. Reservation information formed by the song selection process on the remote control device 1 is sent to the karaoke device 2. The karaoke device 2 registers the received reservation information in a reservation table in the memory 27. The playback process is a process for playing back a reserved song, and is a process in which a performance process, a lyric display process, and a video playback process are executed synchronously.

[0040] The performance process is a process in which the audio control unit 25 performs a performance based on the performance data included in the music data. The music performed by the audio control unit 25 is mixed with audio signals input from the microphones 43a and 43b and is output from the speaker 42. The lyrics display process is a process in which a lyrics video (telop) based on the lyrics data included in the music data is displayed on the monitor 41 to assist singing. A background video display process may be executed in which a background video is superimposed on the lyrics video displayed in the lyrics display process.

[0041] Meanwhile, the remote control device 1 is capable of executing a song selection process, which searches for songs based on instructions from the user and transmits reservation information for the song instructed to be played to the karaoke device 2. The remote control device 1 is also capable of receiving various information from the karaoke device 2 or a server device 5 connected to the Internet and executing various processes. In this embodiment, the remote control device 1 is provided with an operation unit 17 and a touch panel monitor 11 as a user interface for receiving various instructions from the user. The touch panel monitor 11 is configured with a display unit 11a and a touch panel 11b, and displays various information on the display unit 11a and is capable of receiving touch inputs from the user.

[0042] Furthermore, the remote control device 1 is configured with a memory 14 as a storage unit for storing databases required for music selection processing, various programs, and various information generated as the programs are executed, and a remote control control unit for controlling these components in an integrated manner. The remote control control unit includes a CPU 15, an image control unit 13 that forms images to be displayed on the touch panel monitor 11, a video RAM 12 that temporarily stores image data for the images to be displayed, and an operation processing unit 18 that interprets inputs from the touch panel monitor 11 or the operation unit 17 and transmits them to the CPU 15.

[0043] The remote control device 1 is connected to the network formed by the LAN 100 by wirelessly connecting to the access point 130 via the wireless LAN communication unit 16. Each remote control device 1 is pre-associated with a specific karaoke device 2. Various commands output from the remote control device 1 are received by the associated karaoke device 2.

[0044] With this configuration of the remote control device 1, various inputs from the user can be received from the touch panel monitor 11 or the operation unit 17, and various information can be provided by displaying it on the touch panel monitor 11, making it possible to perform various processes such as song selection processing, which transmits reservation information to be output to the karaoke device 2.

[0045] 2 is a flow diagram showing the playback process according to this embodiment. The playback process is executed by the CPU 30. When a song is reserved using the remote control device 1 or the operation unit 21 of the karaoke device 2, the song ID indicating the reserved song is registered in a reservation table managed in the memory 27. In the playback process, this reservation table is checked (S101) to see if there is a song to be played next.

[0046] If there is a song to be played next, song data corresponding to that song is read (S103). The song data may be pre-stored in the karaoke device 2, or may be obtained by being distributed from the server device 5 or the like. The song data in this embodiment is composed of performance data and lyric data (corresponding to the "subtitle source data" according to the present invention). The performance data is data that is played (performed) by the audio control unit 25. The lyric data is data that is displayed on the monitor 41 in synchronization with the performance to assist singing.

[0047] In this embodiment, before a performance based on the performance data is executed, caption data is created based on the lyric data (S104). The caption data is data for displaying captions using the GPU 29. Note that in this embodiment, caption data is created from lyric data, but instead of this, caption data may be included in the music data in advance.

[0048] The lyrics data in this embodiment is composed of text data, and includes spaces, line breaks, and blank lines. Timing data indicating the display start time, erasure time, and singing position are also specified for each character, etc. Conventionally, the CPU 30 installed in the karaoke device 2 uses such lyrics data to display lyrics on the monitor 41.

[0049] However, with the increase in functionality of the karaoke device 2, the processing load on the CPU 30 is also increasing. In this situation, processing related to the display of lyric data is one of the factors that increase the processing load on the CPU 30. Furthermore, having the CPU 30 express new lyrics is likely to increase the processing load on the CPU 30.

[0050] Therefore, in this embodiment, the lyric data that has conventionally been used for displaying lyrics is converted into telop data that can be easily handled by the GPU 29, and the lyrics are displayed using the GPU 29. The CPU 30 generates telop data based on the lyric data.

[0051] The telop data in this embodiment is composed of (1) texture data, (2) vertex data, and (3) reference data. These data will be explained below.

[0052] In the embodiment, a character is used as the smallest unit, but the smallest unit can take various forms, such as multiple characters or a portion of a character. In the present invention, a smallest unit such as a "character" used in the above embodiment is called a "fragment," and a collection of "fragments" such as a "phrase," "line," or "chapter" is called a "fragment collection."

[0053] (1) Texture data FIG. 3 is a diagram for explaining the formation of texture data. This example uses the lyrics of the nursery rhyme "Elephant" shown in FIG. 4 as an example. First, overlapping characters are removed from the text data that forms the lyrics information. Next, using font data or the like, the characters from which overlapping has been removed are rasterized (bitmapped) to form a single piece of image data.

[0054] (2) Vertex data Vertex data is an array of data related to each vertex on a 3D model. In other words, it is an array of data related to each vertex that forms multiple polygons. In this embodiment, one polygon is assigned to one character. Note that one polygon may also be assigned to a portion of a character or multiple characters. Each vertex data includes three-dimensional coordinates, texture coordinates, and index data.

[0055] (2-1) Three-dimensional coordinates Three-dimensional coordinates are data that define the position of polygons placed in three-dimensional space. Figure 4 shows an example of the lyrics of the nursery rhyme "Elephant" placed in three-dimensional space. The three-dimensional coordinates indicate the position, i.e., coordinates, of each vertex of each polygon that corresponds to each character that makes up the lyrics when the lyrics are placed in three-dimensional space. Note that the characters shown within the polygons in the figure are added for ease of understanding; in reality, the characters are visualized by pasting image data extracted from texture data onto the polygons.

[0056] (2-2) Texture coordinates Texture coordinates are data that define the position of the portion of the texture data that is to be applied to each polygon. In other words, texture coordinates indicate the position, or coordinates, that corresponds to each vertex of each polygon in the texture data. By referencing the texture coordinates, it is possible to extract the necessary character image data from the texture data. In other words, the necessary character image data can be extracted from the texture data and applied to each polygon.

[0057] (2-3) Index Data Index data is information assigned to each vertex that forms a polygon. Fig. 5(A) is a diagram for explaining index data. In this embodiment, lyrics are assigned information on "characters" consisting of one character, "phrases" consisting of one or more characters, "lines" consisting of one or more phrases, and "chapters" consisting of one or more lines.

[0058] As described above, the lyrics data in this embodiment is composed of text data that includes spaces, line breaks, and blank lines. When creating index data from lyrics data, for example, a block up to a space is determined as a "phrase," a block up to a line break as a "line," and a block up to a blank line as a "chapter." In addition to this format, various other formats can be adopted, such as tagging lyrics data with "phrase," "line," and "chapter."

[0059] Fig. 5(B) is a diagram for explaining index data assigned to each vertex of a polygon. Fig. 5(B) explains the index data assigned to each vertex of a polygon using the first two phrases "Elephant, Elephant" in Fig. 5(A) as an example. In this embodiment, the initial value for characters, phrases, lines, and chapters is "0."

[0060] For the first character (character 0), "zo", index data (0,0,0,0) is assigned to the top left and bottom left vertices. This index data indicates that it is the 0th character, 0th phrase, 0th line, and 0th chapter. Additionally, index data (0.99,0,0,0,0) is assigned to the top right and bottom right vertices of the first character (character 0), "zo". The value "0.99" is assigned to the character because a smaller value is used, as 1 is assigned to the next character (character 1).

[0061] For the next character (first character), "u", index data (1,0,0,0) is assigned to the top left and bottom left vertices. Index data (1.99,0,0,0,0) is assigned to the top right and bottom right vertices of "u". Index data is assigned to each vertex using the same rule for the second character, "sa", and the third character, "n".

[0062] In the next phrase (first phrase), the fourth character "zo" has index data (4,1,0,0) assigned to the top left and bottom left vertices. This index data indicates that it is the fourth character, the first phrase, the 0th line, and the 0th chapter. Furthermore, the top right and bottom right vertices of "u" have index data (4.99,1,0,0,0). Index data is assigned to each vertex using the same rule for the fifth character "u", the sixth character "sa", and the seventh character "n".

[0063] By assigning index data to each vertex of a polygon in this way, it is possible to easily control subtitles by character, phrase, line, or chapter.In addition, because the index data is a four-dimensional array (character, phrase, line, chapter) just like the data handled by the GPU, it can be processed efficiently in parallel on the GPU.

[0064] (3) Reference data The reference data is data that defines various information about the characters, phrases, lines, and chapters that make up the lyrics. Fig. 6 is a diagram illustrating the reference data of this embodiment. As shown in Fig. 6, the reference data has information for each character T, phrase P, line L, and chapter C.

[0065] The 0th character (the character "zo" in the lyrics example in Figure 5(A)) contains index data, rectangular information indicating the position and size of the character (corresponding to "position information"), and the time range of the character. The index data is the information explained in Figure 5(B), and for this 0th character it is (0,0,0,0).

[0066] The rectangular information indicating the position and size of the character is the same as the vertex data explained in Figure 4, and in the case of the 0th character, it is information indicating the position and size of the polygon that makes up the character "zo." When processing each piece of vertex data, the GPU cannot refer to information on other vertex data. Therefore, by including rectangular information indicating the position and size of the character in the reference data, it becomes possible to refer to the control position of the character to which the vertex data belongs or other characters by referring to the reference data when processing each piece of vertex data.

[0067] The time range of a character is information indicating timing. In this embodiment, (Sn-1_end, Sn_start, Sn_end, Sn+1_start) is assigned as the time range of a character. Sn-1_end indicates the end timing of the previous character Sn-1, Sn_start indicates the start timing of the character Sn, Sn_end indicates the end timing of the character Sn, and Sn+1_start indicates the start timing of the next character Sn+1.

[0068] Here, when displaying the character Sn, two pieces of information are sufficient: the start timing Sn_start of the character Sn and the end timing Sn_end of the character Sn. However, in this embodiment, as with other data handled by the GPU, a four-dimensional array data structure is used, and Sn-1_end and Sn+1_start are included. By including these, it is possible to perform processing using the timing of the previous character and the following character. As a comparative example, if Sn-1_end and Sn+1_start are not included, when performing processing using the timing of the previous character and the following character, it is necessary to refer to the reference data of the character Sn, the reference data of the character Sn-1, and the reference data of the character Sn+1, which results in a large number of references and a high processing load. On the other hand, in this embodiment, it is possible to reduce the number of references compared to the comparative example, thereby reducing the processing load.

[0069] Note that Sn-1_end and Sn+1_start may not be included in the time range. In this case, the time range will have a two-dimensional data structure, so by including two null data, it may be made into a four-dimensional data structure, making it easier to handle on the GPU. This also applies to the time ranges of phrases, lines, and chapters, which will be explained later.

[0070] In the same way as the 0th character, the 1st to mXth (last character) information is defined in the reference data.

[0071] For the 0th phrase (the character "elephant" in the example lyrics of Figure 5(A)), index data for the first character of the phrase, rectangular information indicating the position and size of the phrase, and the time range of the phrase are included. The index data for the first character of the phrase is stored as the index data for the 0th character, (0,0,0,0).

[0072] The rectangular information indicating the position and size of the phrase, in the case of the 0th phrase, indicates the position and size of the area surrounding the "elephant." When processing each vertex data, the GPU cannot refer to the information of other vertex data. Therefore, by including rectangular information indicating the position and size of the phrase in the reference data, it is possible to refer to the control position of the phrase when referencing the reference data.

[0073] The time range of a phrase is information indicating timing. In this embodiment, (Tn-1_end, Tn_start, Tn_end, Tn+1_start) is assigned as the time range of a phrase. Tn-1_end indicates the end timing of the previous phrase Tn-1, Tn_start indicates the start timing of the phrase Tn, Tn_end indicates the end timing of the phrase Tn, and Tn+1_start indicates the start timing of the subsequent phrase Tn+1.

[0074] As with text, when displaying the phrase Tn, two pieces of information are sufficient: the start timing Tn_start of the phrase Tn and the end timing Tn_end of the phrase Tn. However, in this embodiment, by including Sn-1_end and Sn+1_start, it is possible to perform processing using the timing of the previous and following phrases.

[0075] In the same way as the 0th phrase, the reference data defines information on the 1st to mYth phrases (last phrase).

[0076] As with phrases, lines (or chapters) also include index data for the first character of the line (or chapter), rectangular information indicating the position and size of the line (or chapter), and the time range of the line (or chapter).

[0077] In this way, the reference data of this embodiment defines index data, rectangular information indicating the position and size, and time range (timing) for characters, phrases, lines, and chapters, making it possible to easily control the display timing of characters, phrases, lines, and chapters. Note that it is sufficient for the reference data to include information indicating the time range (timing) for characters, phrases, lines, and chapters.

[0078] In this embodiment, the GPU 29 uses the above-described telop data (texture data, vertex data, and reference data) to display telops on the monitor 41. At this time, by changing the shader program for forming the telops in the GPU 29, it is possible to display a variety of telops. In this embodiment, the CPU 30 transmits a shader program corresponding to a piece of music to the RAM 28 for each playback process (S106). For example, by using a shader program corresponding to a piece of music, it is possible to display telops corresponding to the piece of music.

[0079] The shader program of this embodiment (corresponding to the "telop formation program" of the present invention) is a program that defines how telops are displayed on the screen, and is capable of controlling the display position, color, etc. of characters, chapters, lines, and phrases. The shader program of this embodiment is composed of a vertex shader program and a pixel shader program.

[0080] The vertex shader program is a program that specifies the display position and can control display, for example, by character, phrase, line, or chapter. Display control includes layouts such as right-justified, left-justified, stepped, and line breaks, as well as animations such as scrolling and pop-ups. Display control is not limited to the example layouts and animations, and includes various other layouts and animations. In addition, the vertex shader program can express captions as a stereoscopic image captured from a specified camera viewpoint in three-dimensional space.

[0081] On the other hand, the pixel shader program can control the color information for each pixel that forms the caption, and can control various color expressions for the caption to be displayed, such as using a single color or a gradient.

[0082] In this embodiment, a shader program (corresponding to the "telop formation program" according to the present invention) is formed by combining a vertex shader program and a pixel shader program. By preparing multiple vertex shader programs and multiple pixel shader programs in advance and changing the combination, it is possible to form shader programs that express a variety of telops.

[0083] After transmitting the telop data and shader program to RAM 28, enabling the GPU 29 to display telop data, the CPU 30 causes the sound control unit 25 to start playing back the performance data (S107). At this time, the current performance position (playback position) is transmitted to the GPU 29 in synchronization with the playback of the performance data. The current performance position is transmitted to the GPU 29 periodically (for example, for each frame to be drawn). The GPU 29 displays lyrics (telop display) on the monitor 41 based on the telop data and shader program stored in RAM 28 and the current performance position received from the CPU 30.

[0084] The vertex shader program executed by the GPU 29 references the time range (timing) in the reference data, determines the characters (or phrases, lines, or chapters) to be displayed, and determines the display position based on the vertex data. At this time, depending on the vertex shader program, coordinate transformation may be performed. The pixel shader program can also control the color of the character image to be displayed, and can display characters on polygons based on images extracted from texture data. The pixel shader program can also change the color of characters based on the current playing position, or can display characters in a variety of colors.

[0085] FIG. 7 shows an example of caption display on the screen in this embodiment. FIG. 7(A) shows a display format in which captions are displayed horizontally in the lower half of the screen, similar to conventional karaoke machines. The vertex shader program performs coordinate transformation on the vertex data described in FIG. 4 and arranges multiple polygons for a predetermined number of lines (two lines in the example of FIG. 7) on the screen. Next, the pixel shader program displays text on each polygon based on an image extracted from the texture data. The pixel shader program also determines the current singing position based on the current position transmitted from the CPU, and displays captions by changing the color before and after the current singing position.

[0086] Figure 7(B) shows a display format in which telops are displayed across the entire screen. Figure 7(B) also shows a display format in which line breaks are inserted between phrases, and multiple phrases are displayed in a stepped pattern on the screen. In this case, the vertex shader program performs coordinate transformation on the vertex data described in Figure 4, and arranges multiple polygons on the screen so that a predetermined number of phrases (four phrases in the example of Figure 7) are displayed in a stepped pattern. Next, the pixel shader program displays text on each polygon based on an image extracted from the texture data. The pixel shader program also determines the current singing position based on the current position sent from the CPU, and displays the telop by changing the color before and after the current singing position.

[0087] Figures 7(A) and 7(B) illustrate two-dimensional display, but by changing the shader program, it is possible to freely express layouts, such as displaying subtitles viewed from a certain viewpoint in three-dimensional space.

[0088] In this embodiment, the processing related to telop playback can be performed by the GPU 29, and the CPU 30 only needs to transmit the current performance position to the GPU 29, thereby reducing the processing load on the CPU 30. Also, by changing the shader program, it is possible to achieve a variety of telop expressions.

[0089] The CPU 30 continues to transmit the current playing position (S108) until the playback of the performance data is completed. When the playback of the performance data is completed (S109: Yes), the reservation table is checked to play the next song (S101).

[0090] In the above, in this embodiment, the display of lyrics (telop) has been described using a karaoke system as an example, but in this embodiment, by using a shader program, and telop data and shader program optimized for the GPU 29, it is possible to display telops using the GPU 29. Therefore, it is possible to display telops in a variety of ways and to reduce the processing load on the CPU 30.

[0091] [First Modification] Next, a modified example of the image processing device according to the present invention will be described. In the above embodiment, the display of telops has been described by taking the example of displaying lyrics using a karaoke device 2. The display of telops is not limited to displaying lyrics, and may also be used for subtitles of movies, moving images, etc. In this case, the information processing device that plays back the telops corresponds to the image processing device.

[0092] [Second Modification] In the above-described embodiment, a shader program (corresponding to the "telop formation program" according to the present invention) is transmitted to RAM 28 for each playback process. In addition to this configuration, various other configurations can be adopted for changing the shader program. For example, the shader program may be changed by a user operation. In this case, the shader program may be one that is prepared in advance, or may be changeable to a shader program created by the user himself.

[0093] Furthermore, although the above-described embodiment does not specifically mention the correspondence between music and shader programs, shader programs may be associated with at least one of music pieces, music genres, singers, composers, and lyricists, for example. Alternatively, shader programs may be associated with timekeeping information such as seasons, days of the week, and times of day. Alternatively, shader programs may be selected randomly.

[0094] Although various embodiments have been described above using the karaoke machine 2 as an example, the image processing device according to the present invention can be used not only as the karaoke machine 2 but also as various other information processing devices such as a personal computer, a game device, a smartphone, etc. Furthermore, image processing programs or image processing methods executed in various information processing devices also fall within the scope of the invention. [Explanation of symbols]

[0095] 1: Remote control device 24a: LAN communication unit 2: Karaoke device 24b: Wireless LAN communication unit 5: Server device 25: Sound control unit 11: Touch panel monitor 27: Memory 11a:Display section 28:RAM 11b: Touch panel 30: CPU 12: Video RAM 31: Video output section 13: Video control unit 32: Hard disk 14: Memory 33: Touch panel monitor 15: CPU 34: Touch panel 16:Wireless LAN communication section 35:Display section 17: Operation unit 41: Monitor 18: Operation processing unit 42: Speaker 21: Operation unit 43a, 43b: Microphone 22a~22d: Interface 130: Access point

Claims

1. An image processing device that includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, and that plays back telop data, The caption data is vertex data indicating the display position of each fragment of the telop; Texture data to be attached to each subtitle fragment, The reference data is configured as a data dimension array to be handled by the GPU; the reference data includes information indicating the timing of each of the plurality of fragments and information indicating the timing of a fragment collection configured to include a portion of the plurality of fragments; The CPU is a transmission process for transmitting the telop data to a storage unit; a content playback process for playing back the content data; The GPU is executes a telop playback process for playing back telop data using the telop data; The telop playback process is performed in synchronization with the content playback process based on information indicating the timing of the fragment units and fragment aggregate units included in the reference data. Image processing device.

2. The CPU generates at least one of vertex data, texture data, and reference data in the texture data from the telop source data corresponding to the content data. The image processing device according to claim 1 .

3. The timing information for each segment included in the reference data includes the start and end times of each segment. The image processing device according to claim 1 .

4. The timing information for each segment included in the reference data further includes the end time of the previous segment and the start time of the next segment. The image processing device according to claim 3 .

5. The reference data includes information indicating the timing of each of a plurality of fragment sets that are configured to include a portion of the plurality of fragments. The image processing device according to claim 1 .

6. The timing information of the fragment collection included in the reference data includes the start and end times of the fragment collection. The image processing device according to claim 1 .

7. The reference data includes location information for each fragment, The telop playback process plays back the telop based on the position information included in the reference data. The image processing device according to claim 1 .

8. The position information of each fragment included in the reference data includes rectangular information indicating the plane range of each fragment. The image processing device according to claim 7 .

9. The reference data includes location information of a fragment collection that includes a plurality of fragments. The image processing device according to claim 7 .

10. The position information of the fragment collection included in the reference data includes rectangular information indicating the range of the fragment collection on a plane. The image processing device according to claim 9 .

11. An image processing program executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The caption data is vertex data indicating the display position of each fragment of the telop; Texture data to be attached to each subtitle fragment, The reference data is configured as a data dimension array to be handled by the GPU; the reference data includes information indicating the timing of each of the plurality of fragments and information indicating the timing of a fragment collection configured to include a portion of the plurality of fragments; causing a CPU to execute a transmission process for transmitting telop data to a storage unit and a content reproduction process for reproducing content data; causing the GPU to execute a telop playback process for playing back telops using the telop data; The telop playback process is performed in synchronization with the content playback process based on information indicating the timing of the fragment units and fragment aggregate units included in the reference data. Image processing program.

12. An image processing method executed by an information processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a storage unit accessed by the GPU, The caption data is vertex data indicating the display position of each fragment of the telop; Texture data to be attached to each subtitle fragment, The reference data is configured as a data dimension array to be handled by the GPU; the reference data includes information indicating the timing of each of the plurality of fragments and information indicating the timing of a fragment collection configured to include a portion of the plurality of fragments; causing a CPU to execute a transmission process for transmitting telop data to a storage unit and a content reproduction process for reproducing content data; causing the GPU to execute a telop playback process for playing back telops using the telop data; The telop playback process is performed in synchronization with the content playback process based on information indicating the timing of the fragment units and fragment aggregate units included in the reference data. Image processing methods.

Images