Information processing apparatus, method, and program

Through the combination of sound source separation and spectral division data, color information is added to the lyrics, which solves the problem that multiple performers find it difficult to identify the responsible part when singing music, and realizes clear recognition of each performer part and visual of live concerts Improved effect.

CN119054019APending Publication Date: 2024-11-29SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380035229.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-28
Filing Date
2023-04-13
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The prior art is difficult to easily identify the music part that each performer is responsible for when multiple performers sing music, and it is impossible to effectively display the part that each performer is responsible for.

Method used

Through the sound source separation process, the audio data of the music is separated into singing and instrumental sounds, combined with the spectrum division data, add color information to the lyrics, generate spectral division data with color, use color to distinguish the part responsible for each performer, and Displayed in real time in the video.

Benefits of technology

The music part responsible for each performer can be easily identified, reducing operational errors and image capture errors, and improving the visual effect of live concerts and the audience's appreciation experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119054019A_ABST
    Figure CN119054019A_ABST
Patent Text Reader

Abstract

The present technology is an information processing device, method, and program that facilitate understanding a corresponding part in a piece of music in charge of a corresponding player. The information processing apparatus includes a video generation unit that generates a presentation video in which a character string is displayed on the basis of text information including text of sounds of a plurality of sound sources and one or more videos including at least one sound source as a subject, the character string represents a portion, a graphic form, or a sound source corresponding to the emitted sound in the text by a color representing the sound source of the emitted sound. This technique is applicable to a video processing system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present technology relates to an information processing apparatus and method, and a program, and more particularly to an information processing apparatus and method, and a program that enable each performer to easily identify a part of a musical piece for which the performer is responsible. Background Art

[0002] For example, in a live concert venue, a video performance is performed such as displaying a pre-prepared computer graphics (CG) video on a display device called a light emitting diode (LED) visual or LED monitor and displaying a video captured by a camera device that captures the video in real time in the live concert venue on the LED visual.

[0003] In addition, video performances are also performed, such as displaying a video called an embedded video obtained by synthesizing a CG video prepared in advance with a video of a camera device that captures a video in real time on an LED visual, and displaying the lyrics of a piece of music on an LED visual in a manner of superimposing on each of the above videos. In addition, in recent years, there have been cases where such videos are streamed in real time.

[0004] Such video displayed in a live concert venue is generated in what is called a video console. A video console is a place where all video signals are gathered in a live concert venue, such as a video master control room (master tuning room).

[0005] A plurality of video services, switchers, routers, etc. are set in a video console, and an operator at the video console selects a video signal to be output.

[0006] In the example, camera videos captured by about 10 camera operators arranged in the live concert venue are collected in the router of the video console, and the video to be presented (final video) is generated by appropriately performing effect processing and CG synthesis on the camera videos. Then, the final video is visually displayed on the LED installed in the live concert venue. Note that the process of superimposing the lyrics of the music on the final video is also performed on the video console.

[0007] Under current circumstances, video signal control performed by a video console to generate and output a final video is achieved by an operator performing a switching operation or the like according to spectrum division.

[0008] The score division indicates which member will sing and which lyrics part the member will sing for each measure of the music when multiple members (performers) sing the music. In other words, the score division is information indicating the part that each member (performer) in the music is responsible for.

[0009] For example, the spectrum division symbol rules are different for each artist, and the lyrics of the music piece can be shown in the spectrum division, the part of the lyrics to be sung by all members (the harmony part) can be underlined, or the name of the person responsible for the corresponding part of the lyrics (the member to be sung) can be shown.

[0010] Meetings and schedule decisions for live performances are performed based on such spectral divisions. Specifically, for example, the timing of switching camera switches, turning on lighting, activating special effects, etc. during a live concert is determined based on the spectral divisions.

[0011] As described above, the operator (video creation crew member) manually performs video signal control on the video console according to the spectrum division. Similarly, the camera operator inside the live concert venue also performs camera operation in response to voice instructions from other crew members referring to the spectrum division.

[0012] Therefore, the occurrence of errors in operation and errors in image capturing can be reduced by allowing an operator, a camera operator, or the like to clearly recognize at a glance the part in a musical piece that each performer (member) is responsible for.

[0013] In addition, in a case where the lyrics are displayed in a superimposed manner on a performance video to be presented to an audience at a live concert, it is possible for the audience, etc. to instantly identify which performer is singing and which part of the music piece the performer is singing, and to better enjoy the live concert by allowing the audience, etc. to clearly confirm at a glance the part of the music piece that each performer is responsible for.

[0014] For example, an accompaniment playback display device that presents characters of lyric information in different colors according to the pitch of a piece of music has been proposed as a technology for presenting lyrics about a piece of music (for example, see Patent Document 1).

[0015] Furthermore, for example, a technology has been proposed in which lyrics captions are displayed in synchronization with a piece of music by comparing a live video generated in real time with a background video of a performer's performance (see, for example, Patent Document 2).

[0016] Citation List

[0017] Patent Literature

[0018] Patent document 1: JP 2647890 B

[0019] Patent Document 2: JP 2008-145978 A Summary of the invention

[0020] Technical issues

[0021] However, according to the aforementioned technique, it is difficult to easily identify the part in a musical piece for which each performer is responsible.

[0022] For example, although lyrics information can be presented by the technology described in Patent Document 1 so that the pitch of a piece of music can be instantly recognized, it does not take into account that a piece of music is sung by multiple performers, and display about the part each performer is responsible for cannot be performed.

[0023] Similarly, since the technology described in Patent Document 2 does not take into account the fact that a plurality of performers sing a piece of music, it is not possible to perform a display of the part for which each performer is responsible.

[0024] The present technology has been made in view of such circumstances, and is suitable for enabling the part of a musical piece for which each performer is responsible to be easily identified.

[0025] Solution to the problem

[0026] According to the first aspect of the present technology, an information processing device includes: a video generation unit, which generates a presentation video based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted.

[0027] According to the first aspect of the present technology, an information processing method or program includes: generating a presentation video based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic or a character sequence representing the sound source is displayed using the color representing the sound source of the sound being emitted.

[0028] In a first aspect of the present technology, a presentation video is generated based on text information including text of sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part, a graphic, or a character sequence representing the sound source corresponding to the sound being emitted is displayed using the color representing the sound source of the sound being emitted.

[0029] According to the second aspect of the present technology, an information processing device includes: a color processing unit, which generates colored text information based on audio data including sounds from multiple sound sources or metadata of the audio data and text information including sounds from multiple sound sources prepared in advance about the audio data, in which a different color for each of the sound sources is added to the text of the sounds from each of the multiple sound sources.

[0030] An information processing method or program according to the second aspect of the present technology includes: generating colored text information based on audio data including sounds from multiple sound sources or metadata of audio data and text information including text of sounds from multiple sound sources prepared in advance about the audio data, in which a different color for each of the sound sources is added to the text of the sounds from each of the multiple sound sources.

[0031] In a second aspect of the present technology, based on audio data including sounds from multiple sound sources or metadata of audio data and text information including text of sounds from multiple sound sources prepared in advance for the audio data, colored text information is generated, in which a different color for each of the sound sources is added to the text of the sounds from each of the multiple sound sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] [ Figure 1 ] Figure 1 It is a diagram for explaining the present technology.

[0033] [ Figure 2 ] Figure 2 is a diagram showing an example of colored spectrum division data.

[0034] [ Figure 3 ] Figure 3 is a diagram showing a configuration example of an information processing device.

[0035] [ Figure 4 ] Figure 4 is a diagram of an example of metadata.

[0036] [ Figure 5 ] Figure 5 : is a flowchart for explaining the color spectrum division data generation process.

[0037] [ Figure 6 ] Figure 6 is a diagram showing a configuration example of a video processing system.

[0038] [ Figure 7 ] Figure 7 This is a flowchart for explaining the display process.

[0039] [ Figure 8 ] Figure 8 is a diagram showing a configuration example of a synthetic video generating device.

[0040] [ Fig. 9 ] Fig. 9 This is a flowchart for explaining the synthetic video generation process.

[0041] [ Fig.10 ] Fig.10 : is a diagram showing a display example of a camera video and a score.

[0042] [ Fig.11 ] Fig.11 is a diagram showing an example of a return video.

[0043] [ Fig.12 ] Fig.12 is a diagram showing an example of a synthesized video.

[0044] [ Fig.13 ] Fig.13 is a diagram showing an example of a synthesized video.

[0045] [ Fig.14 ] Fig.14 is a diagram showing an example of a synthesized video.

[0046] [ Fig.15 ] Fig.15 is a diagram showing a configuration example of a computer. DETAILED DESCRIPTION

[0047] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0048] <First Embodiment>

[0049] <This technology>

[0050] The present technology is for allowing the part of a musical piece that each performer is responsible for to be easily identified by adding color information corresponding to the performer in charge of the part to the part of the lyrics of the musical piece.

[0051] First, refer to Figure 1 An overview of the present technology is described. Note that the case where the present technology is applied to a live performance will be described below as a specific example.

[0052] exist Figure 1 In the example of , pre-processing and real-time processing are performed in an actual live concert to perform a video performance. The pre-processing is a process performed in advance before the live concert, and the real-time processing is a process performed during the live concert using the result of the pre-processing.

[0053] In the pre-processing, sound source separation processing is performed on the previously prepared sound sources of the music to be performed in the live concert, that is, sound source separation processing is performed on the audio data of the music as needed, and the audio data of the entire music is separated into audio data of instrument sounds and audio data of singing.

[0054] Note that the audio data for reproducing the instrumental sounds in the music, more specifically the sounds other than the performer's singing voice, will also be referred to as the instrumental sound data, while the audio data for reproducing only the performer's singing voice will also be referred to as the singing voice data hereinafter. In the case where the singing voice data is available in advance, there is no need to perform the sound source separation process.

[0055] It is possible to specify which performer among a plurality of performers (members) is singing each section, that is, which performer is in charge of each section of the music piece, based on the singing voice data.

[0056] Next, the vocal data of the music piece is compared with the lyrics information of the music piece prepared in advance, more specifically, the spectrum division data.

[0057] The spectrum division data is data including the lyrics information of the music piece and information indicating the performer responsible for each part of the lyrics of the music piece. In other words, the spectrum division data is text information including information indicating the text of the sound from each of the plurality of sound sources reproduced based on the singing voice data, that is, the text of the lyrics of the music piece sung by the plurality of performers and the sound source emitting the sound at each part of the text (the performer singing each lyrics part).

[0058] Based on the result of comparing the singing voice data with the spectrum division data, color information corresponding to the performer responsible for each part of the lyrics of the music piece is added, and lyrics information to which different colors are applied for each performer in charge is obtained. Here, the colored spectrum division data is obtained as lyrics information to which different colors are applied.

[0059] Although the color corresponding to the performer, that is, the color representing the performer can be defined in any manner, if the member color of the performer (member) is assigned, the part of the music piece that each performer is responsible for can be more easily identified.

[0060] Generally speaking, member colors are usually determined for group artists to identify individual members, and, for example, to prevent mistakes, clothing, microphones, and headphones are colored for members. In addition, during live concerts, fans send messages to members (artists) by flashing LED lights in the colors of the members that the fans support.

[0061] Therefore, not only for working members such as operators and camera operators but also for live concert audiences, viewers of real-time streaming, etc., by assigning the performer's member color to each of a plurality of performers (members), the performer corresponding to the member color can be instantly identified.

[0062] The colored spectrum division data is obtained by the preprocessing as described above. The colored spectrum division data includes lyrics information of a music piece, information indicating a performer responsible for each part of the lyrics of the music piece, and color information indicating the performer responsible for each part of the lyrics of the music piece.

[0063] In other words, it can be explained that the colored spectral division data is colored text information in which, based on the audio data (which is the singing (voice) of multiple performers), different colors are applied for each sound source (performer) to each part of the text information of the sound from multiple sound sources (i.e., each part of the lyrics).

[0064] Here, in Figure 2 A specific example of colored spectrum division data is shown in .

[0065] Figure 2 An example of spectrum division data of one music piece before color information is added is shown on the left side in the figure, and an example of colored spectrum division data obtained by adding color information to the spectrum division data is shown on the right side in the figure.

[0066] In the spectrum division data shown on the left side of the figure, the performer's name is shown on the left column in the spectrum division of the music piece, the number of the chorus part corresponding to the lyrics information is shown in the middle column, and in the right column, the lyrics of the music piece are divided into each measure and shown in black characters (i.e., characters of a single color).

[0067] In this example, the musical composition is sung by a group of three performers named Sato, Tanaka, and Suzuki (a trio).

[0068] In the spectrum division, the performer's name shown on the left side of the lyrics of each shown interval (measure) indicates the performer responsible for that part of the lyrics.

[0069] Specifically, for example, "どんぐりころころどんぶりこ" as the beginning part of the music piece shown at the top of the score division is the part that performer Sato is responsible for. In addition, the part of the music piece where "all members" is described in the column of the performer's name indicates that all three performers are to sing (be responsible for).

[0070] Here, it is assumed that the member colors of the performers "Sato", "Tanaka", and "Suzuki" are "green", "blue", and "orange", respectively.

[0071] In this case, the colored spectrum division data shown on the right side in the figure is obtained from the spectrum division data shown on the left side in the figure and the member colors of the performers.

[0072] The configuration of the spectrum division indicated by the colored spectrum division data, that is, the arrangement of the columns describing the lyrics of the music and the columns describing the names of the performers (members' names) responsible for each part is the same as the case of the spectrum division data shown on the left side of the figure.

[0073] Figure 2 The difference between the spectrum division data shown and the spectrum division data with color is that: in the spectrum division data, the lyrics of the music are described by characters of the same color (single color), regardless of the responsible performer; while in the spectrum division data with color, the lyrics of the music are shown by characters of the color of the member of the responsible performer.

[0074] For example, since performer "Sato" is responsible for the beginning part "Dongurikorokorodonburiko" of the music piece described at the top of the spectrum division indicated by the colored spectrum division data, the lyrics of the beginning part are shown in characters in green, which is the member color of performer "Sato".

[0075] Furthermore, the lyrics of the part for which all the performers are simultaneously responsible are described by characters of a color different from the color of any member of the performers, for which there is a description of "all members" in the column of the performer's name.

[0076] According to such colored spectrum division data to which color information is added, it is possible to instantly and intuitively recognize which performer is in charge, and which part of the music piece the performer is in charge of.

[0077] For example, not only staff members such as operators and camera operators can use colored spectrum division data, but artists (performers) can also use colored spectrum division data, and in this case it is also possible to clearly identify at a glance who is going to sing and which part of the music the performer is going to sing for each performer.

[0078] Return to Figure 1 As described above, once colored spectrum division data is obtained through pre-processing, real-time processing is performed based on the colored spectrum division data during a live concert.

[0079] In real-time processing, image capturing is performed by a plurality of cameras in a live concert venue, and facial recognition processing is performed on the plurality of camera videos obtained as a result.

[0080] Since which performer (member) appears as a subject in the camera video can be specified by facial recognition processing, candidates for camera videos used to generate a video for a performance (presentation) are selected based on the result of the facial recognition processing, the colored spectrum division data, etc. In other words, one or more camera videos are selected as candidates for camera videos used to generate a video for a performance from among a plurality of camera videos obtained by image capture.

[0081] In this case, for example, camera videos including as subjects a performer (member) in charge of a part of a piece of music currently being performed may be selected as candidates.

[0082] The operator or the like appropriately performs final selection from the thus obtained candidates, generates a composite video as a video for the performance based on the finally selected one or more camera videos and the colored spectrum division data, and presents the obtained composite video in a live concert venue.

[0083] For example, in order to generate a composite video, a pre-prepared CG video, some or all of a plurality of camera device videos, etc. are aligned and composited, and the lyrics of a portion of the performer responsible for the piece of music being performed are composited with the composite video using the member color of the performer in that portion (superimposed on the composite video).

[0084] At this time, video creation staff members such as operators can perform operations such as switching, turning on lighting, activating special effects, etc. while referring to the colored spectrum division data and referring to the candidates (camera video) selected and presented based on the colored spectrum division data.

[0085] Therefore, the occurrence of errors made by video creation staff in camera switching timing, special effect activation timing, and the like can be reduced.

[0086] In addition, the following video can be generated as a return video, which is generated based on the colored spectrum division data so that it is visually understood which performer (member) is responsible for a part of the music piece being performed, more specifically, the part immediately after the part being performed. In this case, the return video is provided to each camera device or display device corresponding to the camera device in the live concert venue, and is displayed.

[0087] Note that the return video may be the same video as the composite video presented to the viewer, or may be a different video from the composite video. Furthermore, the return video may be different for each camera.

[0088] Since the camera operator can clearly recognize at a glance which performer (member) should be filmed and at what timing by presenting such return video, the occurrence of failure in capturing video, imaging timing errors, and imaging errors such as focus deviation can be reduced.

[0089] Furthermore, by presenting a composite video generated based on colored spectrum division data and displaying lyrics with color information in a live concert venue, audiences in the live concert venue and viewers of the live video can easily visually identify which performer is singing and which part is being sung.

[0090] <Configuration Example of Information Processing Device>

[0091] Figure 3 : is a diagram showing a configuration example of an information processing device configured by a computer or the like that performs the aforementioned pre-processing. Figure 3 The information processing device 11 shown may be installed in the video console, or may be installed at a location different from the video console.

[0092] The information processing device 11 includes a sound source separation unit 21 , a singing voice analysis unit 22 , a comparison unit 23 , and a color processing unit 24 .

[0093] Sound source data, which is audio data for reproducing the sound of a musical piece, is supplied to the sound source separation unit 21. The sound source separation unit 21 separates instrument sound data and singing voice data from the sound source data by performing sound source separation processing on the supplied sound source data, and supplies the obtained singing voice data to the singing voice analysis unit 22.

[0094] Here, the singing voice data is audio data including the sound of the target sound source (performer) as the performer's singing voice, and the instrument sound data is audio data including other sound sources different from the performer, such as the sound of musical instruments, that is, the sound from the sound source other than the target.

[0095] Note that, in the case where singing voice data exists before the musical instrument sounds are remixed in advance, it is not necessary to perform the sound source separation process, and the sound source separation unit 21 may not be provided in this case.

[0096] The singing voice analysis unit 22 performs singing voice analysis, that is, analysis processing on the singing voice data supplied from the sound source separation unit 21 .

[0097] For example, in singing analysis, which performer is singing and which interval of the sound (singing voice) the performer is singing is specified based on the singing data. In other words, a performer (sound source) is specified for each of the multiple intervals of the singing data, and the performer's sound (singing voice) is included in the interval.

[0098] In a specific example, for example, singing voice analysis can be achieved by matching the frequency of singing voice in singing voice data with the frequency of each performer's voice prepared in advance.

[0099] Furthermore, the singing voice analysis unit 22 generates metadata of the music piece based on the voice analysis result of each section of the singing voice data (result of the singing voice analysis). Furthermore, for example, the singing voice analysis unit 22 generates temporary lyrics information by performing a sound recognition process as singing voice analysis on the singing voice data.

[0100] The singing voice analyzing unit 22 supplies metadata and temporary lyrics information of the music piece obtained through the singing voice analysis to the comparing unit 23 .

[0101] For example, the singing voice analysis unit 22 performs transcription as text conversion of the singing voice data by voice recognition processing on the singing voice (sound) based on the singing voice data, and obtains character information (text information) indicating the result as lyrics information indicating temporary lyrics of the music piece. The lyrics information is information indicating temporary lyrics, and the lyrics indicated by the lyrics information are not necessarily accurate lyrics.

[0102] In addition, metadata of a piece of music is data indicating which performer's voice is included in each interval of the singing voice data. In other words, metadata includes information indicating a performer (sound source) for each of a plurality of intervals of the singing voice data, in which the performer's voice (singing voice) is included. Note that metadata can be used in the case where metadata is included in the sound source data of the piece of music in advance.

[0103] Figure 4 An example of metadata is shown. In this example, the horizontal direction indicates time, and the metadata is a timeline shown in the figure.

[0104] Figure 4 The metadata shown is moving image data including information indicating each of a plurality of intervals aligned in the time direction when the singing voice data is divided into a plurality of time intervals and information indicating a performer whose voice (sound) is included in each interval.

[0105] Specifically, each square aligned in the time direction represents one section (time section), and in this example, color information indicating a performer whose voice is included in the section is added to each section.

[0106] Specifically, one piece of music is sung by three performers, "Sato", "Tanaka", and "Suzuki", and in this example, the member colors of the performers "Sato", "Tanaka", and "Suzuki" are defined as "green", "blue", and "orange".

[0107] If such metadata is reproduced, then for each reproduction time, that is, for each time interval, the color of the performer who is responsible for a part of the music corresponding to the time interval is displayed. In addition, the color (color information) representing the performer can also be used to display the performer's name.

[0108] Specifically, for example, the color “green” indicating the performer “Sato” is present in the section T1. In other words, the color information “green” indicating the performer “Sato” has been added (applied) to the section T1.

[0109] Furthermore, the colors “blue” and “orange” indicating the performers “Tanaka” and “Suzuki” are presented in section T2, and it can be confirmed that the part of the music piece corresponding to section T2 will be sung by the performers “Tanaka” and “Suzuki”.

[0110] In particular, a section in which a plurality of color information pieces are displayed and the number is less than the total number of performers is a section called harmony for which a plurality of performers are responsible at the same time, and in the harmony section, the color representing the performer responsible for the high-pitched part is displayed at the upper part in the square representing the section. In other words, at the position of the harmony, a color information presentation area corresponding to each part is defined in advance for each part at the position.

[0111] In this example, it is possible to instantly visually recognize who is in charge of each part based on the color "blue" displayed in the upper area and the color "orange" displayed in the lower area in interval T2, that is, the positional relationship of the areas. In other words, it is possible to intuitively recognize that the performer "Tanaka" corresponding to the color "blue" is in charge of the upper part, and the performer "Suzuki" corresponding to the color "orange" is in charge of the lower part.

[0112] In addition, each of the colors "green", "blue" and "orange" indicating each of all performers "Sato", "Tanaka" and "Suzuki" is presented in interval T3, and it can be confirmed that the part of the music corresponding to interval T3 will be sung by all performers.

[0113] Note that although an example in which color information is added to each interval is described here, the metadata may be any information as long as it is possible to specify which performer's voice is included in each interval of the singing voice data according to the information. For example, an identifier such as a numerical value for identifying each performer or the performer's name may be added to (displayed in) each interval of the singing voice data in the metadata.

[0114] Return to Figure 3 As described above, the comparison unit 23 compares the singing voice data with the spectrum division data of the music piece prepared in advance.

[0115] In other words, more specifically, the comparison unit 23 compares the metadata and temporary lyrics information of the music piece supplied from the singing voice analysis unit 22 with the spectrum division data prepared in advance, and supplies the comparison result and the metadata to the color processing unit 24 .

[0116] In the comparison unit 23, the temporary lyrics information as a result of the text conversion of the singing voice data is compared with the accurate lyrics indicated by the spectrum division data, for example, to specify the correspondence between each interval in the singing voice data and each measure of the music piece in the spectrum division based on the spectrum division data. In other words, synchronization of the singing voice data and the spectrum division data is performed.

[0117] Furthermore, the comparison unit 23 compares the correspondence between the performer indicated by the color information added to each interval indicated by the metadata and the performer responsible for each measure in the spectrum division based on the spectrum division data. In the comparison between one of the temporary lyrics information and the metadata and the spectrum division data, the comparison result between the other of the temporary lyrics information and the metadata and the spectrum division data may be appropriately referred to.

[0118] By performing such comparison, it is possible to improve the accuracy of specification of each section (measure) of a musical piece based on the specification of the performer in charge of each section of the musical piece and the singing voice data, and obtain more accurate colored spectrum division data.

[0119] The color processing unit 24 generates colored spectrum division data based on the metadata and the comparison result supplied from the comparison unit 23 , and outputs the obtained colored spectrum division data to a recording medium or the like (not shown).

[0120] For example, the color processing unit 24 generates colored spectrum division data by adding color information to the lyrics information included in the spectrum division data prepared in advance for the sound source data of the music piece, that is, by applying color to the lyrics based on the metadata and the comparison result of the comparison unit 23. Note that if the temporary lyrics information and the metadata are corrected based on the comparison result of the comparison unit 23, the color processing unit 24 can generate accurate colored spectrum division data without using the spectrum division data.

[0121] <Description of Colored Spectrum Division Data Generation Process>

[0122] Next, the operation of the information processing apparatus 11 will be described.

[0123] The information processing device 11 performs a signal processing operation for each piece of music performed in the live concert at an appropriate timing before the live concert begins. Figure 5 The colored spectrum division data generation process shown is the aforementioned preliminary process.

[0124] In the following, reference will be made to Figure 5 The colored spectrum division data generation processing performed by the information processing device 11 is described with reference to the flowchart in FIG.

[0125] In step S11 , the sound source separation unit 21 performs a sound source separation process on the supplied sound source data, and supplies singing voice data obtained as a result to the singing voice analysis unit 22 .

[0126] In step S12 , the singing voice analysis unit 22 performs singing voice analysis on the singing voice data supplied from the sound source separation unit 21 to generate temporary lyrics information and metadata, and supplies the temporary lyrics information and metadata to the comparison unit 23 .

[0127] For example, in step S12, metadata of the music piece is generated based on the result of the singing voice analysis, and temporary lyrics information is generated based on the singing voice data by voice recognition processing or the like. In this way, for example, Figure 4 The metadata shown.

[0128] In step S13 , the comparison unit 23 performs comparison processing of comparing the metadata and temporary lyrics information of the music piece supplied from the singing voice analysis unit 22 with the spectrum division data prepared in advance, and supplies the comparison result and metadata obtained by the comparison processing to the color processing unit 24 .

[0129] In step S14 , the color processing unit 24 generates colored spectrum divided data by performing color processing based on the metadata and the comparison result supplied from the comparison unit 23 , and outputs the obtained colored spectrum divided data.

[0130] For example, the result of the comparison processing (comparison result) is taken into consideration in the color processing, the starting position, the ending position, etc. of each interval indicated by the metadata are appropriately corrected, and color information indicating the responsible performer is added to the lyrics information in the spectrum division data, thereby obtaining colored spectrum division data.

[0131] Once the colored spectrum division data is obtained in this way, the colored spectrum division data generation process ends.

[0132] As described above, the information processing device 11 generates colored spectrum division data by performing singing voice analysis, comparison processing, and color processing based on the lyrics data and the spectrum division data.

[0133] By generating colored spectrum division data in this manner, the part of the musical piece for which each performer is responsible can be easily identified by the obtained colored spectrum division data.

[0134] The set list of a live concert, etc., the length of each piece of music, etc. are usually determined a few days before the live preview, and it is difficult to manually write them appropriately into the spectrum division data of many pieces of music. In this regard, by using the present technology, appropriate colored spectrum division data can be obtained in a short period of time without manual effort for many pieces of music.

[0135] <Configuration Example of Video Processing System>

[0136] Figure 6 is a diagram showing a configuration example of a video processing system that performs the aforementioned real-time processing.

[0137] Figure 6 The video processing system 61 shown includes a bird's-eye view camera 71-1, tour cameras 71-2 to 71-N, a serial digital interface (SDI) router 72, a camera switcher 73, a CG / camera video synthesis unit 74, a final video switcher 75, a lyrics synthesis unit 76 and a display unit 77.

[0138] For example, the bird's-eye camera 71-1 is a camera installed at a position where the entire live concert venue can be overlooked in the live concert venue. The bird's-eye camera 71-1 images the entire live concert venue as a subject and supplies the resulting camera video to the SDI router 72.

[0139] The patrol cameras 71 - 2 to 71 -N are arranged at fixed positions in a live concert venue, for example, or move inside the venue together with a camera operator in the live concert venue.

[0140] The patrol cameras 71 - 2 to 71 -N image at least a portion of the interior of the live concert venue as a subject, and supply the resulting camera videos to the SDI router 72 .

[0141] Note that hereinafter, in the case where it is not necessary to particularly distinguish the tour cameras 71 - 2 to 71 -N, the tour cameras 71 - 2 to 71 -N will also be simply referred to as the tour camera 71 .

[0142] In the example, although 8 to 10 patrol cameras 71 are used to capture images in a live concert venue, for example, the number of patrol cameras 71 may be any number.

[0143] In addition, in the case where it is not necessary to particularly distinguish between the bird's-eye camera 71-1 and the patrol camera 71, the bird's-eye camera 71-1 and the patrol camera 71 will also be simply referred to as the camera 71 hereinafter. Hereinafter, it is assumed that in the camera video captured by the camera 71, at least one performer among a plurality of performers providing performances on a stage or the like is included as a subject.

[0144] The camera videos obtained by all the cameras 71 in the live concert venue are collected by the SDI router 72. The SDI router 72 supplies the camera video supplied from each of the plurality of cameras 71 to the camera switcher 73.

[0145] Note that the connection between each camera 71 and the SDI router 72 may be established in a wired manner, may be established in a wireless manner, or may be established in a combination of wired and wireless manners.

[0146] The camera switcher 73 provides any one or more of the multiple camera videos provided from the SDI router 72 to the CG / camera video synthesis unit 74, and provides any one or more of the multiple camera videos provided from the SDI router 72 to the final video switcher 75.

[0147] For example, the camera video supplied to the CG / camera video synthesis unit 74 and the camera video supplied to the final video switcher 75 are different videos from each other.

[0148] The CG / camera video synthesis unit 74 generates a synthesized video (hereinafter also referred to as a synthesized camera video) by synthesizing other camera videos or CG videos prepared in advance with the camera video provided from the camera switcher 73, and provides the synthesized video to the final video switcher 75.

[0149] The final video switcher 75 generates a composite video based on at least any one of the camera video provided from the camera switcher 73 , the composite camera video provided from the CG / camera video synthesis unit 74 , and the CG video prepared in advance, and provides the composite video to the lyrics synthesis unit 76 .

[0150] Furthermore, the final video switcher 75 generates a return video based on the colored spectrum division data generated by the pre-processing. At this time, the final video switcher 75 generates the return video appropriately using the camera video supplied from the camera switcher 73, the synthesized camera video supplied from the CG / camera video synthesis unit 74. As described above, the return video is a presentation video to be presented to the camera operator to check the image capture timing and the like.

[0151] Finally, the video switcher 75 provides the generated return video to the camera 71 or a display device such as an SDI monitor or a tablet computer provided near the camera operator who operates the camera 71, and causes the camera 71 or the display device to display the return video.

[0152] Note that the connection between each camera 71 or the display device corresponding to the camera 71 and the final video switcher 75 may be established in a wired manner, may be established in a wireless manner, or may be established in a combination of wired and wireless manners.

[0153] The lyrics synthesis unit 76 superimposes the colored lyrics information (colored text information), that is, the lyrics to which the color information is added based on the colored spectrum division data generated by the pre-processing, on the synthesized video supplied from the final video switcher 75, thereby obtaining the final synthesized video. The lyrics synthesis unit 76 supplies the final synthesized video to the display unit 77, and causes the display unit 77 to display the final synthesized video as a presentation video.

[0154] For example, the SDI router 72 to the lyrics synthesis unit 76 configuring the video processing system 61 are provided in a video console.

[0155] The display unit 77 is configured by a display device such as an LED vision or a liquid crystal panel, for example, and is set on a stage or the like in a live concert venue. The LED vision is a large display device configured by an LED panel. The display unit 77 presents a synthesized video to the audience of the live concert or the like by displaying the synthesized video provided from the lyrics synthesizing unit 76.

[0156] <Description of Display Processing>

[0157] Subsequently, a general operation flow of the video processing system 61 will be described. Figure 7 The display processing of the video processing system 61 is described with reference to the flowchart in FIG. The display processing is performed as the real-time processing described above.

[0158] In step S41, each of the plurality of cameras 71 images a part or all of the interior of the live concert venue as a subject, and supplies the camera video obtained as a result to the SDI router 72. In addition, the SDI router 72 supplies the camera video supplied from each of the plurality of cameras 71 to the camera switcher 73.

[0159] In step S42 , the camera switcher 73 switches the camera video to be output to the CG / camera video synthesis unit 74 and the final video switcher 75 .

[0160] For example, the operator or the like performs a switching operation of switching the camera video to be output to the CG / camera video synthesis unit 74 and the final video switcher 75, referring to the colored spectrum division data and the rendered camera video, etc. as needed. In other words, the camera video to be provided to the CG / camera video synthesis unit 74 and the camera video to be provided to the final video switcher 75 are selected.

[0161] Then, the camera switcher 73 supplies a designated camera video from among the plurality of camera videos supplied from the SDI router 72 to the CG / camera video synthesis unit 74 in response to a switching operation performed by an operator or the like.

[0162] Similarly, the camera switcher 73 supplies a designated camera video from among the plurality of camera videos supplied from the SDI router 72 to the final video switcher 75 in response to a switching operation performed by an operator or the like.

[0163] In step S43 , the CG / camera video synthesis unit 74 performs video synthesis based on the camera video supplied from the camera switch 73 .

[0164] For example, the CG / camera video synthesis unit 74 generates a synthesized camera video by synthesizing the camera video provided from the camera switcher 73 or synthesizing the camera video provided from the camera switcher 73 and the CG video, and provides the synthesized camera video to the final video switcher 75.

[0165] In step S44 , the final video switcher 75 generates a synthesized video and supplies the synthesized video to the lyrics synthesizing unit 76 .

[0166] For example, the final video switcher 75 generates a synthesized video based on at least any one of the camera video supplied from the camera switcher 73, the synthesized camera video supplied from the CG / camera video synthesizing unit 74, and the CG video prepared in advance. At this time, the final video switcher 75 generates a synthesized video based on an operation such as selection (switching) appropriately performed by an operator or the like and the colored spectrum division data prepared in advance.

[0167] Furthermore, the final video switcher 75 generates a return video based on the colored spectrum division data generated by the pre-processing, using the camera video supplied from the camera switcher 73 and the synthesized camera video supplied from the CG / camera video synthesizing unit 74 as appropriate.

[0168] In step S45 , the lyrics synthesis unit 76 performs lyrics synthesis based on the synthesized video supplied from the final video switcher 75 and the colored spectrum division data prepared in advance.

[0169] In other words, the lyrics synthesis unit 76 superimposes (synthesizes) the colored lyrics information based on the colored spectrum division data onto the synthesized video from the final video switcher 75 , thereby obtaining a final synthesized video.

[0170] In step S46 , the lyrics synthesis unit 76 supplies the final presentation synthesized video generated in step S45 to the display unit 77 , and causes the display unit 77 to display the synthesized video.

[0171] In step S47 , the final video switcher 75 supplies the generated return video to the camera 71 or a display device provided near the camera operator who operates the camera 71 , and causes the camera 71 or the display device to display the return video.

[0172] In step S48 , the final video switcher 75 determines whether or not the process of causing the composite video to be displayed is finished.

[0173] In a case where it is determined in step S48 that the processing has not yet ended, the processing returns to step S41, and the aforementioned processing is repeatedly performed.

[0174] On the other hand, in the case where it is determined in step S48 that the processing is to be ended, each component of the video processing system 61 stops the processing being executed, and ends the display processing.

[0175] As described above, the video processing system 61 generates the composite video and the return video using the colored spectrum division data, and causes the composite video and the return video to be displayed.

[0176] In this way, audience members of a live concert who are presented with a composite video, viewers of a real-time streaming, and camera operators who are presented with a return video can easily identify the part of the piece of music for which each performer is responsible.

[0177] <Configuration Example of Synthetic Video Generating Device>

[0178] Here, a more specific example of the generation of the composite video will be described.

[0179] Figure 8 is a diagram showing a configuration example of a composite video generating device that generates a final composite video to be displayed on the display unit 77 .

[0180] Figure 8 The synthetic video generating device 121 shown in FIG. Figure 6The illustrated information processing device is a camera switcher 73, a CG / camera video synthesis unit 74, a final video switcher 75, and a lyrics synthesis unit 76. Note that the synthesized video generating device 121 may be realized by one device, or may be realized by a plurality of devices.

[0181] The synthetic video generating device 121 includes a face recognition unit 131 , a spectrum division comparison unit 132 , a camera video selection unit 133 , and a video synthesis unit 134 .

[0182] In particular, the camera video selection unit 133 is implemented by the camera switch 73 and functions as a candidate selection unit that selects a candidate of a camera video for generating a composite video as a presentation video. In addition, the video synthesis unit 134 is implemented by the CG / camera video synthesis unit 74, the final video switch 75 and the lyrics synthesis unit 76 and functions as a video generation unit that generates a presentation composite video.

[0183] The plurality of camera videos output from the SDI router 72 are supplied to the facial recognition unit 131 .

[0184] The facial recognition unit 131 performs facial recognition processing on the supplied camera video, specifies a performer (member) included as an object in the camera video, and supplies the specification result as a result of the facial recognition processing to the spectrum division comparison unit 132. In addition, the facial recognition unit 131 also supplies the supplied plurality of camera videos to the spectrum division comparison unit 132.

[0185] Note that the face recognition unit 131 may perform face recognition processing on each of a plurality of camera videos (video signals) individually, or may perform face recognition processing on one group video obtained by merging a plurality of camera videos by the camera switcher 73 .

[0186] For example, the camera switch 73 displays a camera video array obtained by aligning a plurality of camera videos, such as 3×3 or 4×4 as indicated by arrow W11 .

[0187] In the example indicated by arrow W11 , three camera videos are aligned in each of the longitudinal direction and the lateral direction, and a 3×3 (3 rows and 3 columns) camera video array including a total of nine camera videos is formed.

[0188] For example, the camera video P11 forming the camera video array has been captured by one camera 71, and a predetermined performer (member) is included as an object in the camera video P11. Similarly, the camera video P12 is captured by another camera 71 different from the camera 71 that captured the camera video P11, and the same performer as that in the case of the camera video P11 is also included as an object in the camera video P12.

[0189] For example, the camera switch 73 may generate one group video by aligning and synthesizing a plurality of camera videos in the same array (arrangement) as the camera video array indicated by arrow W11, and use the group video as an input to the facial recognition unit 131. In this case, which performer is included as an object and which area in the group video the performer is included in is specified in the facial recognition process, that is, which camera video configuring the group video includes the performer.

[0190] For example, a plurality of feature points of the face of each performer prepared in advance may be compared with a plurality of feature points of the face extracted from the camera video, and facial recognition processing may be performed based on the similarity of the feature points, more specifically, the similarity of the positional relationship of the feature points. In this case, a score indicating the similarity between the face included as an object in the camera video and the predefined performer may also be output as a result of the facial recognition processing.

[0191] In other cases where the camera 71 side has a facial recognition function, such as an artificial intelligence (AI) recognition function installed in the camera 71, the camera 71 may perform facial recognition processing. In this case, for example, the result of the facial recognition processing is added to the camera video as metadata. For example, the result of the facial recognition processing may be performer information indicating a performer (member) included as an object in the camera video.

[0192] The spectrum division comparison unit 132 compares the result of the face recognition processing supplied from the face recognition unit 131 with the colored spectrum division data prepared in advance, and supplies the comparison result and the camera video to the camera video selection unit 133 .

[0193] In this case, for example, a portion of the colored spectrum division data currently being performed by the performer is provided to the spectrum division comparison unit 132. In other words, the spectrum division comparison unit 132 can specify the lyrics part currently being performed and the performer responsible for the lyrics part based on the colored spectrum division data.

[0194] Similarly, the video synthesis unit 134 can also specify the lyrics part currently being performed and the performer responsible for the lyrics part based on the colored spectrum division data.

[0195] For example, the spectrum division comparison unit 132 can specify the camera video including the performer (member) in charge of the part of the music piece currently being performed as a subject by comparing the result of the face recognition processing with the part currently being performed (bar part) in the colored spectrum division data. In other words, it can be specified whether the performer included as a subject in the camera video is currently making a sound (singing).

[0196] The camera video selection unit 133 selects one or more camera videos from the plurality of camera videos supplied from the spectrum division comparison unit 132 as candidates for videos to be used for generating a final synthesized video (presentation synthesized video) based on the comparison result supplied from the spectrum division comparison unit 132 .

[0197] For example, the camera video selection unit 133 selects as a candidate a camera video including at least a performer (member) in charge of a part of a music piece being performed, that is, a performer currently singing as a subject (a sound source of sound being emitted).

[0198] In this case, since the same performer may be imaged as a subject by a plurality of cameras 71, the number of camera videos to be selected as candidates may be one or more. For example, identification of camera videos selected as candidates may be performed based on, for example, a camera number for identifying the camera 71 that has captured the camera video.

[0199] In other cases, the camera video as a candidate may be selected in response to a selection operation performed by an operator, etc. In this case, if a score indicating the similarity to a performer has been obtained as a result of, for example, facial recognition processing, and the score is displayed together with the camera video, the operator, etc. may select the camera video as a candidate with reference to the score and the camera video.

[0200] The camera video selection unit 133 supplies one or more camera videos selected as candidates to the video synthesis unit 134 .

[0201] The video synthesis unit 134 generates a synthesized video based on at least one or more camera videos among the one or more camera videos supplied as candidates from the camera video selection unit 133 and the colored spectrum division data prepared in advance, and supplies the synthesized video to the display unit 77 .

[0202] For example, the video synthesis unit 134 generates a synthesized video by aligning and synthesizing the plurality of camera videos supplied from the camera video selection unit 133 , and superimposes colored lyrics information on the synthesized video based on the colored spectrum division data, thereby obtaining a final synthesized video.

[0203] Note that all of the camera videos selected as candidates by the camera video selection unit 133 may not necessarily be used to generate a synthesized video, and a synthesized video only needs to be generated using one or more of the camera videos selected as candidates.

[0204] For example, one candidate (camera video) selected from a plurality of candidates by an operator etc. may be used as a composite video as it is. In other cases, a composite video may be generated using not only the camera video selected from the candidates but also a CG video prepared in advance.

[0205] As a specific example of a composite video, for example, when a camera video array indicated by arrow W11 is obtained and camera video P11 and camera video P12 including the same performer as an object are selected as candidates, a composite video indicated by arrow W12 is obtained.

[0206] In the composite video, camera video P11 and camera video P12 including a performer currently singing as an object are arranged in an aligned manner, and lyrics information of a portion of the music currently being performed is superimposed using color on the portion corresponding to camera video P11 and camera video P12.

[0207] In this example, in particular, the color of the lyrics information is the member color of the performers (members) included in the camera video P11 and the camera video P12 , that is, the performers who are singing the part being performed.

[0208] In other words, the currently performed part of the lyrics information described in the colored spectrum division data, that is, the part corresponding to the sound being emitted, is displayed in the composite video as the presentation video using the color representing the sound source (performer) of the currently emitted sound.

[0209] The aforementioned processing performed by the face recognition unit 131 to the video synthesis unit 134 corresponds to the reference Figure 7 The processing in steps S42 to S45 in the above-mentioned display processing.

[0210] Note that the video synthesis unit 134 may generate a return video of each camera 71 based on the colored spectrum division data, and supply the obtained return video to each camera 71 or a display device corresponding to the camera 71 .

[0211] <Description of Synthetic Video Generation Processing>

[0212] Then, refer to Fig. 9 The synthetic video generation process performed by the synthetic video generation device 121 is described by referring to the flowchart in FIG. The synthetic video generation process corresponds to Figure 7 Steps S42 to S45 in the display processing in .

[0213] In step S81 , the face recognition unit 131 performs face recognition processing on the supplied camera video, and supplies the result of the face recognition processing and the camera video to the spectrum division comparison unit 132 .

[0214] In step S82 , the spectrum division comparison unit 132 compares the result of the face recognition processing supplied from the face recognition unit 131 with the colored spectrum division data, and supplies the comparison result and the camera video supplied from the face recognition unit 131 to the camera video selection unit 133 .

[0215] In step S83, the camera video selection unit 133 selects a candidate of a camera video to be used for generating a synthesized video from the plurality of camera videos supplied from the spectrum division comparison unit 132, based on the comparison result supplied from the spectrum division comparison unit 132. Then, the camera video selection unit 133 supplies the camera video selected as the candidate to the video synthesis unit 134.

[0216] In step S84 , the video synthesis unit 134 generates a synthesized video based on the camera video supplied from the camera video selection unit 133 and the colored spectrum division data prepared in advance.

[0217] In addition, the video synthesis unit 134 also appropriately generates a return video based on the colored spectrum division data. In the return video, a portion that is a presentation target in the lyrics information described in the colored spectrum division data, an arbitrary graphic, a character sequence (text) of a performer's name, etc. is displayed with a color representing the sound source of the sound emitted (i.e., the member color of the performer).

[0218] Once the final composite video and return video are generated, the composite video generation process ends.

[0219] For example, by Figure 7The composite video generated in step S84 is provided to the display unit 77 and displayed by the display unit 77 by processing in step S46 in step S84. Figure 7 The processing in step S47 in the above operation provides the return video generated in step S84 to the camera 71 or a display device corresponding to the camera 71 and is displayed by the camera 71 or a display device corresponding to the camera 71.

[0220] As described above, the composite video generating means 121 generates a composite video based on the camera video and the colored spectrum division data.

[0221] In this case, if a composite video is generated on which, for example, colored lyric information is superimposed, it can allow audiences at a live concert and viewers of a live streaming broadcast to whom the composite video is presented to easily identify the part of the music for which each performer is responsible.

[0222] Furthermore, a synthesized video can be automatically generated in real time, for example, by performing face recognition processing and using colored spectrum division data, with a switching operation or the like by an operator or the like not always required.

[0223] Similarly, it is also possible to allow a camera operator to easily recognize the part of the musical piece for which each performer is responsible by generating a return video based on the colored spectrum division data.

[0224] <User Interface Example>

[0225] Here, specific examples of the synthesized video and return video described so far, and a user interface (UI) and the like presented to an operator and the like in the camera switcher 73 and the final video switcher 75 will be described.

[0226] First, an example of a user interface presented to an operator or the like by a display unit not shown in the camera switcher 73 will be described.

[0227] In the camera switch 73, that is, in the composite video generation device 121, it is assumed that Fig.10 The camera video array indicated by the arrow W41 in FIG. 1 is displayed on, for example, a display unit not shown. The camera video array indicated by the arrow W41 is displayed on, for example, a display unit not shown. Figure 8 The camera device indicated by arrow W11 in FIG. 1 is similar to the video array.

[0228] In addition, it is assumed that through facial recognition processing, the facial recognition unit 131 has obtained scores such as a score for facial recognition for each camera video (a score for the possibility of a performer) and a score indicating the image angle of the camera video, that is, how good the composition of the camera video is, based on the result of the facial recognition processing.

[0229] Specifically, the score of each camera video is obtained based on the ratio of the area including facial feature points to the area of ​​the entire image angle, that is, the size of the area of ​​the performer's face in the camera video, the number of facial feature points extracted from the camera video, etc.

[0230] At this time, for example, when a face from the side of a performer is captured in a camera video, since the number of feature points of the face extracted from the camera video decreases, the score of composition or the like can be set low.

[0231] In this case, for example, the video synthesis unit 134 can cause an unillustrated display unit, more specifically, cause a UI displayed on the display unit to display each camera video selected as a candidate by the camera video selection unit 133 and the score of the camera video as indicated by arrow W42.

[0232] In this example, three camera videos P31 to P33 are displayed on the UI, and the score of each of the camera videos is displayed below the camera videos P31 and P33 in the figure. In particular, the scores related to the composition (image angle) of the camera videos are displayed here, and it can be confirmed that the score of the camera video P31 that captures the performer's face in a large size is high.

[0233] Note that the score for each camera video may be, for example, a score related to the composition, or may be a score indicating the level of similarity with the face of a performer in facial recognition, i.e., a score indicating the likelihood of the performer, or scores for multiple items, such as both scores, may be displayed.

[0234] The operator etc. refers not only to the camera video itself displayed in this manner but also to the score of the camera video to perform switching operations etc. and ultimately selects (specifies) the camera video to be displayed on the display unit 77, i.e., the camera video to be used to generate the composite video.

[0235] When one or more camera videos are designated by a switching operation of an operator, for example, Fig. 9 In step S84 in , a composite video is generated on the camera video specified by the switching operation, etc. In addition, the camera video specified by the switching operation can be used as the composite video as it is.

[0236] In other cases, the video synthesis unit 134 may select one or more camera videos in descending order of scores from a plurality of camera videos selected as candidates without relying on an operation by an operator or the like, and generate a synthesized video using the camera videos.

[0237] An operator or the like can also select a more appropriate camera video with reference to the scores, and cause a composite video to be generated through the presentation of the scores of the camera videos as described above.

[0238] For example, multiple performers can be captured in a camera video. In this case, if the score of facial recognition is displayed, that is, the score indicating the similarity with the performer, it is easy and instantaneous to specify whether the camera video is the camera video of the performer currently singing.

[0239] Note that Fig.10 As in the example shown, the camera switch 73 (camera video selection unit 133) may cause a display unit not shown to display each candidate (camera video) and the score instead of the video synthesis unit 134. In this case, the camera switch 73 selects the final camera video selected by the operator or the like from the displayed plurality of candidates, and supplies the selected camera video to the video synthesis unit 134.

[0240] <Return to video example>

[0241] Fig.11 An example of a return video displayed on the display unit of the camera 71 is shown.

[0242] In by Fig.11 In the example indicated by the arrow W51 in , for example, on a viewfinder or a monitor as a display unit of the camera 71, colored lyrics information as a return video is displayed in a superimposed manner on a through image including the object.

[0243] Specifically, the lyrics information, i.e., a character sequence (text) representing the lyrics, is displayed in the member color of the performer (member) responsible for the portion of the lyrics information, and is displayed according to the timing of the sound of the music being performed. In other words, the lyrics portion of the music currently being performed or the lyrics portion to be performed immediately after the portion of the music currently being performed is displayed in the member color.

[0244] Furthermore, in the example indicated by the arrow W52 , for example, in a viewfinder or a monitor as a display unit of the camera 71 , a frame K11 surrounding the entire through image is displayed as a return video.

[0245] In particular, frame K11, which is a colored graphic as a return video, is displayed using the member color of the performer (member) in charge of the part of the music currently being performed or the part to be performed immediately after the part of the music currently being performed. In addition, once the performer (member) in charge of the corresponding part changes, the color of the frame displayed as the return video also changes.

[0246] In other words, information indicating the member color of the performer (member) in charge of the portion in the music piece that is the target to be presented to the camera operator is displayed as the return video according to the timing of the sound of the music piece being performed.

[0247] Note that the return video is not limited to the frame K11, and may be any other graphics, or may be colored text information (character sequence) indicating a performer, such as a performer's name, etc. In this case, a graphic or character sequence such as a performer's name as the return video is also displayed using the performer's member color.

[0248] By displaying the return video indicated by arrows W51 and W52 according to the timing of the sound of the music, it is possible to clearly confirm at a glance which performer is to be imaged and at what time the performer is to be imaged. Therefore, the occurrence of failure to capture video, imaging timing errors, and imaging errors such as focus deviation can be reduced.

[0249] Note that even if information related to the performer who is the imaging target is displayed in real time as a return video, there may be a case where image capture cannot be performed in time. Therefore, the occurrence of imaging errors can be further reduced by generating colored lyrics information and the performer's name, graphics, etc. of the part immediately following the part of the music piece currently being performed in the member color of the performer in charge of the part as the return video of each camera device 71, respectively.

[0250] In other cases, a control signal may be provided to the camera 71 based on the colored spectrum division data by the video synthesis unit 134 to turn on (make it emit light) an indicator light as a lighting section provided in the camera 71. In this case, it is assumed that the video synthesis unit 134 has understood which performer is to be imaged and which camera 71 is to image the performer.

[0251] For example, a lighting section called an indicator light is provided at a portion that can be seen from one side of the subject in each camera 71. Generally, the indicator light is used to check which camera is being used for image capture for broadcasting. However, it is impossible to confirm which performer is being imaged by the camera provided with the indicator light from the performer's side.

[0252] Therefore, the video synthesis unit 134 can provide a control signal to the camera 71 that is imaging the performer in charge of the part of the music currently being performed as a subject, so that the indicator light of the camera 71 is turned on with the member color of the performer in charge of the part being performed.

[0253] Therefore, the performer can instantly recognize to which camera 71 the performer himself / herself directed his / her sight, face, etc., and can obtain a camera video with a better composition.

[0254] <Example of composite video>

[0255] For example, a group shot video of multiple performers can be displayed as a composite video in a live concert venue. In addition, each performer in the multiple performers can be displayed one by one as a composite video in a time-division manner.

[0256] Fig.12 An example of a case where a group shot video is displayed as a composite video is shown.

[0257] In this example, five performers are performing on a stage, and a composite video is displayed behind the performers.

[0258] In particular, in this example, the composite video is divided into five member areas corresponding to the performers (members). In addition, the composite video is displayed so that the member area that will display the performer in a large size is arranged at the part behind the performer.

[0259] Fig.13 Another example of the case where the group shot video is displayed as a composite video is shown.

[0260] In this example, the composite video is divided into member areas R11 to R15, the number of which is the same as the number of performers, and a video of one performer (member) corresponding to each member area is displayed in the member areas.

[0261] Specifically, for example, a predetermined performer is displayed in the member region R11 , and other performers are displayed in the member region R12 .

[0262] Furthermore, the lyrics information of the portion of the music piece being performed is displayed in color in the composite video, and in this example, the portion of the lyrics information being displayed is the portion that all the performers are to sing together.

[0263] At this time, each part of the lyrics information displayed in each member area can be displayed in the member color of the performer corresponding to the member area. Specifically, for example, the part of the lyrics information displayed in the member area R11 is displayed using the member color of the performer corresponding to the member area R11.

[0264] In other cases, the parts to be sung by all performers in the lyrics information, that is, the parts for which all performers are responsible, may be displayed with gradation of the member colors of all performers (all member colors).

[0265] For example, such a composite video can be generated by cropping and combining camera videos obtained by a plurality of cameras 71 .

[0266] The realism of the video can be enhanced by dividing the entire screen of the composite video into member areas and displaying performers (members) in the corresponding member areas, or by displaying lyrics information using colors according to the member areas.

[0267] <Other examples>

[0268] With the colored spectrum division data as described above, it is possible to flexibly address the increase / decrease in the number of performers, the change in the colors corresponding to the performers, etc. In other words, the utilization of different colors in the spectrum division can be changed in an adaptable manner.

[0269] For example, even in the case where the configuration of members in a group including a plurality of performers is changed, the colors representing the performers and the configuration of the performers in the colored spectrum division data can be easily changed.

[0270] Furthermore, although an example in which a color is associated with each performer has been described above, a color may be associated with a plurality of performers.

[0271] Specifically, for example, in the case where an idol group includes a large number of members, for example, different colors may be assigned to groups including a plurality of performers, for example, blue for the first season members, green for the second season members, red for the third season members, and yellow for the fourth season members. In this case, too, different color utilizations may still be changed in an adaptable manner.

[0272] In other cases, for example, a user may be able to specify any member color in karaoke. Thus, the use of different colors may be changed according to the user's preferences.

[0273] In a specific example, for example, different colors may be used according to a man's voice or a woman's voice. In addition, for example, in the case where two users sing a piece of music for a five-member group, the number of colors may be reduced from five to two as the number of users (the number of people singing the song). In this case, colored spectrum division data that proposes realistic singing timing may be generated.

[0274] Furthermore, the present technology can also be applied to, for example, 3DCG lyrics generation and 3DCG effect generation using spectrum division using different colors.

[0275] Generally speaking, lyrics display in real-time streaming is usually 2D white letter display. In addition, foreground / background extraction technology using greenback, infrared (IR) / short-wave infrared (SWIR) camera, depth camera, deep learning, etc. has been developed.

[0276] Therefore, it can be Fig.14 As shown, only performer FG11 is extracted from the camera video as the foreground, for example, the video of performer FG11 can be synthesized with a background video such as a CG video, and the colored lyrics information LY11 generated based on the colored spectrum division data can be synthesized with the background video to obtain the final synthesized video.

[0277] In this example, a video of the performer FG11 as a human foreground extracted in real time is superimposed on the background (composite with the background), and the lyrics information LY11 is also displayed as a background video in different colors applied based on the colored spectrum division data. In this example, in particular, the display of the lyrics information LY11 is a gradient display in which the colors are different little by little.

[0278] Note that not only the color of the lyrics information but also the color of effects such as particles and background video can be displayed with differently allocated colors based on the colored spectrum division data.

[0279] As described above, it is possible not only to allow the video to be changed just before being displayed, but also to shorten the video creation time. In addition, new video expressions can be realized.

[0280] Furthermore, the present technology can be applied not only to live concerts, but also to captions for stage performances and television programs, various video distribution services, subtitles for movies, and the like.

[0281] For example, the line of the stage performance of each performer, each gender or each group can be shown in different colors by generating data similar to the colored spectrum division data. Similarly, the TV caption can be shown in different colors for each performer.

[0282] In recent years, each broadcasting station has made various efforts on broadcasting for hearing-impaired persons, teletext broadcasting, and the like, and the present technology which can be applied to television broadcasting and the like is useful.

[0283] Furthermore, in the case of applying the present technology to subtitles of movies, for example, it is only necessary to perform sound recognition processing on sound data of movie contents and show lines in different colors according to characters or translations by audio letters.

[0284] The present technology also enables the use of different colors without any pre-processing, as long as simultaneous real-time processing of voice recognition processing and face recognition processing can be performed.

[0285] <Example of computer configuration>

[0286] The above series of processing can also be performed by hardware or software. In the case where the series of processing is performed by software, the program configuring the software is installed on the computer. Here, the computer includes, for example, a computer built in dedicated hardware and a general-purpose personal computer on which various programs are installed to be able to perform various functions, etc.

[0287] Fig.15 : is a block diagram showing a configuration example of computer hardware that executes the foregoing series of processes using a program.

[0288] In the computer, a central processing unit (CPU) 501 , a read only memory (ROM) 502 , and a random access memory (RAM) 503 are connected to one another via a bus 504 .

[0289] The input / output interface 505 is also connected to the bus 504 . An input unit 506 , an output unit 507 , a recording unit 508 , a communication unit 509 , and a drive 510 are connected to the input / output interface 505 .

[0290] The input unit 506 is a keyboard, a mouse, a microphone, an imaging element, etc. The output unit 507 is a display, a speaker, etc. The recording unit 508 includes a hard disk and a nonvolatile memory. The communication unit 509 includes a network interface. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0291] In the computer thus configured, the CPU 501 loads, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the program, so that a series of processing is performed.

[0292] The program executed by the computer (CPU 501) may be recorded on, for example, the removable recording medium 511 used as a package medium for supply. The program may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0293] In the computer, by mounting the removable recording medium 511 on the drive 510, the program can be installed in the recording unit 508 via the input / output interface 505. In addition, the program can be received by the communication unit 509 through a wired or wireless transmission medium and installed in the recording unit 508. In addition, the program can be pre-installed on the ROM 502 or the recording unit 508.

[0294] Meanwhile, the program executed by the computer may be a program that executes processing in time series according to the sequence described in this specification, or a program that executes processing in parallel or at necessary timing such as timing of execution calling.

[0295] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and may be modified in various ways within the scope of the present technology without departing from the gist of the present technology.

[0296] For example, the present technology may be configured as cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0297] Each step described in the above flowchart may be performed by a single device, and may also be shared and performed by a plurality of devices.

[0298] Furthermore, when one step includes a plurality of processing steps, the plurality of processing steps included in one step may be performed by a single device, and may also be shared and performed by a plurality of devices.

[0299] Furthermore, the present technology can be configured as follows. (1)

[0301] An information processing device includes: a video generation unit, which generates a presentation video based on text information of a text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted. (2)

[0303] The information processing device according to (1), wherein the text information is colored text information in which a different color for each of the sound sources is added to the text of the sound from each of the plurality of sound sources. (3)

[0305] The information processing device according to (1) or (2), wherein, in the presentation video, at least the sound source of the sound being emitted among the plurality of sound sources is displayed as the object. (4)

[0307] The information processing device according to any one of (1) to (3), further comprising: a face recognition unit that performs face recognition processing on the plurality of videos,

[0308] The video generation unit generates the presentation video based on the result of the facial recognition processing and the text information. (5)

[0310] The information processing device according to (4) further includes: a candidate selection unit, which selects a candidate of the video to be used to generate the presentation video from the multiple videos based on the result of the facial recognition processing and the text information, and the video generation unit generates the presentation video based on one or more videos among the one or more videos selected as the candidates. (6)

[0312] The information processing device according to (5), wherein the video generation unit causes the one or more videos selected as the candidates to be displayed together with scores obtained for the videos based on a result of the facial recognition process. (7)

[0314] The information processing device according to (6), wherein the score is a score of possibility of the sound source or a score of composition of the video. (8)

[0316] The information processing device according to (6) or (7), wherein the video generation unit generates the presentation video based on one or more videos selected by an operator from among the one or more videos presented as the candidates together with the score. (9)

[0318] The information processing device according to any one of (5) to (8), wherein the candidate selection unit selects the video including the sound source of the sound being emitted as an object as the candidate. (10)

[0320] The information processing device according to (2), wherein the presentation video is generated for each camera that captures the video including the sound source as the object, and is provided to the camera or a display device corresponding to the camera. (11)

[0322] An information processing device according to (10), wherein the video generation unit generates the presentation video and provides the presentation video to the camera device or the display device, in which a part of the text corresponding to the sound emitted immediately after the sound currently being emitted, the graphic or the character sequence is displayed using the color of the sound source representing the sound emitted immediately after the sound currently being emitted. (12)

[0324] An information processing device according to (2), wherein the video generation unit causes a lighting unit of a camera device that images the sound source of a sound currently being emitted as an object to emit light having a color representing the sound source of the sound currently being emitted based on the colored text information. (13)

[0326] The information processing device according to (2), wherein the colored text information is colored spectrum division data of the music piece. (14)

[0328] An information processing method, comprising using information processing to perform the following operations:

[0329] A presentation video is generated based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic, or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted. (15)

[0331] A program that causes a computer to perform processing, the processing comprising: generating a presentation video based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic, or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted. (16)

[0333] An information processing device includes: a color processing unit, which generates colored text information based on audio data including sounds from multiple sound sources or metadata of the audio data and text information including text of the sounds from the multiple sound sources prepared in advance with respect to the audio data, wherein a different color for each of the multiple sound sources is added to the text of the sounds from each of the multiple sound sources. (17)

[0335] The information processing device according to claim (16), further comprising: a sound source generating unit that performs sound source separation processing on sound source data including sounds from the plurality of sound sources and sounds from other sound sources different from the plurality of sound sources,

[0336] The color processing unit generates the colored text information based on the audio data of the sounds from the plurality of sound sources and the text information obtained by the sound source separation process. (18)

[0338] The information processing device according to (16) or (17), further comprising: an analysis unit that performs analysis processing to specify the sound source for each of a plurality of intervals of the audio data, the sound from the sound source being included in the interval,

[0339] The color processing unit generates the colored text information based on the result of the analysis process and the text information. (19)

[0341] The information processing device according to any one of (16) to (18), further comprising: an analysis unit that performs analysis processing on the text conversion of the audio data,

[0342] The color processing unit generates the colored text information based on the result of the text conversion and the text information. (20)

[0344] The information processing device according to (16), wherein the metadata includes information indicating the sound source for each of a plurality of sections of the audio data, the sound from the sound source being included in the section. (twenty one)

[0346] The information processing device according to any one of (16) to (20), wherein the colored text information is colored spectrum division data of the music piece. (twenty two)

[0348] An information processing method causes an information processing device to perform the following operations:

[0349] Based on audio data including sounds from a plurality of sound sources or metadata of the audio data and text information including text of the sounds from the plurality of sound sources prepared in advance with respect to the audio data, colored text information is generated in which a different color for each of the sound sources is added to the text of the sounds from each of the plurality of sound sources. (twenty three)

[0351] A program causing a computer to execute processing, the processing comprising: generating colored text information based on audio data including sounds from a plurality of sound sources or metadata of the audio data and text information including text of the sounds from the plurality of sound sources prepared in advance with respect to the audio data, wherein a different color for each of the sound sources is added to the text of the sounds from each of the plurality of sound sources.

[0352] Reference numerals list

[0353] 11Information processing device

[0354] 21 sound source separation unit

[0355] 22 Vocal Analysis Unit

[0356] 23. Comparison Unit

[0357] 24 color processing units

[0358] 61 Video Processing System

[0359] 73 Camera switch

[0360] 75 Final Video Switcher

[0361] 76 Lyrics Synthesis Unit

[0362] 121 Synthetic Video Generator

[0363] 131 Facial Recognition Unit

[0364] 132 Spectrum division comparison unit

[0365] 133 Camera video selection unit

[0366] 134 Video Synthesis Unit

Claims

1. An information processing device, comprising: A video generating unit, which generates a presentation video based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted.

2. The information processing device according to claim 1, wherein: The text information is colored text information in which a different color for each of the plurality of sound sources is added to the text of the sound from each of the plurality of sound sources.

3. The information processing device according to claim 1, wherein: In the presentation video, at least the sound source of the sound being emitted among the plurality of sound sources is displayed as the object.

4. The information processing device according to claim 1, further comprising: a facial recognition unit that performs facial recognition processing on the plurality of videos, The video generation unit generates the presentation video based on the result of the facial recognition processing and the text information.

5. The information processing device according to claim 4, further comprising: A candidate selection unit, which selects a candidate of the video to be used to generate the presentation video from the multiple videos based on the result of the facial recognition processing and the text information, and the video generation unit generates the presentation video based on one or more videos among the one or more videos selected as the candidates.

6. The information processing device according to claim 5, wherein: The video generation unit causes one or more videos selected as the candidates to be displayed together with scores obtained for the videos based on a result of the facial recognition process.

7. The information processing device according to claim 6, wherein: The score is a score of the possibility of the sound source or a score of the composition of the video.

8. The information processing device according to claim 6, wherein: The video generation unit generates the presentation video based on one or more videos selected by an operator from among the one or more videos presented as the candidates together with the score.

9. The information processing device according to claim 5, wherein: The candidate selection unit selects the video including the sound source of the sound being emitted as an object as the candidate.

10. The information processing device according to claim 2, wherein: The presentation video is generated for each camera that has captured the video including the sound source as the object, and is provided to the camera or a display device corresponding to the camera.

11. The information processing device according to claim 10, wherein: The video generation unit generates the presentation video and provides the presentation video to the camera device or the display device, in which a part of the text corresponding to the sound emitted immediately after the sound currently being emitted, the graphic or the character sequence is displayed using the color of the sound source representing the sound emitted immediately after the sound currently being emitted.

12. The information processing device according to claim 2, wherein: The video generation unit causes a lighting unit of an imaging device that images the sound source of the currently emitted sound as a subject to emit light having a color representing the sound source of the currently emitted sound based on the colored text information.

13. The information processing device according to claim 2, wherein: The colored text information is colored spectrum division data of the music piece.

14. An information processing method using an information processing device, comprising: A presentation video is generated based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic, or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted.

15. A program for causing a computer to execute a process, the process comprising: A presentation video is generated based on text information of text including sounds from multiple sound sources and one or more videos including at least one sound source as an object, in which a part of the text corresponding to the sound being emitted, a graphic, or a character sequence representing the sound source is displayed using the color of the sound source representing the sound being emitted.

16. An information processing device, comprising: A color processing unit that generates colored text information based on audio data including sounds from a plurality of sound sources or metadata of the audio data and text information including text of the sounds from the plurality of sound sources prepared in advance with respect to the audio data, in which a different color for each of the sound sources is added to the text of the sounds from each of the plurality of sound sources.

17. The information processing device according to claim 16, further comprising: an analyzing unit that performs an analyzing process of specifying the sound source for each of a plurality of sections of the audio data, the sound from the sound source being included in the section, The color processing unit generates the colored text information based on the result of the analysis process and the text information.

18. The information processing device according to claim 16, wherein: The colored text information is colored spectrum division data of the music piece.

19. An information processing method using an information processing device, comprising: Based on audio data including sounds from a plurality of sound sources or metadata of the audio data and text information including text of the sounds from the plurality of sound sources prepared in advance with respect to the audio data, colored text information is generated in which a different color for each of the sound sources is added to the text of the sounds from each of the plurality of sound sources.

20. A program for causing a computer to execute a process, the process comprising: Based on audio data including sounds from a plurality of sound sources or metadata of the audio data and text information including text of the sounds from the plurality of sound sources prepared in advance with respect to the audio data, colored text information is generated in which a different color for each of the sound sources is added to the text of the sounds from each of the plurality of sound sources.