Information processing system, information processing method, and program

Abstract

The present technology relates to an information processing system, an information processing method, and a program, which make it possible to easily generate information that is used for musical direction. An information processing system according to one aspect of the present technology acquires music data and, on the basis of musical piece data included in the music data, acquires musical piece features. Furthermore, the information processing system generates, on the basis of the musical piece features, direction control information that is used for controlling the musical direction, and controls the musical direction on the basis of the direction control information. The present technology can be applied to a computer that controls the direction of a concert being held at a virtual concert venue or a physical concert venue.

Description

Information processing system, information processing method, and program

[0001] The present technology relates to an information processing system, an information processing method, and a program, and more particularly to an information processing system, an information processing method, and a program that enable easy generation of information used in musical performance.

[0002] Virtual concerts are currently being held in which musicians hold concerts in venues prepared in a virtual space called the Metaverse. The musicians use motion capture technology to move their avatars and sing and dance. Audience members can participate in the virtual concerts and watch the musicians' performances using VR (Virtual Reality) devices such as HMDs (Head Mounted Displays).

[0003] Just like in real-life concerts, virtual concerts also feature visual effects such as background lighting and lighting effects behind the musicians. Virtual objects can also be placed in the venue, taking advantage of the unique characteristics of a virtual concert.

[0004] International Publication No. 2015 / 016094

[0005] It takes time and effort to prepare the content for the effects. As mentioned above, in order to make an object appear, you need to prepare a 3D model using a tool. Also, various settings are required, such as how to move the object in the virtual space and when to make it appear.

[0006] The present technology has been made in view of such circumstances, and makes it possible to easily generate information used in musical performance.

[0007] An information processing system according to one aspect of the present technology includes a music data acquisition unit that acquires music data, a feature acquisition unit that acquires features of a song based on the song data included in the music data, a generation unit that generates performance control information used to control the performance of the music based on the features of the song, and a control unit that controls the performance of the music based on the performance control information.

[0008] In one aspect of the present technology, music data is acquired, characteristics of the music are acquired based on the music data included in the music data, performance control information used to control the performance of the music is generated based on the characteristics of the music, and the performance of the music is controlled based on the performance control information.

[0009] 11 is a diagram showing an example of capturing a virtual concert. FIG. 12 is a diagram showing an example of capturing at another timing. FIG. 13 is a diagram showing an example of a stage. FIG. 14 is a block diagram showing an example of a configuration of an information processing system according to an embodiment of the present technology. FIG. 15 is a block diagram showing an example of a configuration of an information processing unit included in a content generation device. FIG. 16 is a diagram showing an example of generating control information. FIG. 17 is a diagram showing an example of generating content for performance. FIG. 18 is a diagram showing an example of a prompt. FIG. 19 is a diagram showing an example of generating a decorative lyric image. FIG. 19 is a diagram showing how a virtual concert is operated using a controller. FIG. 19 is a diagram showing an example of a controller screen. FIG. 11 is an enlarged view of the GRAPH area of ​​FIG. 11. FIG. 19 is a block diagram showing an example of the functional configuration of a distribution side. FIG. 19 is a diagram showing how a real concert is performed. FIG. 19 is a block diagram showing an example of the configuration of an information processing unit that controls the performance of a real concert. FIG. 19 is a diagram showing an example of a market for data used in the progress of a concert. FIG. 19 is a block diagram showing an example of the configuration of a computer.

[0010] Hereinafter, embodiments of the present technology will be described in the following order: 1. Example of a virtual concert 2. Configuration of an information processing system 3. Configuration of a content generation device 4. Editing of performances using a controller 5. Example of application to a real concert 6. Modified examples

[0011] <Example of Virtual Concert> Fig. 1 is a diagram showing an example of capturing a virtual concert. In an information processing system according to an embodiment of the present technology, distribution content including video (movie) and audio of the virtual concert is distributed via the Internet and delivered to users who will be audience members.

[0012] The stage in Fig. 1 is a virtual space constructed as a three-dimensional world. The stage is constructed using a 3D model. Various 3D model objects are placed on the stage. 2D images may also be placed as objects. Avatar A, placed near the center of the stage, is a performer in the concert.

[0013] On the stage, avatar A is depicted singing and dancing. The performance includes singing as well as playing an instrument. Avatar A is a humanoid avatar placed on the stage as a 3D model representing a musician (singer). Many audience avatars are placed around avatar A. The audience avatars move in time with the performance, creating a lively atmosphere for the concert.

[0014] Above avatar A, multiple objects #1 representing a sequence of deformed letters are placed. Object #1 in FIG. 1 represents the letters "Beautiful TOKYO" and "Happy! Bird!". Object #1 changes by moving, changing color, and deforming. The type of letters on object #1 changes in accordance with the music played by avatar A. As will be described later, object #1 represents the lyrics of the song avatar A is singing.

[0015] Furthermore, two objects #2 representing flying birds are placed to the right of object #1. Object #2 changes by moving and changing color. Object #2 represents keywords that are evoked from the lyrics of the song avatar A is singing.

[0016] Beams #3 are emitted into the air from various positions on the stage. The state of Beam #3 also changes, changing direction and color.

[0017] In this way, visual effects using objects, beams, etc. are produced on stage in sync with avatar A's performance. Auditory effects using sound effects such as voices are also produced. The progress of the concert, including avatar A's movements, performance, and stage direction, is controlled by the virtual concert distributor. Users who will be in the audience can also control the direction.

[0018] In addition, the shooting range of the virtual concert, i.e., the camerawork, is controlled by the virtual concert distributor. For example, multiple virtual cameras are set up on the stage. By switching the cameras used for shooting, images with viewpoints from various positions are generated and distributed via the Internet. Users who are audience members of the virtual concert will watch the video of the virtual concert along with audio using devices with displays, such as smartphones, TVs, PCs, and HMDs. Audience users may also be able to move their own avatars and watch images with the avatar's position as the viewpoint.

[0019] Figure 2 shows an example of capture at another timing. By switching the camera in this way, images with various viewpoints can be generated. The image shown in Figure 2 was captured using a camera set in a position close to avatar A. Object #1, which represents the letters "with me," is placed in the background of avatar A.

[0020] FIG. 3 is a diagram showing an example of a stage.

[0021] The stage itself can be switched as needed as the virtual concert progresses. The stage shown in Figure 3A is a virtual space resembling a forest, while the stage shown in Figure 3B is a virtual space resembling an ocean. The type of stage is also controlled by the creator of the virtual concert content.

[0022] <Configuration of Information Processing System> FIG. 4 is a block diagram showing an example configuration of an information processing system 1 according to an embodiment of the present technology.

[0023] As shown on the left side of Fig. 4, the distribution side is configured with a content generator 11, a distribution server 12, and a control device 13. Each device is configured with a computer. The content generator 11 and the distribution server 12, and the content generator 11 and the control device 13 are connected to each other via wired or wireless communication. The devices may also be connected to each other via a network such as the Internet or a LAN.

[0024] The content generation device 11 generates distribution content including video and audio of a virtual concert and outputs the content to the distribution server 12. The content generation device 11 is provided with music data of songs to be performed by avatars, as well as 3D model data of various objects such as a stage and avatars. The content generation device 11 controls the progress of the virtual concert based on, for example, the music data and lyric data that make up the music data, and generates the distribution content. The content generation device 11 is also provided with an AI that generates control information used to progress the virtual concert. The content generation device 11 is also provided with an AI that generates 3D model content used for performance.

[0025] In this way, the content generation device 11 uses AI (Artificial Intelligence) for generating control information, generating content for presentation, and the like. Note that using AI means performing processing using a machine learning model such as a neural network generated by machine learning. The machine learning model generated by machine learning is prepared in the content generation device 11. The machine learning model includes an inference model that analyzes and outputs the characteristics of input data, and a generation model that generates data in a different format based on the input data and outputs the data.

[0026] The distribution server 12 distributes the distribution content generated by the content generator 11. The distribution by the distribution server 12 may be performed in real time, or may be performed using archived distribution content. When the distribution content is distributed in archive format, the distribution content generated by the content generator 11 is recorded, for example, in storage within the distribution server 12 and distributed at any timing. In this way, the distribution server 12 functions as a distribution control unit that distributes the distribution content including the video and audio of the virtual concert in real time or after storing it in storage.

[0027] The control device 13 is a device used by a distributor as a controller for the virtual concert. The control device 13 is used to control the production of the virtual concert. Distributors include, for example, musicians who express their performance through avatars, and producers who manage the progress of the virtual concert and create the content to be distributed.

[0028] As shown on the right side of Fig. 4, the audience side is configured with user terminals 21-1, 21-2, .... When there is no need to distinguish between the user terminals 21-1, 21-2, ... used by users, they are collectively referred to as user terminals 21. The user terminals 21 are devices with displays, such as smartphones, TVs, PCs, and HMDs. A large number of user terminals 21, such as hundreds or thousands, are connected to the distribution server 12 via the Internet.

[0029] The user terminal 21 receives the content distributed by the distribution server 12, displays the video of the virtual concert, and outputs the audio from speakers, earphones, etc. If the virtual concert is distributed in real time, it may be possible for the audience to interact with the distribution side. As an example of audience interaction, a user's avatar placed on the stage may move or the performance may change in response to an operation by the audience user.

[0030] 5 is a block diagram showing an example of the configuration of the information processing unit 101 included in the content generation device 11. The information processing unit 101 is realized by a computer constituting the content generation device 11 executing a predetermined program.

[0031] The information processing unit 101 is composed of a music data acquisition unit 111, a music analysis unit 112, a control information generation unit 113, a control unit 114, a performance content generation unit 115, and a content DB 116. At least a part of the configuration of the information processing unit 101 may be realized in another device.

[0032] The music data acquisition unit 111 acquires music data of a song to be played by an avatar. For example, the music data is acquired by reading it from the storage of a computer that constitutes the content generation device 11.

[0033] Music data includes music data and lyric data. Music data is data of sounds that make up a song. Music data includes at least one of a singer's voice, the sounds of an instrument being played, and sound effects. The singer's voice may be an actual person's voice, or may be voice generated by voice synthesis. Lyric data is text data of lyrics. The music data acquisition unit 111 outputs the music data included in the music data to the music analysis unit 112, and outputs the lyric data to the performance content generation unit 115. Lyric data may be generated by recognizing the voice included in the music data, and output to the performance content generation unit 115.

[0034] The music analysis unit 112 analyzes the music data supplied from the music data acquisition unit 111 and acquires the features of the music. The music analysis unit 112 functions as a feature acquisition unit that acquires the features of the music. The analysis of the music data is performed, for example, using a feature analysis model, which is an inference model for music analysis. The feature analysis model, which takes music data as input and outputs the features of the music, is prepared in the music analysis unit 112.

[0035] By using the feature analysis model, multiple types of features can be acquired, such as musical structures such as sections, intros, and choruses, beats, tempo (BPM), key signatures, and chord progressions. Other features such as the level of excitement at each time in the song and the genre of the song may also be acquired. Information indicating the song features acquired by the music analysis unit 112 is output to the control information generation unit 113. The distributor may be able to edit the song features.

[0036] The control information generation unit 113 generates control information used to control the virtual concert based on the information supplied from the music analysis unit 112. As the control information, camera control information, avatar control information, and performance control information are generated. Control information indicating the start and end times of the virtual concert is also generated. For example, a predetermined calculation is performed using the features of the music data to generate each piece of control information. A generation model that receives the features of the music data as input and outputs each piece of control information may be prepared in the control information generation unit 113, and the control information may be generated using the generation model.

[0037] 6 is a diagram showing an example of generating control information. Based on the characteristics of the music acquired by the music analysis unit 112 using the characteristic analysis model M1, the control information for the camera, the avatar, and the performance control is generated.

[0038] The camera control information is used to control the cameras used to film the virtual concert. As mentioned above, multiple virtual cameras are set up on the stage. The camera control information includes information related to camerawork, such as switching between cameras used for filming, camera movement, zooming in and out, and changing the angle.

[0039] Avatar control information is information used to control an avatar. The avatar control information includes information indicating the avatar's movements at each time (key frame), the type of costume, etc. The movements of an actual singer dancing to the music may be detected using a motion capture device, and the avatar's movements may be controlled based on the detected movement information. In this case, information on the movements of each part of the actual singer's body is generated as avatar control information.

[0040] The production control information is information used to control the production, and includes information indicating the content of control for each production, such as the stage, particles, beams, decorative lyric images, and lighting.

[0041] The stage control content specifies the type of stage, the timing of stage switching, etc. The particle control content specifies the type, movement, and appearance timing of particulate objects used in the performance. Performances using particle-shaped shining objects are also performed on the stage. The beam control content specifies the type, movement, and appearance timing of beams used in the performance.

[0042] The control of the decorative lyrics image specifies the movement and appearance timing of the decorative lyrics image, for example. The appearance timing of the decorative lyrics image specifies the timing to synchronize with the timing when the avatar sings the lyrics portion used to generate the decorative lyrics image, for example. The control of lighting specifies the type and movement of lighting used for the performance, for example.

[0043] Instead of information for controlling all of the stage, particles, beams, decorative lyric images, and lighting effects, information for controlling at least one of these effects may be generated as the effect control information. Information for controlling other effects may be included in the effect control information. In addition to type, movement, and appearance timing, other parameters such as strength (size) may also be generated as information indicating the content of the effect.

[0044] In this way, the content generator 11 automatically generates information indicating the control details of the virtual concert camera, avatars, and performance based on the music data. The distributor can prepare the information necessary to progress the virtual concert simply by preparing the music data of the songs to be played by the avatars, without having to set up camerawork, avatar movements, or performance details. The control information generated by the control information generator 113 is supplied to the controller 114 in FIG. 5.

[0045] The control unit 114 controls the progress of the virtual concert in accordance with the control information supplied from the control information generation unit 113. In the progress of the virtual concert, content recorded in the content DB 116 and production content generated by the production content generation unit 115 are used as appropriate. The content DB 116 stores various data used in the progress of the virtual concert, such as data on 3D models of the stage and avatars, and data on particles, beams, and lighting used in production.

[0046] Camera control unit 114-1 of control unit 114 controls the cameras used to film the virtual concert based on the camera control information supplied from control information generation unit 113. Camera control unit 114-1 controls the cameras to generate images for each time in the virtual concert.

[0047] The avatar control unit 114 - 2 controls the movement of the avatar based on the avatar control information supplied from the control information generation unit 113 .

[0048] The performance control unit 114-3 controls the performance during the performance based on the performance control information supplied from the control information generation unit 113. For example, the performance control unit 114-3 switches the stage and controls performances using particles, beams, and lighting. Furthermore, the performance control unit 114-3 controls the movement and appearance timing of the decorative lyric images and keyword objects generated by the performance content generation unit 115 based on the performance control information.

[0049] The video data of the virtual concert generated under the control of the control unit 114 is output to the outside of the information processing unit 101 and is included in the distribution content and distributed together with the audio data.

[0050] The performance content generation unit 115 generates performance content from the lyric data supplied from the music data acquisition unit 111. Keywords may be extracted from the lyric data supplied from the music data acquisition unit 111, and performance content may be generated based on the extracted keywords. The performance content includes decorative lyric images and keyword objects, which are objects used to produce a virtual concert. The decorative lyric images are generated in the decorative lyric image generation unit 115-1, and the keyword objects are generated in the keyword object generation unit 115-2.

[0051] 7 is a diagram showing an example of generating performance content. As shown in FIG. 7, the performance content is generated using a prompt generation model M11 (third generation model), an ornamental lyrics image generation model M12 (first generation model), and an object generation model M13 (second generation model).

[0052] The prompt generation model M11 is a generation model that receives lyric data, which is text data, as input and outputs a prompt, which is a question sentence containing keywords. A text-to-text generation model is used as the prompt generation model M11.

[0053] FIG. 8 is a diagram showing an example of a prompt.

[0054] For example, keywords expressing the theme or emotion of a song are extracted using the prompt generation model M11, and a prompt containing the extracted keywords is generated. In the example of FIG. 8 , multiple keywords such as "Radiant night," "Savage beauty," etc. are extracted. Words contained in the lyrics may be extracted as keywords, or words evoked by the lyrics may be extracted as keywords. A prompt may be generated that includes all of the keywords extracted by the prompt generation model M11, or a prompt that includes one keyword may be generated. A prompt containing at least one of the keywords is generated based on the output of the prompt generation model M11.

[0055] The prompt generated as the output of the prompt generation model M11 is used as input to the decorative lyrics image generation model M12 and the object generation model M13. The content creator may create the prompt, which is a question sentence containing keywords, from the lyrics data. Alternatively, the content creator may be able to modify the prompt generated by the prompt generation model M11.

[0056] The decorative lyrics image generation model M12 is a generation model that takes a prompt, which is a question sentence containing lyrics keywords, as input and outputs a decorative lyrics image composed of decorated characters. An image generation model such as text-to-image is used as the decorative lyrics image generation model M12. The output of the decorative lyrics image generation model M12 is the decorative lyrics image.

[0057] FIG. 9 is a diagram showing an example of generating a decorative lyrics image.

[0058] As shown in Fig. 9, image data that serves as plain data for each character of lyrics is input to the decorative lyrics image generation model M12 along with a prompt. "Happy! Bird!" shown on the left side of Fig. 9 is plain data for part of the lyrics. For example, when a prompt containing the keyword "bird" is input to the decorative lyrics image generation model M12, an image of the decorated characters "Happy! Bird!" is generated as a decorative lyrics image, as shown on the right side of Fig. 9.

[0059] In the example of FIG. 9 , the decorative lyrics image is generated using line drawing characters decorated with a bird feather pattern texture and an illustration of bird feathers. The bird feather pattern texture and illustration are generated based on the meaning of the keyword "bird" included in the prompt, which is a question sentence containing the input keyword. An image of characters using at least one of a texture and illustration of a predetermined pattern may be generated as the decorative lyrics image. Not only the texture and illustration, but also the color, size, font, etc. of the decorative lyrics image change depending on the prompt.

[0060] The decorated lyrics image generation unit 115-1 generates a decorated lyrics image using all or part of the lyrics. The decorated lyrics image generated by the decorated lyrics image generation unit 115-1 is output to the control unit 114 and used to produce the virtual concert. The decorated lyrics image may be generated in real time when the virtual concert is distributed, or may be generated in advance before the distribution begins. The previously generated decorated lyrics image is recorded in the content DB 116 together with other content.

[0061] On the other hand, the object generation model M13 is a generation model that takes a prompt, which is a question sentence containing lyrics keywords, as input and outputs a 3D model object representing the keyword. As shown in Figure 7, when a prompt containing keywords such as "bird" and "flower" is input, those 3D models are output from the object generation model M13.

[0062] In keyword object generation unit 115-2, keyword objects, which are 3D models representing keywords, are automatically generated based on each prompt. The keyword objects generated by keyword object generation unit 115-2 are output to control unit 114 and used in directing the virtual concert.

[0063] In this way, in the content generation device 11, decorative lyric images and keyword objects, which are production content used to produce a virtual concert, are automatically generated based on lyric data. Distributors can deliver lyrics to each audience member in a way that leaves a lasting impression without having to create production content themselves. Production content may be generated in real time during the distribution of a virtual concert, or may be generated in advance before the start of distribution. Data for the pre-generated production content is recorded in the content DB 116 along with other content.

[0064] <Editing the performance using a controller> The virtual concert progression modes available are an auto mode, in which the performance progresses automatically as described above without the distributor's operation, and a manual mode, in which the performance progresses through the distributor's operation.

[0065] By using the manual mode, the distributor can edit the performance of the virtual concert that is automatically performed in the auto mode. Various operations such as editing the performance and switching the progress mode are performed using the control device 13.

[0066] FIG. 10 shows how a virtual concert is operated using a controller.

[0067] 10, various settings such as the performance settings can be manually edited using a controller screen displayed on the display of the control device 13. The control device 13 may be configured as the same computer as the content generator 11, or may be configured as a computer different from the content generator 11. Editing of the virtual concert settings is performed, for example, while watching the virtual concert and observing the audience, as shown in FIG.

[0068] FIG. 11 is a diagram showing an example of the controller screen.

[0069] A GRAPH area 201 is formed in the upper left corner of the controller screen. The GRAPH area 201 is a display area for information that visually indicates the characteristics of the music.

[0070] FIG. 12 is an enlarged view of the GRAPH area 201.

[0071] The chart 211A displayed in the chart area 211 indicates changes in the excitement level over time. The excitement level is determined based on the characteristics of the song, as described above. The horizontal direction of the chart area 211 corresponds to the time direction, and the vertical direction corresponds to the level of excitement level. The vertical bars aligned in the chart area 211 together with the chart 211A represent, for example, volume. The level of other characteristics, such as beat and tempo (BPM), may also be visually presented using the display in the chart area 211.

[0072] A bar 212 below the chart area 211 indicates the progress of the song. Below the bar 212, a mode switching button 213, a play / stop button 214, and a light switching button 215 are displayed.

[0073] The mode switching button 213 is a button used to select the progress mode of the virtual concert. The broadcaster can select auto mode or manual mode by operating the mode switching button 213. By selecting manual mode, it becomes possible to manually operate the camera, production, etc.

[0074] A semi-automatic mode may be provided as a progress mode for the virtual concert. In the semi-automatic mode, some of the content that can be controlled in the manual mode is controlled manually, and other content is controlled automatically. Alternatively, some time sections may be controlled manually, and other time sections may be controlled automatically. The broadcaster may be able to set which content or which time sections are to be manually controlled.

[0075] The play / stop button 214 is a button used to play or stop a song.

[0076] The light switching button 215 is a button used to switch the type of light illuminating the avatar and the type of ambient light. For example, multiple types of lights with different colors are prepared. In the example of FIG. 12, two types of lights are prepared.

[0077] Returning to the explanation of FIG. 11 , an EFFECTS area 202 is formed in the lower left of the controller screen. The EFFECTS area 202 is a display area for buttons used to select the type of visual effect. A combination of effects such as particles, beams, and lighting is assigned to each button. By operating the buttons in the EFFECTS area 202, effects using particles, beams, lighting, etc. can be switched.

[0078] A COLOR area 203 and a STAGE area 204 are formed in the upper right corner of the controller screen. A circular image 203A displayed in the COLOR area 203 is an image used to select the overall color of the stage, such as the color of the beams and lights. The color of the stage can be selected by operating a predetermined position on the image 203A. The STAGE area 204 is a display area for buttons used to switch the type of stage.

[0079] A CAMERA area 205 is formed below the COLOR area 203 and the STAGE area 204. The CAMERA area 205 is a display area for buttons used to switch cameras.

[0080] Below the CAMERA area 205, a LYRICS area 206 is formed. The LYRICS area 206 is a display area for buttons used to edit decorated lyrics images. For example, it is possible to switch the type of decoration. When a button in the LYRICS area 206 is operated, the parameters input to the decorated lyrics image generation model M12 along with the lyrics data are switched. The type of decoration used for the decorated lyrics image output from the decorated lyrics image generation model M12 is switched depending on the input parameters.

[0081] In this way, it is possible to control not only the effects and camera control but also the design of the decorative lyric image, which is content for the effects, using the controller screen. Other parameters such as the length of the number of characters of the keyword used in the decorative lyric image and the size of the decorative lyric image may also be controlled.

[0082] FIG. 13 is a block diagram showing an example of the functional configuration of the distribution side, including the configuration of the control device 13 that displays the controller screen.

[0083] As shown in FIG. 13 , the control device 13 includes a controller display control unit 131 , a user operation detection unit 132 , and a communication control unit 133 .

[0084] The controller display control unit 131 displays a controller screen on the display of the control device 13. The controller display control unit 131 controls the display of the controller screen in response to an operation by the distributor.

[0085] The user operation detection unit 132 detects operations performed by the distributor on the controller screen and outputs information indicating the details of the distributor's operations. The information indicating the details of the distributor's operations is supplied to the controller display control unit 131 and the communication control unit 133.

[0086] The communication control unit 133 communicates with the content generator 11 and transmits the information supplied from the user operation detection unit 132 to the content generator 11 .

[0087] In addition to the information processing unit 101 described with reference to FIG. 5 , the content generator 11 also includes a communication control unit 102. The communication control unit 102 communicates with the control device 13, receives information transmitted from the control device 13, and outputs the information to the information processing unit 101. The information processing unit 101 switches the progress of the virtual concert, including the effects during the performance, between auto mode and manual mode. When manual mode is selected, the information processing unit 101 controls the progress of the virtual concert in accordance with the distributor's operations transmitted from the control device 13 and received by the communication control unit 102.

[0088] <Example of application to a real concert> The above technology for automatically generating information used to progress a virtual concert can be applied to concerts held in real concert venues. Control information used to control the production at the real concert venue is generated based on the characteristics of the songs that make up the music data. In addition, decorative lyric images to be displayed on a display installed at the real concert venue are generated based on the lyric data that makes up the music data. A concert held in a real concert venue is called a real concert.

[0089] FIG. 14 shows a scene from a real concert.

[0090] The concert venue shown in Figure 14 is located inside a real building. Figure 14 shows the stage of the concert venue as seen from the audience seating side. Person H near the center of the stage is a musician (performer).

[0091] A large display 301 is provided on the wall behind the stage. Display 301 is constructed, for example, by combining a plurality of LED display modules in a tiled pattern. Display 301 displays the words "with me," which are the decorative lyrics image described with reference to FIG. 2. The words "with me" also move and are switched to other words at predetermined times.

[0092] In this way, decorative lyric images automatically generated based on lyric data are used in the production of real concerts. When displays are provided on the sides of the stage, in addition to the wall behind the stage, the side displays are also used in the production as appropriate.

[0093] Fig. 15 is a block diagram showing an example of the configuration of the information processing unit 101 that controls the production of a real concert. Of the components shown in Fig. 15, the same components as those explained with reference to Fig. 5 are given the same reference numerals. Duplicate explanations will be omitted as appropriate.

[0094] The music data acquisition unit 111 acquires music data of songs played by musicians, outputs music data constituting the music data to the music analysis unit 112, and outputs lyric data to the performance content generation unit 115.

[0095] The music analysis unit 112 analyzes the music data supplied from the music data acquisition unit 111 and acquires the characteristics of the music. Information indicating the characteristics of the music is supplied to the control information generation unit 113.

[0096] The control information generation unit 113 generates production control information used to control the production of the real concert based on the information supplied from the music analysis unit 112. For example, the production control information includes information indicating the content of control for each production, such as beams, decorative lyric images, and lighting. For example, beam and lighting production is performed using lighting devices actually installed in the concert venue. Production control information indicating the type of image to be displayed on the display 301, switching timing, and other details may be generated. If the real concert is filmed, camera control information including information such as camerawork may be generated based on the characteristics of the music.

[0097] The decorative lyric image is displayed superimposed on a background image as appropriate. The background image may be a moving image or a still image. If multiple background images are prepared in the content DB 116, the performance control information generated by the control information generator 113 includes information indicating the type of background image to be used as the background for the decorative lyric image.

[0098] The performance control unit 114-3 of the control unit 114 controls the performance of the real concert in accordance with the performance control information supplied from the control information generation unit 113. The performance of the real concert also uses content recorded in the content DB 116 and performance content generated by the performance content generation unit 115 as appropriate. For example, the performance control unit 114-3 controls the display 301 to display an ornamental lyric image superimposed on a background image. The performance control unit 114-3 also controls the lighting device to control performance using beams and lighting.

[0099] The performance content generation unit 115 generates an ornamental lyric image and a keyword object from the lyric data supplied from the music data acquisition unit 111. Keywords may be extracted from the lyric data supplied from the music data acquisition unit 111, and an ornamental lyric image and a keyword object may be generated based on the extracted keywords. The ornamental lyric image and keyword object generated by the performance content generation unit 115 are output to the control unit 114 and used to produce a real concert. A background image for the ornamental lyric image may be generated by the performance content generation unit 115. In this case, for example, a prompt, which is a question sentence including keywords extracted from the lyrics, is input to an image generation model such as text-to-image, and the image output by the image generation model is generated as the background image. The background image generated by the performance content generation unit 115 is output to the control unit 114 together with the ornamental lyric image.

[0100] The output unit 121 controls the performance to display the performance delivery content generated by the control unit 114 on the display 301. The performance delivery content is a performance content used to display an ornamental lyric image and a background image on the display 301. The performance delivery content may be generated in real time during the real concert, or may be generated in advance before the start of the real concert.

[0101] In this way, the technology for controlling the progress of a concert based on music data can also be applied to the progress of a real concert. By displaying decorative lyric images on the display 301, the organizer of the real concert can deliver lyrics in a way that leaves a lasting impression on each audience member. In addition, the organizer of the real concert can prepare content to be used in the performance without any complicated work.

[0102] Although the performer at the real concert is assumed to be person H who is actually present on stage, the performer may be a person displayed on the display 301, or may be a virtual person represented by a 3D model or the like.

[0103] <Modifications> Although the information used to control visual effects is generated based on music data, the information used to control auditory effects may also be generated based on music data. Furthermore, instead of visual content such as decorative lyric images and keyword objects, content used for auditory effects, such as sound effect data, may also be generated based on lyric data.

[0104] Control information including performance control information may be generated based on lyric data. In this case, an inference model that uses lyric data as input and lyric features as output is prepared in the control information generation unit 113. For example, the number of characters in the lyrics, the atmosphere evoked by the lyrics, etc. are analyzed using the inference model and used to generate the control information. Control information may be generated by combining the features of the music piece and the features of the lyrics.

[0105] The effect object may be generated based on music data. In this case, a generation model that receives music data as input and outputs 3D model data as effect content is prepared in the effect content generation unit 115. The effect content may be generated by combining music data and lyric data.

[0106] Production content such as a 3D model of a stage used in producing a virtual concert and a background image used in producing a real concert may be automatically generated based on lyric data of the song used in the performance. In this case, a generation model that inputs lyric data and outputs a 3D model of a stage and a background image is prepared in the information processing unit 101. For example, if the lyrics of the song evoke a forest or nature, a 3D model of a forest stage and a background image that evokes a forest are generated, as shown in A of Fig. 3. Alternatively, if the lyrics of the song evoke the sea or water, a 3D model of an ocean stage and a background image that evokes the sea are generated, as shown in B of Fig. 3.

[0107] It is possible to generate various production contents used in the production of a concert based on lyric data and music data.

[0108] Production content may be generated not only by the distributor but also by various creators and used to produce the concert. For example, production content generated by various creators based on lyric data and music data may be provided to the distributor and used to produce the concert. In addition to the production content, various data used to progress the concert, such as 3D model data of avatars, 3D model data of the stage, data on decorative lyric images, camera control information, avatar control information, and production control information, may also be generated by the creators.

[0109] FIG. 16 is a diagram showing an example of a market for various data used in the progress of a concert.

[0110] The market management server shown in Figure 16 is, for example, a server managed by a certain business. Various data generated by creators and used in the progress of concerts is uploaded to the market management server, and compensation is provided from the market to the creators. The data uploaded to the market management server is downloaded and used by businesses and individuals who distribute virtual concerts as distribution content or hold real concerts.

[0111] Businesses and individuals who stream virtual concerts and hold real concerts using data provided by the market management server will pay the market. In this way, an ecosystem will be created where data used to stream virtual concerts and hold real concerts can be bought and sold. Editing rights for concerts can also be bought and sold.

[0112] - Computer Configuration Example The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed from a program recording medium into a computer built into dedicated hardware or a general-purpose personal computer.

[0113] 17 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. Each device that constitutes the information processing system 1 is realized by a computer having the configuration shown in FIG.

[0114] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0115] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0116] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0117] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0118] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0119] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0120] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0121] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0122] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0123] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0124] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0125] Example of configuration combinations The present technology can also be configured as follows.

[0126] (1) An information processing system comprising: a music data acquisition unit that acquires music data; a feature acquisition unit that acquires features of a song based on music data included in the music data; a generation unit that generates performance control information used to control music performance based on the features of the song; and a control unit that controls music performance based on the performance control information. (2) The information processing system described in (1), wherein the feature acquisition unit inputs the music data to an inference model and acquires the features of the song based on the output of the inference model. (3) The information processing system described in (1) or (2), further comprising a content generation unit that generates performance content expressing the lyrics of the music based on lyric data included in the music data, and the control unit controls performance using the performance content. (4) The information processing system described in (3), wherein the content generation unit generates a lyric image as the performance content from the lyric data using a first generation model. (5) The information processing system described in (4), wherein the lyric image is a decorated lyric image decorated with at least one of a predetermined texture and an illustration based on the lyric data. (6) The information processing system according to any one of (3) to (5), wherein the content generation unit uses a second generation model to generate an object as the performance content from the lyric data. (7) The information processing system according to any one of (4) to (6), wherein the content generation unit uses a third generation model to generate a question including a keyword from the lyric data, and generates the performance content expressing the keyword from the generated question including the keyword. (8) The information processing system according to any one of (4) to (7), wherein the musical performance is a performance on a stage constructed in a virtual space, and the control unit controls a performance performed by an avatar placed on the stage. (9) The information processing system according to (8), wherein the generation unit generates the performance control information including information regarding movement of the lyric image, and the control unit controls the movement of the lyric image based on the performance control information.(10) The information processing system according to (8) or (9), wherein the generation unit generates the performance control information including information indicating a type of the stage, and the control unit switches the stage type in accordance with the performance control information. (11) The information processing system according to any of (8) to (10), wherein the generation unit generates the performance control information including information regarding lighting for the stage, and the control unit controls the lighting in accordance with the performance control information. (12) The information processing system according to any of (8) to (11), wherein the generation unit further generates control information including information regarding movement of the avatar based on characteristics of the music, and the control unit controls the movement of the avatar in accordance with the control information. (13) The information processing system according to any of (8) to (12), wherein the generation unit further generates control information including information regarding camerawork of a virtual camera set on the stage based on characteristics of the music, and the control unit switches the virtual camera in accordance with the control information to generate a video of the concert in the virtual space. (14) The information processing system according to (4) or (5), wherein the musical performance is a performance at a real concert venue, and the control unit controls a performance using an image displayed on a display installed at the concert venue while a person at the concert venue is performing. (15) The information processing system according to (14), wherein the generation unit generates the performance control information including information regarding the movement of the lyric image, and the control unit controls a performance using the lyric image displayed on the display. (16) The information processing system according to any of (1) to (15), wherein the control unit edits the performance based on the performance control information in response to an operation by a distributor who manages the progress of the concert. (17) The information processing system according to (16), wherein the control unit switches between controlling the performance during the music performance based on the performance control information and controlling it in response to an operation by the distributor.(18) The information processing system according to any one of (1), (14) to (17), further comprising an output unit that outputs performance distribution content that is performance content controlled by the control unit. (19) An information processing method in which an information processing device acquires music data, acquires music characteristics based on music data included in the music data, generates performance control information used to control the music performance based on the music characteristics, and controls the music performance based on the performance control information. (20) A program that causes a computer to execute processes in which the computer acquires music data, acquires music characteristics based on music data included in the music data, generates performance control information used to control the music performance based on the music characteristics, and controls the music performance based on the performance control information.

[0127] REFERENCE SIGNS LIST 1 Information processing system, 11 Content generation device, 12 Distribution server, 13 Control device, 101 Information processing unit, 111 Music data acquisition unit, 112 Music analysis unit, 113 Control information generation unit, 114 Control unit, 115 Performance content generation unit, 116 Content DB

Claims

1. An information processing system comprising: a music data acquisition unit that acquires music data; a characteristic acquisition unit that acquires characteristics of a song based on the music data included in the music data; a generation unit that generates performance control information used to control the performance of the music based on the characteristics of the song; and a control unit that controls the performance of the music based on the performance control information.

2. The information processing system according to claim 1, wherein the feature acquisition unit inputs the music data into an inference model and acquires the features of the music based on the output of the inference model.

3. The information processing system according to claim 1, further comprising a content generation unit that generates performance content that expresses the lyrics of the music based on lyric data included in the music data, and the control unit controls the performance using the performance content.

4. The information processing system according to claim 3, wherein the content generation unit generates a lyric image as the performance content from the lyric data using a first generation model.

5. The information processing system according to claim 4, wherein the lyrics image is a decorated lyrics image decorated with at least one of a predetermined texture and an illustration based on the lyrics data.

6. The information processing system according to claim 3, wherein the content generation unit generates an object as the performance content from the lyrics data using a second generation model.

7. An information processing system as described in claim 4 or 6, wherein the content generation unit uses a third generation model to generate a question sentence containing a keyword from the lyrics data, and generates the performance content from the generated question sentence containing the keyword.

8. The information processing system according to claim 4, wherein the musical performance is a performance on a stage constructed in a virtual space, and the control unit controls the performance during a performance by an avatar placed on the stage.

9. The information processing system according to claim 8, wherein the generation unit generates the performance control information including information relating to the movement of the lyric image, and the control unit controls the movement of the lyric image based on the performance control information.

10. The information processing system according to claim 8, wherein the generation unit generates the performance control information including information indicating the type of stage, and the control unit switches the type of stage in accordance with the performance control information.

11. An information processing system as described in claim 8, wherein the generation unit generates the performance control information including information regarding lighting of the stage, and the control unit controls the lighting in accordance with the performance control information.

12. The information processing system according to claim 8, wherein the generation unit further generates control information including information relating to the movement of the avatar based on the characteristics of the music, and the control unit controls the movement of the avatar in accordance with the control information.

13. The information processing system of claim 8, wherein the generation unit further generates control information including information regarding camerawork of a virtual camera set on the stage based on characteristics of the music, and the control unit switches the virtual cameras in accordance with the control information to generate a video of the concert in the virtual space.

14. The information processing system of claim 4, wherein the musical performance is a performance at a real concert venue, and the control unit controls the performance using a display installed at the concert venue while a person at the concert venue is performing.

15. An information processing system as described in claim 14, wherein the generation unit generates the performance control information including information regarding the movement of the lyric image, and the control unit controls the performance using the lyric image displayed on the display.

16. The information processing system according to claim 1, wherein the control unit edits the performance based on the performance control information in response to operations by a distributor who manages the progress of the concert.

17. The information processing system according to claim 16, wherein the control unit switches between controlling the performance during the music performance based on the performance control information and controlling the performance in response to an operation by the distributor.

18. The information processing system according to claim 1, further comprising an output unit that outputs a performance distribution content that is a performance content controlled by the control unit.

19. An information processing method in which an information processing device acquires music data, acquires characteristics of a song based on the music data included in the music data, generates performance control information used to control the performance of the music based on the characteristics of the song, and controls the performance of the music based on the performance control information.

20. A program that causes a computer to execute the following processes: acquire music data; acquire characteristics of a song based on the music data contained in the music data; generate performance control information used to control the performance of the music based on the characteristics of the song; and control the performance of the music based on the performance control information.

Images