Information processing system, information processing method, and program

Abstract

To provide a technique having a high degree of freedom from a creative viewpoint for a user.SOLUTION: According to an aspect of the present invention, there is provided an information processing system, including at least one processor, the processor being configured to execute the following steps by reading a program: In the reception step, an input of request information by the first user is received, the request information includes voice information related to a voice desired by the first user and setting information related to a situation in which the voice is used, in the processing step, the voice information is input to at least one model to cause the model to specify or generate a predetermined voice, and in the output step, first content corresponding to the request information is output, the first content being configured to include the predetermined voice and remind the first user of the situation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] There is an audio playback technology that allows a user to select and listen to a preferred audio from audio in which lines of characters in a scenario, etc. are read aloud by multiple users. For example, Patent Document 1, which takes into consideration the above circumstances, discloses a technology that acquires multiple pieces of audio data, arranges each of the multiple pieces of audio data in a space defined by one or more axes, and plays back each of the multiple pieces of audio data arranged in the space in a playback mode that corresponds to the respective positions of the one or more pieces of audio data displayed on a screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-143296 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional audio playback technologies are based on predefined scenarios and audio, and lack creative freedom.

[0005] In view of the above circumstances, the present invention provides a technique that gives users a high degree of freedom in terms of creativity. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an information processing system comprising at least one processor configured to execute the following steps by reading a program: a reception step of receiving input of request information from a first user, the request information including voice information relating to a voice desired by the first user and setting information relating to a situation in which the voice will be used; a processing step of inputting the voice information into at least one model to cause the model to identify or generate a predetermined voice; and an output step of outputting first content according to the request information, the first content including the predetermined voice and configured to evoke the situation.

[0007] According to the present disclosure, users can have greater freedom in terms of creativity. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a configuration diagram illustrating an information processing system 1. FIG. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of the server device 2. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of a user terminal 3. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the server device 2. [Figure 5] 2 is a flowchart showing an outline of processing executed by the information processing system 1. [Figure 6] 2 is an activity diagram showing a specific example of processing executed by the information processing system 1. FIG. [Figure 7] 10 is an example of a screen 4 provided by this service. [Figure 8] 10 is an example of a screen 4 provided by this service. [Figure 9] This is an example of a music video MV generated based on the request information IF0 shown in FIG. [Figure 10]This is a database 5 that manages at least the user accounts and the generated content in association with each other. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0010] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or may be provided so that it can be downloaded from an external server, or may be provided so that the program is started on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, various information processing according to an embodiment may realize input and output corresponding to the input. Here, the form of information referenced in such information processing (hereinafter referred to as reference information) is not limited as long as an output is obtained as a result of the input. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression formula constructed using a statistical method), a trained model that has previously trained the correlation between input and output, or a large-scale language model that can output a desired result by inputting a prompt.

[0012] In one embodiment, a "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In one embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit set consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculations can be performed on a circuit in the broad sense.

[0013] Furthermore, a circuit in the broad sense is a circuit realized by at least an appropriate combination of a circuit, circuitry, processor, memory, etc. The processor may be a general-purpose processor or a dedicated circuit. That is, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] [Embodiment] 1. Hardware Configuration In this section, a hardware configuration according to one embodiment will be described.

[0015] 1.1 Information Processing System 1 FIG. 1 is a configuration diagram showing an information processing system 1. The information processing system 1 includes a server device 2 and a user terminal 3. The server device 2 and the user terminal 3 are configured to be able to communicate with each other via a telecommunications line. Here, a system exemplified as the information processing system 1 is made up of one or more devices or components. Therefore, even the server device 2 or the user terminal 3 alone is an example of a system. More specifically, the information processing system 1 may include an element selected from the group consisting of the server device 2 and the user terminal 3. Furthermore, multiple server devices 2 or user terminals 3 may be used. The unselected elements may not be included in the information processing system 1, but may be electrically connected to the selected elements as external elements.

[0016] 1.2 Server device 2 2 is a block diagram showing the hardware configuration of the server device 2. The server device 2 includes a communication bus 20, a communication unit 21, a storage unit 22, and a processor 23. The communication unit 21, the storage unit 22, and the processor 23 are electrically connected via the communication bus 20 inside the server device 2.

[0017] The communication unit 21 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, BLUETOOTH (registered trademark) communication, etc. as necessary. In other words, it is more preferable to implement it as a collection of multiple communication means. In other words, the server device 2 may communicate various information from the outside via the communication unit 21 and the network 11.

[0018] The storage unit 22 stores various pieces of information defined above. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the server device 2 executed by the processor 23, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program operations. The storage unit 22 stores various programs, variables, etc. related to the server device 2 executed by the processor 23.

[0019] The processor 23 processes and controls the overall operations related to the server device 2. The processor 23 is, for example, a central processing unit (CPU). The processor 23 realizes various functions related to the server device 2 by reading out predetermined programs stored in the storage unit 22. In other words, information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. Note that the processor 23 is not limited to being single, and multiple processors 23 may be provided for each function. A combination of these may also be used.

[0020] 1.3 User terminal 3 3 is a block diagram showing the hardware configuration of the user terminal 3. The user terminal 3 includes a communication bus 30, a communication unit 31, a storage unit 32, a processor 33, a display unit 34, and an input unit 35. The communication unit 31, the storage unit 32, the processor 33, the display unit 34, and the input unit 35 are electrically connected via the communication bus 30 inside the user terminal 3. The description of the communication unit 31, the storage unit 32, and the processor 33 is the same as the description of each unit in the server device 2, and therefore will be omitted.

[0021] The display unit 34 displays a screen of a graphical user interface (GUI) that can be operated by the user. The display unit 34 may be included in the housing of the user terminal 3 or may be attached externally. Specifically, the display unit 34 may be implemented as a display device such as a CRT display, a liquid crystal display, an organic EL display, or a plasma display. It is preferable that these display devices are implemented by selectively using them depending on the type of the user terminal 3.

[0022] The input unit 35 accepts operation inputs made by the user. The operation inputs are transferred as command signals to the processor 33 via the communication bus 30. The processor 33 can execute predetermined control or calculations based on the transferred command signals as necessary. The input unit 35 may be included in the housing of the user terminal 3 or may be externally attached. For example, the input unit 35 may be implemented as a touch panel integrated with the display unit 34. When the input unit 35 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 35. Instead of a touch panel, a switch button, a mouse, a QWERTY keyboard, etc. can be used as the input unit 35.

[0023] 1, the user terminal 3 may be a general-purpose computer 3a, a mobile terminal 3b, or an electronic musical instrument 3c connectable to a network 11. If the user terminal 3 is an electronic musical instrument 3c, the input unit 35 is an input interface for playing, and sound data may be obtained by the user playing the electronic musical instrument 3c. With this configuration, the user can input request information IF0 (described below) in the form of playing, thereby improving usability when creating desired content (described in detail below).

[0024] Furthermore, a person using the server device 2 may provide a service (described in Section 3 onward) described below by distributing an electronic musical instrument 3c to users. In such a case, the information processing system 1 includes the server device 2 and an electronic musical instrument 3c (an example of a musical instrument). The server device 2 includes a processor 23. Sound data obtained from the electronic musical instrument 3c may be transmitted to the server device 2 via the network 11. Such a configuration allows the information processing system 1 to be realized in a more specific manner. However, such a configuration is not necessarily required, and the information processing system 1 may be provided in other manners.

[0025] 2. Functional configuration Next, each functional configuration of the information processing system 1 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the server device 2. As shown in Fig. 4, the processor 23 executes various programs stored in the storage unit 22, thereby functioning as a reception unit 231, a processing unit 232, an output unit 233, an association unit 234, a calculation unit 235, and a display control unit 236. In other words, information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23.

[0026] The receiving unit 231 is configured to receive various pieces of information as a receiving step. Specifically, the receiving unit 231 is configured to receive information via the communication unit 21 or the storage unit 22 and to be able to read this information into a working memory.

[0027] The processing unit 232 is configured to execute a process of inputting various information into the model MD as a processing step. The model MD here refers to reference information that is not rule-based. That is, the model MD may be a trained model that has been machine-learned to output results according to the input content. This configuration allows the model MD to identify or generate information based on past statistical data, thereby providing content containing such information to the user. Of course, the model MD may also be a large-scale language model that can output desired results by entering a prompt. Large-scale language models are pre-trained using large amounts of text data, etc., and the training data used for training includes text obtained from various sources, such as books, articles, and web pages. This generally allows large-scale language models to understand a wide range of contexts and achieve performance capable of handling a variety of tasks. Furthermore, large-scale language models use generation algorithms such as beam search and top-K sampling to generate text information. This enables the output of more natural and consistent text information.

[0028] In one embodiment, the model MD may be stored in any storage medium, and may be stored, for example, in the storage unit 22 of the server device 2 in a local environment, in the storage unit 32 of the user terminal 3 via the network 11, or in another storage medium (not shown). In particular, the model MD may be provided as an external service to a person who uses the server device 2 (a person who provides a service).

[0029] The output unit 233 is configured to output various information as an output step. It does not matter in which storage medium the output information is stored.

[0030] The associating unit 234 is configured to associate one piece of information with another piece of information as an associating step. It does not matter in which storage medium the associated information is stored.

[0031] The calculation unit 235 is configured to execute, as calculation steps, various calculations related to the server device 2. There is no particular restriction on the type of calculation.

[0032] As a display control step, the display control unit 236 executes processing for displaying various pieces of information on a display medium in a manner recognizable to a user. When the phrase "display" is used, it does not matter whether the display medium to be displayed is in a local environment or whether the processing for displaying is performed via the network 11. As a result of processing by the display control unit 236, various pieces of information are presented to various users operating, for example, the user terminal 3 via the display unit 34. The presented various pieces of information are visual information such as screens, images, icons, and messages. The display control unit 236 may generate the visual information itself, or may generate only rendering information for displaying the visual information on the display unit 34.

[0033] 3. Information Processing Method This section describes an information processing method executed by the information processing system 1. Note that the order of processes included in the information processing method can be changed as appropriate, multiple processes can be executed simultaneously, or some processes can be omitted.

[0034] 3.1 Overview As described above, the information processing system 1 includes at least one processor 23, and the processor 23 functions as each of the following units by reading a program. In other words, the information processing method includes steps corresponding to each unit of the information processing system 1. From another perspective, the program causes a computer to execute steps corresponding to each unit of the information processing system 1. FIG. 5 is a flowchart showing an outline of the processing executed by the information processing system 1. Each step shown in FIG. 5 will be described below.

[0035] First, the receiving unit 231 receives input of request information IF0 by a first user (step S001). The request information IF0 includes voice information IF1 related to a voice desired by the first user and setting information IF2 related to a situation in which the voice will be used. Next, the processing unit 232 inputs the voice information IF1 into at least one model MD, thereby identifying or generating a predetermined voice in the model MD (step S002). Then, the output unit 233 outputs first content according to the request information IF0 (step S003). The first content is configured to include the predetermined voice and evoke a situation.

[0036] This configuration allows efficient provision of content specifically tailored to each user's situation. Furthermore, by enabling the integrated execution of everything from the general design of content tailored to the situation to the detailed design of whose voice or what kind of voice should be used in that content, which has been done in the past, the overall processing load on the computer can be reduced.

[0037] 3.2 Specific examples Specific examples may fall within the scope specified in the above-mentioned overview. In the specific examples, the following situation is assumed: First, an administrator managing a server device 2 provides users with a service (hereinafter referred to as the present service) via the Internet, which allows users to create their desired original music video MVs. The website providing the service is referred to as a specific site. In other words, the present service is a type of SaaS using the specific site, and the administrator is the party providing the service. Various end users using user terminals 3 utilize the service. With the present service, users access the specific site and specify desired information, and a music video MV is generated based on the specified information. Users can download the generated music video MV to their own user terminals 3. With the present service, at least one model MD is used to generate a music video MV; specifically, a first model MD1 and a second model MD2 are used. For example, the first model MD1 is a trained model, and the second model MD2 is a large-scale language model.

[0038] 6 is an activity diagram showing a specific example of processing executed by the information processing system 1. Below, the flow up to when a certain user of this service (hereinafter referred to as the first user) acquires an original music video MV via this service will be described with reference to FIG.

[0039] First, the processor 33 of the user terminal 3 held by the first user accesses a specific site provided by the server device 2 via the network 11 (activity A101).

[0040] Next, within the specific site, user account information corresponding to the first user is input via the input unit 35 of the user terminal 3 (activity A102), and authentication processing is executed, whereby a screen 4 (see FIG. 7 described below) on which various information (request information IF0 described below) can be specified is displayed on the display unit 34 (activity A103). The user account may be registered in advance, or may be newly registered by the first user immediately before activity A102. The user account may be an account having information registered for this service, or may be an account associated with a user account for an existing Web service. Note that this service may be provided to everyone without requiring account registration.

[0041] Next, various pieces of information desired by the first user are input via the input unit 35. Specifically, at least voice information IF1 related to a voice desired by the first user and setting information IF2 related to a situation in which the voice will be used are specified (activities A104 and A105), and preferably, music information IF3 related to a piece of music desired by the first user is further specified (activity A106). Thereafter, the voice information IF1, setting information IF2, and music information IF3 are transmitted to the server device 2 via the network 11, and the receiving unit 231 in the server device 2 receives the voice information IF1, setting information IF2, and music information IF3 as request information IF0 (activity A107). In other words, in the specific example, the request information IF0 further includes music information IF3 related to a piece of music desired by the first user.

[0042] Next, the processing unit 232 of the server device 2 inputs the voice information IF1 included in the request information IF0 into the first model MD1 (activity A108). The first model MD1 is a trained model that has been trained in advance to output voice data corresponding to various voice characteristics when they are input. The voice data may be voice data of a real person such as a singer or voice actor, or may be synthesized voice data. Voice data may be prepared in advance, each with an identification label, and managed in the storage unit 22 of the server device 2. In such a case, the first model MD1 identifies one of the prepared voice data. On the other hand, the first model MD1 may be configured to generate new voice data by synthesizing existing voice data according to the input characteristics. The input of various voice characteristics may be input such as tone, tone of voice, average pitch, or human impression, or may be input directly such as a person's name or an identification label. The input information may be a single piece of information or multiple different pieces of information. Furthermore, the first model MD1 may be composed of two or more models. In such a case, a voice draft data may be generated by one model (e.g., a large-scale language model), and then the draft data may be input to another model to identify or generate a final predetermined voice. In other words, the processing unit 232 inputs the voice information IF1 to a first model MD1 (at least one model MD), causing the first model MD1 to identify or generate a predetermined voice (activity A109). Preferably, the first model MD1 may be a voice conversion model capable of outputting the singing voice of a desired specific person, including the characteristics of the singing voice and singing habits such as changes in timbre depending on the pitch of the sound.

[0043] Next, the processing unit 232 of the server device 2 inputs the setting information IF2 and music information IF3 included in the request information IF0 into the second model MD2 (activity A110). The second model MD2 is a large-scale language model and may be provided by an external server (not shown). In other words, an API for the second model MD2 may be provided, and the second model MD2 may be used via this service. The first user can input the setting information IF2 and music information IF3 as prompts in natural languages ​​such as Japanese and English. The setting information IF2 is information about a situation in which the music video MV will be used or a situation that the user is reminded of when watching the music video MV. The prompt preferably includes content describing the name and situation of a ceremonial occasion. Furthermore, the music information IF3 is information about the details of the song. The prompt preferably includes content describing the lyrics, composition, instruments, and the intended composer or lyricist.

[0044] In response to the input of the setting information IF2 and the music information IF3, music (including lyrics) and images (moving or still images) to be used in the music video MV are identified or generated (activity A111). When an image is generated, only the setting information IF2 may contribute, or the setting information IF2 and the music information IF3 may each contribute in a weighted manner. Similarly, when a song is generated, only the music information IF3 may contribute, or the setting information IF2 and the music information IF3 may each contribute in a weighted manner. In other words, the processing unit 232 inputs the music information IF3 to the second model MD2 (at least one model MD) to cause the model MD to generate a predetermined song. Furthermore, the processing unit 232 inputs at least a portion of the request information IF0 to the second model MD2 (at least one model MD) to cause the model MD to identify or generate a predetermined image.

[0045] Next, the calculation unit 235 performs a calculation to generate a music video MV by combining the predetermined voice, predetermined image, and predetermined music piece identified or generated in activities A109 and A111. The music video MV is preferably generated based on some predetermined reference information. The music video MV (first content) is configured to include the predetermined voice and evoke a specified situation. The output unit 233 then outputs the generated original music video MV (activity A112).

[0046] More specifically, the output unit 233 outputs first content including a predetermined image, and the first content is a music video MV in which a predetermined image is displayed while a predetermined song is played. With this configuration, a user can generate a desired music video MV including a song suitable for a situation, for example, through AI technology. Furthermore, the output unit 233 outputs a music video MV (first content) including a predetermined song, and the predetermined song uses a predetermined voice as vocals at least in part. With this configuration, a user can use AI technology to identify or generate a song suitable for a situation, for example, and have the desired vocalist sing the song, thereby including it in the music video MV, which is the content.

[0047] The generated music video MV is managed in association with the user account of the first user who was involved in the generation (activity A113). That is, information about the generated music video MV may be added to information about the first user in the database stored in storage unit 22. According to this embodiment, the first user can review the music video MVs they have generated after the fact via this service.

[0048] The generated music video MV may then be downloaded to the user terminal 3 via the network 11. In other words, the user terminal 3 may be configured to be able to acquire the music video MV via this service (activity A114). The downloaded music video MV is stored in the storage unit 32 of the user terminal 3. The processor 33 of the user terminal 3 reads out the music video MV stored in the storage unit 32, and the music video MV is played. The first user can view the played music video MV via the display unit 34.

[0049] Furthermore, although not shown in the above-mentioned request information IF0, in this service, it is preferable that a music video MV created in the past can be specified as the request information IF0. In such a case, in activity A107, the receiving unit 231 may receive input specifying second content as request information IF0, and the second content may be content (e.g., a music video MV created in the past) that was output based on request information IF0 input in the past by a second user. Note that the second user is a user different from the first user. In such a case, in activity A112, the output unit 233 outputs first content that makes secondary use of the second content. With this configuration, it is possible to use content that has been output by others, thereby improving usability in creating desired content.

[0050] When such secondary use is performed, information about the secondary use is managed in this service. In this case, it is preferable that the flag indicating the secondary use is changed from off to on in response to the reception unit 231 receiving an input specifying the second content in activity A107. That is, the calculation unit 235 executes a calculation to determine whether or not a music video MV newly output in activity A112 has been generated by secondary use of a music video MV previously generated by another person (activity A115). Specifically, the calculation unit 235 determines whether the flag indicating secondary use is on or off, and if the flag is on (if secondary use has occurred), the information about the secondary use in the database stored in the storage unit 22 is updated (activity A116).

[0051] The above is the flow of processing according to the specific example. Note that the assumptions and flow of processing described above are merely examples and are not limited to these.

[0052] 4. Related technical matters The following describes in detail technical matters related to the above-mentioned information processing method.

[0053] 4.1 Screen examples of this service 7 and 8 are examples of screen 4 provided by this service. Screen 4 is a screen displayed on the display unit 34 of the user terminal 3 during the processing of activity A103 shown in FIG. 6. Screen 4 has areas 41 to 45. Screen 4 is associated with area 42 and has a drop-down 42a and a button 42b. Screen 4 is associated with area 43 and has an input field 43a. Screen 4 is associated with area 44 and has an input field 44a. Screen 4 is associated with area 45 and has a button 45a. Screen 4 also has a button 46.

[0054] Area 41 is an area for allowing the user to visually recognize their own user account, and here, text information "User ID: __okkun" is depicted. That is, the user account in this example is "__okkun."

[0055] Area 42 is an area for prompting the user to input voice information IF1, and in this case, the text information "vocal" is drawn. Furthermore, to the right of area 42, a drop-down menu 42a is arranged, allowing the user to select a predetermined item. Preferably, drop-down menu 42a is a GUI component that also functions as an input field. Drop-down menu 42a may be configured to present a list of items preset by the user's selection, or may be configured to accept input of arbitrary text information by the user. Items included in drop-down menu 42a may include, for example, the singer's name or the aforementioned identification label. When inputting arbitrary text information by the user, the singer's name or identification label may be input, but various information that may be an example of voice information IF1, such as that described in paragraph

[0042] , may also be input.

[0056] For example, as shown in FIG. 8, "Machida Kao" is selected as an item in the drop-down menu 42a. The items in the drop-down menu 42a may be arranged in alphabetical order (the standard Japanese alphabet) or alphabetical order. Furthermore, the screen 4 includes a button 42b that allows the user to upload sample voice data instead of or in addition to specifying the voice information IF1 via the drop-down menu 42a. When the user presses the button 42b (including interactions such as clicking or tapping), a pop-up screen (not shown) may be displayed, allowing the user to select, for example, a directory in the local environment or a URL on the Internet, or to specify data by drag and drop.

[0057] Area 43 is an area for prompting the user to input music information IF3, and in this case, text information "music information" is drawn. Furthermore, below that, an input field 43a is arranged, and input field 43a is configured to accept input of any text information by the user.

[0058] 8, the input field 43a has input therein a prompt written in natural language, such as "I would like a ballad-style song of about three minutes, with a piano solo part. As the vocalist is Machida Kao as set above, I would like the lyrics to be something that Machida himself would write. I would like the lyrics for the chorus part, in which Machida sings with her clear, high notes, to include keywords such as 'thank you' and 'congratulations.'" Such a prompt includes information such as the song structure, instruments, lyrics, and lyricist as music information IF3, and may include content that overlaps with voice information IF1, such as "Machida Kao."

[0059] Area 44 is an area for prompting the user to input setting information IF2, and in this case, the text information "Situation" is drawn. Furthermore, below that, an input field 44a is arranged, and input field 44a is configured to receive any text information input by the user.

[0060] For example, as shown in FIG. 8 , a prompt written in natural language is entered in input field 44a: "I want to create a music video to be shown at a junior high school graduation ceremony. I would like to use comic-style illustrations that evoke school life and the period immediately before graduation. I will upload images of actual students separately as materials, so please use those as a reference." This prompt includes information describing situations such as a junior high school graduation ceremony and school life as setting information IF2. Although not shown, the prompt may also include voice information IF1 or music information IF3. Furthermore, setting information IF2 may include more detailed information, such as the characteristics and appeal of the characters appearing in the music video, the theme to be conveyed through the music in the music video, and the intent and background of the lyrics of the music.

[0061] Area 45 prompts the user to upload material data such as images and audio. Here, the text "Specify Material" is displayed. Below area 45, a button 45a is located, allowing the user to upload sample material. When the user presses button 45a, a pop-up screen (not shown) may be displayed, allowing the user to select, for example, a local directory or an internet URL, or to specify data by drag and drop. The uploaded material may be used directly or indirectly as at least a portion of the music video MV to be subsequently output. For example, if the material is an image, the image may be included in the music video MV. If the material is an existing song (including vocals), at least a portion of the song excluding the vocals may be included in the music video MV. In the above example, actual photos of graduating students may be uploaded to serve as a reference for creating comic-style images. In other words, the receiving unit 231 receives input specified by at least one of text, audio data, image data, and a URL as request information IF0. According to this configuration, the user can input the request information IF0 in various forms, thereby improving usability when creating desired content.

[0062] The input specified as the request information IF0 may be weighted, such as by priority. For example, weighting may be appropriately performed by giving a higher weight to sound data and a lower weight to text. Furthermore, if the sound data is data generated by a user playing the electronic musical instrument 3c, the priority of such data may be increased.

[0063] The button 46 is a GUI component for executing a process of inputting the voice information IF1, setting information IF2, and music information IF3 input via the screen 4 into at least one model MD. To avoid making the user aware of the model MD as a service, the text information "Make a music video" is simply displayed on the button 46. Of course, the button 46 may also display information that makes the user aware of the input to the model MD. In either case, pressing the button 46 inputs the voice information IF1 into the first model MD1, and the setting information IF2 and music information IF3 into the second model MD2. A predetermined voice based on the voice information IF1 and a predetermined image and a predetermined music piece based on the setting information IF2 and / or music information IF3 are then identified or generated, and an original music video MV desired by the user is output.

[0064] In other words, the music information IF3 may include information about a phrase in a song desired by the first user, and the processing unit 232 causes the model MD to generate a predetermined song that includes the phrase in its lyrics. The information about the phrase in the song may be a line, a memorable place, or a motto, or information may be specified as to where in the song the specified phrase is to be inserted. With this configuration, it is possible to have the AI ​​generate a song based on more specific information and include it in the content.

[0065] Furthermore, the music information IF3 may include information regarding the music structure of the music desired by the first user, and the processing unit 232 causes the model MD to generate a predetermined music piece having the music structure. The music structure may be the structure itself, such as an A-melody, a B-melody, and a chorus, or it may be the length of the music piece or the melody of the music piece. With this configuration, it is possible to cause the AI ​​to generate a music piece based on more specific information and include it in the content.

[0066] The music information IF3 may also include information about the instruments used in the music desired by the first user, and the processing unit 232 causes the model MD to generate a predetermined music piece that includes at least the instrument parts. In the above example, a piano is specified as the instrument, but multiple instruments may be specified, and the second model MD2 may select an instrument that was not specified. With this configuration, the AI ​​can generate music based on more specific information and include it in the content.

[0067] Furthermore, the music information IF3 may include information about the composer or lyricist of the music desired by the first user, and the processing unit 232 causes the model MD to generate a predetermined music piece that evokes the involvement of the composer or lyricist. The composer or lyricist may be a historical figure or a currently active celebrity. With this configuration, the AI ​​can generate music based on more specific information and include it in the content.

[0068] 4.2 Music video example 9 is an example of a music video MV generated based on the request information IF0 shown in FIG. 8. As shown in the figure, the music video MV uses images depicting school life in a comic style based on the setting information IF2 described above, and the lyrics LY include the phrases "congratulations" and "thank you" based on the music information IF3 described above. Also, although not shown, the vocals in the music video MV are assigned to the voice of "Machida Kao" based on the voice information IF1 described above, and a piano solo part is preferably included based on the music information IF3. In other words, content is output that includes a specific voice and is configured to evoke a specified situation.

[0069] Of course, the "graduation" featured in the music video shown in Figure 9 is just one example, and in particular, important events in life stages such as entrance ceremonies, company entrance ceremonies, weddings, funerals, the birth of children or grandchildren, travel memories, etc. can be specified as setting information IF2.

[0070] 4.3 Database Example FIG. 10 shows a database 5 that manages at least an association between user accounts and generated content. More specifically, the database 5 stores at least the following items: user account, content ID, source of use, and royalty-related information. A "user account" identifies an individual user. A "content ID" identifies an individual music video that has actually been generated.

[0071] In particular, in the database 5, the second content, which is content of another person, is managed in association with the second user who created it. The associating unit 234 then associates information regarding benefits with the second user when first content that secondary uses the second content is output or has been created. This configuration provides some kind of advantage to the creator who created the content through the information processing system 1, thereby promoting active use of the information processing system 1. The benefits are not limited to money (so-called royalties), but may also be discounts or coupons on the service fees, points or mileage points that are commonly used in society and are equivalent to currency, etc. Below, the simplest example, royalty, as shown in FIG. 10, will be specifically described.

[0072] "Source of use" indicates another music video MV that was secondary reused when generating the music video MV with the corresponding content ID. For example, the music video MV with content ID MV_B2 by user account hamamat0223 secondary reused the music video MV with content ID MV_A3 by user account __okkun when it was generated. Similarly, the music video MV with content ID MV_C1 by user account innovation_road secondary reused the music video MV with content ID MV_B2 by user account hamamat0223 when it was generated.

[0073] In "Royalty Related," "Target Content" indicates another piece of content in which the content with the corresponding content ID has been secondary-used, "Target Account" indicates the user account associated with that other piece of content, and "Amount" indicates the amount of royalties generated by the secondary use. For example, as explained above in "Source of Use," a music video MV with content ID MV_A3 by user account __okkun was secondary-used when a music video MV with content ID MV_B2 by user account hamamat0223 was created. Therefore, the music video MV with content ID MV_A3 is associated with the music video MV with content ID MV_B2 as the target content, with hamamat0223 as the target account, and with the amount of 500 yen.

[0074] Furthermore, the music video MV with content ID MV_B2 by user account hamamat0223 was used secondary when the music video MV with content ID MV_C1 by user account innovation_road was created, and therefore the music video MV with content ID MV_A3 was used tertiary. Database 5 associates information about secondary and tertiary use, respectively; for example, the music video MVs with content IDs MV_A3 and MV_B2 are both associated with the music video MV with content ID MV_C1 as the target content, innovation_road as the target account, and the amount of 250 yen. Note that the amount is an example; for example, the amounts for secondary and tertiary users may be different.

[0075] In addition, financial information such as a bank account, credit card, or virtual currency wallet may be stored in the user account, and in such a case, the calculation unit 235 of the server device 2 may execute royalty payment processing immediately or periodically.

[0076] Furthermore, the associating unit 234 may associate information regarding payment of a secondary use fee with the first user, in contrast to the configuration of the database 5. With such a configuration, when a user uses content that has been output by another person, it becomes possible to collect a secondary use fee from the user and, for example, return a portion of that fee to the creator of the original content.

[0077] When generating the first content by secondary use of the second content, the dialogue history of the second model MD2 when the second content was generated (the history of input prompts and information output as responses) may be referenced. In this case, in consideration of privacy, it is preferable that the first user who performs secondary use cannot refer to the actual dialogue history itself. By adopting a configuration in which the dialogue history is referenced, the accuracy of secondary use can be further improved.

[0078] 4.4 Setting information IF2 The setting information IF2 is not limited to the one exemplified in the embodiment, and various other setting information IF2 may be introduced. An example of the setting information IF2 will be described below.

[0079] (Use of information associated with user accounts) The setting information IF2 may include information associated with the user account (hereinafter referred to as related information). When using this service, various information is associated with the user account. For example, the user's address, age, gender (or gender identity), areas of interest, etc. may be associated, or an account for an existing web service (e.g., an e-commerce site, SNS, etc.) may be associated. Furthermore, for example, if areas of interest include golf, playing musical instruments, creating DTM, etc., a music video evokes these areas may be generated. For example, if the related information includes a purchase history on an e-commerce site, a music video related to the purchased products may be generated.

[0080] (Use of location information) The setting information IF2 may include user location information. The user location information may be acquired by a GPS function of a mobile terminal 3b, which is an example of a user terminal 3, or by a GPS function installed in a car, motorcycle, or the like used by the user. For example, if the location information identifies that the user is at a tourist spot, a music video MV related to the tourist spot may be generated. Furthermore, for example, by comparing the user's address included in the related information with the location information, if the address is more than a predetermined distance (e.g., 100 km) away, the user may be identified as being on a trip. In such a case, even if the location information does not identify a tourist spot (e.g., on an ordinary road in a specific city, town, or village), a music video MV related to the city, town, or village may be generated. Furthermore, a music video MV may be generated based not only on the location information itself but also on changes in the location information (travel route). For example, if a trip from Tokyo to Hamamatsu by car is identified, the music video MV may include information related to places related to the trip between Tokyo and Hamamatsu, such as Yokohama, Ebina, Gotemba, Fuji, Shizuoka, etc.

[0081] [others] The information processing system 1 according to the above embodiment may be configured as follows.

[0082] At least one of the devices included in the information processing system 1 may be installed outside Japan. For example, the server device 2 or a server may be installed outside Japan, and the user terminal 3 may be installed inside Japan. Similarly, a user may access the server device 2 installed inside Japan from outside Japan using their own user terminal 3. According to such an embodiment, a more convenient experience can be provided to the user through various management modes.

[0083] In one embodiment, the reception unit 231, the processing unit 232, the output unit 233, the association unit 234, the calculation unit 235, and the display control unit 236 are described as functional units implemented by the processor 23 of the server device 2, but at least some of them may be implemented as functional units implemented by another server. Alternatively, they may be implemented as functional units implemented by the processor 33 of the user terminal 3. Furthermore, the various pieces of information described in the above example may be stored not only in the storage unit 22 of the server device 2, but also in a distributed manner in other external devices using blockchain technology or the like.

[0084] The model MD is an example, and all request information IF0 may be input to a common model MD, which is a large-scale language model. Conversely, even in a large-scale language model, a separate model MD may be used for each input target. In addition, although the example described above describes a mode in which a prompt is freely entered for a large-scale language model, it is also possible to have the user input only keywords, and have the calculation unit 235 of the server device 2 generate a prompt through back-end processing based on the input keywords and predetermined reference information. It is also possible to use only trained models without using a large-scale language model.

[0085] In one embodiment, the content is described as a music video, but content that does not include a specific song and instead focuses on a reading may also be included. That is, the service does not require the user to input song information IF3, and the service does not require the user to input song information IF3. Even in such a case, the request information IF0 may include at least voice information IF1 and setting information IF2. In other words, the processing unit 232 inputs voice information IF1 into at least one model MD to identify or generate a specific voice in the model MD, and inputs setting information IF2 into at least one model MD to identify or generate specific material appropriate for the situation in the model MD. The output unit 233 outputs a first content including the specific material and the specific voice. This configuration allows the user to determine a desired situation using AI technology, generate material appropriate for the situation, and include the generated material in the content. The content is not limited to the music video MV described above, but may also be a slideshow of images and a reading content consisting of a predetermined voice.

[0086] Furthermore, content of the present service may be presented to the user without user operation. For example, content can be generated by referring to the related information described above without the user having to enter setting information IF2. Therefore, the generated content may be presented to the user in a recommended format while browsing a specific site of the present service. For example, if an interest in golf is identified as related information, a golf-related music video may be recommended while the user is browsing the specific site. In particular, if the present service is included in a more general-purpose portal site, music videos may be presented on the portal site to guide users to the present service.

[0087] Furthermore, a mode in which the request information IF0 does not include information about the voice may be adopted. Even in such a case, the request information IF0 may include setting information IF2 capable of evoking a specific situation. This allows the user to be presented with first content that includes an arbitrary voice or does not include a voice itself and that evokes the situation. In other words, the receiving unit 231 receives input of setting information IF2 related to a desired situation from the first user. Next, the processing unit 232 inputs the setting information IF2 into at least one model MD, causing the model MD to identify or generate material suitable for the situation. Then, the output unit 233 outputs the first content according to the setting information IF2. The first content is configured to evoke the situation.

[0088] In one embodiment, the request information IF0 can be input in various formats, but when the electronic musical instrument 3c is used, the performance information may be transmitted to the server device 2 in real time (here, this means real time in a broad sense that allows for communication delays and a certain amount of buffering) and used as the request information IF0. Specifically, the information may be transmitted in real time directly from the electronic musical instrument 3c via the network 11, or the electronic musical instrument 3c may be connected to a general-purpose computer 3a, and data output via a DAW installed on the computer 3a may be transmitted in real time via the network 11. Such real-time information may be given a higher priority as the request information IF0.

[0089] In one embodiment, the user terminal 3 is exemplified by a general-purpose computer 3a, a mobile terminal 3b, and an electronic musical instrument 3c, but is not limited thereto. The content to be presented, not limited to a music video, may be automatically determined depending on the device (user terminal 3) included in the information processing system 1. Examples of devices that present content other than a music video include a teleconferencing system, an AV amplifier, and a home theater system. When the information processing system 1 includes a teleconferencing system, the output unit 233 may output a video summarizing a conference held via the teleconferencing system (hereinafter referred to as a video summary). In other words, the processing unit 232 inputs a video of the conference as setting information IF2 to the second model MD2, thereby causing the second model MD2 to identify or generate material appropriate for the situation of summarizing the conference. Then, a video summary including the identified or generated material is output as the first content. Furthermore, it is preferable that the video summary be subjected to audio processing, such as emphasizing the speaker's voice. If the information processing system 1 includes an AV amplifier or a home theater system, an original summary video of the viewed movie may be generated.

[0090] Furthermore, it may be provided in the following aspects.

[0091] (1) An information processing system comprising at least one processor, the processor configured to execute each of the following steps by reading a program: a receiving step receiving input of request information from a first user, the request information including voice information relating to a voice desired by the first user and setting information relating to a situation in which the voice will be used; a processing step inputting the voice information into at least one model to cause the model to identify or generate a predetermined voice; and an output step outputting first content according to the request information, the first content including the predetermined voice and configured to evoke the situation.

[0092] This configuration allows efficient provision of content specifically tailored to each user's situation. Furthermore, by enabling the integrated execution of everything from the general design of content tailored to the situation to the detailed design of whose voice or what kind of voice should be used in that content, which has been done in the past, the overall processing load on the computer can be reduced.

[0093] (2) In the information processing system described in (1) above, in the processing step, the setting information is input into at least one model, causing the model to identify or generate material that suits the situation, and in the output step, the first content composed of the material and the specified voice is output.

[0094] With this configuration, the situation desired by the user can be determined using AI technology, and the AI ​​can generate materials that fit the situation and include them in the content.

[0095] (3) In the information processing system described in (1) or (2) above, the request information further includes music information regarding a song desired by the first user, and in the processing step, the music information is input into at least one model to cause the model to generate a specified song, and in the output step, the first content comprising the specified song is output, wherein the specified voice is used at least in part as vocals in the specified song.

[0096] With this configuration, the content can include, for example, using AI technology to identify or generate a song that is appropriate for a situation and then having a desired vocalist sing it.

[0097] (4) In the information processing system described in (3) above, the music information includes information about a phrase in the music desired by the first user, and in the processing step, the model is caused to generate the specified music containing the phrase in its lyrics.

[0098] With this configuration, it is possible to have the AI ​​identify or generate music based on more specific information and include it in the content.

[0099] (5) In the information processing system described in (3) or (4) above, the music information includes information regarding the music structure of the music desired by the first user, and in the processing step, the model is caused to generate the specified music having the music structure.

[0100] With this configuration, it is possible to have the AI ​​identify or generate music based on more specific information and include it in the content.

[0101] (6) In the information processing system described in any one of (3) to (5) above, the music information includes information about an instrument used in the music desired by the first user, and in the processing step, the model is caused to generate the specified music including at least the part of the instrument.

[0102] With this configuration, it is possible to have the AI ​​identify or generate music based on more specific information and include it in the content.

[0103] (7) In the information processing system described in any one of (3) to (6) above, the music information includes information regarding the composer or lyricist of the music desired by the first user, and in the processing step, the model is caused to generate the specified music that evokes the involvement of the composer or lyricist.

[0104] With this configuration, it is possible to have the AI ​​identify or generate music based on more specific information and include it in the content.

[0105] (8) In the information processing system described in any one of (3) to (7) above, in the processing step, at least a portion of the request information is input into the at least one model to cause the model to identify or generate a predetermined image, and in the output step, the first content including the predetermined image is output, wherein the first content is a music video in which the predetermined image is displayed while the predetermined song is played.

[0106] With this configuration, a user can generate a desired music video including music suited to a particular situation through AI technology.

[0107] (9) In the information processing system described in any one of (1) to (8) above, in the receiving step, the request information is received as input specified by at least one of text, sound data, image data, and a URL.

[0108] According to this configuration, the user can input request information in various forms, thereby improving usability when creating desired content.

[0109] (10) In the information processing system described in (9) above, the sound data is data obtained by the first user playing a musical instrument.

[0110] According to this configuration, the user can input request information in the form of a performance, thereby improving usability in creating desired content.

[0111] (11) The information processing system described in (10) above, further comprising a server and the musical instrument, the server comprising the processor, and the sound data obtained from the musical instrument being transmitted to the server via a network.

[0112] With this configuration, it is possible to realize an information processing system in a more specific manner. However, this configuration is not necessarily required, and the information processing system may be provided in other manners.

[0113] (12) In the information processing system described in any one of (1) to (11) above, in the receiving step, an input specifying a second content is received as the request information, wherein the second content is content that was output based on request information previously input by a second user, and in the output step, the first content that makes secondary use of the second content is output.

[0114] According to this configuration, it is possible to use other people's previously outputted content, thereby improving usability in creating desired content.

[0115] (13) In the information processing system described in (12) above, the second content is managed in association with the second user who is the creator, and further, in the associating step, when the first content that makes secondary use of the second content is output or released, information regarding benefits is associated with the second user.

[0116] Such a configuration gives some advantage to creators who create content through the information processing system, and can promote active use of the information processing system.

[0117] (14) In the information processing system described in (13) above, in the associating step, information regarding payment of a secondary usage fee is associated with the first user.

[0118] According to this configuration, when a user uses content that has been output by another person, a secondary usage fee can be collected from the user, and, for example, a portion of the fee can be returned to the creator of the original content.

[0119] (15) In the information processing system described in any one of (1) to (14) above, the at least one model is a trained model that has been machine-learned to output results according to input content.

[0120] This configuration allows the model to identify or generate information based on past statistical data, and thus allows content including such information to be provided to the user.

[0121] (16) An information processing method, comprising the steps of the information processing system according to any one of (1) to (15) above.

[0122] According to this configuration, it is possible to efficiently provide each user with content that is specifically desired for each user depending on the situation.

[0123] (17) A program that causes a computer to execute each step of the information processing system according to any one of (1) to (15) above.

[0124] According to this configuration, it is possible to efficiently provide each user with content that is specifically desired for each user depending on the situation. Of course, this is not the case.

[0125] Finally, while various embodiments of the present disclosure have been described, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0126] 1: Information processing system 11: Network 2: Server device 20: Communication bus 21: Communications Department 22: Storage section 23: Processor 231: Reception 232: Processing section 233: Output section 234: Association section 235: Arithmetic section 236: Display control unit 3: User terminal 30: Communication bus 31: Communications Department 32: Storage section 33: Processor 34:Display section 35: Input section 3a: Computer 3b: Mobile device 3c: Electronic instruments 4: Screen 41 :Area 42 :Area 42a: Dropdown 42b: Button 43 :Area 43a: Input field 44 :Area 44a: Input field 45 :Area 45a: Button 46: Button 5: Database IF0: Request Information IF1: Voice information IF2: Setting information IF3: Song Information LY:Lyrics MD: Model MD1: First model MD2: Second model MV: Music Video

Claims

1. An information processing system, At least one processor is provided, the processor being configured to execute the following steps by reading a program: In the receiving step, input of request information by the first user is received, and the request information is voice information relating to a voice desired by the first user; setting information relating to a situation in which the voice is used; The processing step includes inputting the voice information into at least one model to cause the model to identify or generate a predetermined voice; In the output step, the system outputs first content in response to the request information, wherein the first content includes the predetermined voice and is configured to evoke the situation.

2. 2. The information processing system according to claim 1, In the processing step, the setting information is input to the at least one model, thereby causing the model to identify or generate materials suitable for the situation; In the output step, the system outputs the first content including the material and the predetermined voice.

3. 2. The information processing system according to claim 1, the request information further includes music information regarding a music piece desired by the first user; In the processing step, the music information is input to the at least one model, thereby causing the model to generate a predetermined music piece; In the output step, the first content including the specified music piece is output, and the specified music piece uses the specified voice as vocals at least in part.

4. 4. The information processing system according to claim 3, the music information includes information about a phrase in the music desired by the first user; In the processing step, the system causes the model to generate the predetermined song containing the phrase in its lyrics.

5. 4. The information processing system according to claim 3, the music information includes information about the music structure of the music desired by the first user, In the processing step, the system causes the model to generate the predetermined piece of music having the song structure.

6. 4. The information processing system according to claim 3, the music information includes information about an instrument used in the music piece desired by the first user; In the processing step, the system causes the model to generate the predetermined piece of music including at least the instrument part.

7. 4. The information processing system according to claim 3, the music information includes information about a composer or lyricist of the music desired by the first user; The processing step causes the model to generate the predetermined piece of music that evokes the involvement of the composer or lyricist.

8. 4. The information processing system according to claim 3, In the processing step, at least a portion of the request information is input to the at least one model, thereby causing the model to identify or generate a predetermined image; In the output step, the first content including the specified image is output, wherein the first content is a music video in which the specified image is displayed while the specified song is being played.

9. 2. The information processing system according to claim 1, In the receiving step, the system receives an input specified by at least one of text, sound data, image data, and a URL as the request information.

10. 10. The information processing system according to claim 9, The system, wherein the sound data is data obtained by the first user playing a musical instrument.

11. 11. The information processing system according to claim 10, a server and the musical instrument; the server comprises the processor; The sound data obtained from the musical instrument is transmitted to the server via a network.

12. 2. The information processing system according to claim 1, the receiving step receives an input specifying a second content as the request information, wherein the second content is content that has been output based on request information previously input by a second user; In the output step, the first content that is a secondary use of the second content is output.

13. 13. The information processing system according to claim 12, the second content is managed in association with the second user who is the creator; Furthermore, in the associating step, when the first content that makes secondary use of the second content is output or has been output, the system associates information about a benefit with the second user.

14. 14. The information processing system according to claim 13, In the associating step, information regarding payment of a secondary usage fee is associated with the first user.

15. 2. The information processing system according to claim 1, The system, wherein the at least one model is a trained model that has been machine-learned to output results according to input content.

16. An information processing method, comprising: A method comprising the steps of an information processing system according to any one of claims 1 to 15.

17. A program, A program that causes a computer to execute each step of the information processing system according to any one of claims 1 to 15.

Images