Information processing method, information processing system, and program

Abstract

This information processing system includes: an information generation unit that generates instruction information containing a first character string in which lyrics of a musical piece, sound of content, or text of a comic book is expressed in a first language and a constraint condition related to the length of a character string; and an information acquisition unit that acquires a second character string which is generated as a result of the processing of the instruction information by a trained generation model, the second character string resulting from conversion of the first character string into a second language different from the first language, under the constraint condition.

Description

Information processing method, information processing system, and program

[0001] The present disclosure relates to translation technology.

[0002] Machine translation techniques for converting a string of characters expressed in a specific language into another language have been proposed. For example, Patent Document 1 discloses a technique for generating translations of lyrics using translations corresponding to song information such as the music genre or the gender of the singer.

[0003] Japanese Patent Application Laid-Open No. 2001-075963

[0004] However, simply using translations based on the music genre, the gender of the singer, or the like may result in, for example, an excessively long translation being generated for a song, which may be inappropriate as lyrics to be sung in parallel with the performance of the song. While the above description focuses on translating lyrics, similar problems are anticipated when translating character strings other than lyrics. For example, when translating lines placed in a speech bubble in a manga, a translation that is excessively long for the size of the speech bubble may be generated, resulting in an inability to properly position the translation in the speech bubble. In consideration of the above circumstances, one aspect of the present disclosure aims to reduce the possibility of generating a short string that is excessively long for the environment in which the converted character string is output.

[0005] In order to solve the above problems, an information processing method according to one aspect of the present disclosure generates instruction information including a first string that expresses lyrics of a song, audio of content, or dialogue from a manga in a first language, and a constraint on the length of the string, and obtains a second string generated by a trained generative model processing the instruction information, the second string being obtained by converting the first string into a second language different from the first language under the constraint.

[0006] An information processing system according to one aspect of the present disclosure includes an information generation unit that generates instruction information including a first string of characters that expresses lyrics of a song, audio of content, or dialogue from a manga in a first language and a constraint on the length of the string, and an information acquisition unit that acquires a second string of characters that is generated by a trained generative model processing the instruction information and that is obtained by converting the first string into a second language different from the first language under the constraint.

[0007] A program according to one aspect of the present disclosure causes a computer system to function as an information generation unit that generates instruction information including a first string of characters that expresses lyrics of a song, audio of content, or dialogue from a manga in a first language and a constraint on the length of the string, and an information acquisition unit that acquires a second string that is generated by a trained generative model processing the instruction information and that is converted from the first string into a second language different from the first language under the constraint.

[0008] FIG. 1 is a block diagram of an information system in a first embodiment. FIG. 2 is a block diagram of a terminal device. FIG. 3 is a block diagram illustrating an example of the functional configuration of a terminal device. FIG. 4 is a flowchart of a translation process. FIG. 5 is a block diagram illustrating an example of the functional configuration of a terminal device in a second embodiment. FIG. 6 is a flowchart of a translation process in the second embodiment. FIG. 7 is a block diagram illustrating an example of the functional configuration of a terminal device in a third embodiment. FIG. 8 is an explanatory diagram regarding the relationship between images and dialogue in manga data. FIG. 9 is a flowchart of a translation process in the third embodiment.

[0009] A: First Embodiment Fig. 1 is a block diagram illustrating the configuration of an information system 100 in the first embodiment. The information system 100 includes a terminal device 10 and a translation system 20. The terminal device 10 is an information processing system used by a user U. Examples of the terminal device 10 include portable information devices such as smartphones, tablet terminals, and personal computers. The terminal device 10 can communicate with the translation system 20 via a communication network 200 such as the Internet.

[0010] 2 is a block diagram of the terminal device 10. The terminal device 10 includes a control device 11, a storage device 12, a communication device 13, an operation device 14, a display device 15, and a sound emitting device 16. The terminal device 10 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0011] The control device 11 is composed of one or more processors that control each element of the terminal device 10. For example, the control device 11 is composed of one or more types of processors, such as a central processing unit (CPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). The communication device 13 communicates with the translation system 20 under the control of the control device 11.

[0012] The storage device 12 is one or more memories that store programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 12 may be configured with a combination of multiple types of storage media. Furthermore, a portable storage medium that is detachable from the terminal device 10, or a storage medium (e.g., cloud storage) that the control device 11 can write to or read from via the communication network 200 may be used as the storage device 12.

[0013] The operation device 14 is an input device that accepts instructions from the user U. The operation device 14 is, for example, an operator operated by the user U, or a touch panel that detects contact by the user U. The display device 15 displays images under the control of the control device 11. The display device 15 is configured with a display panel such as a liquid crystal panel or an organic EL panel. The sound emitting device 16 reproduces sound waves under the control of the control device 11. The sound emitting device 16 is, for example, a speaker or headphones.

[0014] 3 is a block diagram illustrating an example of the functional configuration of the terminal device 10. The control device 11 executes a program stored in the storage device 12 to realize multiple functions (a condition specifying unit 31, an information generating unit 32, an information acquiring unit 33, and an output control unit 34).

[0015] The storage device 12 of the first embodiment stores music data X1. The music data X1 is data representing the musical score of a music piece. The music data X1 is, for example, a music file conforming to the MIDI (Musical Instrument Digital Interface) standard. Specifically, the music data X1 includes a sequence of notes N and lyrics L1 of the music piece. The sequence of notes N is a time series of multiple notes that make up the music piece. The music data X1 specifies the pitch and sound duration of each note that makes up the sequence of notes N.

[0016] The lyrics L1 are a time sequence of characters corresponding to each note in the note sequence N. That is, the lyrics L1 correspond in time to the note sequence N. For example, one syllable constituting the lyrics L1 is assigned to one note in the note sequence N. The lyrics L1 in the first embodiment are a character string expressed in a first language. The first language is, for example, Japanese.

[0017] The translation system 20 in FIG. 1 is a machine translation device that converts lyrics L1 expressed in a first language into lyrics L2 expressed in a second language. The second language is different from the first language. The second language is, for example, English. That is, the translation system 20 generates lyrics L2 by converting the language of lyrics L1. Note that lyrics L1 is an example of a "first character string," and lyrics L2 is an example of a "second character string."

[0018] The condition specification unit 31 in FIG. 2 specifies a constraint Cx from the music data X1. The constraint Cx is a condition related to the length of the lyrics L2 after translation. In the first embodiment, the constraint Cx is set according to the duration of each syllable that constitutes the lyrics L1. Specifically, the condition specification unit 31 sets the constraint Cx according to the total duration of the multiple syllables that constitute the lyrics L1 (hereinafter referred to as "singing time"). The duration of each syllable is the duration of the pronunciation period of the note to which that syllable is assigned in the sequence of notes N. For example, the constraint Cx is set to the number of characters that can be uttered within the singing time of the lyrics L1. As explained above, the constraint Cx is a time-related condition related to the lyrics L2.

[0019] As explained above, the note sequence N in the music data X1 is associated information for specifying the constraint Cx. The associated information is a different type of information from the lyrics L1 and is provided in association with the lyrics L1. The note sequence N in the first embodiment can also be expressed as associated information used for playing the lyrics L1, associated information played together with the lyrics L1, or associated information that defines the conditions for playing the lyrics L1. On the other hand, the note sequence N (associated information) is not information that directly specifies the length of the lyrics L2 or information dedicated to setting the length of the lyrics L2.

[0020] The information generating unit 32 generates instruction information P. The instruction information P is a prompt instructing the translation of lyrics L1. Specifically, the information generating unit 32 generates instruction information P including lyrics L1 and constraints Cx. For example, the instruction information P is a natural language prompt requesting that lyrics L1 be converted from a first language to a second language under constraints Cx. For example, the instruction information P is a prompt expressing a request such as "Please translate Japanese lyrics (lyrics L1) into English under (constraints Cx)."

[0021] The information generation unit 32 transmits the instruction information P exemplified above from the communication device 13 to the translation system 20. The translation system 20 generates response information R by processing the instruction information P received from the terminal device 10, and transmits the response information R to the terminal device 10. The response information R is information including lyrics L2. The lyrics L2 are a string of characters obtained by converting lyrics L1 from the first language to the second language under constraints Cx. That is, for example, lyrics L2 in the second language are generated such that the singing time satisfies constraints Cx while retaining the meaning of lyrics L1.

[0022] The translation system 20 generates response information R by processing instruction information P using a trained generative model M. The generative model M is a generative probabilistic model that generates response information R in response to instruction information P. The generative model M has learned the tendency of response information R in response to instruction information P through prior machine learning (pre-training).

[0023] Specifically, the generative model M is an interactive large-scale language model (LLM) that is trained specifically for natural language processing tasks such as machine translation. For example, a natural language processing model realized by a transformer model using a self-attention mechanism is an example of the generative model M.

[0024] As shown in the above example, the response information R (lyrics L2) is a character string generated by the trained generative model M processing the instruction information P. Specifically, the lyrics L2 are a character string in a second language expressed with a singing time that satisfies the constraint Cx for the singing time of the lyrics L1. For example, lyrics L2 with the same singing time as the lyrics L1 are generated. Therefore, even if the lyrics L1 are the same, the longer the singing time of the lyrics L1, the more characters there are in the lyrics L2.

[0025] The information acquisition unit 33 acquires the response information R (lyrics L2). Specifically, the information acquisition unit 33 receives the response information R transmitted from the translation system 20 via the communication device 13. The information acquisition unit 33 also generates new music data X2 by combining the note sequence N of the existing music data X1 with the lyrics L2 of the response information R. As described above, the music data X2 is generated by replacing the lyrics L1 in the first language with the lyrics L2 in the second language while maintaining the note sequence N in the music data X1. The information acquisition unit 33 stores the music data X2 in the storage device 12.

[0026] The output control unit 34 causes the sound emitting device 16 to play the music data X1 or X2. For example, when the user U instructs the playback of the music data X1, the output control unit 34 causes the sound emitting device 16 to play the music represented by the music data X1. The playback sound of the music data X1 is a mixture of the performance sound of an instrument corresponding to the note sequence N in the music data X1 and the singing sound of lyrics L1 in the first language sung at the pitch corresponding to the note sequence N. On the other hand, when the user U instructs the playback of the music data X2, the output control unit 34 causes the sound emitting device 16 to play the music represented by the music data X2. The playback sound of the music data X2 is a mixture of the performance sound of an instrument corresponding to the note sequence N in the music data X2 and the singing sound of lyrics L2 in the second language sung at the pitch corresponding to the note sequence N. Any known voice synthesis technology may be employed to generate the singing sound represented by the lyrics L1 or L2.

[0027] 4 is a flowchart of the process (hereinafter referred to as "translation process") executed by the control device 11. The translation process is initiated in response to an instruction from the user U via the operation device 14. When the translation process is initiated, the control device 11 (condition specification unit 31) specifies constraint conditions Cx from the music data X1 stored in the storage device 12 (Sa1).

[0028] The control device 11 (information generation unit 32) generates instruction information P including lyrics L and constraints C (Sa2) and transmits the instruction information P to the translation system 20 via the communication device 13 (Sa3). The translation system 20 processes the instruction information P using the generation model M to generate lyrics L, and transmits response information R including lyrics L to the terminal device 10.

[0029] The control device 11 (information acquisition unit 33) receives the response information R transmitted from the translation system 20 via the communication device 13 (Sa4). The control device 11 (information acquisition unit 33) generates music data X2 including the note sequence N of the music data X1 and the lyrics L2 of the response information R, and stores the music data X2 in the storage device 12 (Sa5).

[0030] As described above, in the first embodiment, the instruction information P, which includes the lyrics L1 to be converted as well as the constraint Cx, is processed by the generative model M, and lyrics L2 are generated by converting the lyrics L1 from the first language to the second language under the constraint Cx. This reduces the possibility of generating lyrics L2 that are excessively long or short for the sequence of notes N corresponding to the lyrics L1.

[0031] In the first embodiment, lyrics L2 are generated under a singing time constraint Cx. This reduces the possibility of generating lyrics L2 with a singing time that deviates excessively from the singing time of lyrics L1. In other words, lyrics L2 in a second language that can be sung comfortably for an existing song can be generated. For example, lyrics L2 that can be sung within the playback time of the song can be generated.

[0032] In the first embodiment, the constraints Cx are identified from the music data X1, which includes the lyrics L1. That is, the music data X1 is used to identify both the lyrics L1 and the constraints Cx. Therefore, the configuration and processing required to generate the lyrics L2 can be simplified compared to an embodiment in which the constraints Cx are identified from elements separate from the music data X1.

[0033] B: Second Embodiment Fig. 5 is a block diagram illustrating the functional configuration of the terminal device 10 in the second embodiment. The control device 11 executes a program stored in the storage device 12 to realize multiple functions (a condition specifying unit 31, an information generating unit 32, an information acquiring unit 33, and an output control unit 34).

[0034] The storage device 12 of the second embodiment stores content Y1. The content Y1 is a variety of media distributed or sold for viewing by a large number of viewers, and is composed of video V and audio A1. The video V is a live-action video captured by an imaging device, or a video such as an animation created using computer graphics. The audio A1 is sound played in parallel with the video V. In other words, the audio A1 corresponds in time to the video V. The audio A1 is spoken in a first language (e.g., Japanese). Note that the content Y1 may also be audio content (e.g., a podcast or audiobook) that does not include the video V.

[0035] The translation system 20 of the second embodiment converts a character string B1 in a first language represented by a speech A1 into a character string B2 expressed in a second language. Note that the character string B1 is an example of a "first character string," and the character string B2 is an example of a "second character string."

[0036] The condition specification unit 31 specifies a constraint condition Cy from the content Y1. The constraint condition Cy is a condition related to the length of the translated character string B2. In the second embodiment, the constraint condition Cy is a condition related to the duration of the speech period when the character string B2 is spoken. The condition specification unit 31 sets the constraint condition Cy according to the duration of the speech period of the speech A1 (character string B1). For example, the constraint condition Cy is set to be the number of characters that can be spoken within the speech period of the speech A1.

[0037] The speech period of the audio A1 is a period during which the audio A1 is present in the content Y1. For example, the speech period is exemplified by a period during which the audio is observed at a volume exceeding a predetermined threshold. Specifically, the speech period is the period from when playback of one phrase of the audio A1 starts to when playback of that phrase ends. Note that the speech period may also be specified as the period from when playback of one phrase of the audio A1 starts to when playback of the immediately following phrase starts. As explained above, the audio A1 of the content Y1 is associated information for specifying the constraint Cy.

[0038] The information generation unit 32 generates instruction information P. The instruction information P in the second embodiment is a prompt instructing the translation of a character string B1 represented by a speech A1. Specifically, the information generation unit 32 generates the instruction information P including the character string B1 and a constraint Cy. The instruction information P is a natural language prompt requesting that the character string B1 be converted from a first language to a second language under the constraint Cy. For example, the instruction information P is a prompt expressing a request such as "Please translate a Japanese sentence (character string B1) into English under the constraint Cy." The information generation unit 32 estimates the character string B1 by speech recognition of the speech A1 of the content Y1. Any known speech recognition technology may be employed to estimate the character string B1.

[0039] The information generation unit 32 transmits the instruction information P exemplified above from the communication device 13 to the translation system 20. The translation system 20 processes the instruction information P received from the terminal device 10 to generate response information R, and transmits the response information R to the terminal device 10. The response information R is information including a character string B2. The character string B2 is a character string obtained by converting the character string B1 from the first language to the second language under the constraint condition Cy. That is, for example, a character string B2 in the second language is generated that retains the meaning of the character string B1 while satisfying the constraint condition Cy in the duration of speech.

[0040] As shown in the above example, the response information R (character string B2) is a character string generated by the trained generative model M processing the instruction information P. Specifically, the character string B2 is a character string in a second language that can be spoken with a duration that satisfies the constraint Cy with respect to the duration of the speech period in the content Y1. For example, a character string B2 is generated that is spoken with a duration equivalent to the speech period of the speech A1. Therefore, even if the character string B1 is the same, the longer the duration of the speech period of the speech A1, the more characters there are in the character string B2.

[0041] The information acquisition unit 33 acquires response information R (character string B2). Specifically, the information acquisition unit 33 receives the response information R transmitted from the translation system 20 via the communication device 13. The information acquisition unit 33 also generates new content Y2 by combining video V of existing content Y1 with audio A2 representing character string B2 of the response information R. Audio A2 is audio utterance of character string B2. Any known voice synthesis technology may be used to generate audio A2. As described above, content Y2 is generated by replacing audio A1 in the first language with audio A2 in the second language while maintaining the video V in content Y1. The information acquisition unit 33 stores content Y2 in the storage device 12.

[0042] The output control unit 34 plays content Y1 or content Y2. For example, when the user U instructs the playback of content Y1, the output control unit 34 causes the display device 15 to play the video V of content Y1 and the sound emitting device 16 to play the audio A1 in the first language. On the other hand, when the user U instructs the playback of content Y2, the output control unit 34 causes the display device 15 to play the video V of content Y2 and the sound emitting device 16 to play the audio A2 in the second language.

[0043] 6 is a flowchart of the translation process in the second embodiment. When the translation process is started, the control device 11 (condition specifying unit 31) specifies a constraint condition Cy from the content Y1 stored in the storage device 12 (Sb1).

[0044] The control device 11 (information generation unit 32) estimates the character string B1 by speech recognition of the speech A1 (Sb2). The control device 11 (information generation unit 32) generates instruction information P including the character string B1 and the constraint condition Cy (Sb3) and transmits the instruction information P to the translation system 20 via the communication device 13 (Sb4). The translation system 20 processes the instruction information P using the generative model M to generate the character string B2 and transmits response information R including the character string B2 to the terminal device 10.

[0045] The control device 11 (information acquisition unit 33) receives the response information R transmitted from the translation system 20 via the communication device 13 (Sb5). The control device 11 (information acquisition unit 33) generates speech A2 by performing speech synthesis processing on the character string B2 (Sb6). The control device 11 (information acquisition unit 33) generates content Y2 including the video V of content Y1 and the synthesized speech A2, and stores the content Y2 in the storage device 12 (Sb7).

[0046] As described above, in the second embodiment, the instruction information P, which includes the constraint Cy in addition to the character string B2 to be converted, is processed by the generative model M, so that the character string B2 is generated by converting the character string B1 from the first language to the second language under the constraint Cy. Therefore, it is possible to reduce the possibility that an excessively long or short character string B2 is generated for the video V corresponding to the character string B1.

[0047] In particular, in the second embodiment, the character string B2 is generated under a constraint Cy related to the duration of the speech period in the content Y1. Therefore, it is possible to reduce the possibility that an excessively long character string B2 is generated for the video V in the content Y1. In other words, it is easy to temporally correspond the playback of the video V in the content Y1 with the playback of the audio A2 (character string B2).

[0048] In the second embodiment, the constraint Cy is identified from the content Y1 including the audio A1. That is, the content Y1 is used to identify both the character string B1 and the constraint Cy. Therefore, compared to a configuration in which the constraint Cy is identified from an element separate from the content Y1, the configuration and processing required to generate the character string B2 can be simplified.

[0049] C: Third Embodiment Fig. 7 is a block diagram illustrating the functional configuration of the terminal device 10 in the second embodiment. The control device 11 executes a program stored in the storage device 12 to realize multiple functions (a condition specifying unit 31, an information generating unit 32, an information acquiring unit 33, and an output control unit 34).

[0050] The storage device 12 of the third embodiment stores manga data Z1 representing a manga. The manga data Z1 is composed of an image G and lines W1. The lines W1 are character strings representing the words of characters appearing in the manga.

[0051] 8 is an explanatory diagram of an image G and dialogue W1. A text area D is placed within the comic image G. The dialogue W1 is placed within the text area D. The text area D is a speech bubble of an appropriate size and shape placed within the comic image G. The dialogue W1 is a string of characters expressed in a first language (for example, Japanese).

[0052] The translation system 20 of the third embodiment converts a line W1 expressed in a first language into a line W2 expressed in a second language. Note that the line W1 is an example of a "first character string," and the line W2 is an example of a "second character string."

[0053] The condition specification unit 31 in Figure 7 specifies a constraint Cz from the manga data Z1. The constraint Cz is a condition related to the length of the translated line W2. The constraint Cz in the third embodiment is a condition according to the size of the character area D in the image G where the line W1 is to be placed. The condition specification unit 31 specifies the size of the character area D where the line W1 is to be placed, and sets the constraint Cz according to the size of the character area D. For example, the constraint Cz for the line W2 is set to be the number of characters that can be placed in the character area D for the line W1.

[0054] As explained above, image G (text area D) of comic data Z1 is accompanying information for specifying constraint Cz. The accompanying information is a different type of information from the dialogue W1 and is provided in association with the dialogue W1. Image G in the third embodiment can also be expressed as accompanying information used to display the dialogue W1, accompanying information displayed together with the dialogue W1, or accompanying information that defines the conditions for displaying the dialogue W1. On the other hand, image G (text area D) is not information that directly specifies the length of the dialogue W2 or information dedicated to setting the length of the dialogue W2.

[0055] The information generating unit 32 generates instruction information P. The instruction information P in the third embodiment is a prompt instructing the translation of the line W1. Specifically, the information generating unit 32 generates the instruction information P including the line W1 and the constraint Cz. For example, the instruction information P is a natural language prompt requesting that the line W1 be converted from a first language to a second language under the constraint Cz. For example, the instruction information P is a prompt expressing a request such as "Please translate the Japanese sentence (line W1) into English under the constraint Cz."

[0056] The information generation unit 32 transmits the instruction information P exemplified above from the communication device 13 to the translation system 20. The translation system 20 generates response information R by processing the instruction information P received from the terminal device 10, and transmits the response information R to the terminal device 10. The response information R is information including a line W2. The line W2 is a character string obtained by converting the line W1 from the first language to the second language under the constraint Cz. That is, for example, a line W2 in the second language is generated that retains the meaning of the line W1 while satisfying the constraint Cz in the number of characters.

[0057] As shown in the above example, the response information R (line W2) is a character string generated by the trained generative model M processing the instruction information P. Specifically, the line W2 is a character string in a second language expressed with a number of characters that satisfies the constraint Cz for the size of the character area D in which the line W1 is placed. For example, the line W2 is generated with a number of characters that can be placed in a character area D of the same size as the character area D of the line W1. Therefore, even if the line W1 is common, the larger the size of the character area D of the line W1, the more characters the line W2 has.

[0058] The information acquisition unit 33 acquires the response information R (the dialogue W2). Specifically, the information acquisition unit 33 receives the response information R transmitted from the translation system 20 via the communication device 13. The information acquisition unit 33 also generates new manga data Z2 by combining the image G of the existing manga data Z1 with the dialogue W2 of the response information R. As described above, the manga data Z2 is generated by replacing the dialogue W1 in the first language with the dialogue W2 in the second language while maintaining the image G in the manga data Z1. The information acquisition unit 33 stores the manga data Z2 in the storage device 12.

[0059] The output control unit 34 plays back the comic data Z1 or the comic data Z2. For example, when the user U instructs the display device 15 to play back the comic data Z1, the output control unit 34 causes the display device 15 to display the image G of the comic data Z1 and the dialogue W1 in the first language arranged in the text area D. On the other hand, when the user U instructs the display device 15 to play back the comic data Z2, the output control unit 34 causes the display device 15 to display the image G of the comic data Z2 and the dialogue W2 in the second language arranged in the text area D.

[0060] 9 is a flowchart of the translation process in the third embodiment. When the translation process is started, the control device 11 (condition specifying unit 31) specifies constraint conditions Cz from the comic data Z1 stored in the storage device 12 (Sc1).

[0061] The control device 11 (information generation unit 32) generates instruction information P including the line W and the constraint C (Sc2) and transmits the instruction information P to the translation system 20 via the communication device 13 (Sc3). The translation system 20 processes the instruction information P using the generation model M to generate the line W, and transmits response information R including the line W to the terminal device 10.

[0062] The control device 11 (information acquisition unit 33) receives the response information R sent from the translation system 20 via the communication device 13 (Sc4). The control device 11 (information acquisition unit 33) generates manga data Z2 including the image G of the manga data Z1 and the translated dialogue W2, and stores the manga data Z2 in the storage device 12 (Sc5).

[0063] As described above, in the third embodiment, the instruction information P including the line W2 to be converted as well as the constraint Cz is processed by the generative model M, whereby the line W2 is generated by converting the line W1 from the first language to the second language under the constraint Cz. Therefore, it is possible to reduce the possibility that the line W2 that is excessively long or too short for the image G is generated.

[0064] In particular, in the third embodiment, the line W2 is generated under a constraint Cz according to the size of the character area D in which the line W1 is placed. This reduces the possibility of generating a line W2 that is excessively long for the size of the character area D. In other words, it is easy to place the line W2 in the character area D of the image G in the manga data Z1.

[0065] In the third embodiment, the constraint Cz is identified from the comic data Z1 including the dialogue W1. That is, the comic data Z1 is used to identify both the dialogue W1 and the constraint Cz. Therefore, compared to a configuration in which the constraint Cz is identified from elements separate from the comic data Z1, the configuration and processing required to generate the dialogue W2 can be simplified.

[0066] As can be understood from the examples of the first to third embodiments, the music data X1 of the first embodiment, the content Y1 of the second embodiment, and the manga data Z1 of the third embodiment are collectively expressed as target data to be processed by the information system 100.

[0067] D: Modifications Specific modifications that can be added to the above-mentioned embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not mutually contradicting each other.

[0068] (1) The content of the constraints C (Cx, Cy, Cx) for translation using the generative model M is not limited to the examples in the above-mentioned embodiments. In the following description, a character string before translation expressed in a first language is referred to as a "first character string," and a character string in a second language generated by translating the first character string is referred to as a "second character string." The lyrics L1 of the first embodiment, the character string B1 (audio A1) of the second embodiment, and the line W1 of the third embodiment are examples of the "first character string," and the lyrics L2 of the first embodiment, the character string B2 (audio A2) of the second embodiment, and the line W2 of the third embodiment are examples of the "second character string."

[0069] [Aspect 1] Constraint condition C in aspect 1 includes, for example, a condition related to the size of the display surface of display device 15 (hereinafter referred to as "display size"). That is, condition specification unit 31 sets constraint condition C according to the display size of display device 15. For example, constraint condition C is exemplified as a number of characters that is appropriate for the display size of display device 15. As a result of translating a first character string under constraint condition C exemplified above, a second character string is generated with a number of characters according to the display size. For example, even when the first character string is the same, the larger the display size, the greater the number of characters in the second character string.

[0070] [Aspect 2] Constraint condition C in aspect 2 includes a condition related to characters (hereinafter referred to as "characters") appearing in content such as videos or comics. That is, condition specification unit 31 sets constraint condition C according to the characteristics of characters appearing in the content. The characteristics of characters are, for example, features related to the character's personality or appearance. For example, they are specified by user U through an operation on operation device 14. For example, constraint condition C is an appropriate number of characters for the lines of a character. As a result of translating a first character string under constraint condition C exemplified above, a second character string is generated with a number of characters according to the characteristics of the characters appearing. For example, even if the first character string is the same, if the characteristics of the characters appearing are different, the number of characters in the second character string will also be different.

[0071] [Aspect 3] Constraint condition C in aspect 3 includes a condition related to the overall story of content such as a video or manga. That is, the condition specification unit 31 sets constraint condition C according to the story of the content. The story of the content is specified by the user U, for example, by operating the operation device 14. For example, constraint condition C is exemplified as a number of characters that is appropriate for the story of the content. As a result of translating the first character string under constraint condition C exemplified above, a second character string having a number of characters according to the story of the content is generated.

[0072] (2) In the above-described embodiments, the condition specification unit 31 specifies the constraints C (Cx, Cy, Cz) using the target data (music data X1, content Y1, manga data Z1). However, the method of specifying the constraints C is not limited to the above examples. For example, the condition specification unit 31 may set the constraints C in response to an instruction from the user U via the operation device 14. Specifically, the condition specification unit 31 specifies, as the constraints C, a condition selected by the user U from a plurality of conditions prepared in advance. The condition specification unit 31 may also specify, as the constraints C, a character string input by the user U via an operation on the operation device 14.

[0073] (3) In the second embodiment, content Y1 including video V and audio A1 is exemplified, but content Y1 may also be composed of video V and character string B1. Character string B1 is, for example, a subtitle displayed together with video V. When content Y1 includes character string B1, speech recognition (Sb2) for audio A1 is omitted.

[0074] Similarly, content Y2 may be composed of video V and character string B2. Character string B2 is, for example, a subtitle displayed together with video V. When content Y2 includes character string B2, speech synthesis (Sb6) for character string B2 is omitted.

[0075] (4) In the third embodiment, the target data to be processed is not limited to the comic data Z1. The third embodiment can also be applied to various publications other than comics, such as magazines or newspapers. Therefore, the character region D in which the first or second character string is placed is not limited to a comic speech bubble.

[0076] (5) In the second embodiment, the condition specifying unit 31 specifies the constraint condition Cy from the audio A1 of the content Y1. However, the condition specifying unit 31 may specify the constraint condition Cy by analyzing the video V of the content Y1. For example, the condition specifying unit 31 sets the constraint condition Cy according to the length of the playback time of the video V. For example, the constraint condition Cy is set to be the number of characters that can be uttered within the playback time of the video V.

[0077] In the above configuration, the video V of the content Y1 is accompanying information for identifying the constraint Cy. The accompanying information is a different type of information from the audio A1 (character string B1) and is provided in association with the audio A1. The video V in the second embodiment can also be expressed as accompanying information used to play the audio A1, accompanying information played together with the audio A1, or accompanying information that specifies the conditions for playing the audio A1. On the other hand, the video V (accompanying information) is not information that directly specifies the length of the character string B2 or information dedicated to setting the length of the character string B2.

[0078] As can be understood from the above examples, the condition specification unit 31 in some embodiments of the present disclosure uses accompanying information included in the target data to specify the constraint condition C. The accompanying information is a comprehensive concept that includes the sequence of notes N of the music data X1 in the first embodiment, the video V or audio A1 of the content Y1 in the second embodiment, and the image G of the comic data Z1 in the third embodiment.

[0079] (6) In the second embodiment, the translation process may be executed in real time in parallel with the playback of the content Y1. For example, the control device 11 executes the translation process of FIG. 6 for each pronunciation period in parallel with the playback of the content Y1.

[0080] Furthermore, in the second embodiment, the constraint condition Cy may include a condition regarding information that is not to be included in the character string B2. For example, the condition specification unit 31 sets the constraint condition Cy including a condition that information detected from the video V of the content Y1 is not to be included in the character string B2. The information detected from the video V is, for example, information such as the name of an object included in the video V. Any known object detection technology may be employed to detect an object included in the video V. With the above configuration, the number of characters in the character string B2 is reduced, and as a result, the time required to play the audio A2 corresponding to the character string B2 is shortened.

[0081] (7) In the above-described embodiments, a first character string in a first language is converted into a second character string in a second language. However, the first language and the second language may be a common language. That is, the generative model M may convert a first character string expressed in a specific language into a second character string in that language under constraint C. When the number of characters in the second character string exceeds the number of characters in the first character string, the processing by the generative model M corresponds to "filling in" the first character string. When the number of characters in the second character string is less than the number of characters in the first character string, the processing by the generative model M corresponds to "summarizing" the first character string. The above-described embodiments are similarly applicable to filling in or summarizing the first character string.

[0082] (8) In the above-described embodiments, the translation system 20 separate from the terminal device 10 generates the response information R. However, the terminal device 10 may generate the response information R. For example, the terminal device 10 may be equipped with a generative model M. The information acquisition unit 33 generates the response information R by processing the instruction information P generated by the information generation unit 32 using the generative model M. As can be understood from the above explanation, the information acquisition unit 33 is comprehensively expressed as an element that acquires the response information R. The "acquisition" of the response information R includes not only the "reception" of the response information R but also the "generation" of the response information R.

[0083] Furthermore, as can be understood from the above explanation, the “information processing system” in this disclosure may be the terminal device 10 alone in each of the above-mentioned forms, or the entire information system 100 including the terminal device 10 and the translation system 20.

[0084] (9) The generative model M is not limited to the large-scale language model exemplified in the above embodiments. For example, various models such as a recurrent neural network (RNN), a long short-term memory (LSTM), or a bidirectional encoder representations from transformers (BERT) may be used as the generative model M.

[0085] (10) The functions of the terminal device 10 according to each of the above-described embodiments are realized by cooperation between one or more processors constituting the control device 11 and a program stored in the storage device 12. The program according to each of the above-described embodiments can be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium is, for example, a non-transitory recording medium, such as an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transitory, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network 200, the recording medium storing the program in the distribution device corresponds to the non-transitory recording medium described above.

[0086] E: Supplementary Note From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0087] An information processing method according to one aspect (aspect 1) of the present disclosure generates instruction information including a first string of characters that expresses lyrics of a song, audio of a content, or dialogue from a manga in a first language and a constraint on the length of the string, and obtains a second string of characters generated by a trained generative model processing the instruction information, the second string being obtained by converting the first string into a second language different from the first language under the constraint. According to the above aspect, by processing instruction information including a constraint on the length of the string in addition to the first string to be converted using a generative model, a second string is generated by converting the first language into the second language under the constraint. Therefore, it is possible to reduce the possibility of generating a second string that is excessively long for the environment in which the second string is output.

[0088] A "trained generative model" is a probabilistic model that generates a second character string in response to instruction information. The generative model learns the tendency of the second character string in response to instruction information including a first character string and constraints through prior machine learning. Specifically, the "generative model" is a generative large-scale language model (LMM) that is trained specifically for natural language processing tasks such as response generation. For example, a probabilistic model realized by a Transformer model that utilizes a self-attention mechanism is an example of a "generative model."

[0089] A "constraint" is any condition related to the length of a character string. For example, examples of "constraints" include a condition related to the total number of characters (number of characters) constituting the character string, a temporal condition such as the length of time the character string is pronounced, and a spatial condition such as the size at which the character string is displayed. In other words, the "length" of a character string is a concept that includes not only the number of characters, but also the length of time required for pronunciation or the size required for display.

[0090] In a specific example (Aspect 2) of Aspect 1, the first character string is the lyrics, and the constraint is a time constraint related to the second character string. According to the above aspect, lyrics are converted from a first language to a second language under a time constraint. Therefore, it is possible to reduce the possibility that a second character string that is excessively long or too short for a sequence of notes in a piece of music is generated.

[0091] In a specific example (Aspect 3) of Aspect 2, the constraint is a condition according to the duration of each syllable in the first character string. According to the above aspect, the second character string is generated under a constraint according to the duration of each syllable. Therefore, it is possible to reduce the possibility of generating a second character string whose duration deviates excessively from the duration of each syllable in the first character string. In other words, it is possible to generate lyrics (second character string) in a second language that can be sung comfortably for an existing song.

[0092] In a specific example (Aspect 4) of Aspect 1, the first character string is a character string represented by the audio, and the constraint is a condition corresponding to the duration of a speech period in the content. In the above aspect, the second character string is generated under a constraint corresponding to the duration of a speech period in the content. This reduces the possibility of generating a second character string that is excessively long for the content. In other words, it is easy to temporally match the playback of the video in the content with the playback of the audio corresponding to the second character string.

[0093] In a specific example (aspect 5) of aspect 1, the first character string is the dialogue, and the constraint is a condition according to the size of the character area in which the dialogue is placed in the comic. In the above aspect, the second character string is generated under a constraint according to the size of the character area in which the dialogue is placed. Therefore, it is possible to reduce the possibility of generating a second character string that is excessively long relative to the size of the character area. In other words, it is easy to place the second character string in the character area of ​​the comic.

[0094] In a specific example (Aspect 6) of any of Aspects 1 to 5, the constraint is further identified from target data including the first character string, and the instruction information includes the first character string of the target data and the constraint identified from the target data. In the above aspects, the constraint is identified from the target data including the first character string. In other words, the target data is shared for identifying the first character string and the constraint. Therefore, the configuration and processing required to generate the second character string can be simplified compared to an aspect in which the constraint is identified from an element separate from the target data.

[0095] An information processing system according to one aspect (aspect 7) of the present disclosure includes an information generation unit that generates instruction information including a first string of characters that expresses lyrics of a song, audio of content, or dialogue from a manga in a first language and a constraint on the length of the string, and an information acquisition unit that acquires a second string of characters that is generated by a trained generative model processing the instruction information and that is obtained by converting the first string into a second language different from the first language under the constraint.

[0096] A program according to one aspect (aspect 8) of the present disclosure causes a computer system to function as an information generation unit that generates instruction information including a first string of characters that expresses lyrics of a song, audio of content, or dialogue from a manga in a first language and a constraint on the length of the string, and an information acquisition unit that acquires a second string that is generated by a trained generative model processing the instruction information and that is converted from the first string into a second language different from the first language under the constraint.

[0097] 100...information system, 10...terminal device, 11...control device, 12...storage device, 13...communication device, 14...operation device, 15...display device, 16...sound emission device, 20...translation system, 31...condition specification unit, 32...information generation unit, 33...information acquisition unit, 34...output control unit.

Claims

1. An information processing method implemented by a computer system, which generates instruction information including a first string of characters that expresses the lyrics of a song, the audio of a content, or the dialogue of a manga in a first language, and a constraint on the length of the string, and obtains a second string of characters that is generated by a trained generative model processing the instruction information, and that is obtained by converting the first string of characters into a second language different from the first language under the constraint.

2. The information processing method according to claim 1, wherein the first character string is the lyrics, and the constraint is a time condition regarding the second character string.

3. The information processing method according to claim 2, wherein the constraint is a condition according to the duration of each syllable of the first character string.

4. The information processing method according to claim 1, wherein the first character string is a character string represented by the voice, and the constraint condition is a condition according to the duration of a speech period in the content.

5. The information processing method according to claim 1, wherein the first character string is the dialogue, and the constraint condition is a condition according to the size of a character area in which the dialogue is placed in the comic.

6. An information processing method according to any one of claims 1 to 5, further comprising identifying the constraint condition from target data containing the first character string, and the instruction information including the first character string of the target data and the constraint condition identified from the target data.

7. An information processing system comprising: an information generation unit that generates instruction information including a first string of characters that expresses lyrics of a song, audio of content, or dialogue from a manga in a first language and a constraint on the length of the string; and an information acquisition unit that acquires a second string of characters that is generated by a trained generative model processing the instruction information and that is obtained by converting the first string into a second language different from the first language under the constraint.

8. A program that causes a computer system to function as: an information generation unit that generates instruction information including a first string of characters that expresses the lyrics of a song, the audio of content, or the dialogue of a manga in a first language, and a constraint on the length of the string; and an information acquisition unit that acquires a second string that is generated by a trained generative model processing the instruction information and that is obtained by converting the first string into a second language different from the first language under the constraint.

Images