Program, method, and information processor

Abstract

To output a variety of different images to music so that a user will not be bored.SOLUTION: A program causes a computer to execute processing of generating text information including words which express an impression of a music performance from music performance information, inputting the generated text information into a learned model, and acquiring and outputting an image generated by the learned model according to the input text information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The disclosure of this specification relates to a program, a method, and an information processing device. [Background technology]

[0002] There is known a program that generates and outputs images in accordance with music or a performance. A specific example of this type of program is described in, for example, Patent Document 1.

[0003] The program described in Patent Document 1 classifies performance information into those corresponding to its musical elements, reads out drawing patterns stored for each musical element, generates images modified in accordance with each musical event, and displays them on an image display device. The images displayed on the image display device include geometric patterns or characters, and are displayed with attributes such as size, shape, brightness, hue, and saturation changed according to the performance information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2000-276137 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the program described in Patent Document 1, predetermined patterns or characters are displayed on the screen in time with the performance. It has been pointed out that the content displayed on the screen is easy to predict, and therefore users tend to tire of it.

[0006] The present invention has been made in consideration of the above circumstances, and its object is to provide a program, method, and information processing device that can output a variety of images in accordance with the performance so that the user will not become bored. [Means for solving the problem]

[0007] A program according to one embodiment of the present invention causes a computer to execute the following processes: generate text information from performance information, the text information including words expressing the impression given by the performance, input the generated text information into a trained model, and obtain and output an image generated by the trained model in accordance with the input text information. Effect of the Invention

[0008] According to one embodiment of the present invention, there are provided a program, a method, and an information processing device that can output a variety of images in accordance with the performance so that the user is unlikely to become bored. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing the external appearance of an electronic musical instrument system according to an embodiment of the present invention; [Diagram 2] 1 is a block diagram showing a configuration of an electronic musical instrument according to an embodiment of the present invention; [Diagram 3] 1 is a block diagram showing a configuration of an information processing device according to an embodiment of the present invention; [Figure 4] FIG. 11 is a diagram showing a relationship between an evaluation value and text information according to an embodiment of the present invention. [Figure 5A] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 5B] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 5C] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 5D] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 6] 4 is a flowchart showing a process executed by an information processing device in one embodiment of the present invention. [Figure 7] 13 is a flowchart showing a process executed by an information processing device in the first modified example of the present invention. [Figure 8] FIG. 11 is a diagram showing the relationship between the range of evaluation values ​​and text information in the second modified example of the present invention. [Figure 9] 13 is a flowchart showing a process executed by an information processing device in a second modified example of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An information processing device according to an embodiment of the present invention, and a method and a program executed by the information processing device, which is an example of a computer, will be described in detail with reference to the drawings.

[0011] Fig. 1 is a diagram showing the external appearance of an electronic musical instrument system 1 according to an embodiment of the present invention. As shown in Fig. 1, the electronic musical instrument system 1 includes an electronic musical instrument 100 and an information processing device 200. The electronic musical instrument 100 and the information processing device 200 are connected via a wired line or a wireless line.

[0012] The electronic musical instrument 100 is an electronic keyboard instrument, such as an electronic keyboard, an electronic piano, a synthesizer, or an electronic organ. The electronic musical instrument 100 may be another type of electronic musical instrument, such as an electronic percussion instrument, an electronic wind instrument, or an electronic string instrument.

[0013] Fig. 2 is a block diagram showing the configuration of the electronic musical instrument 100. As shown in Fig. 2, the electronic musical instrument 100 includes a processor 110, a keyboard 120, an operation unit 130, a key scanner 140, a speaker 150, and a communication unit 160.

[0014] The processor 110 is an element including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 110 controls the electronic musical instrument 100 by reading out programs and data stored in the ROM and using the RAM as a work area.

[0015] When the user performs a key depression operation, the processor 110 performs a sound generation process to generate a musical tone of a pitch corresponding to the depressed key, and outputs this musical tone from the speaker 150. When the user performs a key release operation, the processor 110 performs a sound erasing process to erase the musical tone that is being generated and corresponds to the released key.

[0016] The keyboard 120 has a plurality of white and black keys as performance operators, each of which corresponds to a different pitch.

[0017] The operation unit 130 is composed of, for example, a mechanical type, a capacitive non-contact type, a membrane type, or other key switch, a button, a touch panel, etc. By operating the operation unit 130, the user can, for example, set the tone or adjust the volume.

[0018] The key scanner 140 detects key depression, key release, and key depression speed (velocity) of the keyboard 120, and outputs performance information including the detected pitch information and velocity to the processor 110. Velocity can also be said to be a value indicating the strength of a sound. Pitch information is also called key number, key number, MIDI (Musical Instrument Digital Interface) key, or note number.

[0019] The communication unit 160 includes a wireless and wired unit for communicating with an external device. The communication unit 160 includes an interface that complies with, for example, the MIDI standard. Under the control of the processor 110, the communication unit 160 transmits performance information including pitch information and velocity to the information processing device 200, which is an example of an external device. The performance information has, for example, a data structure that complies with the MIDI standard.

[0020] The information processing device 200 is, for example, a tablet terminal, but may be another type of device, such as a smartphone, a notebook PC (Personal Computer), a portable game machine, a feature phone, or a PDA (Personal Digital Assistant).

[0021] Fig. 3 is a block diagram showing the configuration of the information processing device 200. As shown in Fig. 3, the information processing device 200 includes a processor 210, a flash ROM 220, an HMI (Human Machine Interface) 230, and a communication interface 240. The information processing device 200 is placed on the music stand 170 of the electronic musical instrument 100, for example.

[0022] The processor 210 is an element including a CPU, a RAM, a ROM, etc. The processor 210 loads various programs including an image output program 220A stored in the flash ROM 220 onto the RAM, which is a work area, and controls the information processing device 200 according to the loaded programs.

[0023] The processor 210 is, for example, a single processor or a multi-processor, and includes at least one processor. When multiple processors are included, the processor 210 may be packaged as a single device, or may be configured as multiple devices that are physically separated within the information processing device 200. The processor 210 may be referred to as a "control unit."

[0024] The flash ROM 220 is a non-volatile semiconductor memory that serves as a secondary storage device or an auxiliary storage device. The flash ROM 220 stores programs and data used by the processor 210 to perform various processes, such as an image output program 220A and a trained model 220B.

[0025] The HMI 230 includes, for example, a touch panel, a mechanical type, a capacitive non-contact type, a membrane type, or other key switch, a button, etc. When a user operates the HMI 230, a signal indicating the operation content is output to the processor 210. The processor 210 controls the information processing device 200 based on the input signal.

[0026] The touch panel includes a screen 230A configured with an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, an LED (Light Emitting Diode) panel, etc. The HMI 230 displays, for example, an image acquired by the processor 210 executing an image output program 220A on the screen 230A.

[0027] The communication interface 240 includes wireless and wired units for communicating with external devices. In this embodiment, the communication interface 240 includes an interface that complies with the MIDI standard. The communication interface 240 accepts input of performance information received from the electronic musical instrument 100, which is an example of an external device. The input performance information is stored in the RAM of the processor 210, for example.

[0028] The image output program 220A stored in the flash ROM 220 causes the processor 210 to execute a series of processes, including generating text information including words expressing the impression given by the performance from the performance information received from the electronic musical instrument 100 via the communication interface 240, inputting the generated text information to the trained model 220B, and acquiring and outputting an image generated by the trained model 220B in accordance with the input text information.

[0029] In other words, the image output program 220A causes the processor 210 to execute a method including the above series of processes.

[0030] It should be noted that "text information" is a text version of words that include at least one part of speech (verb, adjective, judgement word, auxiliary verb, noun, adverb, particle, conjunction, conjunction, interjection, demonstrative, etc.).

[0031] The trained model 220B is an AI (Artificial Intelligence) program (an example of an image generation program) that generates an image from text information, and is realized by, for example, a neural network. When text information that represents the characteristics of performance information is input, the trained model 220B outputs an image that matches the performance.

[0032] As an example, the trained model 220B is an image generation AI "Stable Diffusion" by Stability AI. The trained model 220B may be another publicly known or well-known image generation AI program that generates an image from text information. Hereinafter, the image generated by the trained model 220B from text information is referred to as "image I."

[0033] The trained model 220B may be provided by cloud computing. That is, the information processing device 200 may not include the trained model 220B. In this case, the processor 210 accesses the trained model 220B placed on a network via the communication interface 240 and provides the trained model 220B with text information representing the characteristics of the performance information. The processor 210 acquires the image I generated by the trained model 220B in response to the input text information via the communication interface 240.

[0034] In this embodiment, by converting performance information into text information, the trained model 220B, which is a publicly known or well-known image generation AI program, can generate an image I, which can then be acquired and output. A variety of images I that match the text information (in other words, the performance) can be generated by the image generation AI program. Therefore, to prevent the user from getting bored, a variety of images I can be output to the HMI 230 and displayed on the screen 230A in accordance with the performance.

[0035] The processor 210 includes, as functional blocks, an operation receiving unit 210A, a text generating unit 210B, and an image acquiring unit 210C. Each functional block is realized by an image output program 220A executed by the processor 210, which is an example of a computer. Each functional block may be realized in part or in whole by hardware such as a dedicated logic circuit.

[0036] The operation receiving unit 210A receives a user operation on the HMI 230. Illustratively, the operation receiving unit 210A receives a genre selection operation. The genre selection operation is an operation for selecting a genre of an image I to be displayed on the screen 230A in sync with the performance of the electronic musical instrument 100. The user can select the genre of the image I to be displayed on the screen 230A, for example, by touching one genre from the genre list displayed on the screen 230A.

[0037] Selectable genres include, for example, portraits, landscapes, still lifes, portraits, natural history, religious, historical, genre and realistic paintings.

[0038] The text generating section 210B generates text information from performance information based on the performance of the electronic musical instrument 100, the text information including words expressing the impression given by the performance.

[0039] More specifically, the text generating section 210B analyzes performance information based on the performance of the electronic musical instrument 100, and acquires the feature amount of the performance.

[0040] The text generation unit 210B acquires at least one of the following characteristics of the performance: pitch, velocity, key (e.g., 24 types from C major to B minor), tempo, chord type (e.g., major, minor, sus, arg, dim, 7th, etc.), and number of chords.

[0041] The pitch and velocity can be obtained from the performance information. The key and chord type are not particularly limited, but can be obtained using a method disclosed in, for example, Japanese Patent No. 3211839. The tempo is not particularly limited, but can be obtained using a method disclosed in, for example, Japanese Patent No. 5672280. The number of chords is not particularly limited, but can be obtained based on, for example, the number of simultaneously pressed keys (the number of keys pressed within a specified time).

[0042] The text generating section 210B may obtain, as performance feature quantities, the presence or absence of syncopation, the regularity of beats, chord progressions, and the like, in addition to or instead of the above-mentioned.

[0043] The text generator 210B calculates an evaluation value based on the feature amount obtained from the performance information. The evaluation value is a parameter value for evaluating the impression given by the performance information based on the performance of the electronic musical instrument 100.

[0044] By using the quantitative information of the performance features, the impression given by the performance can be obtained as an evaluation value (i.e., as an objective numerical value).

[0045] The text generating section 210B calculates a Semantic Differential (SD) vector based on the feature amount acquired from the performance information.

[0046] The SD vector includes at least one evaluation value (hereinafter, "SD value") for evaluation using the Semantic Differential (SD) method. The SD value is a value for evaluating whether the performance information based on the performance corresponds to one antonym or the other antonym of a given antonym pair (a pair of words with opposite meanings) in a multi-level classification.

[0047] In this embodiment, there are five antonym pairs for evaluating the impression of a performance based on the performance information: "delicate and powerful," "slow and fast," "sad and happy," "heavy and light," and "regular and free." In another embodiment, one to four of the above five antonym pairs may be used as antonym pairs for evaluating the impression of a performance based on the performance information.

[0048] The antonym pairs listed above are only examples. For example, "delicate and powerful" may be replaced by "weak and strong." "heavy and light" may be replaced by "dark and light."

[0049] The text generation unit 210B calculates corresponding SD values ​​(SD1 to SD5) for each of the antonym pairs, "delicate and powerful," "slow and fast," "sad and happy," "heavy and light," and "regular and free," based on the features acquired from the performance information.

[0050] SD1 (an example of a first evaluation value) corresponding to the antonym pair "delicate and powerful" is calculated based on the average velocity in a predetermined period of the performance information (for example, a period corresponding to the most recent beat).

[0051] SD2 (an example of a second evaluation value) corresponding to the antonym pair "slow and fast" is calculated based on the average tempo of the musical performance information during a predetermined period.

[0052] SD3 (an example of a third evaluation value) corresponding to the antonym pair "sad and happy" is calculated based on the average of the tonality (a value obtained by quantifying the tonality) and the tempo of the performance information during a predetermined period.

[0053] SD4 (an example of a fourth evaluation value) corresponding to the antonym pair "heavy and light" is calculated based on the average pitch, number of chords, and tempo of the performance information during a predetermined period.

[0054] SD5 (an example of a fifth evaluation value) corresponding to the antonym pair "regular and free" is calculated based on the variance or standard deviation of the tempo in a predetermined period of the performance information.

[0055] Each SD value ranges from 0 to 255. The text generating section 210B calculates each SD value by converting the range of values ​​that the performance feature can take so that the minimum value is 0 and the maximum value is 255.

[0056] The stronger (more prominent) the impression of one of the antonyms in the antonym pair, the closer the SD value is to 0, and the stronger (more prominent) the impression of the other antonym, the closer the SD value is to 255. For example, if the antonym pair is "delicate and powerful," the closer SD1 is to 0 the more delicate the performance characteristic is, and the closer SD1 is to 255 the more powerful the performance characteristic is.

[0057] The text generating unit 210B generates text information based on the SD value. In the example of this embodiment, the text generating unit 210B acquires text information corresponding to the SD value from a text group that indicates, in multiple stages, whether one or the other of the antonym pair applies to each antonym pair (in other words, each SD value).

[0058] Through the above series of processes, the text generating unit 210B generates text information including words expressing the impression given by the performance based on the performance of the electronic musical instrument 100, from the performance information. Note that the text generating unit 210B may convert each SD value into text information corresponding to the SD value, for example, by using a conversion algorithm, without obtaining text information corresponding to the SD value from the text group. In other words, there is a degree of freedom in the method of generating text information from the SD value, and various methods can be adopted.

[0059] FIG. 4 is a diagram showing the relationship between each SD value and text information.

[0060] In the example of Fig. 4, the text generating unit 210B acquires text information according to the SD value from a text group that indicates whether it corresponds to one side or the other side of an antonym pair on a four-level scale. As shown in Fig. 4, the text groups are "(very) + (one antonym)", "(somewhat) + (one antonym)", "(somewhat) + (the other antonym)", and "(very) + (the other antonym)".

[0061] When the SD value is greater than or equal to 0 and less than a first threshold value (for example, the SD value is 0 to 63), the performance is evaluated as giving the impression of "(very) + (one antonym)", and so the text generation unit 210B acquires the text information "(very) + (one antonym)". do.

[0062] When the SD value is equal to or greater than the first threshold and less than the second threshold (for example, the SD value is between 64 and 127), the performance is evaluated as giving the impression of "(somewhat) + (one antonym)", and so the text generating unit 210B acquires the text information "(somewhat) + (one antonym)".

[0063] When the SD value is greater than or equal to the second threshold and less than the third threshold (for example, the SD value is between 128 and 191), the performance is evaluated as giving the impression of "(somewhat) + (the other antonym)", and so the text generation unit 210B acquires the text information "(somewhat) + (the other antonym)".

[0064] When the SD value is greater than or equal to the third threshold and less than 255 (for example, the SD value is 192 to 255), the text generation unit 210B acquires the text information "(very) + (the other antonym)" because the performance is evaluated to give the impression of "(very) + (the other antonym)".

[0065] As a specific example, when SD1 is 0 to 63, the text generator 210B obtains text information of "very delicate." When SD1 is 64 to 127, the text generator 210B obtains text information of "somewhat delicate." When SD1 is 128 to 191, the text generator 210B obtains text information of "somewhat powerful." When SD1 is 192 to 255, the text generator 210B obtains text information of "very powerful."

[0066] In the example of FIG. 4, the scale of antonym pairs is four levels, but the scale may be two levels, three levels, or five levels or more.

[0067] In other words, although the value range of 0 to 255 is divided into four value ranges (i.e., value ranges of 0 to 63, 64 to 127, 128 to 191, and 192 to 255), the value range of 0 to 255 may be divided into two, three, or five or more value ranges.

[0068] In the example of FIG. 4, the thresholds for the scale of antonym pairs are set to 64 (first threshold), 128 (second threshold), and 192 (third threshold), but these thresholds may be set to other values.

[0069] In the example of Fig. 4, the threshold is the same for each antonym pair. However, depending on the content of the antonym pair, the SD value may tend to be low, tend to be high, or may not tend to be either low or high. Therefore, a different threshold may be set for each antonym pair.

[0070] The image acquisition unit 210C inputs the text information generated by the text generation unit 210B, which represents the characteristics of the performance information (e.g., "somewhat delicate"), into the trained model 220B together with the text information representing the genre selected by the genre selection operation.

[0071] Trained model 220B generates image I of the input genre according to text information.

[0072] As an example, the image acquisition unit 210C inputs "portrait" as text information representing the genre to the trained model 220B, and inputs "slightly weak," "slightly slow," "very sad," "very heavy," and "slightly regular" as text information representing the characteristics of the performance information to the trained model 220B. In this case, the trained model 220B generates, for example, an image I in which a person with low brightness and saturation and a gloomy expression appears.

[0073] 5A to 5D show examples of image I generated by trained model 220B.

[0074] FIG. 5A is an example of an image that is generated when the text information “landscape,” “somewhat powerful,” “somewhat fast,” “very fun,” “very light,” and “somewhat free” is input into trained model 220B.

[0075] FIG. 5B shows example images that are generated when the text information “landscape,” “somewhat delicate,” “somewhat slow,” “very sad,” “very heavy,” and “somewhat regular” is input into trained model 220B.

[0076] FIG. 5C is an example of an image that is generated when the text information “portrait,” “somewhat powerful,” “somewhat fast,” “very fun,” “very light,” and “somewhat free” is input into trained model 220B.

[0077] FIG. 5D is an example of an image generated when the text information “portrait,” “somewhat delicate,” “somewhat slow,” “very sad,” “very heavy,” and “somewhat regular” is input into trained model 220B.

[0078] In this way, a variety of images are generated compared to the conventional method (for example, simply changing the size, shape, brightness, hue, saturation, etc. of a predefined pattern or character to match the performance). Therefore, users are less likely to get bored.

[0079] For example, when the same text information is input, the trained model 220B may generate different images I each time. This allows a variety of images I to be displayed on the screen 230A so that the user does not get bored.

[0080] The image acquisition unit 210C acquires the image I generated by the trained model 220B, and outputs the acquired image I to the HMI 230 to display it on the screen 230A.

[0081] By using the SD method, the impression given by the performance can be obtained as text information, which is information in a general-purpose format that can be processed by a publicly known or well-known image generation AI program. Using such text information, the trained model 220B, which is a publicly known or well-known image generation AI program, can generate various images I that match the text information (in other words, the performance). This allows the various images I to be output to the HMI 230 and displayed on the screen 230A in accordance with the performance so that the user does not get bored.

[0082] 6 is a flowchart showing the processing of the image output program 220A executed by the processor 210 in one embodiment of the present invention. For example, when a user operation is performed on the HMI 230 to start the image output program 220A, the execution of the processing shown in FIG.

[0083] The processor 210 performs initial settings (step S101). The processor 210 sets, for example, "128" as the initial values ​​of SD1 to SD5.

[0084] Processor 210 waits for a genre selection operation (step S102).

[0085] When processor 210 receives a genre selection operation (step S102: YES), it acquires text information (hereinafter, referred to as "genre text T0") representing the genre selected by this operation (step S103). Genre text T0 for each genre is stored in advance in, for example, ROM or flash ROM 220 of processor 210.

[0086] The user can, for example, specify in advance the genre to be selected by default by operating the HMI 230. In this case, the process of waiting for the genre selection operation (step S102) can be omitted from the process shown in FIG.

[0087] Processor 210 waits for input of performance information from electronic musical instrument 100 (step S104). The performance information includes information such as the pitch, velocity, and timing of a pressed or released key.

[0088] The performance information is not limited to information obtained by the user playing (performing) electronic musical instrument 100. The performance information may be MIDI data obtained from an external input. In other words, even when pre-recorded music data or the like is used, a variety of images I that match the music can be displayed on screen 230A.

[0089] When performance information is input from electronic musical instrument 100 via communication interface 240 (step S104: YES), processor 210 analyzes the input performance information and acquires the characteristics of the performance (step S105). The acquired characteristics include, for example, pitch, velocity, tonality, tempo, chord type, and number of chords.

[0090] Processor 210 determines whether or not the user's performance of electronic musical instrument 100 has ended (step S106). For example, if a first time has not elapsed since the previous input of performance information, processor 210 determines that the performance has not ended (step S106: NO), and proceeds to step S107. If a first time has elapsed since the previous input of performance information, processor 210 determines that the performance has ended (step S106: YES), and proceeds to step S112. The elapsed time is measured, for example, by a counter built into processor 210.

[0091] In step S107, the processor 210, for example, counts the value of a counter and determines whether or not a second time has elapsed since the previous execution of the output process. "The previous execution of the output process" refers to the previous timing at which the image I acquired from the trained model 220B was output to the HMI 230. While the performance continues, the processor 210 repeatedly executes the processes of steps S106 to S107 until the second time has elapsed since the previous execution of the output process.

[0092] The "second time" that is the criterion for the process of step S107 is related to the update frequency of the image I displayed on the screen 230A. The user can set the update frequency of the image I, for example, by operating the HMI 230. When the update frequency setting is changed, the length of this "second time" also changes.

[0093] If a second time has elapsed since the previous input of performance information (step S107: YES), processor 210 calculates SD1 to SD5 based on the features acquired in step S105 in order to output image I corresponding to the currently analyzed performance information to HMI 230 (step S108).

[0094] Processor 210 acquires text information based on each of SD1 to SD5 (step S109). Text information related to the antonyms "delicate and powerful" is written as impression text T1, text information related to the antonyms "slow and fast" is written as impression text T2, text information related to the antonyms "sad and happy" is written as impression text T3, text information related to the antonyms "heavy and light" is written as impression text T4, and text information related to the antonyms "regular and free" is written as impression text T5.

[0095] The processor 210 inputs the genre text T0 acquired in step S103 and the impression texts T1 to T5 acquired in step S109 to the trained model 220B (step S110).

[0096] The trained model 220B generates an image I based on genre text T0 (eg, a portrait) and impression texts T1 to T5 (eg, slightly weak, slightly slow, very sad, very heavy, slightly regular).

[0097] Processor 210 acquires image I generated by trained model 220B (step S111), and outputs acquired image I to HMI 230 (step S112). As a result, image I corresponding to genre text T0 and impression texts T1 to T5 is displayed on screen 230A. After outputting image I, processor 210 returns to the process of step S104 and waits for input of the next performance information from electronic musical instrument 100.

[0098] The impression texts T1 to T5 change in a variety of ways depending on the performance of the user on the electronic musical instrument 100. Therefore, a variety of images I that match the performance are displayed on the screen 230A so that the user is unlikely to tire of the performance.

[0099] When processor 210 determines that the user's performance of electronic musical instrument 100 has ended (step S106: YES), it calculates SD1 to SD5 (step S113).

[0100] In step S113, processor 210 calculates, for example, an average value of SD values ​​calculated from the start of performance to the end of performance for each of SD1 to SD5. Processor 210 may also calculate, for example, an average value of SD values ​​calculated during a portion of the period from the start of performance to the end of performance (for example, a period of n seconds immediately before the end of performance) for each of SD1 to SD5.

[0101] Processor 210 acquires text information based on each of SD1 to SD5 (step S114), similarly to steps S109 to S112, inputs genre text T0 and impression texts T1 to T5 to trained model 220B (step S115), acquires image I generated by trained model 220B (step S116), and outputs acquired image I to HMI 230 (step S117). As a result, for example, immediately after the user finishes playing electronic musical instrument 100, image I according to genre text T0 and impression texts T1 to T5 is displayed on screen 230A.

[0102] Next, a description will be given of Modification 1 of the present invention. In Modification 1, the processor 210 changes the contents of the impression texts T1 to T5 according to the genre selected by the genre selection operation (or the genre set as the default).

[0103] 7 is a flowchart showing the processing of the image output program 220A executed by the processor 210 in the first modified example of the present invention. The flowchart (see FIG. 7) according to the first modified example is the same as the flowchart (see FIG. 6) according to the above embodiment, except for step S109. Therefore, in FIG. 7, parts common to FIG. 6 are omitted as appropriate.

[0104] In the first modification, processor 210 calculates SD1 to SD5 based on the feature amount acquired in step S105 (step S108), and then in the following step S109, acquires text information based on each of SD1 to SD5.

[0105] Specifically, processor 210 determines whether the genre selected by the genre selection operation (or the default genre) is portrait (step S109-A1). If the genre is portrait (step S109-A1: YES), processor 210 changes the text information included in each text group of the antonym pairs "sad and happy" and "slow and fast" to words that are more suitable for portraits (step S109-A2).

[0106] For example, in step S109-A2, the processor 210 changes "very sad", "slightly sad", "slightly happy", and "very happy" to "crying expression", "neutral expression", "smile", and "big smile", respectively. The processor 210 changes "very slowly", "slightly slowly", "slightly fast", and "very fast" to "very static", "slightly static", "slightly dynamic", and "very dynamic", respectively.

[0107] Processor 210 acquires text information corresponding to the SD value from the text group of antonym pairs "sad and happy" and "slow and fast" whose expressions were changed in step S109-A2, and from the text group of antonym pairs "delicate and powerful," "heavy and light," and "regular and free" whose expressions were not changed (step S109-A5).

[0108] Processor 210 inputs genre text T0 and impression texts T1 to T5 to trained model 220B (step S110). More specifically, processor 210 inputs text information of expressions more suitable for portraits for the antonym pairs "sad and happy" and "slow and fast" to trained model 220B. This enables trained model 220B to generate image I of a portrait more suitable for a user's performance of electronic musical instrument 100.

[0109] In step S109-A2, processor 210 may change only a part of the text information included in the text group, rather than all of the text information. As an example, in step S109-A2, processor 210 may change only "very sad" to "crying expression" in the text group of the antonym pair "sad and happy."

[0110] Thus, in step S109-A2, processor 210 changes at least one first text information (e.g., “very sad”) included in the text group to second text information different from the first text information (e.g., “crying expression”) according to the genre (e.g., portrait) selected by the user operation.

[0111] Words related to movement (slow, fast, etc.) are not compatible with landscapes or still lifes that have little movement. Therefore, if the genre selected in the genre selection operation (or the default genre) is landscapes or still lifes (step S109-A3: YES), processor 210 sets not to use SD2 corresponding to the antonym pair "slow and fast" when acquiring text information (step S109-A4).

[0112] Since SD2 is set as unused, the processor 210 does not acquire text information for the antonym pair “slow and fast.” The processor 210 acquires impression texts T1, T3 to T5 excluding the impression text T2 (step S109-A5).

[0113] The processor 210 inputs the genre text T0 and the impression texts T1, T3 to T5 excluding the impression text T2 to the trained model 220B (step S110). In the trained model 220B, an image I is generated based on the impression texts other than the impression text T2 (i.e., the genre text T0, the impression texts T1, and T3 to T5).

[0114] Trained model 220B generates landscape or still-life image I without using text information that is incompatible with landscape or still-life images with little movement. As a result, landscape or still-life image I that is more suitable for the user's performance of electronic musical instrument 100 can be obtained.

[0115] Thus, in step S109-A4, when a specific genre (e.g., landscape painting, still life painting) is selected by user operation, processor 210 does not acquire text information corresponding to the evaluation value (SD2) for at least one specific antonym pair (e.g., “slow and fast”).

[0116] Next, a description will be given of Modification 2 of the present invention. In Modification 2, when a specific SD value falls within a specific value range, the processor 210 additionally inputs specific text information expressing an impression different from the genre text T0 and the impression texts T1 to T5 to the trained model 220B.

[0117] FIG. 8 is a diagram showing the relationship between the value range of the SD value and additional text information in the second modification.

[0118] When SD1 and SD4 calculated by processor 210 are in the range of 192 to 255 (the value range indicated by hatching in FIG. 8, in which the text information is "very powerful" and "very light"), the performance of electronic musical instrument 100 by the user is characterized by "expression with lines (=powerful)" and "vivid colors (=light, bright)," which are reminiscent of ukiyo-e. Therefore, processor 210 additionally inputs "ukiyo-e" as the genre text information into trained model 220B. Trained model 220B thereby generates image I using the text information "ukiyo-e."

[0119] When SD5 calculated by processor 210 falls within the range of 192 to 255 (the value range marked as "Cubism" in FIG. 8, where the text information is "very free"), the performance of electronic musical instrument 100 by the user has a freedom reminiscent of Cubism. Therefore, processor 210 additionally inputs "Cubism" as the text information of the genre to trained model 220B. As a result, trained model 220B generates image I using the text information "Cubism".

[0120] 9 is a flowchart showing the processing of the image output program 220A executed by the processor 210 in the modified example 2 of the present invention. The flowchart (see FIG. 9) according to the modified example 2 is the same as the flowchart (see FIG. 6) according to the above embodiment, except for step S109, as in the modified example 1. Therefore, in FIG. 9, parts common to FIG. 6 are omitted as appropriate.

[0121] In the second modification, processor 210 calculates SD1 to SD5 based on the feature amount acquired in step S105 (step S108), and then in the following step S109, acquires text information based on each of SD1 to SD5.

[0122] Specifically, processor 210 determines whether SD5 calculated in step S108 falls within the value range of 192 to 255 (step S109-B1). If SD5 falls within the value range of 192 to 255 (step S109-B1: YES), processor 210 acquires "Cubism" as text information (step S109-B2). Text information on various genres such as "Cubism" and "Ukiyo-e" is stored in advance in, for example, the ROM of processor 210 or flash ROM 220.

[0123] The processor 210 acquires impression texts T1 to T5 based on SD1 to SD5 (step S109-B5).

[0124] The processor 210 inputs "Cubism" to the trained model 220B in addition to the genre text T0 and the impression texts T1 to T5 (step S110). As a result, the trained model 220B generates an image I that also reflects the element of "Cubism".

[0125] If SD5 is not within the value range of 192 to 255 (step S109-B1: NO), the processor 210 determines whether or not SD1 and SD4 calculated in step S108 are within the value range of 192 to 255 (step S109-B3).

[0126] If SD1 and SD4 are within the value range of 192 to 255 (step S109-B3: YES), processor 210 acquires "Ukiyo-e" as text information (step S109-B4).

[0127] When the processor 210 acquires the impression texts T1 to T5 based on the SD1 to SD5 (step S109-B5), it inputs "ukiyo-e" in addition to the genre text T0 and the impression texts T1 to T5 to the trained model 220B (step S110). As a result, the trained model 220B generates an image I that also reflects the element of "ukiyo-e".

[0128] In the second modification, when a distinctive performance (a performance in which the SD value is close to the minimum (=0) or maximum (=255)) is performed, the processor 210 additionally inputs to the trained model 220B specific text information (such as "Ukiyo-e" or "Cubism") that is of a genre that is evoked by the characteristic and that expresses an impression different from that of the genre text T0 and the impression texts T1 to T5. As a result, an image I that more closely matches the characteristics of the performance of the electronic musical instrument 100 by the user is obtained.

[0129] Thus, in variant example 2, when a particular evaluation value falls within a particular value range (for example, SD5 is in the value range of 192 to 255), processor 210 additionally inputs particular text information (for example, "Cubism") into trained model 220B.

[0130] In addition, the present invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. In addition, the functions executed in the above-mentioned embodiment may be implemented in appropriate combinations as much as possible. The above-mentioned embodiment includes various steps, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if an effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention.

[0131] The antonym pairs for evaluating the impression of a performance based on the performance information are not limited to the above five. For example, another antonym pair such as "random and accurate" may be adopted.

[0132] The SD value corresponding to the antonym pair "random and accurate" is calculated based on the degree of matching of the performance with the teacher data, for example. The closer the performance of the user on the electronic musical instrument 100 is to the performance indicated by the teacher data, the higher the SD value (the text information is "very accurate"), and the more different the performance is from the performance indicated by the teacher data, the lower the SD value (the text information is "very random").

[0133] The SD value corresponding to the antonym pair "random and accurate" may be calculated by a method that does not use teacher data. For example, the SD value may be calculated according to an unusual playing style, the degree of such playing, etc.

[0134] For example, when performance information of an unusual performance is input, such as when a large number of notes (e.g., 11 notes) are input simultaneously or when musical notes of multiple adjacent keys (e.g., five or more consecutive keys) are input simultaneously, the SD value corresponding to the antonym pair "random and accurate" becomes low. When performance information of a performance within the category of a normal performance is input, the SD value corresponding to the antonym pair "random and accurate" becomes high.

[0135] When the trained model 220B receives the input "very random," it generates an image I that resembles, for example, a random doodle.

[0136] Processor 210 may detect a change in the melody of the performance from performance information based on the user's performance of electronic musical instrument 100, and may change at least one piece of text information (an example of third text information) included in the text group to text information different from the third text information (an example of fourth text information) in accordance with the detected change in melody.

[0137] For example, when the text information based on SD4 changes from "very light" to "very heavy" in a short period of time (e.g., eight beats or less), processor 210 changes the text information of "very heavy" and "slightly heavy" to "very dejected" and "slightly dejected," respectively. That is, in order to express the change in the text information from "very light" to "very heavy" in a short period of time with more appropriate text information, "heavy" is changed to "dejected."

[0138] For example, when the text information based on SD1 changes from "very delicate" to "very powerful" in a short period of time (eight beats or less, for example), processor 210 changes the text information of "very powerful" and "slightly powerful" to "very growing" and "slightly growing," respectively. That is, in order to express the change in the text information from "very delicate" to "very powerful" in a short period of time with more appropriate text information, "powerful" is changed to "growing."

[0139] In this way, by changing the text information according to the change in the melody, the tendency of the image I generated by the trained model 220B also changes according to the melody. In addition, by reflecting more appropriate words expressing the change in the melody in the text information, the image I generated by the trained model 220B also has content that more accurately expresses the melody. Therefore, the user can more intuitively imagine the atmosphere of the song.

[0140] For example, if the change in the SD value is small over a long period (for example, 32 beats) (such as when the fluctuation value of SD1 to SD5 is less than the sixth threshold value over a long period), the performance may give an impression of being gentle. Therefore, processor 210 obtains "gentle" as text information.

[0141] The processor 210 inputs "gentle" to the trained model 220B in addition to the genre text T0 and the impression texts T1 to T5 (step S110). As a result, the trained model 220B generates an image I that also reflects the element of "gentle" that represents the impression of the current performance.

[0142] For example, if all SD values ​​change significantly in a short period of time (e.g., 8 beats or less) (such as when the fluctuation values ​​of SD1 to SD5 are equal to or greater than the seventh threshold in a short period of time), it may be recalled that a performance scene has changed during this period. Therefore, the processor 210 changes the current genre text T0 to another genre text T0 that corresponds to the scene change. This allows the trained model 220B to generate an image I of a different genre from the previous one at the timing when a performance scene change is recalled. [Explanation of symbols]

[0143] 1: Electronic musical instrument system 100: Electronic instruments 110: Processor 120: Keyboard 130: Operation section 140: Key scanner 150: Speaker 160: Communications Department 170:Music stand 200: Information processing device 210: Processor 210A: Operation reception section 210B: Text generation unit 210C: Image acquisition section 220: Flash ROM 220A: Image output program 220B: Trained model 230A: Screen 240: Communication interface

Claims

1. generating text information including words expressing the impression given by the performance from the performance information; The generated text information is input into a trained model; Acquire and output an image generated by the trained model in accordance with the input text information; The process is executed by a computer. program.

2. calculating an evaluation value for evaluating an impression given by the performance from the performance information; generating the text information based on the calculated evaluation value; The program according to claim 1.

3. calculating the corresponding evaluation value for each of the predetermined antonym pairs; acquiring, for each of the antonym pairs, the text information corresponding to the evaluation value from a text group indicating whether the text corresponds to one or the other of the antonym pair in multiple stages; The program according to claim 2.

4. Selecting a genre of the image in response to a user operation; Allowing the trained model to generate the images of the selected genre; The program according to claim 3.

5. changing at least one piece of first text information included in the text group to second text information different from the first text information according to the genre selected by the user operation; The program according to claim 4.

6. Calculating the evaluation values ​​for a plurality of the antonym pairs; when a specific genre is selected by the user operation, acquisition of the text information corresponding to the evaluation value is not performed for at least one specific antonym pair. The program according to claim 4.

7. Calculating the evaluation values ​​for a plurality of the antonym pairs; When the specific evaluation value falls within a specific value range, in addition to the text information representing the impression given by the performance, specific text information representing an impression different from the text information is input to the trained model. The program according to claim 3.

8. The antonym pairs include at least one of delicate and powerful, slow and fast, sad and happy, heavy and light, regular and free. The program according to claim 3.

9. calculating a first evaluation value based on an average velocity of the performance information during a predetermined period of time; The antonym pair corresponding to the first evaluation value is delicate and powerful. The program according to claim 3.

10. calculating the second evaluation value based on an average tempo of the performance information during a predetermined period; The antonym pair corresponding to the second evaluation value is "slow and fast"; The program according to claim 3.

11. calculating the third evaluation value based on an average of the key and the tempo of the performance information during a predetermined period of time; the antonym pair corresponding to the third evaluation value is "sad" and "happy"; The program according to claim 3.

12. calculating the fourth evaluation value based on an average of the pitch, the number of chords, and the tempo during a predetermined period of the performance information; the antonym pair corresponding to the fourth evaluation value is heavy and light; The program according to claim 3.

13. calculating the fifth evaluation value based on a variance or standard deviation of the tempo of the performance information during a predetermined period; the antonym pair corresponding to the fifth evaluation value is regular and free; The program according to claim 3.

14. Detecting a change in the melody of the performance from the performance information; changing at least one piece of third text information included in the text group to fourth text information different from the third text information in response to the detected change in melody; The program according to claim 3.

15. Analyzing the performance information to obtain a characteristic amount of the performance; calculating the evaluation value based on the acquired feature amount; The program according to claim 2.

16. The feature amount includes at least one of pitch, velocity, key, tempo, and number of chords. The program according to claim 15.

17. generating text information including words expressing the impression given by the performance from the performance information; The generated text information is input into a trained model; Acquire and output an image generated by the trained model in accordance with the input text information; The process is executed by a computer. method.

18. generating text information including words expressing the impression given by the performance from the performance information; The generated text information is input into a trained model; Acquire and output an image generated by the trained model in accordance with the input text information; A control unit for executing a process is provided. Information processing device.

Images