Program, method, and information processor

Abstract

To suppress a time lag when displaying an image.SOLUTION: A program which outputs each of a plurality of images at different timings causes a computer to execute processing of deriving a first evaluation value for evaluating an impression of a music performance on the basis of input first music performance information, outputting a first image stored in a storage region, corresponding to first information, on the basis of the first information corresponding to a first value range in the first evaluation value, acquiring a second image, which is output when a second evaluation value derived on the basis of second music performance information input later than the first music performance information is included in a second value range, on the basis of second information corresponding to the second value range, and storing the acquired second image into a storage region.SELECTED DRAWING: Figure 7B

Description

[Technical field]

[0001] The disclosure of this specification relates to a program, a method, and an information processing device. [Background technology]

[0002] There is known a program that generates and outputs images in accordance with music or a performance. A specific example of this type of program is described in, for example, Patent Document 1.

[0003] The program described in Patent Document 1 classifies performance information into those corresponding to its musical elements, reads out drawing patterns stored for each musical element, generates images modified in accordance with each musical event, and displays them on an image display device. The images displayed on the image display device include geometric patterns or characters, and are displayed with attributes such as size, shape, brightness, hue, and saturation changed according to the performance information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2000-276137 A Summary of the Invention [Problem to be solved by the invention]

[0005] In this type of program, if an image takes a long time to generate, the image will not appear on the screen in a timely manner.

[0006] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a program, a method, and an information processing device that can reduce the time lag when displaying an image. [Means for solving the problem]

[0007] A program according to one embodiment of the present invention is a program that outputs a number of images at different times, and causes a computer to execute the following processes: deriving a first evaluation value for evaluating the impression given by a performance based on input first performance information; outputting a first image corresponding to the first information, the first image being stored in a memory area, based on first information corresponding to a first value range in which the first evaluation value is included; obtaining a second image, based on second information corresponding to the second value range, that is output when a second evaluation value derived based on second performance information input after the first performance information is included in a second value range; and storing the obtained second image in the memory area. Effect of the Invention

[0008] According to one embodiment of the present invention, a program, a method, and an information processing device are provided that are capable of reducing a time lag when displaying an image. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing the external appearance of an electronic musical instrument system according to an embodiment of the present invention; [Diagram 2] 1 is a block diagram showing a configuration of an electronic musical instrument according to an embodiment of the present invention; [Diagram 3] 1 is a block diagram showing a configuration of an information processing device according to an embodiment of the present invention; [Figure 4] FIG. 11 is a diagram showing a relationship between an evaluation value and text information according to an embodiment of the present invention. [Figure 5A] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 5B] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 5C] FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 5D]FIG. 1 is a diagram illustrating an example of an image generated by a trained model in one embodiment of the present invention. [Figure 6] A figure showing an example of the relationship between the acquisition timing and output timing of an image generated by a trained model in one embodiment of the present invention. [Figure 7A] 4 is a flowchart showing a process executed by an information processing device in one embodiment of the present invention. [Figure 7B] 4 is a flowchart showing a process executed by an information processing device in one embodiment of the present invention. [Figure 8] FIG. 2 is an explanatory diagram relating to generation of an initial image group in one embodiment of the present invention. [Figure 9] FIG. 2 conceptually illustrates a process for generating multiple images in one embodiment of the present invention. [Figure 10] FIG. 7C is a diagram illustrating an example of the process of step S111 shown in FIG. 7B. [Figure 11] FIG. 7C is a diagram illustrating an example of the process of step S115 shown in FIG. 7B. [Figure 12] FIG. 11 is an explanatory diagram regarding image generation in another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An information processing device according to an embodiment of the present invention, and a method and a program executed by the information processing device, which is an example of a computer, will be described in detail with reference to the drawings.

[0011] Fig. 1 is a diagram showing the external appearance of an electronic musical instrument system 1 according to an embodiment of the present invention. As shown in Fig. 1, the electronic musical instrument system 1 includes an electronic musical instrument 100 and an information processing device 200. The electronic musical instrument 100 and the information processing device 200 are connected via a wired line or a wireless line.

[0012] The electronic musical instrument 100 is an electronic keyboard instrument, such as an electronic keyboard, an electronic piano, a synthesizer, or an electronic organ. The electronic musical instrument 100 may be another type of electronic musical instrument, such as an electronic percussion instrument, an electronic wind instrument, or an electronic string instrument.

[0013] Fig. 2 is a block diagram showing the configuration of the electronic musical instrument 100. As shown in Fig. 2, the electronic musical instrument 100 includes a processor 110, a keyboard 120, an operation unit 130, a key scanner 140, a speaker 150, and a communication unit 160.

[0014] The processor 110 is an element including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 110 controls the electronic musical instrument 100 by reading out programs and data stored in the ROM and using the RAM as a work area.

[0015] When the user performs a key depression operation, the processor 110 performs a sound generation process to generate a musical tone of a pitch corresponding to the depressed key, and outputs this musical tone from the speaker 150. When the user performs a key release operation, the processor 110 performs a sound erasing process to erase the musical tone that is being generated and corresponds to the released key.

[0016] The keyboard 120 has a plurality of white and black keys as performance operators, each of which corresponds to a different pitch.

[0017] The operation unit 130 is composed of, for example, a mechanical type, a capacitive non-contact type, a membrane type, or other key switch, a button, a touch panel, etc. By operating the operation unit 130, the user can, for example, set the tone or adjust the volume.

[0018] The key scanner 140 detects key depression, key release, and key depression speed (velocity) of the keyboard 120, and outputs performance information including the detected pitch information and velocity to the processor 110. Velocity can also be said to be a value indicating the strength of a sound. Pitch information is also called key number, key number, MIDI (Musical Instrument Digital Interface) key, or note number.

[0019] The communication unit 160 includes a wireless and wired unit for communicating with an external device. The communication unit 160 includes an interface that complies with, for example, the MIDI standard. Under the control of the processor 110, the communication unit 160 transmits performance information including pitch information and velocity to the information processing device 200, which is an example of an external device. The performance information has, for example, a data structure that complies with the MIDI standard.

[0020] The information processing device 200 is, for example, a tablet terminal, but may be another type of device, such as a smartphone, a notebook PC (Personal Computer), a portable game machine, a feature phone, or a PDA (Personal Digital Assistant).

[0021] Fig. 3 is a block diagram showing the configuration of the information processing device 200. As shown in Fig. 3, the information processing device 200 includes a processor 210, a flash ROM 220, an HMI (Human Machine Interface) 230, and a communication interface 240. The information processing device 200 is placed on the music stand 170 of the electronic musical instrument 100, for example.

[0022] The processor 210 is an element including a CPU, a RAM, a ROM, etc. The processor 210 loads various programs including an image output program 220A stored in the flash ROM 220 onto the RAM, which is a work area, and controls the information processing device 200 according to the loaded programs.

[0023] The processor 210 is, for example, a single processor or a multi-processor, and includes at least one processor. When multiple processors are included, the processor 210 may be packaged as a single device, or may be configured as multiple devices that are physically separated within the information processing device 200. The processor 210 may be referred to as a "control unit."

[0024] The flash ROM 220 is a non-volatile semiconductor memory that serves as a secondary storage device or an auxiliary storage device. The flash ROM 220 stores programs and data used by the processor 210 to perform various processes, such as an image output program 220A and a trained model 220B.

[0025] The HMI 230 includes, for example, a touch panel, a mechanical type, a capacitive non-contact type, a membrane type, or other key switch, a button, etc. When a user operates the HMI 230, a signal indicating the operation content is output to the processor 210. The processor 210 controls the information processing device 200 based on the input signal.

[0026] The touch panel includes a screen 230A configured with an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, an LED (Light Emitting Diode) panel, etc. The HMI 230 displays, for example, an image acquired by the processor 210 executing an image output program 220A on the screen 230A.

[0027] The communication interface 240 includes wireless and wired units for communicating with external devices. In this embodiment, the communication interface 240 includes an interface that complies with the MIDI standard. The communication interface 240 accepts input of performance information received from the electronic musical instrument 100, which is an example of an external device. The input performance information is stored in the RAM of the processor 210, for example.

[0028] The image output program 220A stored in the flash ROM 220 causes the processor 210 to execute a series of processes, including generating text information including words expressing the impression given by the performance from the performance information received from the electronic musical instrument 100 via the communication interface 240, inputting the generated text information to the trained model 220B, and acquiring and outputting an image generated by the trained model 220B in accordance with the input text information.

[0029] Further, the image output program 220A stored in the flash ROM 220 is a program for outputting a plurality of images at different timings, and causes the processor 210 to execute a series of processes, in which a first evaluation value (e.g., an SD value described later) for evaluating the impression given by the performance is derived based on the first performance information input, a first image corresponding to the first information and stored in a memory area (e.g., a partial area of ​​the RAM of the processor 210, the buffer 212) is output based on first information (e.g., text information TXT1 described later) corresponding to a first value range (e.g., value range R1 described later) in which the first evaluation value is included, and a second image is acquired based on second information (e.g., text information TXT2 described later) corresponding to the second value range, to be output when a second evaluation value (e.g., an SD value described later) derived based on second performance information input after the first performance information is included in a second value range (e.g., value range R2 described later), and the acquired second image is stored in the memory area.

[0030] In other words, the image output program 220A causes the processor 210 to execute a method including the above series of processes.

[0031] It should be noted that "text information" is a text version of words that contain at least one part of speech (verb, adjective, judgemental word, auxiliary verb, noun, adverb, particle, conjunction, interjection, demonstrative word, etc.).

[0032] The trained model 220B is an AI (Artificial Intelligence) program (an example of an image generation program) that generates an image from text information, and is realized by, for example, a neural network. When text information that represents the characteristics of performance information is input, the trained model 220B outputs an image that matches the performance.

[0033] As an example, the trained model 220B is an image generation AI "Stable Diffusion" by Stability AI. The trained model 220B may be another publicly known or well-known image generation AI program that generates an image from text information. Hereinafter, the image generated by the trained model 220B from text information is referred to as "image I."

[0034] The trained model 220B may be provided by cloud computing. That is, the information processing device 200 may not include the trained model 220B. In this case, the processor 210 accesses the trained model 220B placed on a network via the communication interface 240 and provides the trained model 220B with text information representing the characteristics of the performance information. The processor 210 acquires the image I generated by the trained model 220B in response to the input text information via the communication interface 240.

[0035] In this embodiment, by converting performance information into text information, the trained model 220B, which is a publicly known or well-known image generation AI program, can generate an image I, which can then be acquired and output. A variety of images I that match the text information (in other words, the performance) can be generated by the image generation AI program. Therefore, to prevent the user from getting bored, a variety of images I can be output to the HMI 230 and displayed on the screen 230A in accordance with the performance.

[0036] The trained model 220B may be replaced with an image generation program that operates on a simpler algorithm, such as that exemplified in Patent Document 1.

[0037] The processor 210 includes, as functional blocks, an operation receiving unit 210A, a text generating unit 210B, and an image acquiring unit 210C. Each functional block is realized by an image output program 220A executed by the processor 210, which is an example of a computer. Each functional block may be realized in part or in whole by hardware such as a dedicated logic circuit.

[0038] The operation receiving unit 210A receives a user operation on the HMI 230. Illustratively, the operation receiving unit 210A receives a genre selection operation. The genre selection operation is an operation for selecting a genre of an image I to be displayed on the screen 230A in sync with the performance of the electronic musical instrument 100. The user can select the genre of the image I to be displayed on the screen 230A, for example, by touching one genre from the genre list displayed on the screen 230A.

[0039] Selectable genres include, for example, portraits, landscapes, still lifes, portraits, natural history, religious, historical, genre and realistic paintings.

[0040] The text generating section 210B generates, from performance information based on the performance of the electronic musical instrument 100, text information (an example of the first information and the second information) including words expressing the impression given by the performance.

[0041] More specifically, the text generating section 210B analyzes performance information based on the performance of the electronic musical instrument 100, and acquires the feature amount of the performance.

[0042] The text generation unit 210B acquires at least one of the following characteristics of the performance: pitch, velocity, key (e.g., 24 types from C major to B minor), tempo, chord type (e.g., major, minor, sus, arg, dim, 7th, etc.), and number of chords.

[0043] The pitch and velocity can be obtained from the performance information. The key and chord type are not particularly limited, but can be obtained using a method disclosed in, for example, Japanese Patent No. 3211839. The tempo is not particularly limited, but can be obtained using a method disclosed in, for example, Japanese Patent No. 5672280. The number of chords is not particularly limited, but can be obtained based on, for example, the number of simultaneously pressed keys (the number of keys pressed within a specified time).

[0044] The text generating section 210B may obtain, as performance feature quantities, the presence or absence of syncopation, the regularity of beats, chord progressions, and the like, in addition to or instead of the above-mentioned.

[0045] The text generator 210B derives an evaluation value based on the feature amount obtained from the performance information. The evaluation value is a parameter value for evaluating the impression given by the performance information based on the performance of the electronic musical instrument 100.

[0046] By using the quantitative information of the performance features, the impression given by the performance can be obtained as an evaluation value (i.e., as an objective numerical value).

[0047] The text generating section 210B calculates a Semantic Differential (SD) vector based on the feature amount acquired from the performance information.

[0048] The SD vector includes at least one evaluation value (hereinafter, "SD value") for evaluation using the Semantic Differential (SD) method. The SD value is a value for evaluating whether the performance information based on the performance corresponds to one antonym or the other antonym of a given antonym pair (a pair of words with opposite meanings) in a multi-level classification.

[0049] In this embodiment, there are five antonym pairs for evaluating the impression of a performance based on the performance information: "delicate and powerful," "slow and fast," "sad and happy," "heavy and light," and "regular and free." In another embodiment, one to four of the above five antonym pairs may be used as antonym pairs for evaluating the impression of a performance based on the performance information.

[0050] The antonym pairs listed above are only examples. For example, "delicate and powerful" may be replaced by "weak and strong." "heavy and light" may be replaced by "dark and light."

[0051] The text generation unit 210B calculates corresponding SD values ​​(SD1 to SD5) for each of the antonym pairs, "delicate and powerful," "slow and fast," "sad and happy," "heavy and light," and "regular and free," based on the features acquired from the performance information.

[0052] SD1 (an example of a first evaluation value) corresponding to the antonym pair "delicate and powerful" is calculated based on the average velocity in a predetermined period of the performance information (for example, a period corresponding to the most recent beat).

[0053] SD2 (an example of a second evaluation value) corresponding to the antonym pair "slow and fast" is calculated based on the average tempo of the musical performance information during a predetermined period.

[0054] SD3 (an example of a third evaluation value) corresponding to the antonym pair "sad and happy" is calculated based on the average of the tonality (a value obtained by quantifying the tonality) and the tempo of the performance information during a predetermined period.

[0055] SD4 (an example of a fourth evaluation value) corresponding to the antonym pair "heavy and light" is calculated based on the average pitch, number of chords, and tempo of the performance information during a predetermined period.

[0056] SD5 (an example of a fifth evaluation value) corresponding to the antonym pair "regular and free" is calculated based on the variance or standard deviation of the tempo in a predetermined period of the performance information.

[0057] Each SD value ranges from 0 to 255. The text generating section 210B calculates each SD value by converting the range of values ​​that the performance feature can take so that the minimum value is 0 and the maximum value is 255.

[0058] The stronger (more prominent) the impression of one of the antonyms in the antonym pair, the closer the SD value is to 0, and the stronger (more prominent) the impression of the other antonym, the closer the SD value is to 255. For example, if the antonym pair is "delicate and powerful," the closer SD1 is to 0 the more delicate the performance characteristic is, and the closer SD1 is to 255 the more powerful the performance characteristic is.

[0059] Through the above series of processes, the text generating unit 210B generates text information including words expressing the impression given by the performance based on the performance of the electronic musical instrument 100, from the performance information. Note that the text generating unit 210B may convert each SD value into text information corresponding to the SD value, for example, by using a conversion algorithm, without obtaining text information corresponding to the SD value from the text group. In other words, there is a degree of freedom in the method of generating text information from the SD value, and various methods can be adopted.

[0060] The text generating unit 210B generates text information based on the SD value. In the example of this embodiment, the text generating unit 210B acquires text information corresponding to the SD value from a text group that indicates, in multiple stages, whether one or the other of the antonym pair applies to each antonym pair (in other words, each SD value).

[0061] FIG. 4 is a diagram showing the relationship between each SD value and text information.

[0062] In the example of Fig. 4, the text generating unit 210B acquires text information according to the SD value from a text group that indicates whether it corresponds to one side or the other side of an antonym pair on a four-level scale. As shown in Fig. 4, the text groups are "(very) + (one antonym)", "(somewhat) + (one antonym)", "(somewhat) + (the other antonym)", and "(very) + (the other antonym)".

[0063] When the SD value is greater than or equal to 0 and less than a first threshold value (for example, the SD value is 0 to 63), the performance is evaluated as giving the impression of "(very) + (one antonym)", and so the text generation unit 210B acquires the text information "(very) + (one antonym)". do.

[0064] When the SD value is equal to or greater than the first threshold and less than the second threshold (for example, the SD value is between 64 and 127), the performance is evaluated as giving the impression of "(somewhat) + (one antonym)", and so the text generating unit 210B acquires the text information "(somewhat) + (one antonym)".

[0065] When the SD value is greater than or equal to the second threshold and less than the third threshold (for example, the SD value is between 128 and 191), the performance is evaluated as giving the impression of "(somewhat) + (the other antonym)", and so the text generation unit 210B acquires the text information "(somewhat) + (the other antonym)".

[0066] When the SD value is greater than or equal to the third threshold and less than 255 (for example, the SD value is 192 to 255), the text generation unit 210B acquires the text information "(very) + (the other antonym)" because the performance is evaluated to give the impression of "(very) + (the other antonym)".

[0067] As a specific example, when SD1 is 0 to 63, the text generator 210B obtains text information of "very delicate." When SD1 is 64 to 127, the text generator 210B obtains text information of "somewhat delicate." When SD1 is 128 to 191, the text generator 210B obtains text information of "somewhat powerful." When SD1 is 192 to 255, the text generator 210B obtains text information of "very powerful."

[0068] In the example of FIG. 4, the scale of antonym pairs is four levels, but the scale may be two levels, three levels, or five levels or more.

[0069] In other words, the value range of 0 to 255 is divided (i.e., divided into multiple parts) into four value ranges (i.e., value ranges of 0 to 63, 64 to 127, 128 to 191, and 192 to 255), but the value range of 0 to 255 may also be divided into two, three, or five or more value ranges.

[0070] In the example of FIG. 4, the thresholds for the scale of antonym pairs are set to 64 (first threshold), 128 (second threshold), and 192 (third threshold), but these thresholds may be set to other values.

[0071] In the example of Fig. 4, the threshold is the same for each antonym pair. However, depending on the content of the antonym pair, the SD value may tend to be low, tend to be high, or may not tend to be either low or high. Therefore, a different threshold may be set for each antonym pair.

[0072] The image acquisition unit 210C inputs the text information generated by the text generation unit 210B, which represents the characteristics of the performance information (e.g., "somewhat delicate"), into the trained model 220B together with the text information representing the genre selected by the genre selection operation.

[0073] Trained model 220B generates image I of the input genre according to text information.

[0074] As an example, the image acquisition unit 210C inputs "portrait" to the trained model 220B as text information representing the genre, and inputs "slightly weak," "slightly slow," "very sad," "very heavy," and "slightly regular" to the trained model 220B as text information representing the characteristics of the performance information. In this case, the trained model 220B generates, for example, an image I in which a person with low brightness and saturation and a gloomy expression appears. For convenience, the set of five pieces of text information representing the characteristics of the performance information is referred to as an "impression text set TS."

[0075] 5A to 5D show examples of image I generated by trained model 220B.

[0076] FIG. 5A shows example images that are generated when “landscape,” “somewhat powerful,” “somewhat fast,” “very fun,” “very light,” and “somewhat free” are input into trained model 220B.

[0077] FIG. 5B shows example images that are generated when “landscape,” “somewhat delicate,” “somewhat slow,” “very sad,” “very heavy,” and “somewhat regular” are input to trained model 220B.

[0078] FIG. 5C shows example images that are generated when “portrait,” “somewhat powerful,” “somewhat fast,” “very fun,” “very light,” and “somewhat free” are input into trained model 220B.

[0079] FIG. 5D shows example images that are generated when “portrait,” “somewhat delicate,” “somewhat slow,” “very sad,” “very heavy,” and “somewhat regular” are input into trained model 220B.

[0080] For example, when the same text information is input, the trained model 220B may generate different images I each time. This allows a variety of images I to be displayed on the screen 230A so that the user does not get bored.

[0081] The image acquisition unit 210C acquires the image I generated by the trained model 220B, and outputs the acquired image I to the HMI 230 to display it on the screen 230A.

[0082] By using the SD method, the impression given by the performance can be obtained as text information, which is information in a general-purpose format that can be processed by a publicly known or well-known image generation AI program. Using such text information, the trained model 220B, which is a publicly known or well-known image generation AI program, can generate various images I that match the text information (in other words, the performance). This allows the various images I to be output to the HMI 230 and displayed on the screen 230A in accordance with the performance so that the user does not get bored.

[0083] FIG. 6 is a diagram showing an example of the relationship between the acquisition timing and output timing of an image I.

[0084] 6, during period P1, trained model 220B generates image I1-1, and image acquisition unit 210C acquires the generated image I1-1 and stores it in buffer 212. At output timing T1, image acquisition unit 210C acquires image I1-1 generated during period P1 from among images I stored in buffer 212, and outputs it to HMI 230 to display it on screen 230A.

[0085] Similarly, during period P2, trained model 220B generates image I1-2, and image acquisition unit 210C acquires and stores generated image I1-2 in buffer 212. At output timing T2, image acquisition unit 210C acquires image I1-2 generated during period P2 from among images I stored in buffer 212, and outputs it to HMI 230 for display on screen 230A.

[0086] The generation of image I by the trained model 220B requires advanced and massive processing. It may take time for the trained model 220B to generate image I, and for example, image I may not be generated by the next output timing of image I. In addition, it may take time for image I to be generated due to other factors (such as analysis of performance information, delays in communication and signal processing, etc.).

[0087] In the example of FIG. 6, since it takes time for image I to be generated due to image generation and other factors described above, the trained model 220B cannot generate image I during period P3. Therefore, if things continue as they are, the image acquisition unit 210C cannot output image I to the HMI 230 at output timing T3. In this case, for example, image I may not be displayed on the screen 230A, or the same image I may continue to be displayed, albeit temporarily. This may cause a user viewing the screen 230A to feel uncomfortable.

[0088] Therefore, the image acquisition unit 210C stores a plurality of images I in advance in the buffer 212. As a result, even if the trained model 220B cannot generate an image I during the period P3, there is a possibility that the corresponding image I has already been stored in the buffer 212. If the corresponding image I is stored in the buffer 212, the image acquisition unit 210C can output it promptly to the HMI 230. Therefore, the time lag when displaying the image I is easily suppressed, and the image I is easily displayed on the screen 230A in a timely manner. This makes it less likely that a user viewing the screen 230A will feel uncomfortable.

[0089] 7A and 7B are flowcharts showing the processing of the image output program 220A executed by the processor 210 in one embodiment of the present invention. For example, when a user operates the HMI 230 to start the image output program 220A, the processing shown in FIG. 7A starts.

[0090] Processor 210 waits for a genre selection operation (step S101).

[0091] When processor 210 receives a genre selection operation (step S101: YES), it acquires text information (hereinafter, referred to as "genre text T0") representing the genre selected by this operation (step S102). Genre text T0 for each genre is stored in advance in, for example, ROM or flash ROM 220 of processor 210.

[0092] The user can specify in advance the genre to be selected by default, for example, by operating the HMI 230. In this case, the process of waiting for the genre selection operation (step S101) can be omitted from the process shown in FIG.

[0093] The processor 210 stores the initial image group in the buffer 212 (step S103).

[0094] Specifically, the processor 210 sets the initial values ​​of SD1 to SD5 to "128."

[0095] The processor 210 provides the impression text set TS, in which the text information of all antonym pairs is the text information TXT1, together with the genre text T0 to the trained model 220B to generate an image I (an example of a first image). Note that the text information TXT1 is an example of the first information, and is text information corresponding to a value range (an example of the first value range, hereinafter referred to as "value range R1") in which the SD value (here, the value 128) falls.

[0096] The processor 210 further provides the impression text set TS, in which the text information TXT1 is replaced with the text information TXT2 in some or all of the antonym pairs, together with the genre text T0 to the trained model 220B to generate an image I (an example of a second image). Note that the text information TXT2 is an example of the second information, and is text information corresponding to a value range (a second value range, hereinafter referred to as "value range R2") different from the value range R1.

[0097] The value range R2 is, for example, a value range adjacent to the value range R1. For example, when the value range R1 is 128 to 191, the value range R2 is, for example, a value range 64 to 127 adjacent to the value range 128 to 191. In this case, the text information TXT1 and the text information TXT2 have relatively similar expressions. Therefore, the images I of both pieces of text information generated by the trained model 220B are relatively similar.

[0098] Fig. 8 is an explanatory diagram regarding the generation of an initial image group. In Fig. 8, the scale of antonym pairs is divided into 16 stages (value range D1 to D16).

[0099] The value 128 falls within the value range D9. In other words, the value range D9 is the value range R1 in which the initial SD value 128 falls.

[0100] In the example of FIG. 8, value ranges D8 and D10 adjacent to value range D9 are value range R2.

[0101] The processor 210 selects one value range from the value ranges R1 and R2 (here, the value ranges D8 to D10) for each antonym pair, obtains text information TXT1 or TXT2 (i.e., the impression text set TS, which is a set of five pieces of text information) corresponding to the selected value range, and provides it to the trained model 220B to generate the image I. The processor 210 performs this for all combinations.

[0102] That is, there are three value ranges to be selected (value ranges D8 to D10) and five antonym pairs (corresponding SD values ​​are SD1 to SD5). Therefore, the processor 210 calculates 243 (=3 5 ) impression text sets TS are obtained and input to the trained model 220B. As a result, 243 images I are generated in the trained model 220B.

[0103] Supplementally, in the example of Fig. 8, the value range D8, which is the value range R2, is a "slightly" range leaning towards "delicate," "slow," "sad," "heavy," and "regular," whereas the value range D10, which is also the value range R2, is a "slightly" range leaning towards "powerful," "fast," "fun," "light," and "free." Therefore, the image I generated based on the text information TXT2 corresponding to the value range D8 and the image I generated based on the text information TXT2 corresponding to the value range D10 will not be the same image.

[0104] The processor 210 stores the 243 images I generated by the trained model 220B as an initial image group in the buffer 212. Each image I is stored in the buffer 212 in association with, for example, the impression text set TS (or SD1 to SD5) that was the basis for image generation.

[0105] Fig. 9 conceptually shows the process of generating 243 images I. In Fig. 9, "R1" indicates the value range R1. "R2[right]" indicates the value range R2 to the right of the value range R1. "R2[left]" indicates the value range R2 to the left of the value range R1.

[0106] As shown in Fig. 9, for each of the five SD values ​​(SD1 to SD5), there are three value ranges to be selected (R1, R2 [right], R2 [left]). Therefore, the combinations of text information for SD1 to SD5 are 3 x 3 x 3 x 3 x 3. The trained model 220B generates images I for all combinations. Therefore, in this embodiment, 243 images I are obtained.

[0107] Processor 210 waits for input of performance information from electronic musical instrument 100 (step S104). The performance information includes information such as the pitch, velocity, and timing of a pressed or released key.

[0108] The performance information is not limited to information obtained by the user playing (performing) electronic musical instrument 100. The performance information may be MIDI data obtained from an external input. In other words, even when pre-recorded music data or the like is used, a variety of images I that match the music can be displayed on screen 230A.

[0109] When performance information (an example of first performance information) is input from electronic musical instrument 100 via communication interface 240 (step S104: YES), processor 210 analyzes the input performance information and acquires features of the performance (step S105). The acquired features include, for example, pitch, velocity, tonality, tempo, chord type, and number of chords.

[0110] Processor 210 determines whether or not the user's performance of electronic musical instrument 100 has ended (step S106). For example, if a first time has not elapsed since the previous input of performance information, processor 210 determines that the performance has not ended (step S106: NO), and proceeds to step S107. If a first time has elapsed since the previous input of performance information, processor 210 determines that the performance has ended (step S106: YES), and proceeds to step S119. The elapsed time is measured, for example, by a counter built into processor 210.

[0111] In step S107, processor 210, for example, counts the value of a counter and determines whether or not a second time has elapsed since the previous execution of output processing. "The previous execution of output processing" refers to the previous timing at which image I was output to HMI 230. Processor 210 repeatedly executes the processes of steps S106 to S107 while the performance continues and until the second time has elapsed since the previous execution of output processing.

[0112] The "second time" that is the criterion for the process of step S107 is related to the update frequency of the image I displayed on the screen 230A. The user can set the update frequency of the image I, for example, by operating the HMI 230. When the update frequency setting is changed, the length of this "second time" also changes.

[0113] If a second time has elapsed since the previous input of performance information (step S107: YES), processor 210 calculates SD1 to SD5 based on the features acquired in step S105 in order to output image I corresponding to the currently analyzed performance information to HMI 230 (step S108).

[0114] As described above, each image I is stored in the buffer 212 in association with, for example, the impression text set TS (or SD1 to SD5) from which the image was generated. The processor 210 refers to this association information and determines whether the image I corresponding to the SD value calculated in step S108 is stored in the buffer 212 (step S109).

[0115] It should be noted that image I already stored in buffer 212 is an example of a first image.

[0116] If image I corresponding to the SD value calculated in step S108 is stored in buffer 212 (step S109: YES), processor 210 reads this image I from buffer 212 and outputs it to HMI 230 (step S110). As a result, image I expressing the impression of the performance is promptly displayed on screen 230A.

[0117] By storing a plurality of images I in advance in the buffer 212, it is possible with a high probability to quickly output the image I expressing the impression of the performance to the HMI 230 and display it on the screen 230A. In other words, it becomes easier to suppress the time lag when displaying the image I, and it becomes easier to display the image I on the screen 230A in a timely manner.

[0118] The processor 210 updates the value ranges R1 and R2 (step S111).

[0119] Specifically, for each antonym pair, the processor 210 sets the value range that includes the SD value calculated in step S108 as a new value range R1, and sets the value ranges on both sides of the new value range R1 as new value range R2.

[0120] Fig. 10 is a diagram for explaining an example of the process of step S111. In the example of Fig. 10, the value ranges R1 and R2 in Fig. 8 are set to the value ranges R1 and R2 before the update.

[0121] In Fig. 10, the value range R1 that includes SD1, SD3, and SD5 is the same value range D9 as in Fig. 8. The value range R1 that includes SD2 and SD4 has changed to value ranges D10 and D8, respectively. In this case, the processor 210 updates the value ranges R1 and R2 only for SD2 and SD4.

[0122] Specifically, for SD2, the processor 210 sets the value range D10 as the new value range R1, and the adjacent value ranges D9 and D11 as the new value range R2. For SD4, the processor 210 sets the value range D8 as the new value range R1, and the adjacent value ranges D7 and D9 as the new value range R2.

[0123] The processor 210 acquires the impression text set TS according to the updated value ranges R1 and R2 (step S112), and inputs it to the trained model 220B together with the genre text T0 (step S113).

[0124] The processor 210 does not have to acquire all (243 combinations) of impression text sets TS according to the updated value ranges R1 and R2. The processor 210 may acquire only a part of the impression text sets TS, excluding impression text sets TS that overlap with the previous time, and input them to the trained model 220B. In this case, the number of images generated by the trained model 220B can be reduced, so that the processing load of the trained model 220B can be suppressed.

[0125] The processor 210 acquires the image I from the trained model 220B (step S114), and updates the buffer 212 with the acquired image I (step S115).

[0126] In this manner, the processor 210 obtains at least one new image I based on the text information TXT1 and TXT2 (i.e., the impression text set TS, which is a set of five pieces of text information), and updates the buffer 212 with the obtained image I.

[0127] The newly acquired image I is an example of a second image that is output when an SD value (an example of a second evaluation value) derived based on performance information (an example of second performance information) input after the immediately preceding performance information (an example of first performance information, see step S104) is included in value range R2 (second value range).

[0128] FIG. 11 is a diagram for explaining the update process (step S115) of the buffer 212 when the value ranges R1 and R2 shown in FIG. 8 are updated to the value ranges R1 and R2 shown in FIG.

[0129] 11, as a result of the value range R1 in which SD2 falls changing from value range D9 to value range D10, in SD2, value range D11 is added as value range R2 and value range D8 is removed from value range R2. As a result of the value range R1 in which SD4 falls changing from value range D9 to value range D8, in SD4, value range D7 is added as value range R2 and value range D10 is removed from value range R2.

[0130] Therefore, the processor 210 acquires only the impression text set TS to be added (step S112), inputs them to the trained model 220B (step S113), and acquires the image I (step S114).

[0131] The impression text sets TS to be added are an impression text set TS including text information TXT2 corresponding to the value range R11 of SD2 (e.g., “somewhat fast”), and an impression text set TS including text information TXT2 corresponding to the value range D7 of SD4 (e.g., “somewhat heavy”).

[0132] The processor 210 overwrites and updates the image I to be excluded with the image I newly acquired in step S114 (step S115).

[0133] The images I to be excluded are images I generated based on an impression text set TS including text information TXT2 corresponding to the value range R8 of SD2 (e.g., "somewhat slowly"), and images I generated based on an impression text set TS including text information TXT2 corresponding to the value range R10 of SD4 (e.g., "somewhat lightly").

[0134] After updating buffer 212, processor 210 returns to the process of step S104 and waits for the input of the next performance information from electronic musical instrument 100.

[0135] If the image I corresponding to the SD value calculated in step S108 is not stored in the buffer 212 (step S109: NO), the processor 210 first causes the trained model 220B to generate only one image I in order to quickly display the image I on the screen 230A.

[0136] Specifically, the processor 210 acquires the impression text set TS corresponding to the SD value calculated in step S108 (step S116), and inputs it together with the genre text T0 to the trained model 220B (step S117).

[0137] Processor 210 acquires image I corresponding to the SD value calculated in step S108 from trained model 220B (step S118), and outputs it to HMI 230 (step S110). As a result, image I expressing the impression of the performance is displayed on screen 230A.

[0138] After that, in order to update the buffer 212, the processor 210 executes the processes of steps S111 to S115.

[0139] In this way, when performance information (an example of first performance information, see step S104) is input, if the corresponding image I (an example of the first image) is not stored in buffer 212 (an example of a memory area) (step S109: NO), processor 210 acquires image I (an example of a third image) corresponding to the SD value (an example of a first evaluation value) (steps S116 to S118), outputs the acquired image I (step S110), and after outputting image I, acquires a new image I (an example of the second image) and stores it in buffer 212 (steps S111 to S115).

[0140] It is possible that it takes time to generate image I in steps S116 to S118. Therefore, while processor 210 executes the processes of steps S116 to S118, it may provisionally display, on screen 230A, an image I (approximate image) corresponding to another SD value that is approximate to the SD value calculated in step S108 from among images I stored in buffer 212, until the process of step S118 is completed.

[0141] In addition, the processor 210 starts counting after performance information is input, for example, and if the count value reaches a predetermined value but image I cannot be acquired, it determines that it is taking a long time for the trained model 220B to generate image I.

[0142] Since an approximate image that is close to the image that is originally desired to be displayed is output, an image that is relatively suitable for playing and does not seem unnatural is displayed on screen 230A.

[0143] When processor 210 determines that the user's performance of electronic musical instrument 100 has ended (step S106: YES), it outputs the last image I to HMI 230, causing it to be displayed on screen 230A.

[0144] To obtain the final image I, the processor 210 calculates, for example, an average value of the SD values ​​calculated from the start of performance to the end of performance for each of SD1 to SD5. The processor 210 obtains text information based on each of the average values ​​of SD1 to SD5, and inputs the genre text T0 and the impression text set TS to the trained model 220B. The processor 210 obtains the image I generated by the trained model 220B as the final image I.

[0145] In addition, the present invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. In addition, the functions executed in the above-mentioned embodiment may be implemented in appropriate combinations as much as possible. The above-mentioned embodiment includes various steps, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if an effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention.

[0146] The antonym pairs for evaluating the impression of a performance based on the performance information are not limited to the above five. For example, another antonym pair such as "random and accurate" may be adopted.

[0147] The SD value corresponding to the antonym pair "random and accurate" is calculated based on the degree of matching of the performance with the teacher data, for example. The closer the performance of the user on the electronic musical instrument 100 is to the performance indicated by the teacher data, the higher the SD value (the text information is "very accurate"), and the more different the performance is from the performance indicated by the teacher data, the lower the SD value (the text information is "very random").

[0148] The SD value corresponding to the antonym pair "random and accurate" may be calculated by a method that does not use teacher data. For example, the SD value may be calculated according to an unusual playing style, the degree of such playing, etc.

[0149] For example, when performance information of an unusual performance is input, such as when a large number of notes (e.g., 11 notes) are input simultaneously or when musical notes of multiple adjacent keys (e.g., five or more consecutive keys) are input simultaneously, the SD value corresponding to the antonym pair "random and accurate" becomes low. When performance information of a performance within the category of a normal performance is input, the SD value corresponding to the antonym pair "random and accurate" becomes high.

[0150] When the trained model 220B receives the input "very random," it generates an image I that resembles, for example, a random doodle.

[0151] In the above embodiment, the value range R2 is a value range adjacent to the value range R1. For example, unless there is a change such as a change in the melody of the performance, the SD value (in other words, the value range) does not change significantly in a short period of time. Therefore, by setting the value range R2 adjacent to the value range R1, it becomes easier to cause the trained model 220B to generate in advance the image I to be output at the next output timing (i.e., the image I corresponding to the SD value calculated in step S108). Therefore, the time lag when displaying the image I is easily suppressed, and the image I is easily displayed on the screen 230A in a timely manner.

[0152] On the other hand, when a change occurs, such as a change in the melody being played, the SD value (or in other words, the value range) may change significantly in a short period of time. If the SD value changes significantly in a short period of time, the value range in which this SD value falls will be significantly different from value range R1.

[0153] Thus, in another embodiment, value range R2 may be a value range that is located between value range R1 and at least one other value range.

[0154] FIG. 12 is an explanatory diagram regarding image generation in another embodiment.

[0155] In another embodiment, as shown in Fig. 12, value range R2 is a value range that is located between value range R1 and at least one value range in between. Specifically, value range R2 is a value range D1 and D16 that is located between value range R1 and at least seven value ranges in between.

[0156] Even if a change occurs, such as a change in the melody being played, causing the value range into which the SD value falls to change significantly (a case in which the value range changes to a value range that is two or more steps away, such as changing from value range D9 to value range D1), image I, which is to be output at the next output timing, can be quickly output to HMI 230, and image I can be displayed on screen 230A in a timely manner.

[0157] In the example of Figure 12, if the value range into which the SD value falls changes, for example, from value range D9 to value range D3, the corresponding image I is not stored in buffer 212, and therefore processor 210 cannot output image I that is optimal for the performance information.

[0158] In this case, while executing the processing of steps S116 to S118, as in the above embodiment, processor 210 may provisionally display on screen 230A an image I corresponding to text information corresponding to a value range close to value range D3 (here, value range D1) from among images I stored in buffer 212, until the processing of step S118 is completed.

[0159] Even when the value range in which the SD value falls changes significantly, an image I that is close to the image I that is optimal for the performance information and is relatively suitable for the performance and does not seem unnatural can be displayed on the screen 230A. Since an image close to the image I that is originally intended to be displayed can be displayed on the screen 230A, the user can feel a change, such as a change in the melody of the performance, by watching the transition of the image I displayed on the screen 230A. [Explanation of symbols]

[0160] 1: Electronic musical instrument system 100: Electronic instruments 110: Processor 112: Buffer 120: Keyboard 130: Operation section 140: Key scanner 150: Speaker 160: Communications Department 170:Music stand 200: Information processing device 210: Processor 210A: Operation reception section 210B: Text generation unit 210C: Image acquisition section 220: Flash ROM 220A: Image output program 220B: Trained model 230A: Screen 240: Communication interface

Claims

1. A program for outputting a plurality of images at different times, deriving a first evaluation value for evaluating an impression given by the performance based on the input first performance information; outputting a first image corresponding to the first information, the first image being stored in a storage area, based on first information corresponding to a first value range including the first evaluation value; acquiring a second image based on second information corresponding to the second value range, the second image being output when a second evaluation value derived based on second performance information input after the first performance information is included in the second value range; storing the acquired second image in the storage area; The process is executed by a computer. program.

2. The second value range is a value range adjacent to the first value range. The program according to claim 1.

3. The second value range is a value range located between the first value range and at least one of the value ranges. The program according to claim 1.

4. the first information and the second information are text information including words expressing impressions given by the first performance information and the second performance information, respectively; The program according to claim 1.

5. inputting said text information into an image generating program; When the image generating program generates the image based on the input text information, the generated image is acquired; storing the acquired image in the storage area; The program according to claim 4.

6. Analyzing the performance information to obtain a characteristic amount of the performance; Deriving the evaluation value based on the acquired feature amount. The program according to claim 1.

7. The feature amount includes at least one of pitch, velocity, key, tempo, and number of chords. The program according to claim 6.

8. When the first performance information is input, if the first image is not stored in the storage area, a third image corresponding to the first evaluation value of the first performance information is acquired; Outputting the acquired third image; After outputting the third image, acquiring the second image and storing it in the storage area; The program according to claim 1.

9. A method for outputting a plurality of images at different times, comprising the steps of: deriving a first evaluation value for evaluating an impression given by the performance based on the input first performance information; outputting a first image corresponding to the first information, the first image being stored in a storage area, based on first information corresponding to a first value range including the first evaluation value; acquiring a second image based on second information corresponding to the second value range, the second image being output when a second evaluation value derived based on second performance information input after the first performance information is included in the second value range; storing the acquired second image in the storage area; The process is executed by a computer. method.

10. An information processing device that outputs a plurality of images at different times, deriving a first evaluation value for evaluating an impression given by the performance based on the input first performance information; outputting a first image corresponding to the first information, the first image being stored in a storage area, based on first information corresponding to a first value range including the first evaluation value; acquiring a second image based on second information corresponding to the second value range, the second image being output when a second evaluation value derived based on second performance information input after the first performance information is included in the second value range; storing the acquired second image in the storage area; A control unit for executing a process is provided. Information processing device.

Images