Music visualisation method

Abstract

A method of producing a graphical representation of a musical composition, the method including: (a) receiving music data indicative of at least the harmony and rhythm of the musical composition; (b) processing the music data to provide a set of harmonic data including a time series with one or more common harmonic features of the musical composition; (c) processing the music data to provide a set of rhythmic data including a time series with one or more common rhythmic features of the musical composition; and (d) forming a graphical representation of the musical composition including a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data. The first and second shaded regions of the grid may overlap.

Description

[0001] MUSIC VISUALISATION METHOD

[0002] FIELD OF THE INVENTION

[0003] This invention relates to a method and computer system for producing a graphical representation of a musical composition.

[0004] BACKGROUND

[0005] Programmable electronic devices or software that record, playback and edit musical compositions typically display the musical compositions to users on a screen. Such devices or software include a musical instrument digital interface (MIDI) sequencer or a digital audio workstation (DAW). Musical notes can be graphically displayed to the user in a "piano roll" format on the screen. In this format, the notes appear as bars of varying lengths typically superimposed on a lined grid - the longer the bar length, the longer the note duration. The appearance is reminiscent of piano rolls made of paper with holes punched into them for operating self-playing pianos.

[0006] As a musical notation system, the piano roll format has several advantages over traditional sheet music. In particular, the piano roll format is more accessible to a broader range of users as it is more intuitive to users who cannot read sheet music. However, the piano roll format also suffers from several disadvantages. First, the grid lines create visual clutter but may not be able to be removed because the style of each line denotes the music's key. Secondly, only a limited number of line styles (e.g., varying the width and / or colour of a line) are available to differentiate between different keys.

[0007] It is an object of at least preferred embodiments of the present invention to provide a method and computer system for producing a graphical representation of a musical composition that overcomes the abovementioned disadvantages, and / or to at least provide the public with a useful alternative.

[0008] SUMMARY OF THE INVENTION

[0009] According to a first aspect of the invention, there is provided a method of producing a graphical representation of a musical composition, the method comprising:

[0010] (a) receiving music data indicative of at least the harmony and rhythm of the musical composition; (b) processing the music data to provide a set of harmonic data comprising a time series with one or more common harmonic features of the musical composition;

[0011] (c) processing the music data to provide a set of rhythmic data comprising a time series with one or more common rhythmic features of the musical composition; and

[0012] (d) forming a graphical representation of the musical composition comprising a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data.

[0013] In one embodiment, the first and second shaded regions of the grid partially or fully overlap each other.

[0014] In one embodiment, the one or more common harmonic features comprise one or more properties regarding a consistent key of at least a portion of the musical composition, and the one or more common rhythmic features comprise one or more properties regarding a consistent measure of at least a portion of the musical composition.

[0015] In one embodiment, the grid comprises a plurality of cells, at least one of the cells having a border surrounding an interior portion of the cell, the interior portion comprising at least one of the first shaded regions of the grid and at least one of the second shaded regions of the grid.

[0016] In one embodiment, the grid comprises a pitch axis and a time axis.

[0017] In one embodiment, pitch is represented by one or more rows in the grid and time is represented by one or more columns in the grid.

[0018] In one embodiment, the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for a pitch domain and the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a time domain.

[0019] In one embodiment, the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for an octave. In one embodiment, the intensity function for the octave comprises a weighted base function for each semitone in the octave.

[0020] In one embodiment, the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a bar.

[0021] In one embodiment, the intensity function for the bar comprises a weighted base function for each beat in the bar.

[0022] In one embodiment, the music data is also indicative of the melody of the musical composition.

[0023] In one embodiment, one or more notes or a sonograph are superimposed onto the grid.

[0024] In one embodiment, each of the one or more notes is represented as a cylinder with a tapered end.

[0025] In one embodiment, the music data is also indicative of the structure of the musical composition.

[0026] In one embodiment, the method further comprises processing the music data to provide a set of structural data comprising a time series with one or more common structural features of the musical composition.

[0027] In one embodiment, the one or more common structural features comprise one or more properties regarding a consistent phrase of at least a portion of the musical composition.

[0028] In one embodiment, the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a phrase.

[0029] In one embodiment, the method is computer implemented.

[0030] In one embodiment, the steps in the method are implemented by a processor or computer system.

[0031] According to a second aspect of the invention, there is an electronically-implemented method comprising software code or coded instructions that are executable or implemented by a computer, processor, or controller to carry out the first aspect of the invention.

[0032] According to a third aspect of the invention, there is provided a non-transitory computer- readable medium having stored thereon computer executable instructions that, when executed on a processing device or devices, cause the processing device or devices to perform a method of producing a graphical representation of a musical composition, the method comprising:

[0033] (a) receiving music data indicative of at least the harmony and rhythm of the musical composition;

[0034] (b) processing the music data to provide a set of harmonic data comprising a time series with one or more common harmonic features of the musical composition;

[0035] (c) processing the music data to provide a set of rhythmic data comprising a time series with one or more common rhythmic features of the musical composition; and

[0036] (d) forming a graphical representation of the musical composition comprising a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data.

[0037] According to a fourth aspect of the invention, there is provided a computer system configured to produce a graphical representation of a musical composition, the system comprising:

[0038] (a) a microprocessor;

[0039] (b) a data storage device for storing music data indicative of at least the harmony and rhythm of the musical composition; and

[0040] (c) a visual output device for displaying the graphical representation of the musical composition; wherein the microprocessor is configured to:

[0041] (i) receive music data indicative of at least the harmony and rhythm of the musical composition;

[0042] (ii) process the music data to provide a set of harmonic data comprising a time series with one or more common harmonic features of the musical composition;

[0043] (iii) process the music data to provide a set of rhythmic data comprising a time series with one or more common rhythmic features of the musical composition; (iv) form a graphical representation of the musical composition comprising a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data; and

[0044] (v) display the grid on the visual output device.

[0045] In one embodiment, the first and second shaded regions of the grid partially or fully overlap each other.

[0046] In one embodiment, the one or more common harmonic features comprise one or more properties regarding a consistent key of at least a portion of the musical composition, and the one or more common rhythmic features comprise one or more properties regarding a consistent measure of at least a portion of the musical composition.

[0047] In one embodiment, the grid comprises a plurality of cells, at least one of the cells having a border surrounding an interior portion of the cell, the interior portion comprising at least one of the first shaded regions of the grid and at least one of the second shaded regions of the grid.

[0048] In one embodiment, the grid comprises a pitch axis and a time axis.

[0049] In one embodiment, pitch is represented by one or more rows in the grid and time is represented by one or more columns in the grid.

[0050] In one embodiment, the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for a pitch domain and the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a time domain.

[0051] In one embodiment, the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for an octave.

[0052] In one embodiment, the intensity function for the octave comprises a weighted base function for each semitone in the octave.

[0053] In one embodiment, the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a bar. In one embodiment, the intensity function for the bar comprises a weighted base function for each beat in the bar.

[0054] In one embodiment, the music data is also indicative of the melody of the musical composition.

[0055] In one embodiment, one or more notes or a sonograph are superimposed onto the grid.

[0056] In one embodiment, each of the one or more notes is represented as a cylinder with a tapered end.

[0057] In one embodiment, the music data is also indicative of the structure of the musical composition.

[0058] In one embodiment, the microprocessor is further configured to process the music data to provide a set of structural data comprising a time series with one or more common structural features of the musical composition.

[0059] In one embodiment, the one or more common structural features comprise one or more properties regarding a consistent phrase of at least a portion of the musical composition.

[0060] In one embodiment, the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a phrase.

[0061] In one embodiment, the system further comprises an input device and an audio output device.

[0062] In one embodiment, the system further comprises a graphical user interface comprising a main window, a sub-window, and a timeline, wherein the main window shows a more detailed view of the graphical representation of the musical composition than the subwindow.

[0063] In one embodiment, the graphical user interface is configured to synchronise the graphical representation of the musical composition with playback of the musical composition. In one embodiment, the sub-window contains a rectangular box highlighting a portion of the graphical representation of the musical composition shown in the main window, the graphical user interface being configured to allow a user to adjust a focus position indicated by the box.

[0064] In one embodiment, the graphical user interface is configured to allow a user to edit the music data.

[0065] In this specification, where reference has been made to patent specifications, other external documents, or other sources of information, this is generally for the purpose of providing a context for discussing the features of the invention. Unless specifically stated otherwise, reference to such external documents or such sources of information is not to be construed as an admission that such documents or such sources of information, in any jurisdiction, are prior art or form part of the common general knowledge in the art.

[0066] The term 'comprising' as used in this specification and claims means 'consisting at least in part of'. When interpreting statements in this specification and claims which include the term 'comprising', other features besides the features prefaced by this term in each statement can also be present. Related terms such as 'comprise' and 'comprised' are to be interpreted in a similar manner.

[0067] As used herein, the term 'musical composition' refers to any piece or work of music, whether partially or fully complete, and includes pieces or works of music that are instrumental, vocal, or both.

[0068] As used herein, the term 'music data' refers to data about a musical composition used to produce a graphical representation of the musical composition. The music data comprises semantic data, for example, semantic data relating to one or more of a musical composition's structure, rhythm, harmony, and melody. The music data may optionally also include audio data or visual data.

[0069] As used herein, the term '(s)' following a noun means the plural and / or singular form of that noun.

[0070] As used herein, the term 'and / or' means 'and' or 'or', or where the context allows both. It is intended that reference to a range of numbers disclosed herein (for example, 1 to 10) also incorporates reference to all rational numbers within that range (for example, 1, 1.1, 2, 3, 3.9, 4, 5, 6, 6.5, 7, 8, 9 and 10) and also any range of rational numbers within that range (for example, 2 to 8, 1.5 to 5.5 and 3.1 to 4.7) and, therefore, all sub-ranges of all ranges expressly disclosed herein are hereby expressly disclosed. These are only examples of what is specifically intended and all possible combinations of numerical values between the lowest value and the highest value enumerated are to be considered to be expressly stated in this application in a similar manner.

[0071] To those skilled in the art to which the invention relates, many changes in construction and widely differing embodiments and applications of the invention will suggest themselves without departing from the scope of the invention as defined in the appended claims. The disclosures and the descriptions herein are purely illustrative and are not intended to be in any sense limiting.

[0072] This invention may also be said broadly to consist in the parts, elements and features referred to or indicated in the specification of the application, individually or collectively, and any or all combinations of any two or more said parts, elements or features, and where specific integers are mentioned herein which have known equivalents in the art to which this invention relates, such known equivalents are deemed to be incorporated herein as if individually set forth.

[0073] The invention consists in the foregoing and also envisages constructions of which the following gives examples only.

[0074] BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention will now be described by way of example only and with reference to the accompanying drawings in which:

[0076] Figures 1A-1D show four prior art configurations of time (t) and pitch (p) axes for piano roll scores.

[0077] Figure 2 shows a block diagram of a computer system according to an embodiment of the present invention.

[0078] Figure 3 shows a flow chart depicting a method according to an embodiment of the present invention.

[0079] Figures 4A-4C show three graphs of an example intensity base function with different parameter values depending upon the semitone. Figure 5 shows a graph of an example intensity base function for an octave resulting from combining a base function spanning the whole octave with a weighted base function for each of the 12 semitones within the octave, such as those shown in Figures 4A-4C.

[0080] Figure 6 shows a column representing the key of C major, with each of the 13 spacedapart horizontal lines in the column representing one of the 12 pitches in the key or a reference pitch and the shading intensity of each line being set by a weighted base function for each pitch.

[0081] Figure 7 shows a row representing a 4 / 4 measure, with each of the five spaced-apart vertical lines representing a beat and the shading intensity of each line being set by a weighted base function for each beat.

[0082] Figure 8 shows a graphical representation of a musical composition as a grid of columns and rows, including the column of Figure 6 and the row of Figure 7, with pitch along the vertical axis and time along the horizontal axis.

[0083] Figure 9 shows notes superimposed onto a portion of the grid of Figure 8.

[0084] Figure 10 shows a schematic diagram of an example graphical user interface for synchronising the graphical representation of the musical composition with playback.

[0085] DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

[0086] A preferred embodiment of a method for producing a graphical representation of a musical composition will be described in the context of a preferred embodiment of a computer system 100 configured to do so. Figure 2 shows that the computer system 100 comprises an optional input device 110 for receiving a user's input, a visual output device 120 for displaying the graphical representation of the musical composition to the user, an optional audio output device 130 for producing sound or an audio signal, a microprocessor 140, a data storage device 150 for storing music data, and a programmable memory unit 160 for storing a computer program to control the operation of the microprocessor 140 and implement the steps in the method. The microprocessor 140 is configured to interface with the data storage device 150 to import and receive music data stored in the device. The microprocessor 140 is also configured to read and write data to the programmable memory unit 160.

[0087] The computer system 100 may be any suitable system, including but not limited to a MIDI sequencer, a DAW, a karaoke machine, a disc jockey (DJ) controller, and a personal computer, laptop, tablet, or smartphone for creating, producing, or editing music or playing music-based video games. Depending upon the availability of input devices in the computer system 100, the optional input device 110 may comprise one or more of a mouse, keyboard, touchscreen, physical controls such as buttons or dials on specialised music hardware, and a MIDI controller. The visual output device 120 and the optional audio output device 130 will depend on the availability of output devices in the computer system 100. The visual output device 120 comprises a display, such as a liquid crystal display (LCD) or a touchscreen. The optional audio output device 130 comprises one or more speakers for converting audio signals into sound or an external MIDI device that can turn a MIDI signal into an audio signal.

[0088] The input device 110 and the audio output device 130, shown in dashed lines in Figure 2, are advantageous but not essential to the computer system 100. In one embodiment of the computer system 100, music data relating to a musical composition is imported from the data storage device 150, and the graphical representation of the musical composition is produced for output on the visual output device 120 according to the method of the invention. However, playing the music data with synchronised highlighting in the musical score is desirable where the music data includes audio data. Accordingly, in another embodiment of the computer system 100, the system includes an audio output device 130 to produce the sound and an input device 110 for controlling the playback. It can also be desirable to allow navigation independent of the playback, for example, to inspect details of the musical score. Navigation is usually implemented by panning and zooming with a mouse or touchscreen as the input device 110, but it can also be realised with physical controls on specialised music hardware. In one embodiment, the computer system 100 allows the user to edit the music data, for example, to create, delete or modify notes, key changes, and phrases. This embodiment requires a complex Graphical User Interface (GUI), which usually uses a combination of mouse and text input via a keyboard or a touchscreen with a virtual keyboard as the input device 110. Alternatively, an external MIDI controller, such as a MIDI keyboard, can be used as the input device 110 for more convenient note input.

[0089] A flow chart illustrating the steps in the method 200 for producing a graphical representation of a musical composition is shown in Figure 3. At steps 210 or 220, the music data is imported from a file (e.g., MIDI or MusicXML file format) or created by the user with music editing software and stored in the data storage device 150. The music data comprises semantic data relating to one or more of the musical composition's structure, rhythm, harmony, and melody. In the method 200 shown in Figure 3, the music data comprises all four semantic data types. The music data does not need to contain any signal-based audio data. The semantic data relating to the structure of the musical composition specifies each phrase's start time, duration, and label (e.g., "intro", "verse", or "chorus"). The semantic data relating to the rhythm of the musical composition measures changes in the placement of sounds in time and in the nominator or denominator of the measure, e.g., a 4 / 4 measure. The semantic data relating to the harmony of the musical composition is concerned with key changes, specifically the time, type (e.g., major or minor), or base pitch (e.g., C). Finally, the semantic data relating to melody specifies each note's start time, duration, and pitch (e.g., C# 5 or numeric equivalent) in the composition. Depending upon the amount of semantic data in the file imported at step 210, it may be necessary for the user to add supplementary semantic data at step 220. For example, in MIDI files, rhythmic and harmonic semantic data is only optionally stored, and only major or minor keys can be expressed.

[0090] At steps 230, 240 and 250, the duration of the music data is divided into structural data, rhythmic data, and harmonic data, each of these three data sets comprising an interval or time series having one or more common or constant structural, rhythmic, or harmonic features of the musical composition, such as properties regarding phrase (step 230), measure (step 240) or key (step 250). At steps 260, 270, 280, 290 and 300, intensity base functions are used to calculate an intensity function per phrase (step 260), per beat (step 270), per bar (step 280), per octave (step 290), and per semitone (step 300). An intensity base function is used to map intervals in the time or pitch domain to intensity values. For scalability, it is feasible to use a normalised function. Some desirable properties of an intensity base function are listed below. First, it is desirable that the intensity base function y - f x) be over real numbers, where x e [0,1] and y e [0,1], Secondly, as shown in Figures 3A-C, it is also desirable that there be a single maximum xj in the middle with (x = 1, preferably at x = 0.5. This is the maximum intensity and corresponds to the visual impression of a ridge in the graphical representation of the musical composition. Thirdly, the intensity base function contains weight parameters o0and which control the values at the start and end of the function, respectively, so that / (o) = 1 - o0and / (l) = e[0, 1]- These values are the intensities corresponding to the visual impression of grooves on either side of the ridge with a higher weight parameter corresponding to a deeper groove. Fourthly, the intensity base function should be strictly monotonically increasing for x < xj and strictly monotonically decreasing for x > xlrwith the exception that if o0= 0 or wj = o, the function remains constant at y - 1. This corresponds to the visual impression of no groove being present. Finally, the intensity base function contains an additional parameter a to control the curvature of the function between its maximum and minima. An example of an intensity base function that meets the above criteria is contained in equations 1 and 2: y - 1 - o0■ (2 ■ (0.5 - x))“{o < x < 0.5} Equation 1 y - 1 - a)1■ (2 ■ (x - 0.5))“{0.5 < x < 1} Equation 2 Figures 4A-4C show graphs of the intensity base function defined by equations 1 and 2 (e.g., for a semitone) across the function's domain [0, 1] with different parameter values (e.g., depending upon the semitone). In Figure 4A, the weight parameters are 1 and 0.5, respectively, to set the start (0) and end (0.5) values of the function, and the curvature parameter a is 2. In Figure 4B, the weight parameters o0and are 0 and 1, respectively, to set the start (1) and end (0) values of the function, and the curvature parameter a is 3. In Figure 4C, the weight parameters o0 are 0.25 and 1, respectively, to set the start (0.75) and end (0) values of the function, and the curvature parameter a is 4. An additive combination of these intensity base functions can create grids combining different resolutions. It will be appreciated that equations 1 and 2 provide an example of an intensity base function and that other intensity base functions with similar properties can also be used.

[0091] A musical key is defined by its fundamental pitch and by the set of 12 semitones within an octave that are being used. The same pattern is repeated within each octave. Therefore, an intensity function for an octave can be created by combining a base function spanning the whole octave with a weighted base function for each semitone. Figure 5 shows a graph of an example intensity base function for an octave resulting from combining a base function spanning the whole octave (40%) from step 290 with a weighted base function for each of the 12 semitones within the octave (60%) from step 300, such as those shown in Figures 4A-4C. The intensity per octave and the intensity per semitone are combined using a blending function to produce an intensity function for the pitch domain at step 340.

[0092] Figure 6 shows a column 400 representing the key of C major, with the 13 parallel, spaced-apart horizontal lines 410-421 representing the 12 pitches in the key (initial pitch 410 is shown twice). The shading intensity of each line is set by a weighted base function for each pitch, with the harmonic weight given to each pitch depending upon the key. The harmonic weights for each pitch within the octave are provided for each key at step 320. Table 1 below shows example weights for the pitches in C major. In this embodiment, the chromatic pitches not belonging to the scale (i.e., the black keys on a piano) are assigned a lower weight, corresponding to the impression of a shallower groove. Table 1 : Harmonic weights for each of the 12 pitches in C major as depicted in Figure 6

[0093] The weights are only specific to the type of scale but not to its fundamental, so the same weights can be applied to any major scale by shifting the pitches accordingly. The choice of blending function to combine the intensities of each octave at step 290 with each semitone at step 300 to produce the intensity for the pitch domain at step 340 can be a simple multiplication. However, using other functions allows different weightings depending upon which harmonic aspect is being emphasised. The evaluation of the intensity function at step 340 is repeated for all time intervals with different keys.

[0094] Evaluating the intensity function for the time domain at step 330 works in the same way as for the pitch domain at step 340 as described above, but with weighted base functions for each phrase at step 260 being combined with weighted base functions for each beat at step 270 and weighted base functions for each bar at step 280.

[0095] Additionally, at step 310, rhythmic weights are defined for each measure. All measure types have in common that the first beat (downbeat) is the strongest. Furthermore, measures with an even denominator (e.g., 4 / 4 or 6 / 8) usually have a slightly stronger beat in the middle of the bar. Figure 7 shows a row 500 representing a 4 / 4 measure, with each of the four different parallel, spaced-apart vertical lines 510-540 representing a beat and the shading intensity of each line being set by a weighted base function for each beat (the downbeat 510 is shown twice). Table 2 below shows example weights for the beats in a 4 / 4 measure. Table 2: Rhythmic weights for each of the 4 beats in a 4 / 4 measure, as depicted in Figure 7

[0096] The visualisation of phrase structure is optional and adds another level of granularity to the time domain, where each phrase is represented by a scaled base function. As in the pitch domain, the intensities of phrases (at step 260), beats (at step 270), and bars (at step 280) are combined by a blending function to produce an intensity function for the time domain (at step 330). The evaluation of the intensity function at step 330 is repeated for all time intervals with different measures or phrases.

[0097] At step 350, the intensity functions for the time domain and the pitch domain are combined to produce a shaded, two-dimensional grid representing the musical composition. Figure 8 shows a graphical representation of a musical composition as a rectangular grid 600 comprising rectangular cells arranged in columns and rows, with pitch along the vertical axis and time along the horizontal axis. Each cell has a border (e.g., formed from the intersection of lines 410-421 in Figure 6 with lines 510-540 in Figure 7) surrounding an interior portion of the cell. Accordingly, the interior portion of each cell can comprise a first shaded region having a shading intensity based at least in part on the harmonic data and a second shaded region having a shading intensity based at least in part on the rhythmic data. The grid 600 spans two octaves 610, 620 in the pitch domain along the vertical axis and two sections 630, 640 of four bars each in the time domain along the horizontal axis. The four bars 631-634 in the first section 630 are each in the key of C major and measure 4 / 4. Each of these four bars, 631-634, uses the weights given in Tables 1 and 2 and is graphically represented in grid 600 with the column shown in Figure 6 and the row shown in Figure 7. At the beginning of the second section 640, there is a key change to F major and a measure change to 3 / 4.

[0098] Accordingly, the four bars 641-644 in the second section 640 are visually different from the four bars 631-634 in the first section 630, allowing the user to quickly and easily discern the change in key and measure. The horizontal white line 650 marks the middle C (C 4) as the reference pitch. The reference pitch is needed to denote the absolute pitch since the shading only visualises the semitones' relative weight. To make it comparable to a traditional musical score, choosing a pitch that can be easily located, such as the middle C between systems with a treble clef and ones with a bass clef, is appropriate. Because the grid 600 is itself is not lined, the reference pitch line 650 stands out visually.

[0099] At step 360, as shown in Figure 9, notes 660 can be superimposed onto the grid 600 as geometric primitives. In a piano roll score, notes are usually depicted as rectangles. Because in the graphical representation of the present invention, the background of the grid 600 appears to have depth from its shading, it is possible to apply shading to the notes 660. By applying a circular shading function, the notes 660 appear to have a cylindrical shape. This creates a visual analogy of cylinders resting in grooves, where deeper grooves represent more stable harmonies. The notes 660 may be coloured differently (for example, to distinguish voices and / or instruments) or be sized differently (for example, different thicknesses depending upon loudness or velocity). A common problem in piano roll scores is that several consecutive shorter notes with the same pitch are difficult to distinguish from one longer note. This can be avoided by adding a tapered or triangular tip 670 to the end of the shapes. As a result of the shading, the triangular tip 670 appears to have a conical shape. If several voices and / or instruments are combined in a single score, problems of overlap and occlusion can occur. To avoid this, the notes 660 can be drawn transparently so that notes in the background remain visible. Slurs in traditional music notation are additional arcs connecting several notes. Slurs indicate that the notes are supposed to be played as one unit, even though they have different pitches. This can be visualised by connecting the individual note shapes with vertical lines 680. The image with the superimposed notes 660 can either be presented to the user on a display of the visual output device 120 at step 370 or be saved as a file for further use.

[0100] Although the grid 600 in Figure 8 has pitch along the vertical axis and time along the horizontal axis, it will be appreciated that different orientations of the axes are possible. Similarly, as shown in Figure 1A, in a typical piano roll score, the time axis (t) is depicted horizontally and the pitch axis (p) vertically, just as in a traditional music score. However, in software applications, other axis configurations, such as those shown in Figures 1B-1D, are common, and the grid 600 could be reconfigured to adopt any one of these orientations. Depending on the chosen orientation, each pixel on a display of the visual output device 120 corresponds to a two-dimensional coordinate in time or pitch space. The intensity functions for each domain yield an intensity for each point combined with another blending function. This can be achieved by multiplying the intensities. In practice, the simplest way of representing the evaluation result of an intensity function is by using textures. Being one-dimensional, these need to be only one pixel wide because they can be stretched to cover a rectangle. In this case, blending can be achieved by simply drawing the different textures on top of each other using transparency. The technical advantage lies in the blending function being executed on the GPU (Graphics Processing Unit) instead of the CPU (Central Processing Unit), which is much more efficient.

[0101] In a computer application that can play back the musical composition, it is desirable to synchronise the visualisation with the playback. Figure 10 shows a schematic diagram of an example graphical user interface (GUI) 700 for synchronising the graphical representation of the musical composition with playback. The GUI 700 comprises a main window 710 for displaying a graphical representation of a portion of the musical composition to the user, and a smaller sub-window 720 for displaying a graphical representation of the entire musical composition. The main window 710, therefore, provides a more detailed view of the graphical representation than the sub-window 720. For example, the main window 710 might show both the octaves and the semitones in the pitch domain, and the bars and beats in the time domain. In contrast, the subwindow 720 might only show the octaves in the pitch domain and the phrases and bars in the time domain. The main window 710 contains a bar counter 730 showing the bar numbers, while the sub-window 720 contains a timeline 740 showing ticks per bar and a rectangular box 750 highlighting the portion of the graphical representation of the musical composition shown in the main window 710. By clicking (dragging, scrolling) on the timeline 740, the user can adjust the focus position indicated by the box 750. The main window 710 and the sub-window 720 each contain a vertical scanline 760, which moves along the timeline 740, marking the current playback position. If only a portion of the musical composition is shown in the main window 710, the focus area on the timeline 740 needs to be scrolled accordingly to view other portions of the musical composition. Additionally, the notes currently being played in the musical composition can be highlighted in the graphical representation in a different colour. The graphical user interface can also be configured to allow a user to edit the music data.

[0102] In traditional music notation, it is common practice to divide the whole score into separate systems for each voice or instrument. Only for choral pieces, is it common to combine several voices into one system. For example, in an SATB (Soprano, Alto, Tenor, Bass) score, the two higher and lower parts usually share a system. This partition can be applied to the described graphical representation of a musical composition. In this case, the systems use a common time axis but has an individual pitch axes according to the displayed pitch range. In traditional music notation, lyrics are usually displayed below the systems. When several voices are combined in a system, this can make it difficult to see the association of lyrics to notes. In karaoke machines or players, whole phrases are usually displayed instead of individual words. During playback, the current word is highlighted, and the whole phrase is replaced once it has finished the playback. Accordingly, it is possible to let the user choose a preferred method for displaying lyrics, for example, attached to individual notes, below the system and synchronised with the time axis, or below the system and showing whole phrases in a karaoke-style.

[0103] Sonographs are acoustic spectrograms commonly used for the analysis of audio signals. Sonographs are usually plotted with time on the horizontal axis against pitch on the vertical axis, just like piano roll diagrams. The combination of both is a useful approach for analysing features of a musical composition, for example, the features of a singing voice. However, when overlaying a sonograph with a piano roll grid consisting of lines, it creates occlusion and the potential problem of visual clutter. These problems can be avoided by using the graphical representation described herein. When superimposed onto the graphical representation of the present invention, the intensity of the sonograph can be mapped solely to the hue of the colours, while the grid is only mapped to the brightness.

[0104] Musical compositions are usually analysed considering the three main aspects of melody, harmony, and rhythm. However, customary methods of depicting musical compositions focus solely on the melody. The invention presented here allows for the visualisation of harmonic and rhythmic aspects with a special kind of shaded grid that avoids the problem of visual clutter caused by grid lines. Additionally, the invention provides the advantages of an intuitive visual metaphor where visually perceived groves and ridges correspond to stronger or weaker musical features, as well as the ability to miniaturise a musical score while retaining the readability of macroscopic information such as tempo changes, key changes, and phrase structure. Furthermore, the invention allows a user to overlay a sonograph onto a graphical representation of a musical composition without occlusion or visual clutter.

[0105] Preferred embodiments of the invention have been described by way of example only, and modifications may be made thereto without departing from the scope of the invention.

Claims

CLAIMS1. A method of producing a graphical representation of a musical composition, the method comprising:(a) receiving music data indicative of at least the harmony and rhythm of the musical composition;(b) processing the music data to provide a set of harmonic data comprising a time series with one or more common harmonic features of the musical composition;(c) processing the music data to provide a set of rhythmic data comprising a time series with one or more common rhythmic features of the musical composition; and(d) forming a graphical representation of the musical composition comprising a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data.

2. The method of claim 1, wherein the first and second shaded regions of the grid partially or fully overlap each other.

3. The method of claim 1 or 2, wherein the one or more common harmonic features comprise one or more properties regarding a consistent key of at least a portion of the musical composition, and the one or more common rhythmic features comprise one or more properties regarding a consistent measure of at least a portion of the musical composition.

4. The method of any one of claims 1-3, wherein the grid comprises a plurality of cells, at least one of the cells having a border surrounding an interior portion of the cell, the interior portion comprising at least one of the first shaded regions of the grid and at least one of the second shaded regions of the grid.

5. The method of any one of claims 1-4, wherein the grid comprises a pitch axis and a time axis.

6. The method of any one of claims 1-5, wherein pitch is represented by one or more rows in the grid and time is represented by one or more columns in the grid.

7. The method of any one of claims 1-6, wherein the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for a pitch domain and the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a time domain.

8. The method of any one of claims 1-7, wherein the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for an octave.

9. The method of claim 8, wherein the intensity function for the octave comprises a weighted base function for each semitone in the octave.

10. The method of any one of claims 1-9, wherein the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a bar.

11. The method of claim 10, wherein the intensity function for the bar comprises a weighted base function for each beat in the bar.

12. The method of any one of claims 1-11, wherein the music data is also indicative of the melody of the musical composition.

13. The method of claim 12, wherein one or more notes or a sonograph are superimposed onto the grid.

14. The method of claim 13, wherein each of the one or more notes is represented as a cylinder with a tapered end.

15. The method of any one of claims 1-14, wherein the music data is also indicative of the structure of the musical composition.

16. The method of claim 15, wherein the method further comprises processing the music data to provide a set of structural data comprising a time series with one or more common structural features of the musical composition.

17. The method of claim 16, wherein the one or more common structural features comprise one or more properties regarding a consistent phrase of at least a portion of the musical composition.

18. The method of any one of claims 15-17, wherein the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a phrase.

19. The method of any one of claims 1-18, wherein the method is computer implemented.

20. The method of any one of claims 1-19, wherein the steps in the method are implemented by a processor or computer system.

21. An electronically-implemented method comprising software code or coded instructions that are executable or implemented by a computer, processor, or controller to carry out any one of claims 1-20.

22. A non-transitory computer-readable medium having stored thereon computer executable instructions that, when executed on a processing device or devices, cause the processing device or devices to perform a method of producing a graphical representation of a musical composition, the method comprising:(a) receiving music data indicative of at least the harmony and rhythm of the musical composition;(b) processing the music data to provide a set of harmonic data comprising a time series with one or more common harmonic features of the musical composition;(c) processing the music data to provide a set of rhythmic data comprising a time series with one or more common rhythmic features of the musical composition; and(d) forming a graphical representation of the musical composition comprising a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data.

23. A computer system configured to produce a graphical representation of a musical composition, the system comprising:(a) a microprocessor;(b) a data storage device for storing music data indicative of at least the harmony and rhythm of the musical composition; and(c) a visual output device for displaying the graphical representation of the musical composition; wherein the microprocessor is configured to:(i) receive music data indicative of at least the harmony and rhythm of the musical composition;(ii) process the music data to provide a set of harmonic data comprising a time series with one or more common harmonic features of the musical composition;(iii) process the music data to provide a set of rhythmic data comprising a time series with one or more common rhythmic features of the musical composition;(iv) form a graphical representation of the musical composition comprising a grid with a first shaded region of the grid having a shading intensity based at least in part on the harmonic data and a second shaded region of the grid having a shading intensity based at least in part on the rhythmic data; and(v) display the grid on the visual output device.

24. The computer system of claim 23, wherein the first and second shaded regions of the grid partially or fully overlap each other.

25. The computer system of claim 23 or 24, wherein the one or more common harmonic features comprise one or more properties regarding a consistent key of at least a portion of the musical composition, and the one or more common rhythmic features comprise one or more properties regarding a consistent measure of at least a portion of the musical composition.

26. The computer system of any one of claims 23-25, wherein the grid comprises a plurality of cells, at least one of the cells having a border surrounding an interior portion of the cell, the interior portion comprising at least one of the first shaded regions of the grid and at least one of the second shaded regions of the grid.

27. The computer system of any one of claims 23-26, wherein the grid comprises a pitch axis and a time axis.

28. The computer system of any one of claims 22-27, wherein pitch is represented by one or more rows in the grid and time is represented by one or more columns in the grid.

29. The computer system of any one of claims 22-28, wherein the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for a pitch domain and the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a time domain.

30. The computer system of any one of claims 22-29, wherein the shading intensity of the first shaded region of the grid is calculated at least in part using an intensity function for an octave.

31. The computer system of claim 30, wherein the intensity function for the octave comprises a weighted base function for each semitone in the octave.

32. The computer system of any one of claims 22-31, wherein the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a bar.

33. The computer system of claim 32, wherein the intensity function for the bar comprises a weighted base function for each beat in the bar.

34. The computer system of any one of claims 22-22, wherein the music data is also indicative of the melody of the musical composition.

35. The computer system of claim 34, wherein one or more notes or a sonograph are superimposed onto the grid.

36. The computer system of claim 35, wherein each of the one or more notes is represented as a cylinder with a tapered end.

37. The computer system of any one of claims 22-36, wherein the music data is also indicative of the structure of the musical composition.

38. The computer system of claim 37, wherein the microprocessor is further configured to process the music data to provide a set of structural data comprising a time series with one or more common structural features of the musical composition.

39. The computer system of claim 38, wherein the one or more common structural features comprise one or more properties regarding a consistent phrase of at least a portion of the musical composition.

40. The computer system of any one of claims 37-39, wherein the shading intensity of the second shaded region of the grid is calculated at least in part using an intensity function for a phrase.

41. The computer system according to any one of claims 22-40, wherein the system further comprises an input device and an audio output device.

42. The computer system according to claim 41, wherein the system further comprises a graphical user interface comprising a main window, a sub-window, and a timeline, wherein the main window shows a more detailed view of the graphical representation of the musical composition than the sub-window.

43. The computer system according to claim 42, wherein the graphical user interface is configured to synchronise the graphical representation of the musical composition with playback of the musical composition.

44. The computer system according to claim 42 or 43, wherein the sub-window contains a rectangular box highlighting a portion of the graphical representation of the musical composition shown in the main window, the graphical user interface being configured to allow a user to adjust a focus position indicated by the box.

45. The computer system according to any one of claims 42-44, wherein the graphical user interface is configured to allow a user to edit the music data.

Images