Methods for training neural networks to reflect emotional perception, and related systems and methods for classifying and discovering associated content and associated digital media files with embedded multi-dimensional property vectors - Patents.com

Abstract

Property vectors representing extractable measurable properties, such as musical properties, of a file are mapped to the file's semantic properties. This is accomplished using an artificial neural network (ANN) whose weights and biases are trained to inversely align distance dissimilarity measures in the property space of files for pairwise comparisons to corresponding semantic distance dissimilarity measures in the semantic space of those same files. As a result, once optimized, the ANN can process any files parsed with their properties to identify other files that share common features that reflect emotional perceptions, thereby providing more reliable and realistic results of similarity / dissimilarity. This is in contrast to simply training a neural network to consider extractable measurable properties, which alone do not provide reliable contextual relationships in the real world.

Description

[Technical field]

[0001] The present invention generally relates to the ability of artificial intelligence and neural networks trained to reflect human subjective responses to sensory stimuli, particularly audio in the form of music or image data (whether static or video) and / or inputs with subjective context, such as scent descriptions or text snippets. More specifically, but not exclusively, the present invention relates to artificial intelligence to realize a discrimination system capable of classifying and reliably identifying perceived commonalities or dissimilarities between properties with musical or image contexts - in an emotionally perceived semantic sense - to enable users of a typical multimedia device to explore and identify ranked orders of data files sharing commonalities from a vast catalog. Even more specifically, the present invention relates to a system and operating methodology for identifying selectable audio content, objectively classifying or critiquing an initial subjective assessment of a track (or a portion thereof), and facilitating access to classified content that commonly shares a user-selectable perception of a musical subject, in the sense of selectively "pushing" the content. [Background technology]

[0002] One of the most challenging long-term goals in artificial intelligence (AI), typically based on artificial neural network architectures, is to mimic human intelligent behavior. This is a complex task, especially because human opinions are based on subjective responses to stimuli, and existing AI approaches often do not correlate well with emotionally perceived responses. Rather, the rationality of computational architectures in AI is to suggest the most likely responses based on the assimilation of large amounts of data with objectively identifiable characteristics.

[0003] The refinement or training of a deep neural network (DNN), as will be seen below, is often based on the concept of "backpropagation", or backward propagation of errors, to calculate the gradients required for the DNN to calculate the weights used in the network. The DNN therefore travels through its layers, as will be seen below, to calculate the probability of each output, attempting to find the correct mathematical operations that transform the inputs into outputs, whether linear or nonlinear relationships.

[0004] As a practical example of the current limited approach to music interpretation in AI, the identification of perceived similarities between different musical pieces is constrained to absolute similarity, not to semantically perceived similarity. This may not seem problematic on first investigation, but it remains fundamentally problematic from an intellectual and practical standpoint, since "there is no music other than that which is created, perceived and experienced in the human mind. In other words, "music does not exist in itself...because neither music nor language can be studied as purely surface forms, because their perception generates information that is not contained in the surface form." This proposition is established in the paper "On the non-existence of music: why music theory is a figment of the imagination" by Gerant A. Wiggins et al. (ESCOM European Society for the Cognitive Sciences of Music, Musicae Scientiae, Discussion Form 5, 2010, pages 231-255).

[0005] Therefore, existing AI modeling, which is based on a certain degree of absoluteness (based on the interpretation of measured parameters) from the outset, is fatally flawed and may produce inconsistent and / or erroneous results, for example in relation to music search tools.

[0006] The same problem exists in identifying and classifying other forms of representations, such as actual interpretations of images such as paintings or photographs or medical CT scans, or other purely descriptive representations (such as descriptions of scents, medical reports or plot summaries of novels), to identify and assess the relevance of completely unrelated or otherwise potentially interrelated searchable electronic images and / or data in terms of having an acceptably approximate set of subjective attributes, characteristics or features to a defined starting point (e.g., a specific description of the scent or the tonality, rhythm and timbre of a musical piece).

[0007] Indeed, existing AI systems cannot resolve semantically related attributes and may therefore miss semantic similarities while accepting or suggesting that perceptually distinct differences are closely related.

[0008] The music, movie and game industries and aspects of content delivery in particular are evolving. In this regard, the sale or distribution of (for example) music or soundtracks as streamed or downloaded digital files is becoming dominant in those markets. This contrasts with the established but now declining sale of compact disc or DVD technology (or historically, vinyl records) through consumer retail stores.

[0009] While music sales are commercial and content-perceptual and aesthetic in nature, there is no direct and reliable mechanism for identifying tracks that share common musical characteristics that are targeted to an individual's particular tastes. To support this view, music is loosely classified into its genres, such as jazz, rock, classical, and blues, to name a few, but within each of these genres there are usually a huge number of subcategories or sub-genres. For example, jazz apparently has at least 30 different forms, including free funk, crossover, hard bop, and swing. These sub-genres may share some similarities in the compositional architecture that are discernible to the user that define the genus, but there are often noticeable differences that are fully audible or musically manifested. To provide further context, two different kinds of jazz may be perceptually and sonically very different to the extent that a particular listener may conclude that they like one but not the other. On the same (but opposite) grounds, a listener may prematurely reject (or simply not recognize) a classical music piece, based on the mistaken perception that the listening characteristics (in both musical space and musical sense) of a classical music piece must be quite different from an orchestral arrangement of a hard rock piece, when in fact these two different audio tracks are substantially identical in terms of their proximity in musical space.

[0010] In typical online music libraries, each containing millions of songs, e.g., the iTunes® and Tidal® music libraries, each reportedly containing about 50 million tracks, problems exist as to how these databases can be effectively searched to identify common musical themes, features, or characteristics perceived by users among countless tracks across potentially disparate genres. As a result, searches for similar music and to date have often excluded entire genres [or at least genre sub-species] from consideration, and / or failed to correlate highly related music content within different tracks from different genres. Commercial libraries can utilize "collaborative filtering," in which recommendations are made based on playlists of other users who have listened to the same songs, but this approach relies heavily on stored user data and statistical usage. Although collaborative filtering can reflect the personal tastes of the listener / library user, it is constrained by the amount of available user data and is therefore not inherently a perfect solution.

[0011] There is also the problem of a "cold start" that occurs when a new artist (in the sense of unknown or nearly unknown) [i.e. a beginner, new face or "newcomer" potentially signed by a recording studio or label] releases their first audio track or first album. The problem is that the artist is unknown and therefore has virtually no following, whether online or in other media, e.g. listeners acquired from promotion via radio broadcast or television. In other words, the lack of listening history is an obstacle both to making recommendations, e.g. via collaborative filtering, or to building a reputation and boosting the rising star. The problem for distributors, e.g. record labels, is how to raise awareness of the new artist, how to categorize the nature of the new artist's music [which may possibly change since it is user perceptible], and how to link / insert it into the existing music library to ensure that the music is actually listened to, downloaded or streamed for commercial reasons and thus maximizes exposure. The problem for the listening and / or streaming public or radio stations with respect to these newcomers is that "they don't know what they don't know," making the odds of finding a newcomer's first foray into the music world low and random, and more often based on luck than judgement.

[0012] For distributors, effective exposure and access to an artist's music tracks equates to increased sales potential. Indeed, from a commercial standpoint, it is also desirable to avoid "late bloomers" and thus rapidly build the reputation of new artists.

[0013] In short, new artists must enter the market with new works that they have no track record of. In contrast, fans of existing artists are always following, i.e., paying attention to newly released songs, and generally tend to buy from existing artists, regardless of whether such newly released songs are good or bad. In fact, the "cold start" problem does not exist for existing artists with established followers and listener bases, because newly released songs from popular artists are streamed, listened to, and / or purchased, even if they are poorly received by critics. The cold start problem therefore hampers the spread of music and the potential evolution of new forms of music.

[0014] Also, the nature of user perception and music appreciation is a personal attribute that is being adopted recently. In particular, listeners evaluate whether a track is to their liking and preferably their personal preference for the track (or part thereof) being played / listened to within a few seconds. As a result, any track-finding recommendation scheme, for example implemented as a downloadable app, must be inherently fast (in terms of identifying recommendations) and any recommendations it gives must be reliable in that they must satisfy the user's perceived musical value, i.e., personal musical taste. Any track-finding recommendation tool that outputs seemingly random tracks, such as existing systems that utilize statistical analysis of demographic data of other users identified as having common tastes or circumstances (e.g., age range 30-40, married with 2 children, working as an accountant, living in a property purchased with a mortgage in Staten Island, New York), is ultimately poor and its use is ignored or downplayed. Perceptual classification of audio tracks, similar to musicology, is an important consideration for an effective audio track-finding technique, as it is genre-independent.

[0015] The problems introduced above are not solved by existing apps such as Shazam® and SoundHound® because these apps focus on identifying sampled audio tracks in real time, or conversely, they list tracks that others in the community have discovered.

[0016] Using SoundHound®, one can sing or hum a piece of music and attempt to identify it. These apps may therefore identify the track being played / sampled, or recommend further listening based on a reported hard number, but in many cases, do not relate to the identification at all. Musical similarities are not perceived by these existing apps across the myriad tracks in a music library.

[0017] Another problem facing the music industry is how to best enhance the listener / user experience, especially at an individual / personal level. Indeed, it has long been recognized that the contextual relevance or relationship of a song to an event can bring about recognition or elicit complementary emotional responses, such as fear or suspense while watching a movie or the familiarity with a product that occurs in a television commercial.

[0018] The identification of common musical features is desirable because it is recognized that the appropriate use of musical content supports the emotional, physiological and / or psychological engagement of the listener, thus enhancing the listener's sensory experience. This is relevant for example to game developers and / or advertisement or movie trailer creators / editors whose task is to quickly edit a suitable multimedia product that matches the associated musical subject matter, e.g. increasing musical intensity (meaning increasing drama and tension, not necessarily in the sense of absolute audio output level), with the video output. In providing at least one resulting "proof" for review, the developer or editor already spends a significant amount of time identifying potentially suitable music and then adapting / matching the selected music to the video. To reduce the need to identify commercially usable audio tracks, content developers can currently make use of so-called "temp tracks", which are often well-known tracks with rights that are not easy to obtain, but this is only a stop-gap measure, since it requires a search to identify a suitable commercially viable track for which usage rights can be obtained. Further time delays are then incurred due to the directing client having to determine whether the edit matches the initial preview or not. Thus, an effective track search tool would facilitate the identification of alternative music track selections to align with the visual sequence or structure of a music program, for example (such as occurs in a "spin" class where cycling movements are choreographed to music to enhance momentum).

[0019] Technologies exist on the web for searching images with the same or similar visual characteristics, including identification of websites that represent the same or related images. For example, Google® supports a computer program application [sometimes abbreviated to the term “app”] called “reverse image search” (see https: / / support.google.com / websearch / answer / 1325808?hl=en) in which an uploaded image is apparently broken down on the server side into groups of constituent bits, and the bit groups are searched to identify related images according to some form of logical distance measure within a predefined parameter space. The identified related images are then provided to the user who uploaded the original image using the app.

[0020] Image comparison requires complex computations (typically based on neural networks), but it is known that the underlying source document can be divided into shape, color and / or dimensions such as angles or lengths. By contrasting one or more of these coefficients, associations can be established, for example through relative scaling. In contrast, the critique of musical features, again making use of neural networks, has until now generally been hampered by the difficulty of perceptually resolving more subtle differences in musical structure. Summary of the Invention [Means for solving the problem]

[0021] According to a first aspect of the present invention, there is provided a method of training an artificial neural network "ANN" in a system configured to identify similarities or dissimilarities of content of a first data file to content of a different data file, the method comprising: for a selected pair of different data files, extracting measurable signal characteristics from each of the first data file and the different data file and identifying a plurality of properties from these measurable signal characteristics; at the output of the ANN tasked with processing the properties, for each property of the first data file and the different data file of the selected pair, generating a corresponding property vector in a property space; generating a first multidimensional vector for the first data file and a separate second multidimensional vector for the different data file. the method includes assembling multi-dimensional vectors, each multi-dimensional vector resulting from assembling a respective property vector from an output of an ANN, determining a distance measure between the first multi-dimensional vector and the second multi-dimensional vector, adjusting weights and / or bias values ​​in the ANN by a backpropagation process in response to the determined distance measure to account for identified discrepancies occurring between the determined distance measure in property space and a quantified semantic dissimilarity distance measure evaluated in semantic space for the first data file relative to the different data files, thereby training the system by evaluating the semantic perception reflected in the quantified semantic dissimilarity distance measure over the property evaluation reflected by the distance measure between the first multi-dimensional vector and the second multi-dimensional vector.

[0022] The quantified semantic dissimilarity distance measure evaluated in the semantic space may be based on a vector representation of a textual description associated with each of the first data and the distinct data file, the textual description being encoded in metadata of the respective files.

[0023] In a preferred embodiment, evaluating the quantified semantic dissimilarity distance measure includes applying natural language processing "NLP" to the text descriptions to generate semantic vectors for a plurality of N data files in a training set; calculating, for the training set and for each pair, a separation distance between each semantic vector; identifying, for each of the files in the training set, a minimum and maximum separation distance relative to other files in the training set; generating, for each file in the training set, a set of pairs, where a first pair has a minimum separation distance and a second pair has a maximum separation distance; assigning a first value representing semantic closeness to the first pair and a second value representing semantic dissimilarity to the second pair, where the second value is different from the first value.

[0024] The first pair may be a first data file and a different data file.

[0025] The training method may include, for a subset that includes the m smallest separation distances and the m largest separation distances, assigning a first value to the m smallest and a second value to the m largest, where m is a positive integer less than N.

[0026] In a second aspect of the present invention, there is provided a method for training and operating an artificial neural network "ANN" configured to identify content similarities or dissimilarities between data files comprising one or more audio and image contents, the method comprising relating quantified semantic dissimilarity measures of the content of the data files in a semantic space to associated property separation distances in a property space of measurable properties extracted for the content, and training and operating the system to evaluate the semantic dissimilarity measures across the measurable properties.

[0027] The quantified semantic dissimilarity measure evaluated in the semantic space may be based on a vector representation of the textual descriptions associated with the different data files.

[0028] In a preferred embodiment, the data files are audio files and the measurable properties are musical properties defined by properties relating to rhythm, tonality, timbre and musical texture.

[0029] The quantified semantic dissimilarity measure is obtained by applying natural language processing "NLP" to the text descriptions to generate semantic vectors for a plurality of N data files in a training set; calculating, for the training set and for each pair, a separation distance between each semantic vector; identifying, for each of the files in the training set, a minimum and maximum separation distance relative to other files in the training set; generating, for each file in the training set, a set of pairs, where a first pair has a minimum separation distance and a second pair has a maximum separation distance; assigning a first value representing semantic closeness to the first pair and a second value representing semantic dissimilarity to the second pair, where the second value is different from the first value.

[0030] The first pair may be a first data file and a different data file.

[0031] In one particular embodiment, for a subset that includes the m smallest separation distances and the m largest separation distances, the method may include assigning a first value to the m smallest and a second value to the m largest, where m is a positive integer less than N.

[0032] The method for training and operating an ANN further comprises storing in a database file identifiers enabling identification of individual files whose dissimilarity has been assessed according to the method of any of the preceding claims, and recording for the stored file identifiers at least one of the distance relations with other identified files in the database and the calculated property vectors for the individual files.

[0033] The database may be a network accessible database or in a memory or storage device local to the user device.

[0034] In some embodiments, the method further includes, during training of the neural network having multiple processing branches, differentiating different processing branches assigned to process different properties of the files to generate respective property vectors in the property space, the differentiating including identifying relative similarities and dissimilarities of the respective property vectors for the pairwise contrasted files subjected to the pairwise comparison, and facilitating adjustment of at least one of bias values ​​and weights within the particular processing branch in which the greatest dissimilarity is found in the generated property vectors of the pairwise comparison.

[0035] Relatively greater importance may be assigned to a particular processing branch of the multiple processing branches if the properties in the pairwise comparison are rated as relatively similar rather than relatively dissimilar.

[0036] The quantified semantic dissimilarity distance measure is derived from natural language processing of the files' associated descriptive content, which may be included in the files' metadata.

[0037] In particular embodiments of various aspects, the data files include audio, and the properties are musical properties, and the measurable signal characteristics define properties indicative of rhythm, tonality, timbre, and musical texture.

[0038] In another aspect of the invention, a method is provided for identifying files sharing common user-perceivable characteristics, the method including evaluating a target file for proximity to stored files in a file library including a plurality of uniquely identified files having associated property vectors, the method including processing the target file in a neural network to generate a multidimensional property vector in a property space, the multidimensional property vector being composed of a plurality of property vectors derived from a plurality of different sets of measurable signal characteristics selectively extracted from the target file, each of the plurality of property vectors being weighted by the neural network to a measure of semantic dissimilarity; and generating an ordered list of files from the library based on the proximity between the multidimensional property vector of the target file and the property vectors of files in the library.

[0039] In a third aspect of the present invention, there is provided a method for identifying semantically similar data files using a neural network, the method comprising assessing in the neural network that semantic perceptions reflected in a quantified semantic dissimilarity distance measure are more important in identifying semantically similar files than property ratings reflected by distance measures (330) of extracted measurable signal characteristics.

[0040] In a further aspect of the present invention there is provided a computer program comprising code which, when executed by a processor intelligence, performs the methods of the various aspects set out in this specification, particularly the claims.

[0041] In another aspect of the present invention, there is provided an artificial neural network "ANN" comprising layers of interconnected neurons arranged to apply configurably selected weights and biases by backpropagation to content presented to the ANN in the form of at least one of audio content and image content, the ANN correlating a quantified semantic dissimilarity measure of said content in a semantic space with associated property separation distances in a property space of measurable properties extracted for the content to provide an output that is adapted over time to align results in the property space with results in the semantic space, the ANN being configured to evaluate the semantic dissimilarity measure across the measurable properties during the adaptation of said weights and biases.

[0042] In yet another aspect of the present invention, there is provided an artificial neural network "ANN" comprising a layer of interconnected neurons arranged to apply weights and biases selectively configured by backpropagation to content presented to the ANN in the form of at least one of audio content and image content to correlate quantified semantic dissimilarity measures of said content in a semantic space with associated property separation distances in a property space of measurable properties extracted for the content and processed by the neurons of the ANN, thereby configured to evaluate semantic dissimilarity measures across measurable properties in its application of said weights and biases.

[0043] In yet another aspect of the present invention, there is provided an artificial neural network "ANN" including an input layer of neurons coupled to an output layer of neurons, the neurons arranged to apply weights and biases to data received thereby, the ANN configured to generate weights and biases for the neurons to correlate a consistency of an output of the ANN in a property space with a pre-measured baseline semantic dissimilarity measure for a baseline comparison descriptive content in a semantic space, the property space being determined by processing by the ANN measurable properties extracted from audio and / or image content of the files applied to the input.

[0044] The input layer is typically separated from the output layer by at least one hidden layer.

[0045] In a further aspect of the invention, there is provided a database comprising: a) a plurality of artificial neural network outputs arranged to provide a plurality of property vectors, the property vectors being extracted measurable signal characteristics from the contents of a first data file; b) at least one convolutional neural network arranged to provide further property vectors, the further property vectors being derived from a two-dimensional representation of the contents of the first data file; and c) a database comprising a plurality of uniquely identifiable data files each having a reference vector, each reference vector being a representation of (a) the extracted measurable signal characteristics obtained from the contents of that data file. and (b) a database assembled from property vectors derived from a two-dimensional representation of the contents of the data file, and further property vectors derived from the two-dimensional representation of the contents of the data file; and d) processing intelligence configured to: generate, for a first data file, a multidimensional property vector from the plurality of property vectors and the further property vector; compare the multidimensional property vector to the plurality of reference vectors in the database; and identify at least one unique file identifier having a reference vector identified as measurably similar to that of the multidimensional property vector of the first file, thereby identifying a distinct second data file in the database that is semantically closer to the first data file.

[0046] The multiple property vectors are each weighted by a neural network to reflect semantic dissimilarity.

[0047] The system may include a network connection and a communication unit, and the processing intelligence causes the communication unit to transmit the different second data file to the interactive user equipment over the network connection. A user interface may be provided for the user to select preferred characteristics for the search.

[0048] In another aspect of the invention, there is provided a system including: a) a plurality of artificial neural network outputs arranged to provide a plurality of property vectors for data files, the property vectors being extracted measurable signal characteristics from content of the data files; b) at least one convolutional neural network arranged to provide a further vector that is assembled from the plurality of property vectors to generate a multidimensional property vector; c) processing intelligence arranged to receive first and second different multidimensional property vectors for a pair of data files comprising a first data file and a different data file, and in response generate a distance measure between the first and second different multidimensional property vectors; and d) processing intelligence configured to resolve whether the distance measure in the property space correlates with a reference distance generated for the first and second vectors in a semantic space, the reference distance being based on a semantic dissimilarity measure of descriptive content for a reference comparison of the pair of data files.

[0049] The processing intelligence functions to relate quantified semantic dissimilarity measures of said content in semantic space to associated property separation distances in a property space of measurable properties extracted for the content.

[0050] In yet another aspect of the present invention, there is provided a prediction system trained by generating a plurality of pairs of independent vectors representing human perceived characteristics and measurable characteristics of electronic source files, a first vector being generated in semantic space and based on measured dissimilarity in human generated descriptions between a pair of source files such that the first vector provides a user subjective perception of pairwise closeness, and a second vector being obtained from a pairwise comparison of measurable properties extracted from content of the same pair of source files, and the prediction system includes processing intelligence arranged to adapt a generation process of the second vector such that the pairwise distances of the first vector approach the pairwise distances of the second vector.

[0051] The electronic source files include at least one of digital audio, image data, and contextual bibliographic data.

[0052] In yet another aspect of the present invention, there is provided a digital audio file having embedded therein a multi-dimensional property vector including a plurality of constructed property vector components each associated with a musical property derived from a plurality of sets of quantized representations of signal characteristics of the digital audio file.

[0053] The embedded multi-dimensional vectors are vectors correlated with musical properties defined by measurable signal characteristics indicative of rhythm, tonality, timbre and musical texture.

[0054] The assembled vector components (OR x , O.T.O. x , O.T.I. x and OTX x ) reflects a measure in vector dissimilarity space of a musical property evaluated against a reference vector in a semantic distance separation space.

[0055] Also provided is a digital image file having embedded therein a multi-dimensional property vector including a plurality of assembled property vector components each associated with a visual property derived from a plurality of sets of quantized representations of signal characteristics of the digital image file.

[0056] Rather than being directly embedded, the multi-dimensional property vector (of either an audio or image / video file) can be linked or matched with a unique attribute, such as the filename or international truck identification number of a digital audio file.

[0057] In another aspect there is provided a database comprising a plurality of digital audio files according to any one of claims 41 to 43 or a plurality of digital image files according to claim 44.

[0058] Fundamentally, the present approach differs from current data scientific approaches that go back to hard and / or absolute data. Rather, the present invention emphasizes the output results from a neural network whose task is to evaluate the dissimilarity of extracted measurable properties of pairwise contrasted source files in a vector space back to the human perception of similarity / dissimilarity expressed in semantic space between the same pairwise contrasted source files. This semantic space represents a context, but it is a different vector space in which subjective descriptive context is mapped to measurable vectors expressed in a manipulable mathematical form. In other words, the embedding process is designed such that semantically similar subjective descriptions are considered correspondingly similar in the resulting vector (semantic) space.

[0059] Advantageously, the present invention provides a novel methodology for data classification, and more specifically, a system and method that enables rapid assimilation of user-perceivable characteristics between source data and retrieved data that may be related, e.g., detection of snippets of audio or audio files that are likely to match the interests of a listener.

[0060] In particular, the preferred embodiment provides a track finder or track recommendation tool that is able to consistently characterize a sampled musical piece by extracting identifiable musical properties within a piece of music, and then identify other musical pieces that share musical characteristics and / or subjective musicological traits.

[0061] Given the number of music files accessible within a music library (whether a personal library containing hundreds or thousands of audio tracks or a commercial library with millions of tracks for commercial streaming or download), including remixes of tracks that can slightly or significantly alter the original musical work, the present invention provides a useful and effective recommendation tool that can narrow search results of tracks based on rankings of perceived musicological similarity, thus eliminating random genre classifications and focusing on perceived audio characteristics / similarity.

[0062] The track finding tool of various embodiments thus beneficially reduces the need for a listener to listen to tracks intensively in order to identify music that matches the user's specific and subjective tastes, i.e., the track finding tool narrows the search space by identifying perceptually relevant tracks for the user. Furthermore, through an objective and technically guaranteed evaluation of musicology, embodiments of the present invention aim to improve end-user selection and access to music through guaranteed recommendations by making more frequent and faster access to music libraries, particularly a wider range of songs stored in or accessible via online libraries or server stores accessible to subscribers. Embodiments of the present invention thus alleviate the cold start problem by promoting new audio tracks and artists to a more selective and perhaps more receptive listening base based on perceptually similar musical properties.

[0063] The same principles apply to the identification of other contextually descriptive subjective works, including images, text and / or video, that serve as sources for computer-implemented data analysis.

[0064] Various aspects and embodiments of the present invention as outlined in the appended claims and the following description may be implemented as hardware solutions and / or software, including downloadable code or web-based apps.

[0065] Exemplary embodiments of the invention will now be described below with reference to the accompanying drawings. [Brief description of the drawings]

[0066] [Figure 1] FIG. 1 shows a flow diagram of a preferred process for assessing dissimilarity of files, particularly audio files, according to the present invention, and a process by which an artificial neural network can be trained. [Diagram 2] FIG. 1 is a schematic diagram of a system architecture for training an artificial neural network, according to a preferred embodiment. [Diagram 3] FIG. 3 is a flow diagram of an exemplary process for training the neural network of FIG. 2 to assimilate a semantic vector space with a property vector space to identify property similarities and property dissimilarities between source files. [Figure 4] A typical Mel spectrum presentation of an audio track. [Diagram 5] 1 shows convolutional and pooling layers in an artificial neural network assigned to the Mel spectrum interpretation. [Figure 6] FIG. 3 is a representation of the artificial neural networks used within the various ANN chains of FIG. [Figure 7] 1 is a flow process used by a preferred embodiment to evaluate an affectively-perceived measure of file dissimilarity, particularly with respect to audio files. [Figure 8]1 is a network architecture including an accessible database containing vector representations according to a preferred embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0067] It has been found that in order to provide a tool that can be accessed via a web browser or a local app and that assesses the semantic similarity or dissimilarity between (for example) audio tracks, it is necessary to utilize deep learning and artificial intelligence to identify similarities between semantic meanings from which measurable properties of the content of the same data source have been extracted, which are processed to provide a first measure in the semantic space, and in different measurable spaces, such as Euclidean space (other dimensional spaces may be used). This process effectively provides a translational mapping between the similarity of semantic meanings in one space and the similarity of measurable properties extracted in another space.

[0068] More specifically, it has been found that measures of emotionally perceived similarity or dissimilarity (particularly in the illustrative sense of digital audio files, image files, or other perceived aesthetic creations in digital form) cannot be derived solely from hard data fields, e.g., quantized representations of signal characteristics, because such hard data does not provide any interpretation as experienced by a human critic, e.g., a listener. In other words, feature extraction alone does not provide a sufficiently accurate and objective assessment of emotionally perceived similarity or dissimilarity, because the quantized representations of signal characteristics (either alone or grouped) bear no relation to the emotional real world.

[0069] The present invention therefore works in conjunction with one or more trained artificial neural networks (ANN) that are functionally configured to improve the currently existing disconnect between subjectively evaluated / perceived data, in particular feature extraction of similarities or dissimilarities in music, and human intuition and / or human emotional perception (compared to insufficient conventional, quite disconnected, objectively evaluated approaches), by first mapping, i.e. relating or combining, subjectively derived content descriptions represented in a semantic space to measurable properties extracted for the same paired contrast files represented in a Euclidean space.

[0070] The effect of the neural network function is to generate two independent vectors that represent the emotionally perceptible or documented dissimilarities in digital audio and / or image data and / or literary works together, but in different vector spaces. The first vector in the semantic space is based on a human-generated description of the source file and therefore has a significantly higher context weight. The first vector is then used to evaluate and modify, for example, the second vector in the Euclidean space, so that the output of the different neural network can be converged to the semantic result of the first neural network through changing the weights in the ANN. The Euclidean vector is also derived from selected subjective properties extracted from the original source data during deep learning of the artificial neural network, for example pairwise comparison of songs.

[0071] Following training, a convergence process ultimately provides tunable capabilities within the ANN that allow any data file to be evaluated against other pre-evaluated data files to assess similarity of semantic and emotionally perceptible content.

[0072] Thus, at least during the training phase of the artificial neural network, two independent vectors are generated for the common source. The first vector is based on semantics and is derived from the (typically) relevant metadata of the source data / file, and the second vector is extracted from the main content (e.g., payload) of the source / data file. These two vectors should be identical, the first based on human judgment and the second extracted from hard, identifiable and absolute measurable properties, but they do not have to be. Consequently, to generate a truly representative predictive tool for assessing emotional / perceived dissimilarity or closeness, it is necessary that the processing of absolute measurable properties ultimately leads to the same results as the processing of human judgment, i.e., semantic properties. To reflect true emotional perception, the evaluation regarding human judgment is given more weight and exceeds the absolute evaluation of identifiable and measurable tangible properties, both of which are derived from the common source. By imposing changes in the weights and bias values ​​applied in the artificial neural network processing identifiable and measurable tangible properties, it is possible to get closer to reality, reflecting human intelligence, judgment and reasoning through perception.

[0073] 1. Similarity / dissimilarity assessment of context descriptions in semantic space An initial semantic description of the nature of a file, such as the contextual description and use of specific words within a sentence, is first converted or "embedded" into a multi-dimensional semantic vector, such as using Natural Language Processing "NLP" techniques. The contextualized description becomes a measure of human judgment based on subjectivity, perception, and / or emotion.

[0074] NLP supported by (for example) Google®'s Universal Sentence Encoder and especially Tensorflow™ Hub encodes text into high-dimensional vectors that can be used for text classification, semantic similarity, clustering and other natural language processing tasks. In practice, NLP processing of two semantically similar descriptions will produce similar vector representations.

[0075] There may be some differences in the textual descriptions from different annotators, but given the nature of the processing performed, these are not considered statistically significant.

[0076] The choice of transformation process of text and vector representation is a design choice, for example the Tensorflow™ process can be based on a transformer encoder or alternatively on training with a deep averaging network (DAN). Vectors related in semantic space are technically important from a global training perspective.

[0077] The semantic vectorization process can be applied to other forms of media data, such as image data in the form of paintings or films that have semantic properties and corresponding aesthetic descriptors that can be converted into numerical representations.

[0078] During the training sequence, the NLP-derived multi-dimensional vectors are compared pairwise with other NLP-derived vectors to identify a separation distance representation of pairwise semantic closeness in a semantic vector space. This first establishes a user-subjected perception of pairwise closeness. In this sense, it will be appreciated that the use of the terms "semantic" and "semantic space" etc. reflects that the origin of any corresponding vectors or values ​​arises from a subjectively prepared description of human perceptual or affective (i.e., semantic) characteristics of the content of a file, e.g., an audio track.

[0079] A preferred use of NPL provides an initial mapping between textual descriptors and vector values ​​in a semantic space. The same principles could be applied to the classification of other media, such as video, film, painting, fashion in the exemplary sense of clothing and decoration (where properties are color and pattern and texture such as covering), as well as medical records that contain images.

[0080] To provide context from a musicological perspective, take Rimsky-Korsakov's "Flight of the Bumblebee" as the first audio training track, which can be described in terms of the dimensions of two words "noisy" and "light," and NLP maps a vector representation of 1004512112 to the track containing only these two NLP-resolved terms. Of course, the number of linguistic dimensions can be more than two, and so the description of the audio track can be extended to include other semantic associations that occur with, for example, (i) temporal events such as dusk, Easter, cloudy, etc., and / or (ii) emotions, and / or (iii) themes, such as fables or facts, and / or (iv) environments, etc.

[0081] The vector "1004512112" is provided merely as an arbitrary example; in reality the generated multi-dimensional vectors may take entirely different forms, especially since the number of word / sentence dimensions is limited only by the semantic associations that can be derived from the descriptive sentences in the audio track.

[0082] This process is repeated for a large number of independent samples, typically thousands and preferably at least about 10,000 or more, to build a multidimensional matrix for audio track search applications that is used to provide contextual examples. Thus, semantic similarities / dissimilarities between all training tracks, such as the above-mentioned "Flight of the Bumblebee" and, for example, the electronic song "White Flag" by Delta Heavy or "Boulevard of Broken Dreams" performed by Green Day, are established. However, the size of the training set is a design choice depending on processing power, time and the desired level of achievable reliability / accuracy. It is also an option to train the ANN by choosing extreme variations in the pairwise distance measure rather than evaluating all pairs.

[0083] The resulting semantic first vector is, in a preferred embodiment, constructed from at least a multiple of 64 separate dimensional components (although the exact number reflects implementation choice and desired accuracy). When using the Tensorflow™ universal sentence encoder, processing of the semantic description results in a 512-dimensional vector (in the semantic space). Consequently, the exact semantic vector length is a design choice and may vary.

[0084] It does not matter whether the semantic vector and the property vector (described in more detail below) are the same size, as the system assumes that dissimilarity is evaluated pairwise.

[0085] 2. Distance Estimation Based on Extracted Properties In generating second independent vectors in a second training process based on the derived "properties" of the files (as opposed to the semantic description of the files used for pairwise semantic closeness outlined immediately above and detailed in Chapter 3 below), the weighting coefficients applied to the nodes in the layers of the neural network are varied by backpropagation to converge the results in the property distance space (typically Euclidean) back to the results in the (typically Euclidean) semantic separation distance (in semantic space) and thus essentially back to the original semantic description.

[0086] As indicated above, the vector spaces of the first and second vectors are different in the sense that they differ in the input characteristics of the input data they process, despite coming from a common source and from one file. The processing of subjectively descriptive material by NLP can thus be considered as resulting in a first vector in a semantic space (or a semantic distance space), whereas the processing of absolute values ​​on the identified properties (even though these properties can be represented as a selectable number of different signal properties) results in a second vector in a "property space" as output of the ANN.

[0087] In a preferred embodiment, an easily understood variant is used, namely Euclidean space rather than non-Euclidean geometry.

[0088] The artificial neural network functions to transform the measurable properties of the source file into an operational vector representation. The transformation produces a second independently generated vector, i.e., the second vector. The transformation is considered to be "feature extraction". In a preferred embodiment (in the exemplary case of audio processing), the feature extraction is realized using the Essentia™ app developed by the Music Technology Group at Pomepu Fabra University (see https: / / essentia.upf.edu / documentation / streaming_extractor_music.html).

[0089] Essentia™ (or its functional equivalent) is an existing library that provides a foundation for the analysis of source audio files to identify multiple audio descriptors, such as band energy, band histograms, and other measurable musical properties of the source track. In Essentia™, there are up to 127 of these audio descriptors. Each audio descriptor can be considered to be a quantized representation of a measurable parameter of the audio signal.

[0090] Returning to the exemplary context of audio files, the processing intelligence supporting Essentia™ performs feature extraction from source files, as well as equivalent classification mechanisms. Selecting appropriate ones of the audio descriptors within the subsets defines broader musical aspects or characteristics of each audio track, for example a programmer might combine a first subset of measured quantized representations [nominal] from audio descriptor bins 1, 15, 32, 33 and 108 (from the possible total set of 127 audio descriptors within Essentia) to define "rhythm", while a subset of measured quantized representations from audio descriptors 5-21, 43, 45, 50, 71-77 and 123-127 might define "timbre", and a third different subset might define tonality, i.e., the tonal characteristics of the performance. These subsets thus provide further musical and semantic properties of the sampled source audio track.

[0091] For other types of source files, such as video or image files, alternative measurable parameters are parsed from the source file to define alternative properties that can be used.

[0092] As mentioned above, with respect to audio, and audio properties in particular, music can be described using timbre, rhythm, tonality and texture, with timbre, rhythm and tonality properties being of particular importance.

[0093] 3. Measurable Musical Properties In this regard, it will be appreciated that: "texture" is generally reflected by a two-dimensional pattern in time-frequency space relating to the temporal evolution of the spectral content. Texture is therefore seen in a melspectograph or mel spectrum, which plots the frequency domain against the time domain. Within such a mel spectrum, the evolving texture can be learned by a neural network (as described below) by identifying patterns that evolve over time, such as (i) horizontal spectral lines interrupted in the high / mid frequency range, (ii) parallel vertical spectral lines stretching throughout the mid and high frequency range, and (iii) ascending or descending steps in the low to mid frequency range. Texture thus provides a further complementary semantic property that can be utilized in the assessment of track similarity / dissimilarity in the context of the present invention, through the provision of a further measurable measure in the property space.

[0094] "Rhythm" can be thought of as the arrangement of notes according to their relative duration and relative accentuation (see https: / / www.naxos.com / education / glossary.asp?char=PR#). As is evident, rhythm can be expressed as (but is not limited to): i) Beat loudness, calculated from beat and musical spectrograms, where aggregate values ​​reflect the mean and variance (https: / / essentia.upf.edu / documentation / reference / std_BeatsLoudness.html); and See https: / / essentia.upf.edu / documentation / reference / std_BeatTrackerMultiFeature.html), ii) Beats per minute "BPM" (https: / / see essentia.upf.edu / documentation / reference / std_BpmHistogramDescriptors.html, and See https: / / essentia.upf.edu / documentation / reference / std_BeatTrackerMultiFeature.html), iii) a BPM histogram (https: / / essentia.upf.edu / documentation / reference / std_BpmHistogramDescriptors.html), calculated from the signal and whose aggregate values ​​reflect the height and width of the first and second peaks; and See https: / / essentia.upf.edu / documentation / reference / std_BeatTrackerMultiFeature.html), iv) Danceability (see https: / / essentia.upf.edu / documentation / reference / std_Danceability.html); v) Onset Rate (see https: / / essentia.upf.edu / documentation / reference / std_OnsetRate.html), vi) Beat loudness per band calculated from the beat and musical spectrograms and reflected by the mean and variance over 6 bands (see https: / / essentia.upf.edu / documentation / reference / std_BeatsLoudness.html and https: / / essentia.upf.edu / documentation / reference / std_BeatTrackerMultiFeature.html).

[0095] Although the properties of rhythm are suggested from an Essentia perspective as a collection of six measurable attributes, it will be recognized that in practice more than seven measurable attributes may contribute to the properties, as reflected by (for example) reference to the mean and variance of certain musicological attributes. Those skilled in the art will appreciate that the multi-dimensional vector compiled for the "rhythm" property can thus be formed from other measurable attributes that provide a musicologically useful definition of rhythm, other than the suggested Essentia parameters. In the preferred embodiment, nominally 19 measurable attributes are assigned to the concept of rhythm, although other numbers of attributes may be used.

[0096] "Tonality" is the arrangement of pitches and / or chords of a musical work in a hierarchy of perceived relationships, stability, appeal, and directionality. In this hierarchy, the single pitch or triad with the greatest stability is called the tonic. Tonality is thus a woven system of tones in which one tone (the tonic) takes a central position with respect to the rest of the tones, which can be defined in terms of their relationships to each other (e.g., the tones of a major or minor scale). Harmony is a property of perceived tonality.

[0097] As will be appreciated below, tonality can be expressed as, but is not limited to, the following: i) chord change ratios calculated from the spectral harmonic pitch class profile (HPCP) (see https: / / essentia.upf.edu / documentation / reference / std_ChordsDescriptors.html); ii) chord number rates calculated from HPCP (see https: / / essentia.upf.edu / documentation / reference / std_ChordsDescriptors.html); iii) chord strengths, calculated from the HPCP, where aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_ChordsDescriptors.html); iv) HCPC entropy (https: / / essentia.upf.edu / documentation / reference / std_HPCP.html), which is calculated from the HPCP and whose aggregate values ​​reflect the mean and variance; and Please refer to https: / / essentia.upf.edu / documentation / reference / std_Entropy.html. v) key strength calculated from HPCP (see https: / / essentia.upf.edu / documentation / reference / std_KeyExtractor.html); vi) Tuning of diatonic intensities calculated from HPCP (see https: / / essentia.upf.edu / documentation / reference / std_TuningFrequency.html), vii) Tuning of equal temperament deviations calculated from HPCP (see https: / / essentia.upf.edu / documentation / reference / std_TuningFrequency.html), viii) Tuning of the untuned energy rate calculated from the HPCP (see https: / / essentia.upf.edu / documentation / reference / std_TuningFrequency.html), and ix) Chord histograms calculated from HPCP (see https: / / essentia.upf.edu / documentation / reference / std_ChordsDescriptors.html).

[0098] Although the tonality property is suggested from an Essentia perspective as a collection of 10 measurable attributes, it will be recognized that in practice more than 10 measurable attributes may contribute to the property, as reflected by the reference to the mean and variance of certain musicological attributes. Those skilled in the art will appreciate that the multi-dimensional vector compiled for the tonality property may thus be formed from other measurable attributes that provide a musicologically useful definition of tonality, other than the suggested Essentia parameters. In the preferred embodiment, nominally 33 measurable attributes are assigned to the notion of tonality, although other numbers of attributes may be used, these being obtained by applying greater or lesser granularity of the quantized measure. For example, a "chord histogram" is implemented as a 23-dimensional vector.

[0099] In terms of Essentia's treatment of another measurable attribute "chord strength", this is calculated by parsing the audio file by a moving window (frames) and extracting values ​​from each window (frame) to generate a sequence of numbers (one number per frame). The sequence is then used to calculate its mean and variance. Thus, in a preferred embodiment, the measure "chord strength" is assigned just two numbers, the mean and variance of the above sequence. This example shows how far the measurements used to evaluate the identified properties may deviate from the recommendations made in Essentia, even if the multi-dimensional vectors generated to reflect properties such as rhythm or tonality contain enough spectral information to give a definition of the properties that is acceptable to the user.

[0100] "Sound quality" is a relatively subtle measure and manifests itself in the complexity of sound measurable via the spectrogram of the sound. Sound quality is the perceived audio characteristics of a musical note, voice or tone. Sound quality distinguishes between different types of sound productions, such as choral voices and musical instruments such as strings, winds and percussion. It also allows a listener to distinguish between different instruments of the same category (e.g., an oboe and a clarinet, both of which are woodwind instruments). The physical characteristics of sound that describe the perception of sound quality include the sound spectrum and the signal envelope, and sound quality provides the ability to distinguish between sounds even if they have the same pitch and volume.

[0101] As will be appreciated below, sound quality can be expressed as (but is not limited to) the following: i) the Bark band crest (https: / / essentia.upf.edu / documentation / reference / streaming_crest.html), calculated from the Bark band filtered spectrogram and tallying the mean and variance of the identified Bark frequency range; and See https: / / en.wikipedia.org / wiki / Bark_scale#Bark_scale_critical_bands), ii) the Bark-band flatness db (see https: / / essentia.upf.edu / documentation / reference / std_FlatnessDB.html), calculated from the Bark-band filtered spectrograms and summarizing the mean and variance of the identified Bark frequency ranges; iii) Bark-band kurtosis, calculated from the Bark-band filtered spectrogram and tabulating the average of the identified Bark frequency ranges (see https: / / essentia.upf.edu / documentation / reference / std_DistributionShape.html); iv) Bark-band skewness, calculated from the Bark-band filtered spectrogram and summing the mean and variance of the identified Bark frequency range (see https: / / essentia.upf.edu / documentation / reference / std_DistributionShape.html); v) Bark Bandwidth, calculated from the filtered spectrogram and collating the average of the identified Bark frequency ranges (see Spectral Complexity, calculated from the spectrogram of a speech signal, where the collating value reflects the mean and variance) (see https: / / essentia.upf.edu / documentation / reference / std_DistributionShape.html), vi) Spectral Dissonance, calculated from the spectrogram of the audio signal, where the aggregated values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_Dissonance.html); vii) Dynamic Complexity calculated from the RMS envelope of the audio signal (see https: / / essentia.upf.edu / documentation / reference / std_DynamicComplexity.html); viii) High frequency content, calculated from the spectrogram of the audio signal and tabulated as the average (see https: / / essentia.upf.edu / documentation / reference / std_HFC.html); ix) Pitch salience, calculated from the spectrogram of the speech signal, where the aggregated values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_PitchSalience.html); x) Spectral Complexity, calculated from the spectrogram of the speech signal and whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_SpectralComplexity.html), xi) Spectral Energy High Frequency, calculated from the spectrogram of the speech signal and whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_EnergyBand.html); xii) Spectral Energy Low Frequency, calculated from the spectrogram of the speech signal, whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_EnergyBand.html); xiii) Spectral energy mid-high frequencies, calculated from the spectrogram of the speech signal and whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_EnergyBand.html); xiv) Spectral energy mid-low frequencies calculated from the spectrogram of the audio signal, the aggregated values ​​of which reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_EnergyBand.html); xv) Spectral entropy, calculated from the spectrogram of the speech signal and whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_Entropy.html); xvi) Spectral flux, calculated from the spectrogram of the audio signal and whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / streaming_Flux.html); xvii) Spectral kurtosis, calculated from the spectrogram of the speech signal and tabulated as the mean (see https: / / essentia.upf.edu / documentation / reference / std_DistributionShape.html); xviii) Spectral Strong Peaks, calculated from the spectrogram of the speech signal and whose aggregate values ​​reflect the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_StrongPeak.html); xix) The zero-crossing rate calculated from the speech signal and tallying the mean and variance (see https: / / essentia.upf.edu / documentation / reference / std_ZeroCrossingRate.html); xx) MFCCs calculated from the spectrogram of the audio signal and averaged together (see https: / / essentia.upf.edu / documentation / reference / std_MFCC.html), and xxi) Spectral contrast, calculated from the audio signal and collating the mean and variance of both peaks and valleys (see https: / / essentia.upf.edu / documentation / reference / std_SpectralContrast.html).

[0102] The sound quality property is suggested in terms of Essentia as a collection of 21 measurable attributes, although it will be recognized that in practice more than 22 measurable attributes may contribute to the property, as reflected by the reference to mean and variance values ​​of certain musicological attributes. It will be understood by those skilled in the art that the multi-dimensional vector compiled for the "sound quality" property may thus be formed from other measurable attributes that provide a musicologically useful definition of sound quality other than the suggested Essentia parameters. In the preferred embodiment, nominally 75 measurable attributes are assigned to the concept of sound quality, although, as noted above and as will be understood by musicologists, other numbers of attributes may be used, which may be obtained by applying a greater granularity of the quantized measures.

[0103] In the context of audio track evaluation and track finding, the properties of tonality, rhythm, and timbre are important in that they provide a basis on which measurements of the subjective characteristics of the source file can be objectively evaluated. These properties can be derived from the Essentia™ attributes, as described above, or a subset of these Essentia™ signal attributes, or an equivalent library that identifies suitable audio descriptors. In fact, as will be recognized below, the present invention selects 19, 33, and 75 quantized representations for the properties of rhythm, tonality, and timbre, some of which overlap with the Esssentia™ toolbox, while others are variants or different signal measures. As a result, the number of quantized representations is not fixed, but rather varies according to the musicologist's belief about which signal attributes are required to define the particular property to be evaluated.

[0104] Given the above, it is a design choice how or which measurable attributes one skilled in the art will choose to define properties suitable for use in assimilation processes. It will be appreciated that, in some respects, the properties being evaluated are technical in nature and measurable by existing technical processes, so that the lack of a consistent definition of what constitutes a "property" is technically inconsistent, if not surprising, since rhythmic properties can be checked, for example, to include or exclude certain Essentia measurements. Rather, file content properties are to some extent esoteric and subjective. What is important, however, is the mapping of deterministic but subjectively constructed measurable objects in property space to semantic evaluations in an independent but fully related and corresponding semantic space.

[0105] 4. Artificial Neural Networks (ANN) In accordance with concepts of various aspects and embodiments of the present invention, pairwise similarities / dissimilarities in property space are back-mapped to initial semantic similarities / dissimilarities in semantic space (e.g., expressive and subjective linguistic descriptions). This is a multi-stage process that may include multiple neural networks operating in parallel. The use of multiple parallel ANNs allows for control of musical modalities, although a single ANN could also be used. Aspects of the present invention relate to training a neural network that processes the extracted properties to evaluate dissimilarities in property space.

[0106] Figure 1 shows a flow diagram of a preferred process 100 for assessing dissimilarity of files, particularly audio files, and a process by which an artificial neural network can be trained in accordance with the present invention. Figure 1 thus corresponds to and extends the process described above with respect to Chapter 1, "Similarity / Dissimilarity Assessment of Context Descriptions in Semantic Space."

[0107] Audio files are used as an example of the underlying processing because audio files, especially music files, can be subjectively interpreted from the perspective of individual human perception.

[0108] Pairs of files are selected (102) as a training set of hundreds (preferably thousands) of source files and semantically compared through ANN evaluation. In a first path, using NLP, the artificial neural network extracts (104), i.e., performs a generating / embedding process, a representation vector for the semantic meaning contained in (or as an accompanying description of) the text-form metadata associated with each file, e.g., each audio track of the pair. As a result, typically, a 512-dimensional vector is generated (106) from TensorFlow (trademark), etc., representing the derived semantic meaning as an operable value that can be the object of evaluation.

[0109] The ANN can thus effectively represent in tabular form the separation distance in vector form between all N files in the training set, where N is typically 500, generally more files than a few thousand. Although the processing complexity increases as the number of samples increases, the more samples in the training sequence, the higher the granularity and the accompanying reliability. In short, the more samples, the better. However, as an ANN training option, this process can make a secondary selection of pairs indicating that the distance separation is very similar or very dissimilar, i.e., the training can be based on extreme situations.

[0110] At this point, representing the relative distance separation in tabular form is abstract in that it does not reflect a scaled value of similarity / semantic proximity in a multi-dimensional space, although an absolute distance exists from the perspective of the separation distance value (e.g., 5 measurement units, 57 measurement units, or 1013 units). Assuming N is large enough, for each file (track), it can be seen that at least a reasonably specific number m (a positive integer, m << N) of the N files are similar or dissimilar.

[0111] In a preferred embodiment, for each source file in the training set, e.g. song "A", an arbitrary number, e.g. 10, of the closest vectors in the semantic distance vector space are selected. This forms groups or clusters of closely semantically related songs. Statistically, clustering 0.1% of the total in a training set of thousands or perhaps tens of thousands of source files is statistically acceptable in terms of possible semantic closeness. In practice, for the entire set of songs in a reasonable training sequence, closeness can be considered to be in the range of about 0.05% to about 1%, although it is likely that the user-perceived phonetic dissimilarity increases with increasing percentage values.

[0112] For a song "A", the system intelligence is configured to consider the "m" (e.g., 10 and m>1) closest songs to be semantically similar in the sense that the user perceives them as close. This is reflected by setting the distance between the m songs around the vector of song "A" to zero, which is then recorded in the data record. For all songs other than the m closest ones, the system intelligence is configured to consider these songs dissimilar, i.e., semantically dissimilar in the sense that the user perceives these other (non-m) songs to be far apart. As a result, songs that are dissimilar to song "A" are identified as having a distance of 1. Thus, for each audio track evaluated, 2*m pairs of records are generated and stored as searchable and accessible records by the system. Choosing a value equal to m ensures that the training of the neural network is not biased towards one or the other extreme values ​​(in terms of similarity or dissimilarity).

[0113] The processing load of the ANN can in most cases be streamlined (114) at some point in the training process, as will be seen below. Specifically, optimized training of the ANN is achieved through training using extreme cases rather than a large number of similar values. As a result, by using the furthest and nearest separation distances for any pairwise associations, the time to refine the adaptive weights applied to the neurons of the ANN is reduced.

[0114] As outlined above in Chapter 1 “Similarity / Dissimilarity Assessment of Context Descriptions in Semantic Space”, a first semantic criterion in the form of a “first vector” is therefore established as the basis for ANN training.

[0115] Returning to the original source files (e.g., audio tracks), a second pass 126 for assessment and evaluation again searches pairwise for symbolic patterns across the entire training space of N files (e.g., N audio tracks). In particular, as described above in Chapter 2, "Distance Assessment Based on Extracted Properties," this process performs feature extraction 130 of signal attributes by parsing source (audio) file pairs to generate bins of quantized representations of signal characteristics as described above in Chapter 3, "Semantic Properties" [in the specific exemplary context of speech / music]. The individual bins of the quantized representations of signal characteristics can then be appropriately identified and selectively grouped 132 to define semantic / subjective musical properties, i.e., rhythm, tonality, timbre, and texture, that can be more absolutely assessed and manipulated within the property space.

[0116] Please refer to the processes of FIGS.

[0117] Figure 2 is a schematic diagram of a system architecture for training a system including an artificial neural network according to a preferred embodiment. Figure 3 is a flow diagram of a preferred process for training the neural network of Figure 2 to assimilate a semantic vector space with a property vector space to identify property similarities and dissimilarities between source files.

[0118] For each pair, two of the N files (e.g., digital audio files 302, 304) are selected from a training database of files 306 for evaluation and interpretation by the system 300. The system 300 may be implemented within a more general system intelligence supported by a distributed processing system of servers or interactive processors, and includes multiple artificial neural networks.

[0119] As described above, initial processing of each selected audio file in a feature extractor 301 (such as Essentia or its functional equivalent, whether source file of a different type, such as a context or an image in the exemplary case of audio file processing) generates bins of quantized representations of signal characteristics, which are selectably grouped to define each of a number of outputs numerically representing different semantic properties P, e.g., timbre "PTi", tonality "PTo" and rhythm PR. The numerical representations of these subjective properties for each audio track (e.g. PTo2 is the tonality property extracted from track 2) are commonly applied as inputs to a dedicated parallel neural network that optimizes the weights in the evaluation process for each property.

[0120] In the illustrative embodiment of the audio file and track search system, the rhythm "NN R ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture NN TX There is an independent ANN for each of the 318.

[0121] Musical textures are a special case and require a different processing flow, and are described in more detail below.

[0122] There may be more or fewer parallel ANN chains for processing and evaluation of other training data such as images. The ANN chains, numbered 4 in FIG. 2, can be considered as independent processing paths, branches or routes, and thus sub-networks of the network). The number relates only to the number of semantically distinguishable properties. The system may actually work with only a single chain, processing the data in multiple passes to arrive at a synthetic result suitable for evaluation.

[0123] Rhythm "NN R The ANN for "Tonality" 310 therefore receives an input representation of only the "Rhythm" property, which is assembled (in the preferred embodiment) from a vector of 19 components, i.e., 19 extracted signal attributes. TO The ANN for "Tonality" 312 therefore receives an input representation of only the "Tonality" property, which is (in the preferred embodiment) assembled from a vector of 33 components, i.e., 33 extracted signal attributes. TI The ANN for 314 therefore receives an input representation of only the "tonality" property, which is assembled from (in the preferred embodiment) 75 components, ie a vector of 75 extracted signal attributes.

[0124] As mentioned above, the definition of each property may vary in terms of the number of extracted signal representations for each bin and / or the nature of the attributes. Thus, in the explicit context of audio files and in the use of Essentia, all of the available attribute signal bins (including, for example, the Bark band flatness db and dynamic complexity of sound quality) may be used, some may be used, or others not mentioned above may be used instead or alternatively in an expanded number. The definition of a "property" is therefore (to some extent) subjective, but this subjectivity is irrelevant if a consistent approach to the definition of the property is adopted. In other words, the programmer can decide how to define the subjective property by identifying and selecting the desired measures of the signal attributes.

[0125] Rhythm "NNR ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture NN TX The ANN for 318 therefore determines and refines weight values ​​that account for the differences in these properties, and the weights and biases are refined by an iterative process involving a backpropagation algorithm whose task is to search the entire training set and appropriate adjustments to each trainable parameter. Because the backpropagation process will be understood by those skilled in the art, it is important to point out the intent of what is being matched, as well as the objectives and advantages achieved by the architecture and process described herein.

[0126] Musical textural issues have also been found to have a role to play in assimilating content property measures (derived from vector representations of the measurable properties of each track by pairwise comparison) to semantic measures (derived from vector representations of the semantic descriptions of each track by pairwise comparison).

[0127] The approach taken by some embodiments of the present invention therefore places emphasis on human perception rather than statistical mapping based on interpretation of absolute numerical data in the operation of the ANN, emphasizing the importance of human emotional perception over strict machine learning.

[0128] Briefly referring to FIG. 4, a typical Mel spectrum 500 of an audio track is shown. As will be understood below, a Mel spectogram (also known or referred to as a Mel spectrum) is pseudo-logarithmically spaced roughly similar to the resolution of the human auditory system, and is therefore a more "biologically inspired" measure of musical perception. A Mel spectrum is a representation of the short-term power spectrum of audio across the frequency spectrum, based on a linear cosine transform of the logarithmic power spectrum shown in a non-linear Mel scale of frequency. In a Mel spectrum, considering the power spectrum in a frequency bin (nominally) 50 Hz to 100 Hz would be equivalent to considering the power spectrum over a wider frequency range at higher frequencies, e.g. 10 kHz to 20 kHz, as well as 400 Hz to 800 Hz. The process of how a Mel spectrum is generated is known, as these frequency bins are perceptually equally important in terms of musical interpretation.

[0129] Furthermore, while keeping in mind that audio tracks may have musical themes that vary from fragment to fragment and thus affect the Mel spectrum, for purposes of describing the preferred embodiment, the audio, and therefore the themes, within the excerpted window are assumed to be relatively constant. Of course, an audio track such as Queen's "Bohemian Rhapsody" could instead be divided into fragments that are each subjected to a discrete evaluation process within the semantic space.

[0130] The Mel spectrum is not only a partial sample, but is also complex in nature in that it has dimensions in both the time and frequency domains. In the resulting two-dimensional matrix of time and frequency domain components, the subject matter can be identified by isolation of patterns of interest. Such patterns of interest can be seen in the spectral components plotted as frequency (ordinate) against time (abscissa): i) parallel vertical lines 502 extending throughout the mid-high frequency range, ii) horizontal lines 504 interrupted in the mid-high frequency range, iii) rising (506) or falling (508) steps in the low-mid frequency range. Other patterns also exist with the Mel spectrum having these discoverable events, as will be seen below.

[0131] Property textures can thus be derived from analysis of the mel spectrum and the identification of patterns and trends by the ANN, which in particular provide additional vector components in the property space for use in training the system 300 of FIG.

[0132] The output from each ANN, including the contribution to the texture, is then divided into multiple outputs for each track used in the training sequence / training data set, OR x , O.T.O. x , O.T.I. x and OTX x (x represents the associated track number, i.e., track 1 or track 2) into a concatenated or otherwise assembled multi-dimensional output vector as outputs in property space. Note that the exact length of each output vector has some design freedom, and the length is chosen to be sufficient to allow objective evaluation and differentiation within the property space. In a preferred embodiment, the output from each essentially parallel processed ANN chain is combined into a 64-dimensional output vector OR for each of the properties of rhythm, tonality, timbre and texture (the latter requiring different processing as described below). x , O.T.O. x , O.T.I. x and OTX x Contribute to.

[0133] Referring again to FIG. 2, a Mel spectrum 500 is generated for each of the selected pair of files (digital audio tracks in this exemplary case) 302, 304. This process will be well understood by those skilled in the art. Both tracks are first processed in a convolutional neural network "CNN" 320 to obtain individual vector outputs for each track, which are then fed to the assigned ANN (CNN) 320 for texture estimation. Tx 316) and is processed and interpreted by NN Tx 316 is therefore in parallel with other neural networks whose role it is to estimate and embed the rhythm, tonality and timbre vectors. Tx Each vector output OTX1, OTX2 for tracks 1, 2 from 316 is also a 64-dimensional vector in the preferred form, and each of these outputs is then input to each track (OR x , O.T.O. x , O.T.I. x ) to produce a 256-dimensional vector for each track 1, 2. This 256-dimensional vector, whose exact length is again a design choice as discussed above, is the "second vector in Euclidean space" discussed above.

[0134] System intelligence is a distance measure in the semantic space (four outputs OR x , O.T.O. x , O.T.I. x and OTX x The system includes a comparator 330 that functions to evaluate the distance between the composite second vector assembled for each pair of tracks assembled from the first vector and the second vector assembled from the second vector, along with a corresponding distance measure in the property space. The system intelligence thus establishes a relationship between the two spaces. As an example of how the system may operate to compare distances between vectors, the system intelligence may utilize a squared absolute distance calculation.

[0135] The system intelligence then serves to compare the first and second vectors to the behavioral view and align the second vector to an evaluation of the closeness of the first vector. In other words, the system intelligence compares semantic distance (based on the evaluation of the textual form) to property distance. Stated differently, by using the first vector in the semantic space (based on the human-generated description of the source file) to evaluate the second vector in the property space (associated with the extracted measurable properties of the content), the output of the second neural network can be converged to the semantic result of the first neural network through changing the weights in the ANN. The objective is that the recombined concatenated output [and in particular the Euclidean property vector evaluated for the difference 330 between the training tracks] can also be expressed on a scale of 0 to 1, and rhythmically "NN" R ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture NN TX The neural network weights of each of the ANNs for 318) are adjusted so that the Euclidean property distance measure 330 tends to, and preferably does, replicate the semantic quantized distance. In the quantization approach, other scalings may be applied rather than hard levels.

[0136] In particular, the rhythm "NN R ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture NN TX The weighting coefficients applied in each ANN 318 are adjusted by the known process of backpropagation so that the resulting Euclidean property distance measure 330 between each pair of compared tracks / files converges to a distance measure in semantic space that ideally and ultimately exhibits a high degree of accuracy in correlation. As will be understood below, the backpropagation process thus trains each neural network by adjusting the applied weights based on contrasting objectively measurable signal attributes used to define discriminatory file properties.

[0137] The effect of evaluating two independent paths, the first path processed entirely within a semantic space and the second path forced into a measured property space based on measurable characteristics of the subjectively evaluated properties, produces an emotionally perceived system that more closely aligns with human perception of either closeness or dissimilarity. An illustratively relevant effect in exploring tracks between different genres of music is that quantitatively more and qualitatively better associations are made between the different tracks, even if they may appear objectively to be unrelated music genres on initial inspection, because the differences between them are prominent. This represents a step forward in solving problems such as cold starts in providing improved and more reliable recommendation tools that can recommend related content to new or existing users. In fact, the processing and system architecture is emotionally perceived to the extent that it allows for language independent embedding of semantic meanings. This means, for example, that Chinese and English can be overlaid without affecting the semantic interpretation or results.

[0138] As a further component of the assessment of the semantic properties of audio works within an objective Euclidean space, the Melspectograph is processed through a convolutional neural network (CNN) to generate a vector component representation of a subjective, yet complementary, notion of musical "texture".

[0139] 5 illustrates the convolution and pooling layers within the artificial neural network allocated to the mel-spectral interpretation, in particular the deep learning required for identifying important musical patterns and trends within the tracks being evaluated. The convolution process targets the two-dimensional nature of the spectral input matrix 600.

[0140] As mentioned above, the mel spectrum contains time-varying patterns reflecting texture that serve as an additional component for the property similarity / dissimilarity assessment in the property space. To identify these textural trends in the two-dimensional mel spectrogram, filters in the convolutional neural network are trained to identify patterns in the mel spectrogram, specifically to identify optimized parameter values ​​in each of these filters that produce filter outputs that represent a high degree of confidence in identifying patterns / trends in the input matrix. In this way, the parameters in each filter are adjusted, as can be seen from the nature of the operation of the ANN, such that each filter is able to detect specific inputs that are related to the desired subjective properties, e.g. rhythmic and / or melodic patterns contained in the mel spectrum of the track under study.

[0141] In this regard, the processing chain of an ANN for texture includes successive convolutional layers. For example, layers 1, 3 and 5 may be implemented as convolutional layers with 128, 128 and 64 neurons, respectively, with each filter having a kernel size (i.e., the size of the filter matrix) of 3. During training, a filter 602 is developed stepwise [first with an untrained and then with a refined set of parameters] over the spectral input matrix 600. By applying the filter 602 to the input data, the output matrix 604 gives a positive matching result between the input values ​​in the overlap matrix. For example, a simple example is as follows:

[0142] [Table 1]

[0143] In an iterative phase, the values ​​of the parameters in the filter are then changed and the 2D input is re-run to determine whether the new filter coefficients yield better or worse matching results for the same input data.

[0144] [Table 2]

[0145] By going through all possible filter positions in the 2D input data, a further result matrix 604 of positive results is obtained, which represents the ANN attempting to optimize the filter coefficients / parameters to maximize matching. In Figure 5, the result matrix shows a value of 4, indicating a high correlation with filter 602, and therefore a high matching rate, and a higher chance of identifying interesting patterns in the input data, compared to poor matching rates indicated by 0 and 1.

[0146] As with any CNN, more filters allow for more patterns to be identified, but at the cost of more parameters and more training data.

[0147] Preferably, for reasons of convenience, each convolution is followed by a max-pooling layer with an appropriate kernel size, for example a 2x2 matrix / kernel. The effect of the max-pooling approach is shown in the lower part of Fig. 5, where the result matrix 606 is decimated to generate a new smaller input matrix that is processed in the successive convolution phase. As will be seen, max-pooling focuses on one block of output and rejects all values ​​except the maximum value in the analyzed block on the assumption that the smaller values ​​are statistically irrelevant for further processing. In Fig. 5, applying the 2x2 max-pooling approach to the 4x4 input matrix from the previous convolution stage results in four independent blocks, each of which contains four (result) values. The max-pooling result is thus a first 2x2 max-pooled matrix 608 in which only the maximum value obtained is kept. This first 2x2 max-pooled matrix 608 is then input to the successive convolution layers. Consequently, max pooling reduces the operating dimension of the matrix, reducing the dimensionality across different (successive) layers of the ANN.

[0148] Using a max pooling approach improves computational efficiency since each neuron introduces a parameter that needs to be learned and the input matrix size constraint reduces the amount of data that would otherwise be required to mitigate inadequate granularity and imprecision in computing the parameters / weights.

[0149] A CNN therefore typically contains several convolutional layers interspersed with max pooling layers.

[0150] The output of the last max-pooled layer is flattened, i.e. all matrix columns are concatenated, to generate the texture estimation, i.e. the musical texture NN. TX 318 to form a single vector that serves as input to a dedicated neural network.

[0151] Before discussing the general form and operation of the ANN, particularly as shown in relation to FIG. 6, the flattened output from the CNN 230 is preferably fed to a dedicated texture neural network NN having an optimized deep learning modified linear unit ("ReLU") activation function. TX Note that the texture is applied as a (say) 64-dimensional vector that is input to a hidden layer of 256 neurons of 318. TX 318, at its output, the Mel spectrum components O TX1 , O TX2 , and provide a 64-dimensional vector (in the form of an activated linear function) representing each of the vectors O TX1 , O TX2 is assembled with other output vectors representing the evaluated properties of each file, namely tonality, timbre and rhythm. The resulting 256-dimensional vectors for each of the two paired files are then subjected to a distance evaluation in Euclidean space, as described above and shown in FIG.

[0152] The early / top convolutional layers of the CNN function to identify filter weights that are applied across neural nodes to define parameter functions that can be used to enable identification of those patterns of interest within the Mel-spectrum [i.e., the input of the CNN]. The values ​​of the parameters 612-620 of the filter matrix are thus learned by iteration and backpropagation that tests the viability of alternative values ​​and optimizes the output, with the optimization being performed through successive passes across the source input data and the varying source inputs of the training set.

[0153] FIG. 6 is a representation of an artificial neural network 700 that may be used within the various ANN property processing chains of FIG.

[0154] Rhythm "NN R ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture (Post-convolution) NN TX Each of the ANNs for 318 includes at least one and typically multiple (1st through kth) hidden neuron layers that include at least the same number of individual neurons 704-718 as the multi-neuron input layer or level 702 following the multi-neuron input layer or level 702. The kth hidden layer provides an output level 720, with the number of neurons at the output typically being less than the number of neurons in the preceding kth hidden level.

[0155] In terms of basic neuron mapping, the output from each neuron (e.g., in the first input layer) is many-to-many mapped as an input to each neuron in the immediately succeeding (e.g., first hidden) layer. The kth hidden layer, i.e., the penultimate layer of each ANN, provides outputs O1 through O m is a linear function, we set multiple inputs to each output (O1 to O m ) (such as that described in https: / / towardsdatascience.com / activation-functions-neural-networks-1cbd9f8d91d6).

[0156] Each quantized signal representation extracted for each identified property (in the case of rhythm, tonality and timbre) or the flattened output from the CNN function (in the case of texture) is fed to one of the neurons of the input layer 702 (i1 to i n )

[0157] Take neuron 712 as an example. In FIG. 6 (boxed representation on the left), the neuron has multiple weighted inputs w i,1 , w i,2 , w i,3 , w i,r It can be seen that the summation function 730 receives a second bias input b, which is in fact just a learned constant for each neuron in each layer. i The processing intelligence is weighted by w i and bias b i The output a from the summation function 730 is estimated and then modified via a backpropagation process that employs the pairwise Euclidean property distance measure 330 as an influence on how it is assimilated / mapped to the pairwise target distance in the corresponding semantic space. i is subjected to a nonlinear activation function f (reference numeral 734). i The output of is propagated to the next layer.

[0158] In an exemplary sense of a pairwise comparison of audio data signals, inputs i1 to i n can be derived from the set of Essentia features mentioned above for timbre, tonality and rhythm, whereas CNN Melspectrum is derived from a texture-specific artificial neural network NN TX The final output o1~o m forms a 64-dimensional embedding vector for each specific property, e.g., sound quality OTI1 and texture OTX2.

[0159] For the preferred implementation of Figure 6, there are at least two hidden layers. The first hidden layer contains 512 neurons. The second hidden layer contains 1024 neurons. The activation functions of both of these hidden layers are preferably ReLU functions as described in https: / / en.wikipedia.org / wiki / Rectifier_(neural_networks).

[0160] Referring now in detail to FIG. 3, there is disclosed in general terms a training process by which the system of FIG. 2 is trained.

[0161] From the entire training set of audio tracks (or a selected subset of pairs), a pair of tracks is selected (402) for semantic and musical property comparison. Both tracks are then subjected to feature extraction 404 to identify a set of measurable descriptors that can be used to define a property, e.g. rhythm. Texture is subjected to a modified process given the mel-spectral nature, as described above. For each pair, the properties are jointly processed by the system intelligence to train a network to determine the rhythm "NN". R ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture NN TX The weights and biases applied (406) in each of the parallel artificial neural networks for 318 are refined. Whether the ANN process includes a CNN or not, each of the multiple parallel neural networks refines the embedding vector output 350, 352 [contribution vector OR ] in a (typically Euclidean) property space for each pair of files under consideration. x , O.T.O. x , O.T.I. x and OTXx]. An evaluation / determination (410) of the Euclidean property distance between the vector outputs 350, 352 is then performed for each file. The determined Euclidean distances calculated by the neural network are then mapped / contrasted with the semantic distances (in semantic space) between the same files (as described with respect to FIG. 1).

[0162] If it is assessed (418) that there is a general numerical correspondence (416) between the property distances and the quantized semantic distances, which is unlikely for the initial weight and bias values ​​at the start of training with initial tens / hundreds of pairwise comparisons, then a determination can be made whether the weights and biases of the contributing ANNs satisfy an agreeable rule. This allows ANN training to be shortcutted without exhausting all pairwise comparison options, although optimization in each NN improves as the number of pairwise evaluations and revisions of weights and biases continues to increase.

[0163] From a practical standpoint, systems are typically configured to perform several runs or "epochs" through the training set. Training can be stopped if (a) the training loss does not improve over several epochs, or (b) the validation loss (on unseen data) does not improve. Note also that overfitting is indicated when the training loss improves but the validation loss does not.

[0164] However, at the beginning of training, there will probably be a noticeable difference as well as improved filter weights w i and bias b i It may be necessary to improve the performance of the ANN in terms of parameter identification using a method such as the above. This is accomplished by optimizing the performance of the ANN through use of the entire set of training data. As a result, the training process iteratively follows a path of pairwise evaluation of all elements in the training set. This is represented by decision block 414 and the negative or positive paths therefrom.

[0165] If there is a close correspondence (a positive path between the quantized semantic distance and the (typically used) property distance obtained from the vector outputs 350, 352) iterated over consecutive files, it can be assumed that weight and bias optimization is achieved (at least to an appreciable and acceptable degree).

[0166] Returning to paths where there is a significant number of discrepancies between the semantic and property space distance measures (i.e., negative outcome 420), it is necessary to adjust the filter parameters and in particular the weights and biases applied to one or more neural networks. The aim of this adjustment is to achieve numerical convergence between the vector distance dissimilarity measures in the associated property space, i.e., the corresponding distance dissimilarity measures in the semantic space. In this regard, it should be noted that values ​​in the property space will always vary from the hard values ​​between 0 and 1 in the semantic distance space, since there are perceptual and absolute differences between dissimilar pairs of tracks (even if the compared tracks are cover versions of the same song). Examining the loss or overfitting after each epoch is a typical approach.

[0167] The system's processing intelligence then adjusts (422) the weights and biases through backpropagation to seek convergence between the (numerical) semantic and property distances. These adjusted weights are then applied to the neurons of the various neural networks, as shown in Figure 2, in order to improve the match of the next file pair in the training set.

[0168] The training of the ANN results in distance values ​​in a property metric space that reflect track dissimilarity based on pairwise comparisons. As a result, when trained, any distance in the property metric space is accurately and reliably mapped to an actual perceptible difference in semantic space. The modification of the weights and biases in the neurons of the ANN is a transformation function or mechanism that maps the property space into an abstract semantic space.

[0169] If the training set is exhausted, then it is determined that the neural network is optimized, which is reflected by the yes path 424 from decision block 414.

[0170] As will be understood below, each processing chain for each extracted property is a machine. In the exemplary case of audio evaluation, there are four machines, one each for rhythm, tonality, timbre and texture. To optimize the training process, it has been found that each independent machine makes an independent and separate contribution to the final vector representation 350, 352 in the property space. As a result, a preferred approach based on pairwise evaluation for semantic evaluation in the semantic space employs a weighting of importance between each of these approximately parallel individual machines. In other words, the training process determines the relative importance between the specific audio descriptors (associated with each property) in each input to each ANN. This means that each machine learns which of the specific contributing extracted measurables has the greatest impact in modifying the final result reflecting the desired human subjective evaluation (in the semantic space). To achieve this, the system operates to evaluate two tracks in each machine. Each machine is then configured to identify similarities or dissimilarities between the sets of quantized representations used to define each property evaluated by the particular machine. In adjusting its bias and weight factors in the backpropagation process, the machine operates in a preferred embodiment to downplay, i.e., lower the relative importance of a property (e.g., rhythm) if there is a dissimilarity (in the property distance space) with the corresponding property simultaneously evaluated in a particular pairwise comparison in the semantic space. In other words, the identified dissimilarity does not contribute to generating a set of biases and weights that results in a better match with the semantic evaluation and semantic differences between the pairwise audio tracks evaluated in the semantic space. Thus, across all machines, the system intelligence implicitly values ​​other properties (of both tracks) for a particular machine because other properties are evaluated as having a greater impact on the alignment with the semantic evaluation, i.e., the rhythm vector components OR x But the tonal vector component OTO xIn fact, by extending this principle to individual quantizations, dissimilarity between the individual quantizations identified by the machine in each pair of tracks (such as the Bark band top values ​​that contribute to the property timbre in Essentia) means that such individual quantizations are less important in matching property-based vectors to semantic-based values.

[0171] It will be appreciated that the accuracy of the resulting neural network's transformation function is dictated by the robustness of the training data, particularly the size of the matrix, and thus, while 10,000 audio files may be evaluated and 10,000 corresponding vectors generated, significantly fewer or significantly more files may be evaluated by the NLP for embedding.

[0172] Now, to build a library for comparison, it is necessary to non-comparatively simply process (426) each file in the training set through an ANN to generate a Euclidean vector for the track. The vector can then be stored (430) in a database as a value matched against the filename, e.g., song title and artist or other form of identifier. Because the vector contains distinct components that can be attributed to specific file properties, the vector itself can be parsed to make the specific identified properties searchable. For example, if rhythmic commonality is a priority requirement, then a specific (preferred, but exemplary case) contributing 64-dimensional output OR x If there is any numerical closeness between the source of and the reference file, then the semantic closeness of the rhythm is decisive.

[0173] In other words, the rhythm "NN R ”310, tonality NN TO 312, sound quality NN TI 314 and Musical Texture NN TXWhen the individual artificial neural networks for 318 are optimized, the measurable properties of the (exemplary) audio tracks are reliably reflected in the multi-dimensional vectors generated by processing samples of the audio tracks (e.g., parts or whole pieces of music) through various NNs with optimized weights and biases. As a result, based on an absolute value scale, the actual perceptible dissimilarity or similarity can be evaluated for each track, including new tracks that were not used in the training data set. At this point, the semantic distances used for training can therefore be ignored, since the semantic space is mapped to an absolute scale where large numerical distances represent user-discernible dissimilarity, whereas close numerical values ​​accurately represent contextual similarity.

[0174] FIG. 7 is a flow process 800 used by a preferred embodiment for evaluating measures of affectively perceived file dissimilarity, particularly in the context of audio files.

[0175] Once the neural network of FIG. 2 has been trained, an audio track (or file of an appropriate category) is selected (802). The selection is typically made by a user, such as an owner or subscriber of a music library or service. Alternatively, the selection may be in the form of an upload of a song or file containing an original work. The selected or uploaded "initial" audio file is then processed to perform feature extraction 804 of identifiable properties, such as tonality. The neural network of FIG. 2 then processes (806) the extracted features with optimized weights and biases to generate a first file vector V (in Euclidean property space or some other suitable property space) that represents a number of user-identifiable or user-selectable, system-measurable properties of the particular file. FILE The file vector V FILEBy querying (810) a first audio file in a library indexed by both the file identifier and the associated file vectors (of those other files), the files registered in the library can be listed (812) in order of decreasing semantic similarity to the first audio file. This can be achieved or supplemented using kNN analysis.

[0176] FIG. 8 is a system or network architecture 900 that includes an accessible database 902 that contains vector representations reflecting file similarity / dissimilarity measures according to an embodiment of the present invention.

[0177] Typically, a network 902 (such as the Internet) allows communication to and from devices such as a server 904, a home computer 906, and a smartphone 908. These three types of devices represent both the processing intelligence within the system 900 and the access points to / from the system 900, without limitation. The server 904 typically supports the artificial neural network 905 described above, particularly with respect to Figs. 2 and 6. However, the system intelligence may be more distributed, including cloud-based or distributed across multiple interconnected servers. For purposes of clarity only, the system intelligence is shown as simply a block within the server, but it will be readily recognized that there is computing power within smartphones and computers as well. The server, like other interactive devices, includes general control firmware and software 914, for example, to support web-based access and / or to control user registration to services managed by the server or other service providers 912, and / or to support communication protocols. The server may restrict access to information loaded or extracted from a source database 306 coupled to the server, for example, via a LAN or WAN. This access may be via a computer 906, a smartphone 908, or the like.

[0178] The source database may actually be an existing file library, such as a catalog of audio files. Files in the source database are thus extracted and processed by the server over time to generate file attributes 922 (such as track name and artist) and generated Euclidean vector measures (V FILE )920.

[0179] A user interface 930, for example a touch screen of a Graphic User Interface "GUI" on a smart phone, is provided to allow access to a search tool software application that allows searching for tracks sharing semantically close properties in accordance with the present invention. The software may be local or accessed via a web browser that allows interaction with the server 904, database 306 or a separate service provider (such as a social media company that has access to the content). Alternatively, the software may be hosted as a web-based service. Preferably, the GUI 930 presents the user with a number of "soft" slider controls for selectable properties or listening / search preferences, for example a first slider may be for rhythm. The position of the sliders may then be selected by the user to select the individual contributing multi-dimensional vectors OR ... x , O.T.O. x , O.T.I. x and OTX x The settings of the sliders in the GUI can thus be targeted to specific vector aspects of the processed tracks 920 stored in the system.

[0180] Insofar as certain configurations are not mutually exclusive, the various embodiments described herein may be combined to generate complementary functions or systems that enhance the functionality of the system and / or assist in the effective identification of user-perceivable similarities and dissimilarities. Such combinations will be readily recognized by those skilled in the art given the entirety of the above description. Similarly, aspects of the preferred embodiments may be implemented in independent configurations where more limited functional configurations are suitable. Indeed, unless the features of certain preferred embodiments are expressly identified as mutually incompatible or the surrounding circumstances suggest that they are mutually exclusive and cannot be readily combined in a complementary and / or supporting sense, it will be understood that the entire disclosure contemplates and assumes that certain features of these complementary embodiments may be selectively combined to provide one or more comprehensive, but slightly different, technical solutions. In terms of the process flows suggested in the accompanying drawings, these features may vary in terms of the exact execution points of steps within the process, so long as the overall effect or reordering achieves the same objective result that allows for progress to the next logical step. The nature of the process of the flow is therefore logical, not absolute.

[0181] Aspects of the present invention may be provided separately in a downloadable format or on a computer readable medium such as a CD-ROM containing program code which, when instantiated, performs a link embedding function on a web server or the like.

[0182] It will of course be appreciated that the above description is given for purposes of example only and that modifications in detail may be made within the scope of the invention. For example, the principles of training neural networks and the way in which semantically evaluated features, represented by scaled distances in a semantic vector space, can be mapped to objectively generated (typically Euclidean) vectors in a property space, can be applied to multiple forms of searchable data, including audio, visual and / or cinematic, literature and scientific reports (medical reports requiring collation for trend analysis).

[0183] Properties that can be extracted for such different source data include brightness, contrast, color, intensity, shape and relative size, as well as relative feature positions and rates of change of some or all of these properties. Since there are other measurable properties in such files, including word frequency (in the case of text analysis) or motion-related measures (derived from sensors), the above description is provided as a non-limiting example of how the property space can be enriched with semantic vectors [in the property space] that can be contrasted and matched with those presented in the semantic space. In the case of images or videos, the entire pixelated image or a series of frames can be used to correspond to a musical "texture," with every pixel of the sampled image providing a two-dimensional matrix for the convolution process. Indeed, it will be recognized that there is some overlap between properties that can be derived from static images or videos and the music modality, as described in the detailed but exemplary embodiment above.

Claims

1. A method for training an artificial neural network "ANN" (NNR 310, NNTO 312, NNTI 314, NNTX 318) to identify relative content similarities between different data files (302, 304), comprising: exposing the ANN to multiple pairs of training data files that share a common modality, and for each pair of training data files, determining two independently derived separation distances: a first independently derived separation distance representing a measure of relative distance between a first pair of training data files in a semantic embedding space, the first independently derived separation distance being obtained from natural language processing "NLP" of human-expressed descriptive semantic content associated with each one of the first pairs of training data files; and a second independently derived separation distance representing a measure of relative distance similarity between the first pair of training data files in a property embedding space, the second independently derived separation distance being a property distance derived from measurable signal properties extracted from each one of the first pair of training data files. There is a step, iteratively adjusting weights and / or bias values ​​of the ANN (NNR 310, NNTO 312, NNTI 314, NNTX 318) in a backpropagation process that utilizes output vectors generated within the ANN from processing the plurality of pairs to adapt the ANN during training to converge the distances of generated output vectors in a property embedding space towards the corresponding pairwise semantic distances in a semantic embedding space. wherein the common modality relates to one of: (i) video data file to video data file, (ii) audio data file to audio data file, (iii) static image file to static image file, and (iv) text file to text file that are comparable.

2. 2. The method of training an artificial neural network of claim 1, wherein said human-expressed descriptive semantic content is encoded within metadata of a data file.

3. identifying a plurality of characteristics from the measurable signal properties; Further comprising:

3. A method for training an artificial neural network as claimed in claim 1 or 2, wherein the data files contain audio, and the properties are musical properties, and the measurable signal properties define characteristics relating to rhythm, tonality, timbre and musical texture.

4. The evaluation of quantified semantic dissimilarity distance measures is identifying, for each of the files in a training set, a minimum and maximum separation distance from other files in the training set; generating, for each file in the training set, a set of pairs, a first pair having the minimum separation distance and a second pair having the maximum separation distance; assigning a first value representing semantic closeness to the first pair and a second value representing semantic dissimilarity to the second pair, the second value being different from the first value; A method for training an artificial neural network according to any one of claims 1 to 3, comprising:

5. 5. The method of training an artificial neural network of claim 4, wherein for a subset that includes the m smallest separation distances and the m largest separation distances, the first value is assigned to the m smallest and the second value to the m largest, where m is a positive integer less than N.

6. 1. A composite method of evaluating data files to reflect common user-perceivable semantic characteristics and, once evaluated, ordering the data files based on the semantic characteristics with respect to subject data of a given modality, and then providing recommendations to at least one selected associated reference data file that is semantically related to the subject data, comprising: training an ANN according to any one of claims 1 to 5; storing in a database a plurality of reference data files having a common modality with the subject data and stored associations between each reference data file and an individual property vector specifically related to each reference data file, each related individual property vector being obtained from processing file properties extracted from its respective reference data file in the trained ANN, whereby the database stores searchable property vectors embodying semantic characteristics of the semantic content of each of the plurality of reference data files; generating a file vector (VFile) in a property space for the target data based on processing, in the trained ANN, file properties extracted from the target data in response to the target data being input to the trained ANN for which a relative semantic similarity assessment is to be performed; accessing said database and comparing said file vector of said subject data with individual property vectors of each of said plurality of reference data files in said database in order to generate an order within said plurality of reference data files and to identify said at least one associated reference data that is measurably similar to said file vector (VFile) of said subject data and thus semantically similar to said subject data and its semantic content; transmitting, over a communications network and in response to providing the subject data to the input, the first reference file as a recommendation for the semantically related file to a network-connected device; A composite method including:

7. An artificial neural network "ANN" having an output capable of generating property vectors in a property embedding space, trained by exposing said ANN to multiple pairs of training data files that share a common modality, and for each pair of training data files, two independently derived separation distances, namely: a first independently derived separation distance representing a measure of relative distance between a first pair of training data files within a semantic embedding space, the first independently derived separation distance being obtained from natural language processing "NLP" of a semantic description of a property of data associated with each one of the first pairs of training data files; and a second independently derived separation distance representing a measure of relative distance similarity between the first pair of training data files in a property embedding space, the second independently derived separation distance being a property distance derived from measurable properties extracted from each one of the first pair of training data files. There are training the ANN by a backpropagation process, using output vectors generated at the output of the ANN from processing the plurality of pairs to adjust weighting coefficients to iteratively adapt the ANN during training to converge the distances of generated output vectors in a property embedding space towards the corresponding pairwise semantic distances in a semantic embedding space; The common modality is an artificial neural network "ANN" that relates to one of: (i) video data file to video data file, or alternatively (ii) audio data file to audio data file, or alternatively (iii) static image file to static image file, or alternatively (iv) text file to text file that is comparable.

8. 8. The ANN of claim 7, wherein for subject data, when in the modality of an audio data file, the measurable properties relate to musical characteristics indicative of rhythm, tonality, timbre and musical texture.

9. A prediction system including an artificial neural network "ANN" trained by exposing the ANN to a plurality of pairs of training data files sharing a common modality, wherein the independence vectors represent both human perceived semantics and derivable measurable characteristics derived from natural language processing "NLP" of the descriptive content of each one of the pairs of training data files and measurable characteristics extracted from the media content of each one of the pairs of training data files, and wherein the construction of the ANN through training of the ANN maps pairwise similarities in a property space for each pair of training data files towards corresponding pairwise semantic similarities in a semantic space for each pair of training data files, thereby causing the ANN to evaluate a measure of semantic similarity across a measure of property separation distance, converging the property separation distance to a corresponding semantic dissimilarity measure; A prediction system, wherein once trained on a pair of training data files, the prediction system is configured to provide a measured quantification, expressed in a property space, of similarity / dissimilarity in media content between a review data file presented to the prediction system and processed by the ANN and a reference file processed by the ANN, the quantification being based on extracted file properties extracted from each of the review data file and the reference file.

10. 1. A system for providing recommendations of semantically related files, comprising: a) An artificial neural network "ANN" having an output capable of generating a property vector in a property space, trained by exposing said ANN to multiple pairs of training data files that share a content modality, and for each pair of training data files, two independently derived separation distances, namely: a first independently derived separation distance representing a measure of relative distance between a first pair of training data files within a semantic embedding space, the first independently derived separation distance being obtained from natural language processing "NLP" of a semantic description of a property of data associated with each one of the first pairs of training data files; and a second independently derived separation distance representing a measure of relative distance similarity between the first pair of training data files in a property embedding space, the second independently derived separation distance being a property distance derived from measurable properties extracted from each one of the first pair of training data files. There are training the ANN by a backpropagation process using output vectors generated at the output of the ANN from processing the plurality of pairs to adjust weighting coefficients to adapt the ANN during training to converge the distances of generated output vectors in a property embedding space towards the corresponding pairwise semantic distances in a semantic space; an artificial neural network "ANN"; the shared content modality is (i) a video data file, or alternatively (ii) an audio data file, or alternatively (iii) a static image file; b) a database in which a plurality of reference data files having content modalities with subject data and stored associations between each reference data file and an associated individual property vector are stored, each associated individual property vector resulting from processing file properties extracted from its respective reference data file in the trained ANN, and each associated individual property vector encoding the semantic description of its respective reference data file; c) a communications network; and d) a user device coupled to said communications network; e) processing intelligence, generating a file vector (VFile) in a property space for the target data based on the trained ANN receiving as input target data and the target data being assessed for relative semantic similarity of its contents, and based on the trained ANN processing file properties extracted from the target data; accessing said database; comparing the file vector of the subject data with individual property vectors of the plurality of reference data files in the database to generate an ordered list that identifies relevant reference data files having property vectors that are measurably similar to the file vector (VFile), and thus identifying relevant reference data files that are semantically similar to the subject data; and transmitting an associated reference data file to said user device over said communications network; and a processing intelligence configured to Including, The user device is configured to receive the associated reference data file and output its content.

11. The system of claim 10 , wherein the user device includes an interface that enables selection of configurable user preference properties associated with the semantically related file recommendations to be provided to the user device in the request for rating.

12. 12. The system of claim 10 or 11, wherein for subject data, when in the modality of an audio data file, the measurable properties relate to musical characteristics indicative of rhythm, tonality, timbre and musical texture.

13. 1. A method for providing semantically related file recommendations in a system including an artificial neural network "ANN" having an output capable of generating a property vector in a property space, comprising: a) training the ANN by exposing it to multiple pairs of training data files that share content modality, where for each pair of training data files, two independently derived separation distances are calculated, namely: a first independently derived separation distance representing a measure of relative distance between a first pair of training data files within a semantic embedding space, the first independently derived separation distance being obtained from natural language processing "NLP" of a semantic description of a property of data associated with each one of the first pairs of training data files; and a second independently derived separation distance representing a measure of relative distance similarity between the first pair of training data files in a property embedding space, the second independently derived separation distance being a property distance derived from measurable properties extracted from each one of the first pair of training data files. wherein the content modality to be shared is (i) a video data file, or alternatively (ii) an audio data file, or alternatively (iii) a static image file; b) in a backpropagation process within the ANN, adjusting weighting coefficients within the ANN using output vectors generated at the output of the ANN from processing the plurality of pairs, thereby adapting the ANN during training to converge the distances of generated output vectors in a property embedding space towards the corresponding pairwise semantic distances in a semantic space; c) storing in a database a plurality of reference data files having content modalities with subject data and stored associations between each reference data file and an associated individual property vector, each associated individual property vector resulting from processing file properties extracted from its respective reference data file in the trained ANN, and each associated individual property vector encoding the semantic description of its respective reference data file; d) generating a file vector (VFile) in a property space for the target data in response to the trained ANN receiving as input target data and an assessment of the relative semantic similarity of the content of the target data and based on processing within the trained ANN file properties extracted from the target data; e) comparing the file vector of the subject data with individual property vectors of the plurality of reference data files in the database to generate an ordered list identifying relevant reference files having property vectors that are measurably similar to the property vector, and thus identifying relevant reference data files that are semantically similar to the subject data; f) transmitting the relevant reference data file to the user device over a communications network; g) receiving the associated file-of-reference in the user device and outputting its contents; The method includes:

14. The method of claim 13 , wherein the semantic description of the nature of the data is encoded within metadata of the data file.

15. The step of identifying and selecting a plurality of characteristics from the measurable signal properties. Further comprising:

15. The method of claim 13 or 14, wherein the data files contain audio and the measurable signal properties define musical characteristics relating to rhythm, tonality, timbre and musical texture.

16. The evaluation of the distance measure is identifying, for each of the files in a training set, a minimum and maximum separation distance from other files in the training set; generating, for each file in the training set, a set of pairs, a first pair having the minimum separation distance and a second pair having the maximum separation distance; assigning a first value representing semantic closeness to the first pair and a second value representing semantic dissimilarity to the second pair, the second value being different from the first value; The method according to any one of claims 13 to 15, comprising:

17. 17. The method of claim 16, wherein for a subset that includes the m minimum separation distances and the m maximum separation distances, the first value is assigned to the m minimum and the second value to the m maximum, where m is a positive integer less than N.

Images