Representation learning methods and related devices

Abstract

Embodiments of the present application disclose a representation learning method and related equipment in the field of artificial intelligence. The method comprises obtaining a data set of to-be-learned data, the data set comprising at least one sub-set, the sub-set comprising data segments of K scales, and in the sub-set, an i-th scale data segment being a part of an (i+1)-th scale data segment. The data set is input to an encoder, feature extraction is performed on the data segments based on parameters of the encoder, and representation vectors corresponding to the data segments of various scales are obtained. The representation vectors are input to an interaction module, and based on parameters of the interaction module, information interaction is performed on the representation vectors corresponding to data segments of adjacent scales in the sub-set, and fused representation vectors corresponding to the data segments of various scales are respectively obtained. A target function is constructed according to the fused representation vectors. The target function is optimized to adjust the parameters of the encoder and the parameters of the interaction module, so that the encoder and the interaction module learn high-quality representation vectors of the to-be-learned data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of Artificial Intelligence (AI), and in particular, to a representation learning method and related equipment. BACKGROUND

[0002] Artificial intelligence needs a large amount of data as the basis for "thinking" and "decision-making", for example, a large amount of data as "learning materials" for deep learning, from which computers can find rules. The data we obtain, such as audio, video, text, etc. are very complex, lengthy and variable, how to effectively extract features and express them becomes an important challenge.

[0003] In machine learning, representation learning is a collection of techniques that learn a feature: convert raw data into a form that can be effectively exploited by machine learning. It avoids the trouble of manually extracting features, allowing computers to learn to use features while also learning how to extract features: learning how to learn.

[0004] However, the quality of the representation vector learned by the existing representation learning is not high. For example, in the representation learning scenario of music, the song file is usually divided into audio segments of the same length, and then each audio segment is learned to obtain the representation vector of the audio segment, and then all audio segment representation vectors are aggregated to obtain the representation vector of the whole song. However, the aggregation method is affected by high-frequency features, and the obtained whole song representation vector is not discriminative. Alternatively, the whole song is directly learned to obtain the whole song representation vector, but the representation vector obtained by this method lacks local feature information. SUMMARY

[0005] The present application provides a representation learning method and related equipment, by information interaction between representation vectors corresponding to data segments of different scales and adjacent scales, to excavate the potential of deeper information, so that the fine-grained representation vector of the i-th scale data segment includes the information of the coarse-grained representation vector of the i+1-th scale data segment, and the coarse-grained representation vector of the i+1-th scale data segment includes the information of the fine-grained representation vector of the i-th scale data segment, and thus a high-quality representation vector can be obtained.

[0006] In a first aspect, the present application provides a representation learning method, which can be applied to a training device. The method comprises obtaining a data set of data to be learned, wherein the data set comprises at least one sub-set, and the sub-set comprises data segments of K scales, K being an integer greater than or equal to 2. In the sub-set, the data segment of the i-th scale is a part of the data segment of the (i+1)-th scale, i being less than or equal to K-1. The data set is input into an encoder, and based on the parameters of the encoder, feature extraction is performed on the data segments to obtain representation vectors corresponding to the data segments of various scales, respectively. The representation vectors are input into an interaction module, and based on the parameters of the interaction module, information interaction is performed on the representation vectors corresponding to the data segments of adjacent scales in the sub-set to obtain fused representation vectors corresponding to the data segments of various scales, respectively. A target function is constructed according to the fused representation vectors. The target function is optimized to adjust the parameters of the encoder and the parameters of the interaction module, so that the encoder and the interaction module learn the representation vectors of the data to be learned.

[0007] It should be noted that the order of the i-th scale and the (i+1)-th scale is not limited, and it can be understood that the data segment of the i-th scale is a part of the data segment of the (i+1)-th scale, i.e., the i-th scale is smaller than the (i+1)-th scale. In the sub-set, the data segment of a small scale is a part of the data segment of a large scale.

[0008] In the sub-set, the number of data segments of each scale is one or more. When performing information interaction on the representation vectors corresponding to the data segments of adjacent scales, information interaction can be performed on all data segments corresponding to the two scales, or information interaction can be performed on part of the data segments corresponding to the two scales. It can be understood that the more data segments perform information interaction, the more fused representation vectors corresponding to each data segment are output.

[0009] For example, the number of data segments of the i-th scale is M, and the number of data segments of the i+1-th scale is N. When the data segments of the i-th scale and the i+1-th scale are interacted, the M data segments can be interacted with the N data segments, and the fused representation vector corresponding to each of the M data segments and the fused representation vector corresponding to each of the N data segments are output. Alternatively, part (for example, m, m is less than M) of the M data segments can be interacted with the N data segments, and the fused representation vector corresponding to each of the m data segments and the fused representation vector corresponding to each of the N data segments are output. Alternatively, the M data segments can be interacted with part (for example, n, n is less than N) of the N data segments, and the fused representation vector corresponding to each of the M data segments and the fused representation vector corresponding to each of the n data segments are output.

[0010] In the embodiments of the present application, the representation vectors corresponding to the data segments of different scales and adjacent scales in the same subset are interacted, so that the representation vector of the data segment of the i-th scale includes the information of the representation vector of the data segment of the i+1-th scale, and the representation vector of the data segment of the i+1-th scale includes the information of the representation vector of the data segment of the i-th scale. That is, for the representation vector of fine granularity (for example, the representation vector of the data segment of the i-th scale), the fused representation vector includes the information of the representation vector of coarse granularity (for example, the representation vector of the data segment of the i+1-th scale), and for the representation vector of coarse granularity, the fused representation vector includes the information of the representation vector of fine granularity. Then, the target function is constructed according to the fused representation vector, and the encoder and the interaction module are trained through the target function, so that the encoder and the interaction module can mine deeper information in the to-be-learned data, and then a high-quality representation vector can be obtained. That is, the obtained representation vector contains as much data information as possible, which is useful for subsequent tasks, so that it is easier to extract useful information for the subsequent construction of a classifier or other prediction tasks, and the performance in the downstream task (for example, classification) is improved. Moreover, the representation vectors corresponding to the data segments of different scales of the to-be-learned data can be learned. For example, for a song, the representation vector of the whole song can be learned, and the representation vector of the audio segment can also be learned. Thus, different granularity representation vectors can be provided for different tasks.

[0011] Optionally, the representation vectors are input into the interaction module, and information interaction is performed on the representation vectors corresponding to the data segments of the adjacent scales in the subset based on the parameters of the interaction module, to obtain the fused representation vectors corresponding to the data segments of various scales respectively, including: inputting the representation vectors into the interaction module; determining the scales to be processed in the n-th time as the i-th scale and the i+1-th scale, and assigning an initial value of n as 1; and performing the following operations in a loop until n=K-1, where n=i, and n is an integer greater than or equal to 1; performing information interaction on the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the subset in the n-th time based on the parameters of the interaction module, to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

[0012] In the embodiments of the present application, the information overlap and association between the data segments of the adjacent scales in the same subset are used to make the representation vectors more matched in the information interaction process. Further, the order of the information interaction of the pairs of representation vectors is determined according to the order of the scales from small to large, the smallest scale and the second smallest scale in the subset are first subjected to information interaction, then the second smallest scale and the third smallest scale in the subset are subjected to information interaction, and so on. The information interaction is performed layer by layer according to the order of the scales from small to large, which can ensure that the information is gradually fused from fine granularity to coarse granularity in the interaction process, realize the layering and refinement of the information interaction, and further improve the quality of the representation vectors.

[0013] Optionally, when n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after the information interaction of the data segment of the i-th scale in the n-1-th time.

[0014] In the embodiments of the present application, when n is an integer greater than or equal to 2, the learned fused representation vector is continuously used for learning representation in the next time, which can ensure that the information is gradually fused from fine granularity to coarse granularity in the interaction process, realize the layering and refinement of the information interaction, and further improve the quality of the representation vectors.

[0015] Optionally, the information interaction between the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set based on the interaction module in the n-th time comprises: taking the representation vector corresponding to the data segment of the i-th scale as a first representation vector, and taking the representation vector corresponding to the data segment of the i+1-th scale as a second representation vector; based on the parameters of the interaction module, the first representation vector is mapped into a first query vector, and the second representation vector is mapped into a first key value vector and a first value vector respectively; the attention score of the first query vector and the first key value vector is calculated through the attention mechanism to obtain a first attention score; the fused representation vector corresponding to the data segment of the i-th scale is obtained according to the first value vector, the first attention score and the first representation vector; based on the parameters of the interaction module, the second representation vector is mapped into a second query vector, and the fused representation vector corresponding to the data segment of the i-th scale is mapped into a second key value vector and a second value vector respectively; the attention score of the second query vector and the second key value vector is calculated through the attention mechanism to obtain a second attention score; the fused representation vector corresponding to the data segment of the i+1-th scale is obtained according to the second value vector, the second attention score and the second representation vector.

[0016] In the embodiments of the present application, the information interaction between the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale based on the attention mechanism can quickly extract important features of the to-be-learned data.

[0017] Optionally, the obtaining of the data set of the to-be-learned data comprises: obtaining the to-be-learned data and a preset scale category number; segmenting the to-be-learned data to obtain data segments; and segmenting the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

[0018] In the embodiments of the present application, the to-be-learned data is randomly segmented, and each segmentation is to segment a smaller scale data segment from the current smallest scale data segment, which increases the hierarchy of the same to-be-learned data in scale and guarantees the correlation between different scale data segments, thereby providing a basis for subsequent fusion of different scale data segments.

[0019] Optionally, the constructing of the target function according to the fused representation vector comprises: obtaining an overall representation vector of other to-be-learned data, wherein the other to-be-learned data and the to-be-learned data are in the same batch, and the overall representation vector is a representation vector corresponding to a data segment with the largest scale in the other to-be-learned data; calculating the similarity between each fused representation vector and the overall representation vector; screening out the fused representation vector corresponding to the lowest similarity for each scale; and constructing the target function according to the screened fused representation vector.

[0020] In the embodiments of the present application, by screening out data segments of different scales with the lowest similarity to other to-be-learned data, data segments with better distinguishability between different to-be-learned data are obtained, so as to avoid redundancy in subsequent contrast learning and accelerate the convergence of the objective function.

[0021] Optionally, constructing the objective function according to the screened fused representation vector comprises: mapping the screened fused representation vector as a positive sample; mapping the overall representation vector corresponding to other to-be-learned data in the same batch as a negative sample; based on contrast learning, taking the positive sample and the negative sample as sub-elements of the objective function to construct the objective function.

[0022] In the embodiments of the present application, the latent representation vector is learned by maximizing the mutual information between the positive sample and the negative sample through contrast learning.

[0023] Optionally, the method further comprises: taking the positive samples corresponding to two scales adjacent in scale as a positive sample pair; calculating a first similarity between the positive sample corresponding to the largest scale and the negative sample and a second similarity of the positive sample pair through the objective function; and optimizing the objective function according to the first similarity and the second similarity to update the parameters of the encoder and the parameters of the interaction module.

[0024] In the embodiments of the present application, a multi-level contrast learning is adopted, that is, the representation vectors of data segments of adjacent lengths are pulled closer and gradually transitioned to the representation vector of a data segment of the largest scale (such as a whole song), which can achieve a better similar clustering effect.

[0025] Optionally, the to-be-learned data is audio data, text data or video data. It can be understood that the representation learning method provided in the embodiments of the present application can be applied in various subfields of the field of artificial intelligence, such as the field of speech processing, the field of computer vision, the field of text processing, the field of natural language processing, etc.

[0026] In a second aspect, the embodiments of the present application provide a representation learning method, which comprises: obtaining a data set of to-be-learned data, wherein the data set comprises at least one sub-set, the sub-set comprises data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the i-th scale data segment is part of the (i+1)-th scale data segment, i is less than or equal to K-1; the number of data segments of each scale is one or more; inputting the data set into a trained encoder, and performing feature extraction on the data segments based on the parameters of the encoder to obtain representation vectors corresponding to the data segments of each scale respectively; inputting the representation vectors into a trained interaction module, and performing information interaction on the representation vectors corresponding to the data segments of adjacent scales in the sub-set based on the parameters of the interaction module to obtain fused representation vectors corresponding to the data segments of each scale respectively, so as to complete the learning of the representation vectors of the to-be-learned data.

[0027] It should be noted that the order of the i-th scale and the (i+1)-th scale is not limited, and it can be understood that the i-th scale data segment is part of the (i+1)-th scale data segment, that is, the i-th scale is smaller than the (i+1)-th scale. In the sub-set, the data segment of a small scale is part of the data segment of a large scale.

[0028] In the sub-set, when performing information interaction on the representation vectors corresponding to the data segments of adjacent scales, information interaction can be performed on all data segments corresponding to two scales for interaction, or information interaction can be performed on part of the data segments corresponding to the two scales. It can be understood that the more data segments perform information interaction, the more fused representation vectors corresponding to each data segment are output.

[0029] For example, the number of data segments of the i-th scale is M, and the number of data segments of the i+1-th scale is N. When the data segments of the i-th scale and the i+1-th scale are interacted, the M data segments can be interacted with the N data segments, and the corresponding fused representation vectors of each of the M data segments and each of the N data segments are output. Alternatively, part (for example, m, m is less than M) of the M data segments can be interacted with the N data segments, and the corresponding fused representation vectors of each of the m data segments and each of the N data segments are output. Alternatively, the M data segments can be interacted with part (for example, n, n is less than N) of the N data segments, and the corresponding fused representation vectors of each of the M data segments and each of the n data segments are output.

[0030] In the embodiment of the present application, the trained encoder and the interaction module interact the representation vectors corresponding to the data segments of different scales and adjacent scales in the same subset, so that the representation vector of the data segment of the i-th scale includes the information of the representation vector of the data segment of the i+1-th scale, and the representation vector of the data segment of the i+1-th scale includes the information of the representation vector of the data segment of the i-th scale. Thus, a high-quality representation vector can be obtained, that is, the obtained representation vector contains as much data information as possible, which is useful for subsequent tasks, so that it is easier to extract useful information for subsequent tasks such as classifier construction or other prediction tasks, and the performance in downstream tasks (such as classification) is improved. Moreover, the representation vectors corresponding to different scale data segments of the to-be-learned data can be learned, for example, for a song, the representation vector of the whole song can be learned, and the representation vector of the audio segment can also be learned. Thus, representation vectors of different granularities can be provided for different tasks.

[0031] Optionally, the representation vectors are input into the trained interaction module, and the representation vectors corresponding to the data segments of adjacent scales in the subset are interacted based on the parameters of the interaction module, to obtain the fused representation vectors corresponding to the data segments of various scales. The method includes: inputting the representation vectors into the trained interaction module; determining that the scales to be processed in the n-th time are the i-th scale and the i+1-th scale, and assigning an initial value of n as 1; and performing the following operations in a loop until n=K-1, where n=i, and n is an integer greater than or equal to 1; and based on the parameters of the trained interaction module, the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the subset are interacted in the n-th time, to obtain the fused representation vectors corresponding to the data segments of various scales.

[0032] In the embodiments of the present application, the interaction module performs information interaction layer by layer in the order of the scales from small to large, which can ensure that the information gradually fuses from fine granularity to coarse granularity in the process of interaction, realize the layering and refinement of information interaction, and further improve the quality of the representation vector.

[0033] Optionally, when n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after the data segment of the i-th scale performs information interaction in the n-1th time.

[0034] In the embodiments of the present application, the learned fused representation vector is continuously used for learning representation in the next time, which ensures that the information gradually fuses from fine granularity to coarse granularity in the process of interaction, realizes the layering and refinement of information interaction, and further improves the quality of the representation vector.

[0035] Optionally, based on the parameters of the trained interaction module, the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set are performed information interaction in the n-th time, and the fused representation vectors corresponding to the data segments of various scales are obtained, including: taking the representation vector corresponding to the data segment of the i-th scale as a first representation vector, and taking the representation vector corresponding to the data segment of the i+1-th scale as a second representation vector; based on the parameters of the trained interaction module, the first representation vector is mapped into a first query vector, and the second representation vector is mapped into a first key-value vector and a first value vector, respectively; the attention score of the first query vector and the first key-value vector is calculated through the attention mechanism to obtain a first attention score; the fused representation vector corresponding to the data segment of the i-th scale is obtained according to the first value vector, the first attention score and the first representation vector; based on the parameters of the trained interaction module, the second representation vector is mapped into a second query vector, and the fused representation vector corresponding to the data segment of the i-th scale is mapped into a second key-value vector and a second value vector, respectively; the attention score of the second query vector and the second key-value vector is calculated through the attention mechanism to obtain a second attention score; the fused representation vector corresponding to the data segment of the i+1-th scale is obtained according to the second value vector, the second attention score and the second representation vector.

[0036] In the embodiments of the present application, the interaction module performs information interaction on the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale based on the attention mechanism, which can quickly extract important features of the to-be-learned data.

[0037] Optionally, obtaining the data set of the to-be-learned data includes: obtaining the to-be-learned data and a preset scale category number; segmenting the to-be-learned data to obtain data segments; and segmenting the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

[0038] In the embodiments of the present application, the to-be-learned data is randomly divided each time, and each division is to divide a data segment of a smaller scale from a data segment of a current smallest scale, thereby increasing the hierarchy of the to-be-learned data in the scale and ensuring the correlation between data segments of different scales, thereby providing a basis for subsequent fusion of data segments of different scales.

[0039] In a third aspect, the embodiments of the present application provide a training device, which comprises an acquisition module, an encoder, an interaction module, a construction module and an optimization module. The acquisition module is configured to acquire a data set of to-be-learned data, wherein the data set comprises at least one sub-set, the sub-set comprises data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is part of the data segment of the (i+1)-th scale, i is less than or equal to K-1; the number of data segments of each scale is one or more; the encoder is configured to receive the data set and perform feature extraction on the data segments based on parameters of the encoder to obtain representation vectors corresponding to the data segments of various scales respectively; the interaction module is configured to receive the representation vectors and perform information interaction on the representation vectors corresponding to the data segments of adjacent scales in the sub-set based on parameters of the interaction module to obtain fused representation vectors corresponding to the data segments of various scales respectively; the construction module is configured to construct an objective function according to the fused representation vectors; and the optimization module is configured to optimize the objective function to adjust the parameters of the encoder and the parameters of the interaction module, so that the encoder and the interaction module learn the representation vectors of the to-be-learned data.

[0040] Optionally, the interaction module is specifically configured to: receive the representation vectors; determine that the scale to be processed in the n-th time is the i-th scale and the (i+1)-th scale, and assign an initial value of n as 1, and perform the following operations in a loop until n=K-1, wherein n=i, n is an integer greater than or equal to 1; perform information interaction on the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the sub-set in the n-th time based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

[0041] Optionally, when n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after the information interaction in the (n-1)-th time.

[0042] Optionally, based on the parameters of the interaction module, the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set are interacted the n-th time, and the fused representation vectors corresponding to the data segments of various scales are obtained, specifically including: taking the representation vector corresponding to the data segment of the i-th scale as a first representation vector, and taking the representation vector corresponding to the data segment of the i+1-th scale as a second representation vector; based on the parameters of the interaction module, the first representation vector is mapped into a first query vector, and the second representation vector is mapped into a first key value vector and a first value vector, respectively; the attention score of the first query vector and the first key value vector is calculated through the attention mechanism to obtain a first attention score; the fused representation vector corresponding to the data segment of the i-th scale is obtained according to the first value vector, the first attention score and the first representation vector; based on the parameters of the interaction module, the second representation vector is mapped into a second query vector, and the fused representation vector of the data segment of the i-th scale is mapped into a second key value vector and a second value vector, respectively; the attention score of the second query vector and the second key value vector is calculated through the attention mechanism to obtain a second attention score; the fused representation vector corresponding to the data segment of the i+1-th scale is obtained according to the second value vector, the second attention score and the second representation vector.

[0043] Optionally, the obtaining module is specifically configured to: obtain the to-be-learned data and a preset scale category number; segment the to-be-learned data to obtain data segments; and segment the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

[0044] Optionally, the constructing module is specifically configured to: obtain an overall representation vector of other to-be-learned data, wherein the other to-be-learned data and the to-be-learned data are in the same batch, and the overall representation vector is a representation vector corresponding to a data segment with the largest scale in the other to-be-learned data; calculate the similarity between each fused representation vector and the overall representation vector; for each scale, filter out the fused representation vector corresponding to the lowest similarity; and construct a target function according to the filtered fused representation vector.

[0045] Optionally, constructing a target function according to the filtered fused representation vector includes: mapping the filtered fused representation vector as a positive sample; mapping the overall representation vector corresponding to the other to-be-learned data in the same batch as a negative sample; based on contrast learning, taking the positive sample and the negative sample as sub-elements of the target function to construct the target function.

[0046] Optionally, the optimization module is specifically configured to: take the positive samples corresponding to two scales adjacent in scale as a positive sample pair; calculate a first similarity between the positive sample corresponding to the largest scale and the negative sample and a second similarity of the positive sample pair through the target function; and optimize the target function according to the first similarity and the second similarity, so as to update the parameters of the encoder and the parameters of the interaction module.

[0047] Optionally, the to-be-learned data is audio data, text data or video data.

[0048] In a fourth aspect, an embodiment of the present application provides an execution device, which comprises: an acquisition module, an encoder and an interaction module. The acquisition module is configured to acquire a data set of to-be-learned data, wherein the data set comprises at least one sub-set, the sub-set comprises data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is part of the data segment of the (i+1)-th scale, i is less than or equal to K-1; the number of data segments of each scale is one or more; the encoder is configured to receive the data set and perform feature extraction on the data segments based on the parameters of the encoder, to obtain representation vectors corresponding to the data segments of various scales respectively, wherein the encoder is trained; and the interaction module is configured to receive the representation vectors and perform information interaction on the representation vectors corresponding to the data segments adjacent in scale in the sub-set based on the parameters of the interaction module, to obtain fused representation vectors corresponding to the data segments of various scales respectively, wherein the interaction module is trained.

[0049] Optionally, the interaction module is specifically configured to: receive the representation vectors; determine that the scale to be processed in the n-th time is the i-th scale and the (i+1)-th scale, and assign an initial value of n as 1, and perform the following operations in a loop until n=K-1, wherein n=i, n is an integer greater than or equal to 1; perform information interaction on the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the sub-set in the n-th time based on the parameters of the trained interaction module, to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

[0050] Optionally, when n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after the information interaction in the (n-1)-th time.

[0051] Optionally, based on the parameters of the trained interaction module, the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set are interacted for the n-th time, and the fused representation vectors corresponding to the data segments of various scales are obtained, including: taking the representation vector corresponding to the data segment of the i-th scale as a first representation vector, and taking the representation vector corresponding to the data segment of the i+1-th scale as a second representation vector; based on the parameters of the trained interaction module, the first representation vector is mapped into a first query vector, and the second representation vector is mapped into a first key-value vector and a first value vector, respectively; the attention score of the first query vector and the first key-value vector is calculated through the attention mechanism to obtain a first attention score; the fused representation vector corresponding to the data segment of the i-th scale is obtained according to the first value vector, the first attention score and the first representation vector; based on the parameters of the trained interaction module, the second representation vector is mapped into a second query vector, and the fused representation vector corresponding to the data segment of the i-th scale is mapped into a second key-value vector and a second value vector, respectively; the attention score of the second query vector and the second key-value vector is calculated through the attention mechanism to obtain a second attention score; the fused representation vector corresponding to the data segment of the i+1-th scale is obtained according to the second value vector, the second attention score and the second representation vector.

[0052] Optionally, the obtaining module comprises: obtaining the to-be-learned data and a preset scale category number; segmenting the to-be-learned data to obtain data segments; and segmenting the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

[0053] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory is configured to store computer readable instructions; and the processor is configured to read the computer readable instructions and implement the method provided in the first aspect and the optional implementation manners thereof or the method provided in the second aspect and the optional implementation manners thereof.

[0054] In a sixth aspect, an embodiment of the present application provides a computer storage medium, which stores computer readable instructions, and the computer readable instructions, when executed by a processor, implement the method provided in the first aspect and the optional implementation manners thereof or the method provided in the second aspect and the optional implementation manners thereof.

[0055] In a seventh aspect, an embodiment of the present application provides a computer program product, which comprises computer readable instructions, and when the computer readable instructions are executed by a processor, the method provided in the first aspect and the optional implementation manners thereof or the method provided in the second aspect and the optional implementation manners thereof are implemented.

[0056] In an eighth aspect, an embodiment of the present application provides a computer program product, which comprises computer readable instructions. When the computer readable instructions are executed by a processor, the method provided in the first aspect and the optional implementation manners thereof or the method provided in the second aspect and the optional implementation manners thereof are implemented.

[0057] The technical effects obtained by the second aspect, the third aspect, the fourth aspect, the fifth aspect, the sixth aspect, the seventh aspect and the eighth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and thus are not described herein. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 A schematic diagram of an artificial intelligence subject framework is provided for the embodiments of the present application.

[0060] Figure 2 A schematic diagram of a learning system is provided for the embodiments of the present application.

[0061] Figure 3 A schematic diagram of a representation learning method is provided for the embodiments of the present application.

[0062] Figure 4 A schematic diagram of segmentation of to-be-learned data is provided for the embodiments of the present application.

[0063] Figure 5 A structural schematic diagram of a convolutional neural network is provided for the embodiments of the present application.

[0064] Figure 6 A schematic diagram of a flow of an information interaction method is provided for the embodiments of the present application.

[0065] Figure 7 Another schematic diagram of a flow of an information interaction method is provided for the embodiments of the present application.

[0066] Figure 8 A structural schematic diagram of an interaction module is provided for the embodiments of the present application.

[0067] Figure 9 A schematic diagram of a flow of a method of constructing an objective function is provided for the embodiments of the present application.

[0068] Figure 10Another flowchart of a method for constructing a target function provided in an embodiment of the present application is shown in FIG. 6;

[0069] Figure 11 Another flowchart of a method for optimizing a target function provided in an embodiment of the present application is shown in FIG. 7;

[0070] Figure 12 A process diagram of a method for optimizing a target function provided in an embodiment of the present application is shown in FIG. 8;

[0071] Figure 13 Another flowchart of a method for representation learning provided in an embodiment of the present application is shown in FIG. 9;

[0072] Figure 14 A flowchart of a method for information interaction provided in an embodiment of the present application is shown in FIG. 10;

[0073] Figure 15 Another flowchart of a method for information interaction provided in an embodiment of the present application is shown in FIG. 11;

[0074] Figure 16 A structural diagram of an execution device provided in an embodiment of the present application is shown in FIG. 12;

[0075] Figure 17 Another structural diagram of an execution device provided in an embodiment of the present application is shown in FIG. 13;

[0076] (a) in FIG. 18 is a music classification scenario diagram provided in an embodiment of the present application;

[0077] (b) in FIG. 18 is a song cover version retrieval scenario diagram provided in an embodiment of the present application;

[0078] (c) in FIG. 18 is another song cover version retrieval scenario diagram provided in an embodiment of the present application;

[0079] (d) in FIG. 18 is a similar song recommendation scenario diagram provided in an embodiment of the present application;

[0080] Figure 19 A structural diagram of a training device provided in an embodiment of the present application is shown in FIG. 14;

[0081] Figure 20 A structural diagram of a training device provided in an embodiment of the present application is shown in FIG. 15. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0083] First, the overall workflow of the artificial intelligence system is described, Figure 1 An artificial intelligence subject framework is shown, which describes the overall workflow of the artificial intelligence system and is suitable for general artificial intelligence field requirements.

[0084] The above artificial intelligence subject framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0085] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom".

[0086] The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0087] (1) Infrastructure:

[0088] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the external world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0089] (2) Data

[0090] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, voice, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0091] (3) Data processing

[0092] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision, and the like.

[0093] Among them, machine learning and deep learning can model, extract, preprocess, train, and the like of symbolic and formalized intelligent information of data.

[0094] Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0095] Decision refers to the process of decision-making after intelligent information is reasoned, and generally provides functions such as classification, sorting, and prediction.

[0096] (4) General capabilities

[0097] After data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, and the like.

[0098] (5) Intelligent products and industry applications

[0099] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are encapsulations of overall solutions of artificial intelligence, and realize application landing by productizing intelligent information decision-making. The application fields mainly include intelligent manufacturing, intelligent transportation, smart home, intelligent medical care, intelligent security, autonomous driving, safe city, intelligent terminal, and the like.

[0100] Embodiments of the present application can be applied to the optimization design of representation learning models. The optimized representation learning models of the present application can be applied in various subfields of the field of artificial intelligence, such as the field of speech processing, the field of computer vision, the field of text processing, the field of natural language processing, and the like. Specifically, in combination with the field of speech processing, the field of computer vision, the field of text processing, the field of natural language processing, and the like, the data in the data set obtained by the infrastructure can be different types of data obtained by a sound collecting device, a camera, and the like, such as audio data, image data, text data, or video data, and the like. The optimized representation learning model of the present application is used to convert the data in the data set into a representation vector, while making the representation vector contain as much data information as possible that is useful for subsequent tasks, so that it is easier to extract useful information when a classifier or other prediction task is constructed subsequently. Figure 1

[0101] The system architecture provided by the embodiments of the present application is introduced below.

[0102] Referring to the accompanying drawings Figure 2 ​The embodiment of the present application provides a representation learning system 200. The representation learning system 200 comprises an execution device 210, a training device 220, a database 230, a client device 240, a data storage system 250 and a data collection device 260, and the execution device 210 comprises a calculation module 211 and an input / output (I / O) interface 212.

[0103] In the training phase, the data collection device 260 can be used to collect a data set (i.e. a training set) of to-be-learned data, and store the training set in the database 230. The training set can comprise samples (which can be referred to as training data or training samples) from multiple source domains, and the data of the training samples in the training set can be audio data, text data, image data or video data, etc., as long as the training set meets the function of iteratively training the representation learning model, which is related to the target task to be performed, and is not limited here.

[0104] The training device 220 generates a representation learning model 201 for converting to-be-learned data into a representation vector, iteratively trains the representation learning model 201 based on the training set maintained in the database 230, and obtains a mature representation learning model 201. The representation learning model 201 is a representation learning model constructed in the following embodiments of the present application, and the representation learning model can be used to implement the representation learning method of the embodiment of the present application, and obtain a representation vector of to-be-learned data (such as audio data, video data or text data). How the training device 220 obtains the representation learning model 201 based on the data set of to-be-learned data will be described in more detail below, and reference can be made to the related description in the following Figures 3-12

[0105] The representation learning model 201 in the embodiment of the present application can be a deep learning model, for example, a convolutional neural network model. It should be noted that in actual application, the training data maintained in the database 230 can not all be collected from the data collection device 260, but can also be received from other devices. In addition, it should be noted that the training device 220 can not completely train the representation learning model 201 based on the training data maintained in the database 230, but can also obtain training data from the cloud or other places for model training, and the above description should not be regarded as a limitation of the embodiment of the present application.

[0106] The representation learning model 201 obtained by the training device 220 can be applied in different systems or devices. For example, it can be applied in the execution device 210, the client device 240, the data storage system 250, etc. Figure 2 ​The execution device 210 shown can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) terminal, a vehicle-mounted terminal, a television, and the like, and can also be a server or a cloud terminal, and the like. The execution device 210 interacts with external devices through an I / O interface 212. A "user" can input data to the I / O interface 212 through a client device 240. The input data in the embodiments of the present application can include audio data, text data, or video data input by the client device to be learned.

[0107] In the inference phase, the execution device 210 can call data, code, and the like in the data storage system 250, or store data, instructions, and the like in the data storage system 250. The data storage system 250 can be placed in the execution device 210, or can be an external memory of the execution device 210. The computing module 211 can process the to-be-learned data input by the execution device 210 through the mature representation learning model 201 to obtain a representation vector of the to-be-learned data.

[0108] It should be noted that the devices, components, modules, and the like shown in the accompanying drawings are only schematic representations for purposes of illustration. Figure 2 The system architecture shown in the accompanying drawings is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, and the like shown in the accompanying drawings does not constitute any limitation.

[0109] As can be seen from the description in the foregoing Figure 2 The embodiments of the present application include a training phase and an inference phase, which are described as follows.

[0110] I. Training phase

[0111] In the embodiments of the present application, the training phase is a process in which the training device 220 performs a training operation on the representation learning model using the to-be-learned data in the training set. Figure 2

[0112] As shown in the foregoing Figure 3 A representation learning method provided by the embodiments of the present application includes the following steps:

[0113] Step S31: The training device obtains a data set of to-be-learned data, wherein the data set includes at least one sub-set, the sub-set includes data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segments of the i-th scale are part of the data segments of the (i+1)-th scale, i is less than or equal to K-1, and the number of data segments of each scale is one or more.

[0114] ​The training device can generate a corresponding representation learning model based on different types of to-be-learned data for different targets to provide better results for the user. For example, if a representation learning model for converting audio data into a representation vector is needed, the training device obtains a training set of audio data.

[0115] In the embodiments of the present application, the data set of the to-be-learned data includes at least one sub-set, and each sub-set can include data segments of K scales. The scale can be understood as a measure of the size of the data, and the information amount contained in data segments of different scales is different. It can be understood that the information amount contained in data segments of large scales is more than that contained in data segments of small scales.

[0116] For different types of to-be-learned data, the scale of the to-be-learned data can be determined according to actual conditions. For example, for audio data, the time dimension can be used as the basis for scale division, and audio segments of different scales can be divided according to the time length, such as 10s of audio data corresponding to one scale and 20s of audio data corresponding to another scale. For video data, video segments of different scales can be divided according to the time length, such as 10s of video segments corresponding to one scale and 20s of video segments corresponding to another scale. In other embodiments, video segments of different scales can also be divided according to the number of video frames, such as video segments including 100 consecutive image frames corresponding to one scale and video segments including 1000 consecutive image frames corresponding to another scale. For text data, strings can be used as the basis for scale division, and text segments of different scales can be divided according to the number of strings, such as text segments including 100 consecutive strings corresponding to one scale and text segments including 1000 consecutive strings corresponding to another scale. It can be understood that a string is a finite sequence composed of zero or more characters. The segmented string includes English strings, numerical strings, symbol strings, pinyin strings, and Chinese strings, etc. For example, “Good Luck”, “2001”, “Nihao”, “https: / / www.huawei.com / cn / ”, “mobile phone”, etc.

[0117] In the embodiments of the present application, the information amount contained in data segments of large scales in the sub-set covers the information amount contained in data segments of small scales. The i-th scale data segment in the sub-set is part of the i+1-th scale data segment, that is, the information amount contained in the i+1-th scale data segment covers the information amount contained in the i-th scale data segment. The K scales in the sub-set are sorted, and the sizes of the 1st scale to the i+1-th scale are from small to large.

[0118] In the embodiments of the present application, the to-be-learned data in the training set can also be regarded as a data segment, that is, the to-be-learned data not segmented in the training set is also a data segment. For each sub-set of to-be-learned data, the sub-set includes the to-be-learned data and part of the to-be-learned data.

[0119] In the embodiments of the present application, the training device obtains a training set including a plurality of to-be-learned data, and segments each to-be-learned data in the training set to obtain a data set of each to-be-learned data.

[0120] Before training, the training personnel can set the segmentation parameter according to the task and the training needs, and the segmentation parameter can be a scale type. The training device obtains the preset scale type, and can obtain the preset scale type quantity according to the set scale type. For example, the training device obtains three different scale types, and can obtain a preset scale type quantity of three.

[0121] In other embodiments, the segmentation parameter can be a scale type quantity, and the training device can set different scale types according to the set scale type quantity, for example, the preset scale type quantity is three, and the training device can set three different scale types. It should be noted that in the embodiments of the present application, the scale of the to-be-learned data itself is one scale by default. The training device sets three different scale types according to the preset scale type quantity of three, which includes the scale of the to-be-learned data itself.

[0122] In other embodiments, the segmentation parameter can also include the quantity of data segments corresponding to each scale type. For example, the quantity of data segments corresponding to the first scale type is a, the quantity of data segments corresponding to the second scale type is b, and the quantity of data segments corresponding to the third scale type is c. a, b, and c can be set according to actual conditions, and are all integers greater than or equal to 1. a, b, and c can be the same, such as all being 2, or all being different, such as a being 1, b being 2, and c being 3, or part of them being the same, such as a and b being the same and a and c being different.

[0123] In the embodiments of the present application, after the training device obtains the training set and the preset scale type quantity, for each to-be-learned data in the training set, the to-be-learned data is segmented according to the segmentation parameter to obtain a data segment. Each segmentation is a random segmentation of the data segment with the smallest scale at present, until the scale type quantity in the sub-set of the to-be-learned data reaches the preset scale type quantity.

[0124] Specifically, please refer to Figure 4, take the 140s song A as an example, the segmentation parameters input to the training device are: the scale types are the first scale (8s), the second scale (30s), and the third scale (50s), and the number of data segments corresponding to each scale type is 2, and the training device also includes the fourth scale (140s) by default, so the scale types of the song A are 140s, 50s, 30s, and 8s. Figure 4 The line segment A in FIG. 13 represents the 140s song A in the training set, and two audio segments (audio segments B and C) of the third scale (50s) are randomly segmented from the song A, and the audio segments B and C correspond to the line segments B and C in FIG. 14, respectively. Figure 4 At this time, the scale types are 140s and 50s, and the audio segments of the smallest current scale are the two audio segments of the third scale.

[0125] Two audio segments of the second scale (30s) are randomly segmented from each audio segment of the third scale, such as two audio segments (audio segments D and E) of 30s in length segmented from the line segment B, which correspond to the line segments D and E in FIG. 15, respectively. Figure 4 Two audio segments (audio segments F and G) of 30s in length segmented from the line segment C correspond to the line segments F and G in FIG. 16, respectively. Figure 4 At this time, the scale types are 140s, 50s, and 30s, and the audio segments of the smallest current scale are the two audio segments of the second scale.

[0126] Two audio segments of the first scale (8s) are randomly segmented from each audio segment of the second scale, such as two audio segments (audio segments d1 and d2) of 8s in length segmented from the line segment D, which correspond to the line segments d1 and d2 in FIG. 17, respectively. Figure 4 Two audio segments (audio segments e1 and e2) of 8s in length segmented from the line segment E correspond to the line segments e1 and e2 in FIG. 18, respectively. Figure 4 Two audio segments (audio segments f1 and f2) of 8s in length segmented from the line segment F correspond to the line segments f1 and f2 in FIG. 19, respectively. Figure 4 Two audio segments (audio segments g1 and g2) of 8s in length segmented from the line segment G correspond to the line segments g1 and g2 in FIG. 20, respectively. Figure 4

[0127] Figure 4 ​​As shown, the data set of the song A includes four sub-sets, the first sub-set is {song A, audio segment B, audio segment D, audio segment d1, audio segment d2}, the second sub-set is {song A, audio segment B, audio segment E, audio segment e1, audio segment e2}, the third sub-set is {song A, audio segment C, audio segment F, audio segment f1, audio segment f2}, and the fourth sub-set is {song A, audio segment C, audio segment G, audio segment g1, audio segment g2}. The audio segment with small scale in each sub-set is a part of the audio segment with large scale, that is, the audio segment with large scale contains the audio segment with small scale. In the sub-set in the embodiment of the present application, the audio segments with different scales have a containing relationship, and the audio segments with the same scale can have a coincident part or can have no coincident part.

[0128] In the embodiment of the present application, by setting the scale types and the number of data segments corresponding to each scale type, the sample amount is reduced and the training efficiency is improved. Specifically, as shown in FIG. 2, two 50s audio segments are segmented from the 140s song A, and the contents of the two 50s audio segments do not cover the 140s song A. Figure 4

[0129] It can be understood that, based on the random segmentation of the to-be-learned data, the two or more audio segments segmented can be coincident. For example, in the above example, the audio segments B and C randomly segmented from the song A can be coincident, and the audio segments D and E randomly segmented from the audio segment B are coincident. The various data segments are obtained by random sampling, which is simple and easy to implement.

[0130] In the embodiment of the present application, for each to-be-learned data, the data set of the to-be-learned data is reacquired after one forward propagation and one backward propagation in the representation learning model. Based on the random segmentation of the to-be-learned data by the training device, the data set of the to-be-learned data reacquired will be different. For example, in the above example, after one forward propagation and one backward propagation in the representation learning model, the training device reacquires the data set of the song A, and the training device randomly segments two audio segments with the third scale from the song A. The two 50s audio segments can no longer correspond to the line segments B and C in FIG. 1; by random segmentation, each information in the song A can be randomly collected. Thus, by reacquiring the data set of the to-be-learned data each time and using the random sampling method, the amount of information in each to-be-learned data can be ensured to be acquired, while reducing the training sample amount and improving the training efficiency. Figure 4

[0131] ​​In the embodiments of the present application, each time of segmentation is to segment a data segment with smaller scale from a data segment with current smallest scale. As the example above, a 30s audio segment is segmented from a 50s audio segment, and then an 8s audio segment is segmented from the 30s audio segment, which increases the hierarchy of the data set of the same to-be-learned data in scale, and guarantees the correlation between different scale data segments, and provides a basis for subsequent fusion of different scale data segments.

[0132] Step S32: The training device inputs the data set to the encoder, and extracts features of the data segment based on the parameters of the encoder, to obtain the representation vector corresponding to the data segment of each scale.

[0133] In step S31, for each to-be-learned data in the training set, the training device obtains the data set of the to-be-learned data. The training device extracts features of the audio segment in the data set, to obtain the audio features (such as the log-mel spectrum of the audio segment) corresponding to each audio segment, inputs the extracted audio features to the encoder, and maps each audio feature to the same feature space through the encoder, so that the learning of the representation vector of the to-be-learned data can be performed in the feature space through subsequent operations.

[0134] It can be understood that each audio segment is input to the encoder, and the encoder outputs the audio features corresponding to each audio segment.

[0135] In the embodiments of the present application, the encoder can adopt a convolutional neural network (CNN), please refer to Figure 5 , the convolutional neural network (CNN) 500 can include an input layer 510, a convolutional layer / pooling layer 520, and a neural network layer 530, wherein the pooling layer is optional.

[0136] As shown in Figure 5 , the convolutional layer / pooling layer 520 can include layers such as 521-526, in one implementation, 521 is a convolutional layer, 522 is a pooling layer, 523 is a convolutional layer, 524 is a pooling layer, 525 is a convolutional layer, and 526 is a pooling layer; in another implementation, 521 and 522 are convolutional layers, 523 is a pooling layer, 524 and 525 are convolutional layers, and 526 is a pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0137] For example, the convolutional layer 521 can include a number of convolution operators, also known as kernels, which function as filters that extract specific information from the input audio or semantic information. The convolution operators can be essentially weight matrices, which are usually predefined. The convolutional layer 521 can also include bias parameters. The weight parameters in the weight matrices and the bias parameters are obtained through a large amount of training in actual applications. The weight matrices formed by the weight parameters obtained through training and the bias parameters can extract information from the input audio segment, thereby helping the convolutional neural network 500 make correct predictions.

[0138] When the convolutional neural network 500 has multiple convolutional layers, the initial convolutional layer (for example, 521) often extracts more general features, which can also be referred to as low-level features. As the depth of the convolutional neural network 500 increases, the features extracted by the later convolutional layers (for example, 526) become more and more complex, such as high-level semantic features and the like. The higher the semantic features, the more suitable they are for the problem to be solved.

[0139] Pooling layer:

[0140] Because it is often necessary to reduce the number of training parameters, a pooling layer is often periodically introduced after a convolutional layer, that is, each layer in 521-526 shown in 520 can be a convolutional layer followed by a pooling layer, or a plurality of convolutional layers followed by one or more pooling layers. Figure 5

[0141] Neural network layer 530:

[0142] After processing by the convolutional layer / pooling layer 520, the convolutional neural network 500 is still not sufficient to output the required output information. As mentioned earlier, the convolutional layer / pooling layer 520 only extracts features and reduces the parameters brought by the input data. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 500 needs to use the neural network layer 530 to generate an output of the number of required classes. Therefore, the neural network layer 530 can include a plurality of hidden layers (for example, 531, 532, and 53n shown in 531) and an output layer 540, and the parameters included in the plurality of hidden layers can be pre-trained according to the related training data of a specific task type, for example, the task type can include speech or semantic recognition, classification, or generation, and the like. Figure 5

[0143] ​​After the multiple hidden layers in neural network layer 530, the final layer of the entire convolutional neural network 500 is the output layer 540. This output layer 540 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 500 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 5 The propagation from 510 to 540 is the forward propagation, and the backward propagation (such as...) is completed. Figure 5 The propagation from 540 to 510 (backpropagation) will begin to update the weight parameters and bias parameters of each layer mentioned above, in order to reduce the loss of the convolutional neural network 500 and the error between the output of the convolutional neural network 500 through the output layer and the ideal result.

[0144] by Figure 5 The convolutional neural network 500 shown serves as the encoder in this embodiment of the application, and the parameters of the encoder are also the parameters in the convolutional neural network model 500 (e.g., weight parameters and bias parameters).

[0145] It should be noted that, as Figure 5 The convolutional neural network 500 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0146] In some optional implementations, the encoder used in this application embodiment may include multiple sequentially connected two-dimensional convolutional processing units (not shown). The training device inputs the dataset to the encoder, which extracts audio features. When extracting audio features, the encoder may first filter the audio segments to obtain a two-dimensional Mel spectrogram. Specifically, a Mel-scale filter bank can be used to transform the one-dimensional audio data spectrogram into a two-dimensional Mel spectrogram. Then, a logarithmic operation is performed on the Mel spectrogram to obtain two-dimensional spectral information for quantifying sound intensity (in dB). The two-dimensional spectral information is then convolved by the two-dimensional convolutional processing unit to obtain the audio features of the audio segments. The audio features are used as input to the input layer 510, and the encoder outputs the representation vectors corresponding to each audio segment based on the convolutional / pooling layer 520 and the neural network layer 530. The Mel spectrogram is a frequency domain representation of sound. Mel spectrograms emphasize low frequencies that are crucial for speech intelligibility, while neglecting high frequencies that are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity.

[0147] Exemplarily, the encoder obtains an audio feature corresponding to an audio segment as a feature vector sequence X = [x1, x2, … xk], encodes the feature vector sequence X = [x1, x2, … xk] to obtain a representation vector. The representation vector is a feature vector sequence H = [h1, h2, … hk], where xk represents the kth frame of the speech feature vector, hk is a feature vector output by the last neural network layer of the encoder, and hk corresponds to the result of the xk after being transformed by the encoder module.

[0148] Step S33: The training device inputs the representation vector into the interaction module, and performs information interaction on the representation vectors corresponding to the data segments of adjacent scales in the sub-set based on the parameters of the interaction module, to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

[0149] In the embodiment of the present application, for the representation vector A corresponding to each data segment a in the sub-set, the interaction module is configured to perform information interaction on the representation vector A and the representation vector B corresponding to another data segment b, and output a fused representation vector C. That is, the interaction module converts the representation vector A into the fused representation vector C. The another data segment b and the data segment a belong to the same sub-set, the scales of the data segment b and the data segment a are different, and the scales of the data segment b and the data segment a are adjacent. That is, the data segment b is a part of the data segment a, or the data segment a is a part of the data segment b. For example, the scale of the data segment a is the i-th scale, and the scale of the data segment b is the i+1-th scale or the i-1-th scale. The information interaction is to fuse the information related to the representation vector B into the representation vector A to obtain a fused representation vector C. The information interaction mode can be linear transformation, summation, product or dot product.

[0150] In the embodiment of the present application, please refer to Figure 6 Step S33 can specifically include the following steps:

[0151] Step S61: The training device inputs the representation vector into the interaction module.

[0152] In step S32, the encoder outputs the representation vector corresponding to each audio segment, and in step S61, the training device inputs the representation vector output by the encoder into the interaction module.

[0153] Step S62: The training device determines that the scale to be processed in the n-th time is the i-th scale and the i+1-th scale, and assigns the initial value of n as 1, and performs the following operation in a loop until n = K-1, where n = i, and n is an integer greater than or equal to 1.

[0154] In the embodiment of the present application, the training device needs to determine two representation vectors that interact with each other and determine the order of information interaction of each pair of representation vectors after inputting the representation vectors corresponding to each audio segment into the interaction module.

[0155] In the embodiment of the present application, the training device determines two representation vectors that interact with each other according to the scale adjacency, that is, the audio segments corresponding to the two representation vectors that interact with each other are adjacent in scale. The training device determines the order of information interaction of each pair of representation vectors according to the order from small to large, that is, the training device determines that the scales that interact first (that is, the first time to be processed) are the smallest scale and the second smallest scale in the sub-set, and then the scales that interact (that is, the second time to be processed) are the second smallest scale and the third smallest scale in the sub-set, and so on.

[0156] Step S63: The training device performs information interaction on the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set for the n-th time based on the parameters of the interaction module, to obtain the fused representation vectors corresponding to the data segments of each scale.

[0157] In the embodiment of the present application, when n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after the data segment of the i-th scale is interacted in the n-1-th time. That is, when the number of information interaction is the second time and above, the training device interacts the fused representation vector of the i-th scale data segment obtained after the n-1-th time with the representation vector of the i+1-th scale data segment. Step S63 is specifically: the training device performs information interaction on the fused representation vector of the i-th scale data segment output by the interaction module and the representation vector of the i+1-th scale data segment output by the encoder based on the parameters of the interaction module, to obtain the fused representation vectors corresponding to the data segments of each scale, wherein the fused representation vector of the i-th scale data segment output by the interaction module is the representation vector obtained after the i-th scale data segment is interacted in the n-1-th time.

[0158] Specifically, as the above example, please refer to Figure 4 Take the training device performing step S62 and step S63 on the first sub-set of song A as an example, and take the representation vectors of song A, audio segment B, audio segment D, audio segment d1 and audio segment d2 output by the encoder as A1, B1, D1, d11 and d21 respectively for illustration, which are specifically:

[0159] The training device sets the initial value of n as 1, and determines that the scales to be processed for the first time are the first and second scales. The training device performs information interaction between the representation vectors corresponding to the data segments of the first and second scales in the first subset based on the parameters of the interaction module. That is, the training device performs information interaction between the representation vector d11 and the representation vector D1, and information interaction between the representation vector d21 and the representation vector D1, so as to obtain a fused representation vector d13 corresponding to the audio segment d1, a fused representation vector d14 corresponding to the audio segment d2, and a fused representation vector D11 corresponding to the audio segment D, where the fused representation vector d13 represents information obtained by fusing the representation vector d11 and the representation vector D1, the fused representation vector d14 represents information obtained by fusing the representation vector d21 and the representation vector D1, and the fused representation vector D11 represents information obtained by fusing the representation vector d11, the representation vector d21 and the representation vector D1.

[0160] The training device detects that n is not equal to K-1, sets the initial value of n as 2, and determines that the scales to be processed for the second time are the second and third scales. The training device performs information interaction between the fused representation vector D11 and the representation vector B1 based on the parameters of the interaction module, so as to obtain a fused representation vector D12 corresponding to the audio segment D and a fused representation vector B11 corresponding to the audio segment B, where the fused representation vector D12 represents information obtained by fusing the fused representation vector D11 and the representation vector B1, and the fused representation vector B11 represents information obtained by fusing the representation vector B1 and the fused representation vector D11.

[0161] The training device detects that n is not equal to K-1, sets the initial value of n as 3, and determines that the scales to be processed for the third time are the third and fourth scales. The training device performs information interaction between the fused representation vector B11 and the representation vector A1 based on the parameters of the interaction module, so as to obtain a fused representation vector B12 corresponding to the audio segment B and a fused representation vector A11 corresponding to the song A, where the fused representation vector B12 represents information obtained by fusing the fused representation vector B11 and the representation vector A1, and the fused representation vector A11 represents information obtained by fusing the representation vector A1 and the fused representation vector B11.

[0162] The training device detects that n is equal to K-1, and stops the loop operation.

[0163] Similarly, the training device performs corresponding operations on other subsets of song A, which will not be described here. It should be noted that the training device performs information interaction on data in the same subset, and does not perform information interaction on data in different subsets. For example, the representation vector corresponding to the first scale audio segment d1 in the first subset does not perform information interaction with the representation vector corresponding to the second scale audio segment E in the second subset.

[0164] As in the above example, song A as a data segment of the largest scale can appear in each subset. After the training device calculates the fused representation vectors corresponding to each third scale data segment (data segments B and C) in all subsets, the fused representation vectors corresponding to the data segments B and C are taken as the representation vectors of the i-th scale data segment, and the representation vector of song A output by the encoder is taken as the representation vector of the i+1-th scale data segment. According to the above information interaction mode, the information interaction is performed to obtain the fused representation vector of song A, that is, the overall representation vector of song A. The overall representation vector is the representation vector obtained after information interaction of the data segment of the largest scale in the to-be-learned data, that is, the fused representation vector obtained after information interaction of song A and the data segment of a smaller scale.

[0165] In the embodiments of the present application, the interaction module can be implemented as a neural network, and the interaction module can implement information interaction in the manner of an attention mechanism, so that the encoder part of the neural network can implement information interaction of the representation vectors corresponding to the data segments of adjacent scales by using the attention mechanism, to realize information fusion between data segments of different scales. In some embodiments, the interaction module can be implemented as a fully connected neural network and a neural network of a transformer structure.

[0166] The attention mechanism simulates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external feeling to increase the observation accuracy of a part of the region, and can quickly filter out high-value information from a large amount of information using limited attention resources. The attention mechanism can quickly extract important features of sparse data, and is therefore widely used in natural language processing tasks, especially machine translation. The essential idea of the attention mechanism can be rewritten as the following formula:

[0167] Wherein, Lx=||Source|| represents the length of Source, Source represents the input to-be-learned data, and the formula means that a plurality of elements included in Source are imagined to be constituted by a series of data pairs. When a query of a certain element in the given target Target is given, the weight coefficient of the value corresponding to the Key of each element in Source is obtained by calculating the similarity or correlation between the Query of the aforementioned certain element and the Key of all elements in Source, and then the value of each element in Source is weighted and summed, so that the final attention score value (Attention value) of the aforementioned certain element is obtained. Therefore, the essence of the attention mechanism is to weight and sum the value of each element in Source, and Query and Key are used to calculate the weight coefficient of the corresponding value. Conceptually, the attention mechanism can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on these important information, ignoring a large amount of unimportant information. The focusing process is reflected in the calculation of the weight coefficient. The greater the weight, the more focused on the corresponding value of the value, that is, the weight represents the importance of the information, and the value is the corresponding information.

[0168] The following describes that the interaction module adopts the attention mechanism to interact information, please refer to Figure 7 Based on the parameters of the interaction module, step S63 can specifically include the following steps:

[0169] Step S71: The training device maps the representation vector corresponding to the data segment of the i-th scale as the first representation vector, and the representation vector corresponding to the data segment of the i+1-th scale as the second representation vector, and maps the first representation vector into the first query vector, and maps the second representation vector into the first key vector and the first value vector, respectively, based on the parameters of the interaction module.

[0170] In the embodiment of the present application, the training device uses the interaction module to perform attention function operation on the input representation vector, and outputs the corresponding request (query) vector, key (key) vector and value (value) vector. The above attention function can be implemented as linear transformation. Linear transformation can map vectors belonging to one vector space to another vector space. A vector space is a set composed of a plurality of vectors of the same dimension.

[0171] In specific implementation, the interaction module includes three different learnable parameter matrices WQ, WK and WV. The training device uses the three learnable parameter matrices to perform linear transformation on the representation vector input to the interaction module, to obtain the corresponding query vector, key vector and value vector. The learnable parameter matrices WQ, WK and WV are parameters of the interaction module.

[0172] Step S72: The training device calculates the attention score of the first query vector and the first key-value vector through the attention mechanism, to obtain a first attention score.

[0173] In the embodiment of the present application, the training device takes the representation vectors corresponding to the audio segments of the same sub-set and adjacent scales (such as the i-th and the i+1-th) as inputs, and obtains the learnable parameter matrices WQ, WK and WV corresponding to the audio segments of the i-th scale. The training device maps the representation vector corresponding to the audio segment of the i-th scale (i.e., the first representation vector) into the first query vector through the learnable parameter matrix WQ, maps the representation vector corresponding to the audio segment of the i+1-th scale (i.e., the second representation vector) into the first key-value vector through the learnable parameter matrix WK, and maps the representation vector corresponding to the audio segment of the i+1-th scale (i.e., the second representation vector) into the first value vector through the learnable parameter matrix WV. The training device performs attention score calculation on the first query vector and the first key-value vector, to obtain the first attention score between the first representation vector and the second representation vector.

[0174] The first attention score calculation formula is as follows:

[0175]

[0176] wherein, and represents a linear transformation with a bias term, is used to map the input into the first query vector, is used to map the input into the first key-value vector. is the first representation vector. is the second representation vector. T in the above formula represents the transpose of a matrix, is the first attention score. represents the exponential function with base e.

[0177] Step S73: The training device obtains the fused representation vector corresponding to the data segment of the i-th scale according to the first value vector, the first attention score and the first representation vector.

[0178] In the embodiment of the present application, the training device performs content query in the value vector of the i+1-th scale data segment according to the first attention score obtained above, and sums the content queried from the value vector of the i+1-th scale data segment and the first representation vector of the i-th scale data segment according to a certain weight, to obtain the fused representation vector corresponding to the data segment of the i-th scale.

[0179] The fusion formula of the fused representation vector corresponding to the data segment of the i-th scale is as follows:

[0180]

[0181] wherein, represents a linear transformation with bias term, for mapping the input into the first value vector. is the fused representation vector corresponding to the i-th scale data segment. is the first representation vector. is the second representation vector. is a weight hyper-parameter, for measuring the importance between the first representation vector and the interaction query to the content (i.e., in the above fusion formula, ).

[0182] , and correspond to the learnable parameter matrices WQ, WKand WVof the interaction module, i.e. , and are the parameters to be learned by the interaction module.

[0183] Step S74: The training device maps the second representation vector into the second query vector, and maps the fused representation vector corresponding to the i-th scale data segment into the second key vector and the second value vector, respectively, based on the parameters of the interaction module.

[0184] In the embodiments of the present application, the interaction module includes three learnable parameter matrices Wq, Wkand Wvcorresponding to the i+1-th scale data, and the training device utilizes the three learnable parameter matrices to perform linear transformation on the representation vector input into the interaction module, to obtain the corresponding second query vector, second key vector and second value vector. The learnable parameter matrices Wq, Wkand Wvare the parameters of the interaction module.

[0185] The learnable parameter matrices WQ, WKand WVand the learnable parameter matrices Wq, Wkand Wvcan be different, i.e., for different scales of data segments or different data segments, the corresponding learnable parameter matrices are different.

[0186] Step S75: The training device calculates the attention score of the second query vector and the second key vector through the attention mechanism, to obtain the second attention score.

[0187] In the embodiment of the present application, the training device takes the fused representation vector corresponding to the data segment of the i-th scale and the second representation vector obtained in step S74 as input, and obtains the learnable parameter matrix Wq, Wk and Wv corresponding to the audio segment of the i+1-th scale, the training device maps the representation vector (i.e., the second representation vector) corresponding to the audio segment of the i+1-th scale into the second query vector through the learnable parameter matrix Wq, maps the fused representation vector corresponding to the data segment of the i-th scale into the second key-value vector through the learnable parameter matrix Wk, and maps the fused representation vector corresponding to the data segment of the i-th scale into the second value vector through the learnable parameter matrix Wv. The second query vector and the second key-value vector are subjected to attention score calculation to obtain the second attention score between the second representation vector and the fused representation vector corresponding to the data segment of the i-th scale.

[0188] The second attention score calculation formula is as follows:

[0189]

[0190] Among them and represents a linear transformation with a bias term, is used to map the input into the second query vector, is used to map the input into the second key-value vector. is the first representation vector. is the second representation vector. T of represents the transpose of the matrix. is the fused representation vector corresponding to the data segment of the i-th scale. is the second attention score. represents the exponential function with base e.

[0191] Step S76: The training device obtains the fused representation vector corresponding to the data segment of the i+1-th scale according to the second value vector, the second attention score and the second representation vector.

[0192] In the embodiment of the present application, the training device queries the content in the value vector of the i-th scale data segment according to the second attention score obtained above, and sums the content queried from the value vector of the i-th scale data segment and the second representation vector of the i+1-th scale data segment according to a certain weight to obtain the fused representation vector corresponding to the data segment of the i+1-th scale.

[0193] The fusion formula of the fused representation vector corresponding to the data segment of the i+1-th scale is as follows:

[0194]

[0195] Among them, represents a linear transformation with bias term, used to map the input into a second representation vector. is the fused representation vector corresponding to the data segment of the i+1th scale. is the second representation vector. is a weight hyper-parameter, used to measure the importance between the second representation vector and the interaction query to the content (i.e. ) in the above fusion formula. is the second attention score.

[0196] 、 and correspond to the learnable parameter matrices Wq, Wk and Wv of the interaction module, i.e. 、 and are the parameters to be learned by the interaction module.

[0197] Please refer to Figure 8 , Figure 8 for the structure diagram of the interaction module provided by the embodiments of the present application.

[0198] As shown in Figure 8 , the interaction module includes at least two encoder parts. As an example, the information interaction between the representation vectors d11, d21 corresponding to the data segments of the first scale in the first sub-set of the song A and the representation vector D1 corresponding to the data segments of the second scale is taken as an example.

[0199] The training device inputs the representation vectors d11, d21 corresponding to the data segments of the first scale and the representation vector D1 corresponding to the data segments of the second scale in the first sub-set to the first encoder part, and the representation vectors d11, d21 and the representation vector D1 perform information interaction respectively to obtain the fused representation vectors respectively.

[0200] The training device obtains the learnable parameter matrices WQ1, WK1 and WV1 (i.e. the parameters of the interaction module) corresponding to the data segment d1, maps the representation vector d11 into a first query vector Qd11 according to the matrix WQ1 (i.e. d11*WQ1), maps the representation vector D1 into a first key-value vector KD1 according to the matrix WK1 (i.e. D1*WK1), and maps the representation vector D1 into a first value vector VD1 according to the matrix WV1 (i.e. D1*WV1).

[0201] The training device calculates the first attention score Then, the training device calculates the fusion representation vector corresponding to representation vector d11 based on the fusion formula of the fusion representation vector corresponding to the data segment of the i-th scale, the representation vector d11, the first attention score, and the first value vector. .

[0202] Similarly, the training device processes the representation vectors d21 and D1 based on the learnable parameter matrices WQ2, WK2, and WV2 (i.e., the parameters of the interaction module) corresponding to the data segment d2, and can obtain the fused representation vector corresponding to the representation vector d21. The learnable parameter matrices WQ1, WK1, and WV1 corresponding to d1 can be different from the learnable parameter matrices WQ2, WK2, and WV2 corresponding to d2, depending on the specific circumstances.

[0203] The training device will use the fused representation vector corresponding to the representation vector d11. The fused representation vector corresponding to the representation vector d21 The representation vector D1 is input to the second encoder to obtain the learnable parameter matrices Wq, Wk, and Wv associated with the representation vector D1. The representation vector D1 is mapped to the second query vector qD1 (i.e., D1*Wq) through the learnable parameter matrix Wq, and the fused representation vector is obtained through the learnable parameter matrix Wk. Mapped to the second key value vector kd1 (i.e. *Wk), the fused representation vector is obtained through the learnable parameter matrix Wv. Mapped to the second value vector vd1 (i.e. *Wv), through the learnable parameter matrix Wk, the fused representation vector Mapped to the second key value vector kd2 (i.e. *Wk), the fused representation vector is obtained through the learnable parameter matrix Wv. Mapped to the second value vector vd2 (i.e. *Wv).

[0204] The training device calculates the second attention score between the second query vector qD1 and the second key value vector kd1 according to the first attention score calculation formula described above. Calculate the second attention score between the second query vector qD1 and the second key value vector kd2. Then, the training device uses the fusion formula, representation vector d11, and second attention score corresponding to the data segment of the (i+1)th scale mentioned above as the basis for its training. Second attention score The second value vector vd1 and the second value vector vd2 are used to calculate the fused representation vector corresponding to the representation vector D1. Specifically, it is as follows:

[0205]

[0206] wherein, represents a linear transformation with bias term, for mapping the input into the second value vector. is the fused representation vector corresponding to vector d11. is the fused representation vector corresponding to vector d21. i.e. the second value vector vd1, i.e. the second value vector vd2. is the fused representation vector corresponding to the i+1th scale data segment. is the second attention score between the second query vector qD1 and the second key value vector kd1. is the second attention score between the second query vector qD1 and the second key value vector kd2. D1 is the second representation vector corresponding to the second scale data segment. is a weight hyper-parameter for measuring the importance between the second representation vector and the interactive query to the content (i.e. the in the above fusion formula). is a weight hyper-parameter for measuring the importance between the second representation vector and the interactive query to the content (i.e. the in the above fusion formula).

[0207] In the embodiments of the present application, for the first scale audio segment d1, its fused representation vector fuses the information of the second scale audio segment D. For the first scale audio segment d2, its fused representation vector fuses the information of the second scale audio segment D. For the second scale audio segment D, its fused representation vector fuses the information of all the first scale audio segments (such as audio segments d1, d2) in its corresponding sub-set.

[0208] Similarly, the fused representation vector corresponding to the second scale audio segment E also fuses the information of all the first scale audio segments (such as audio segments e1, e2) in its corresponding sub-set. The fused representation vector corresponding to the second scale audio segment F also fuses the information of all the first scale audio segments (such as audio segments f1, f2) in its corresponding sub-set. The fused representation vector corresponding to the second scale audio segment G also fuses the information of all the first scale audio segments (such as audio segments g1, g2) in its corresponding sub-set.

[0209] The training device takes the fused representation vectors corresponding to the audio segments D and E of the second scale as input, and continues to interact with the audio segment B of the third scale. For the audio segment D of the second scale, the fused representation vector corresponding thereto further fuses the information of the audio segment B of the third scale. For the audio segment E of the second scale, the fused representation vector corresponding thereto further fuses the information of the audio segment B of the third scale. For the audio segment B of the third scale, the fused representation vector corresponding thereto fuses the information of all the audio segments (such as the audio segments D and E) of the second scale in the corresponding sub-set.

[0210] The training device takes the fused representation vectors corresponding to the audio segments F and G of the second scale as input, and continues to interact with the audio segment C of the third scale. The fused representation vector corresponding to the audio segment C of the third scale further fuses the information of all the audio segments (such as the audio segments F and G) of the second scale in the corresponding sub-set.

[0211] The training device takes the fused representation vectors corresponding to the audio segments B and C of the third scale as input, and continues to interact with the audio segment (i.e., the whole song A) of the fourth scale. For the audio segment B of the third scale, the fused representation vector corresponding thereto further fuses the information of the whole song A. For the audio segment C of the third scale, the fused representation vector corresponding thereto further fuses the information of the whole song A. For the whole song A, the fused representation vector corresponding thereto fuses the information of all the audio segments (such as the audio segments B and C) of the third scale in the corresponding sub-set, and the information of all the audio segments (such as the audio segments B and C) of the third scale further fuses the information of the audio segments of a smaller scale, that is, finally the whole song A fuses the information of all the audio segments in the corresponding sub-set.

[0212] Step S34: The training device constructs a target function according to the fused representation vectors.

[0213] The learning representation model provided in the embodiments of the present application can use a neural network. Since it is desired that the output of the neural network is as close as possible to the value that is actually desired to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the value that is actually desired to be predicted by comparing the predicted value of the current network with the value that is actually desired to be predicted (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer in the neural network are pre-configured). For example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower. The adjustment is continuously made until the neural network can predict the value that is actually desired to be predicted. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the neural network becomes a process of trying to minimize the loss.

[0214] In the embodiments of the present application, the objective function is constructed according to the fused representation vector. Please refer to Figure 9 , step S34 can be the following steps:

[0215] Step S91: The training device obtains the overall representation vector of other to-be-learned data, wherein the other to-be-learned data and the to-be-learned data are in the same batch, and the overall representation vector is the representation vector corresponding to the data segment with the largest scale in the other to-be-learned data.

[0216] It should be noted that in the training process, iteration is to train part of the to-be-learned data in the training set once, that is, to perform one forward propagation and one backward propagation on part of the to-be-learned data in the training set in the to-be-trained representation learning model. Batch is a part of data sent into the to-be-trained representation learning model. Batch size is a hyperparameter used to define the number of to-be-learned data to be trained before updating the parameters of the to-be-trained representation learning model.

[0217] In the embodiments of the present application, the to-be-learned data input into the to-be-trained representation learning model for training in the same batch can include multiple to-be-learned data. For example, in the above example, the to-be-learned data in the same batch can also include song B, song C, etc. The training device inputs the to-be-learned data in the same batch (such as song A, B and C) into the encoder and the interaction module, and obtains the overall representation vector of song A, B and C through step S33, wherein the overall representation vector represents the representation vector corresponding to the data segment with the largest scale of the song, which fuses the information of all audio segments in the corresponding sub-set.

[0218] Step S92: The training device calculates the similarity between each fused representation vector and the overall representation vector of other to-be-learned data.

[0219] In the embodiments of the present application, in step S33, the training device can obtain the fused representation vector corresponding to all audio segments in the data set of each to-be-learned data in the training set. For each fused representation vector, the similarity between the fused representation vector and the overall representation vector of other to-be-learned data in the same batch is calculated. For example, for the fused representation vector corresponding to each audio segment in the data set of song A, the cosine similarity between the fused representation vector and the overall representation vector of song B is calculated, and the cosine similarity between the fused representation vector and the overall representation vector of song C is calculated.

[0220] The cosine similarity calculation formula is as follows:

[0221]

[0222] wherein, and respectively represent the representation vectors corresponding to different two to-be-learned data in the same batch output by the interaction module, and wherein, is the fused representation vector corresponding to a data segment in a to-be-learned data, is the overall representation vector of another to-be-learned data, and ‖.‖ is the modulus of the vector. is the cosine similarity.

[0223] In the embodiments of the present application, when M to-be-learned data are included in the same batch data, for a data segment of a certain scale of a certain to-be-learned data, the training device calculates the cosine similarity between the fused representation vector of the data segment and the overall representation vector of another M-1 to-be-learned data, and then processes all the obtained cosine similarities, such as averaging all the cosine similarities, and the obtained average value is taken as the similarity between the fused representation vector of the data segment and the overall representation vector of other to-be-learned data. In other embodiments, one other to-be-learned data in the same batch data can be randomly selected, and the training device calculates the cosine similarity between the fused representation vector of the data segment and the overall representation vector of the selected to-be-learned data, and the obtained cosine similarity is taken as the similarity between the fused representation vector of the data segment and the overall representation vector of other to-be-learned data.

[0224] In the embodiments of the present application, for each to-be-learned data in the training set, the training device can obtain, in step S92, the similarity between the fused representation vector of all scale data segments in the data set of the to-be-learned data and the overall representation vector of other to-be-learned data.

[0225] Step S93: For each scale, the training device screens out the fused representation vector corresponding to the lowest similarity.

[0226] In the embodiment of the present application, for each to-be-learned data, if the to-be-learned data includes M scales, the training device screens out, for each scale, the fused representation vector corresponding to the M lowest similarities based on the lowest similarity value.

[0227] As in the above example, for song A, the training device respectively calculates, in step S92, the similarity of the fused representation vector of song A, audio segments B, C, D, E, F, G, d1, d2, e1, e2, g1 and g2 with the overall representation vector of other to-be-learned data in the same batch. For the first scale, the training device determines the lowest similarity from the similarity of the fused representation vector of audio segments d1, d2, e1, e2, g1 and g2 with the overall representation vector of other to-be-learned data in the same batch, and selects the fused representation vector corresponding to the lowest similarity. For example, in the first scale, the fused representation vector d13 corresponding to audio segment d1 has the lowest similarity with the overall representation vector of other to-be-learned data, and the fused representation vector d13 is selected. Similarly, for the second scale, the training device selects the fused representation vector D11. For the third scale, the training device selects the fused representation vector B11. Based on the fact that there is only song A as a complete song in the data set of song A, the training device directly obtains the overall representation vector output from the interaction module for song A for the fourth scale.

[0228] In the embodiment of the present application, by screening out the data segments of different scales with the lowest similarity to other to-be-learned data, more distinguishable data segments between different to-be-learned data are obtained, so as to avoid redundancy in subsequent contrast learning and accelerate the convergence of the objective function.

[0229] Step S94: The training device constructs an objective function according to the screened fused representation vector.

[0230] It should be noted that machine learning, as an important branch of the AI field, has been widely applied in many fields. From the perspective of learning methods, machine learning can be divided into supervised learning, self-supervised learning, semi-supervised learning, and reinforcement learning. Supervised learning refers to learning an algorithm or establishing a pattern based on training data, and predicting new instances based on the algorithm or pattern. The model of machine learning (i.e., learning representation model) is also called machine learning model, and its expected output is called label, which can be a predicted classification result (called classification label). The difference between self-supervised learning and supervised learning is that the training sample of self-supervised learning does not have a given label, and the machine learning model obtains certain results by analyzing the training sample. Self-supervised learning can be regarded as an "ideal state" of machine learning, in which the model learns directly from unlabeled data without the need for labeled data. The core of self-supervised learning is how to automatically generate labels for data.

[0231] The representation learning model provided by the embodiments of the present application can be trained by a self-supervised learning method. The self-supervised learning method can be divided into three categories: context-based, temporal-based, and contrast-based. In the embodiments of the present application, the representation learning model is trained by using contrastive learning. The contrastive learning method constructs positive samples and negative samples, and compares the positive samples and the negative samples in the feature space to learn the latent feature representation of the model. This method aims to maximize the mutual information between positive and negative samples through contrastive learning to learn the latent representation vector.

[0232] Please refer to Figure 10 , step S94 can specifically include the following steps:

[0233] Step S101: The training device maps the filtered fused representation vector to a positive sample.

[0234] Step S102: The training device maps the overall representation vector corresponding to the other to-be-learned data in the same batch to a negative sample.

[0235] Step S103: Based on contrastive learning, the training device constructs a target function by taking the positive sample and the negative sample as sub-elements of the target function.

[0236] Step S35: The training device optimizes the target function to adjust the parameters of the encoder and the parameters of the interaction module, so that the encoder and the interaction module learn the representation vector of the to-be-learned data.

[0237] In this embodiment, a multi-level contrastive learning approach is adopted, which involves narrowing down the representation vectors of data segments at adjacent scales and gradually transitioning to the representation vector of the largest scale data segment (i.e., the entire song). The objective function is optimized as follows:

[0238]

[0239] Where N is the number of scale types. i represents the i-th song in the same batch. This is the predicted value output by the objective function after multiple layers of contrastive learning, i.e., the loss value. τ is a temperature hyperparameter used to control the uniformity of the generated representation. It is a hyperparameter, j is less than or equal to N-1, and j represents the weight value of the j-th adjacent scale pair in the data to be learned. This represents an exponential function with base e. `sim()` is the function for calculating similarity. `B` represents the batch size. It is a function whose value is 0 when k=i, and 1 otherwise. and Let i and k represent the representations of the entire song in the contrastive learning space, respectively; that is, the overall representation vectors of the i and k songs, respectively. The vector representing the audio segment at the j-th scale in the contrastive learning space. The vector representing the audio segment at the (j+1)th scale in the contrastive learning space.

[0240] In this embodiment, after the training device selects the corresponding fused representation vector in step S93, it outputs the selected representation vector to a multi-layer perceptron (MLP). The MLP mapper maps the fused representation vector to the multi-layer perceptron in contrastive learning, enabling multi-level contrast in the contrastive learning space. That is, the MLP maps the fused representation vector in the above fusion formula... Convert to ,Will Convert to Map the overall representation vector of the song to .

[0241] In this embodiment, the representation vector input to the interaction module can be changed by adjusting the weight and bias parameters in the encoder, and the representation vector output by the interaction module can be changed by adjusting the parameter matrices in the interaction module, such as WQ, WK, and WV. , This correspondingly changes the representation vector in the contrastive learning space. and This makes the predicted value It is close to the target value we really want.

[0242] Please refer to Figure 11 , step S35 can specifically include the following steps:

[0243] Step S111: The training device takes the positive samples corresponding to two scales adjacent in scale as a positive sample pair.

[0244] Step S112: The training device calculates the first similarity between the positive sample corresponding to the maximum scale and the negative sample, and the second similarity of the positive sample pair, through the target function.

[0245] Step S113: The training device optimizes the target function according to the first similarity and the second similarity to update the parameters of the encoder and the parameters of the interaction module.

[0246] In the embodiments of the present application, the information amount between the complete to-be-learned data and the segmented data segments is greatly different, for example, the information amount between song A and audio segment d1 is greatly different, and forcibly narrowing the distance between the whole song A and the audio segment d1 will destroy the global feature. The embodiments of the present application adopt a multi-level contrastive learning, and the training device narrows the representation vectors of audio segments adjacent in length. In some embodiments, the representation vectors of small scales can be first narrowed and gradually transitioned to the representation vectors of large scales.

[0247] In the embodiments of the present application, for a to-be-learned data, the training device calculates the first similarity between the positive sample corresponding to the maximum scale of the to-be-learned data and the negative sample, as in the above example, the first similarity between the whole representation vector of song A and the representation vectors of other songs B in the same batch is calculated, and the first similarity between the whole representation vector of song A and the representation vectors of other songs C in the same batch is calculated. The training device calculates the second similarity between the positive samples corresponding to two scales adjacent in scale in song A.

[0248] As in the above example, please refer to Figure 12For song A, the training device screens the representation vector d13 corresponding to the first scale, the representation vector D11 corresponding to the second scale, the representation vector B11 corresponding to the third scale, and the overall representation vector A2 corresponding to the fourth scale in step S34. The training device narrows the distance between the representation vector d13 corresponding to the first scale and the representation vector D11 corresponding to the second scale, narrows the distance between the representation vector D11 corresponding to the second scale and the representation vector B11 corresponding to the third scale, and finally narrows the distance between the representation vector B11 corresponding to the third scale and the representation vector A11 corresponding to the fourth scale, and widens the distance between the overall representation vector A2 corresponding to the fourth scale and the overall representation vectors of other to-be-learned data in the same batch. Finally, the obtained loss is added together to optimize the representation learning model. It can be understood that narrowing the distance means increasing the similarity between two representation vectors, and widening means reducing the similarity between two representation vectors.

[0249] In the embodiments of the present application, the data segments between different scales of the same to-be-learned data are taken as positive samples, the correlation between the positive samples is ensured, and good feature expression can be learned. Other to-be-learned data in the same batch are taken as negative samples, so that the workload can be reduced without constructing additional negative samples. Based on the low similarity between the selected positive samples and negative samples, better similar clustering effect can be achieved in the positive and negative sample comparison learning.

[0250] The training device trains the representation learning model in the training stage described above, and the representation learning model at least includes a trained encoder and a trained interaction module. The process of applying the model is described below, that is, the process of performing inference by the execution device.

[0251] II. Inference stage

[0252] In the embodiments of the present application, please refer to Figure 13 , Figure 13 Another flowchart of the representation learning method provided in the embodiments of the present application. The representation learning method provided in the embodiments of the present application can include the following steps:

[0253] Step S131: The execution device acquires a data set of to-be-learned data, wherein the data set includes at least one sub-set, the sub-set includes data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is part of the data segment of the i+1-th scale, i is less than or equal to K-1, and the number of data segments of each scale is one or more.

[0254] In the embodiment of the present application, if the data set of the to-be-learned data itself includes data segments of K scales, the execution device can directly use the data set. If not, the execution device can obtain the to-be-learned data and the preset number of scale categories, and then segment the to-be-learned data to obtain data segments. In the segmentation, the execution device segments the data segment of the smallest current scale until the number of scale categories in the sub-set reaches the preset number of scale categories. For details, refer to Figure 4 , which will not be repeated here.

[0255] Step S132: The execution device inputs the data set to the trained encoder, and extracts features of the data segments based on the parameters of the encoder to obtain the representation vectors corresponding to the data segments of various scales respectively.

[0256] In the embodiment of the present application, the trained encoder includes the content of the encoder in the training stage described above, which will not be repeated here.

[0257] Step S133: The execution device inputs the representation vectors to the trained interaction module, and performs information interaction on the representation vectors corresponding to the data segments of adjacent scales in the sub-set based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively, so as to complete the learning of the representation vectors of the to-be-learned data.

[0258] Please refer to Figure 14 , step S133 can include the following steps:

[0259] Step S141: The execution device inputs the representation vectors to the trained interaction module.

[0260] Step S142: The execution device determines that the scale to be processed in the n th time is the i th scale and the i+1 th scale, and assigns the initial value of n as 1, and performs the following operations in a loop until n=K-1, where n=i, and n is an integer greater than or equal to 1.

[0261] Step S143: The execution device performs information interaction on the representation vectors corresponding to the data segments of the i th scale and the i+1 th scale in the sub-set in the n th time based on the parameters of the trained interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

[0262] In the embodiment of the present application, when n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i th scale is the fused representation vector obtained after the information interaction of the data segment of the i th scale in the n-1 th time.

[0263] Please refer to Figure 15 , step S143 can include the following steps:

[0264] Step S151: The execution device takes the representation vector corresponding to the data segment of the i-th scale as the first representation vector, takes the representation vector corresponding to the data segment of the i+1-th scale as the second representation vector, and maps the first representation vector into the first query vector and maps the second representation vector into the first key-value vector and the first value vector respectively based on the parameters of the trained interaction module.

[0265] Step S152: The execution device calculates the attention score of the first query vector and the first key-value vector through the attention mechanism to obtain the first attention score.

[0266] Step S153: The execution device obtains the fused representation vector corresponding to the data segment of the i-th scale according to the first value vector, the first attention score and the first representation vector.

[0267] Step S154: The execution device maps the second representation vector into the second query vector and maps the fused representation vector corresponding to the data segment of the i-th scale into the second key-value vector and the second value vector respectively based on the parameters of the trained interaction module.

[0268] Step S155: The execution device calculates the attention score of the second query vector and the second key-value vector through the attention mechanism to obtain the second attention score.

[0269] Step S156: The execution device obtains the fused representation vector corresponding to the data segment of the i+1-th scale according to the second value vector, the second attention score and the second representation vector.

[0270] It can be understood that the representation learning model obtained in the reasoning stage inherits the content and functions of the related modules (encoder, interaction module) in the training stage. The trained encoder includes the content of the encoder in the above training stage, and the trained interaction module includes the content of the interaction module in the above training stage, which will not be described here.

[0271] In the embodiments of the present application, the trained encoder and the interaction module perform information interaction on the representation vectors corresponding to the data segments with different scales and adjacent scales in the same sub-set, so that the representation vector of the data segment of the i-th scale includes the information of the representation vector of the data segment of the i+1-th scale, and the representation vector of the data segment of the i+1-th scale includes the information of the representation vector of the data segment of the i-th scale, and then a high-quality representation vector can be obtained, that is, the obtained representation vector contains as much data information as possible which is useful for subsequent tasks, so that it is easier to extract useful information for subsequent tasks such as constructing a classifier or other prediction tasks, and the performance in downstream tasks (such as classification) is improved. Moreover, the representation vectors corresponding to different scale data segments of the to-be-learned data can be learned, for example, for a song, the representation vector of the whole song can be learned, and the representation vector of the audio segment can also be learned, so that different granularity representation vectors can be provided for different tasks.

[0272] Please refer to Figure 16 , Figure 16 The structure schematic diagram of the execution device provided in the embodiments of the present application is shown in FIG. 1.

[0273] The execution device 160 includes an acquisition module 161, an encoder 162, and an interaction module 163.

[0274] The acquisition module 161 is configured to acquire a data set of to-be-learned data, wherein the data set includes at least one sub-set, the sub-set includes data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is part of the data segment of the i+1-th scale, i is less than or equal to K-1; and the number of data segments of each scale is one or more.

[0275] The encoder 162 is configured to receive the data set, and perform feature extraction on the data segments based on parameters of the encoder, to obtain representation vectors corresponding to the data segments of various scales respectively, wherein the encoder is trained.

[0276] The interaction module 163 is configured to receive the representation vectors, and perform information interaction on the representation vectors corresponding to the data segments with adjacent scales in the sub-set based on parameters of the interaction module, to obtain fused representation vectors corresponding to the data segments of various scales respectively, wherein the interaction module is trained.

[0277] In some embodiments, the representation learning module can include the acquisition module 161, the encoder 162, and the interaction module 163.

[0278] It should be noted that the information interaction, execution process, and the like between the modules / units in the execution device are the same as those in the present application. Figures 13 to 15Corresponding various method embodiments are based on the same concept, and specific content can be referred to the description of the method embodiments shown in the foregoing of the present application, which will not be repeated here.

[0279] The present application also provides an execution device, please refer to Figure 17 , Figure 17 A structural schematic diagram of an execution device provided by the present application is shown in FIG. 17, wherein the execution device 170 can be deployed with Figures 13 to 15 The representation learning model described in the corresponding embodiments is used to realize Figures 13 to 15 the functions of the execution device in the corresponding embodiments. Specifically, the execution device 170 includes a receiver 171, a transmitter 172, a processor 173 and a memory 174 (wherein the number of processors 173 in the execution device 170 can be one or more, Figure 17 taking one processor as an example in the present application), wherein the processor 173 can include an application processor 1731 and a communication processor 1732. In some embodiments of the present application, the receiver 171, the transmitter 172, the processor 173 and the memory 174 can be connected through a bus or other means.

[0280] The memory 174 can include read-only memory and random access memory, and provide instructions and data to the processor 173. A part of the memory 174 can also include non-volatile random access memory (NVRAM). The memory 174 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0281] The processor 173 controls the operation of the execution device. In specific applications, various components of the execution device are coupled together through a bus system, wherein the bus system can include a data bus, a power supply bus, a control bus and a state signal bus, etc. However, in order to clearly illustrate, various buses are referred to as a bus system in the figure.

[0282] The method disclosed in the embodiments of the present application can be applied to the processor 173 or implemented by the processor 173. The processor 173 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 173. The processor 173 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller. The processor 173 can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 173 can implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor to execute, or a combination of hardware and software modules in the code processor to execute. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the memory 174, and the processor 173 reads the information in the memory 174 and combines the hardware to complete the steps of the method.

[0283] The receiver 171 can be configured to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 172 can be configured to output digital or character information through the first interface; the transmitter 172 can also be configured to send instructions to the disk group through the first interface to modify the data in the disk group; and the transmitter 172 can further include a display device such as a display screen.

[0284] In the embodiments of the present application, in one case, the application processor 1731 is configured to execute Figures 13 to 15 the functions of the execution device in the corresponding embodiments. It should be noted that the specific implementation manner of the application processor 1731 executing Figures 13 to 15 the functions of the execution device in the corresponding embodiments and the beneficial effects brought by the specific implementation manner can be referred to the description in the corresponding method embodiments, which will not be repeated here. Figures 13 to 15

[0285] ​The following takes the application of the representation learning model provided in the application to representation learning of audio data as an example to introduce the application scenario example of the representation learning model in the embodiments of the application. (a) in FIG. 18 is a schematic diagram of a music classification scenario provided in the embodiments of the application. (b) in FIG. 18 is a schematic diagram of a song cover version retrieval scenario provided in the embodiments of the application. (c) in FIG. 18 is a schematic diagram of a similar song recommendation scenario provided in the embodiments of the application.

[0286] As shown in (a) in FIG. 18, the music application 182a is deployed in the client 181a, and the music classification service 184a is deployed in the cloud server 183a. The music classification service 184a includes the music representation module 185a and the music classification module 186a. The music representation module is internally deployed with the representation learning model provided in the embodiments of the application, and the representation vector of the audio file is extracted through the representation learning model, for example, the representation vector can be 512 dimensions. The music classification module obtains the representation vector of the audio file from the music representation module, and then performs category judgment using the representation vector.

[0287] The music representation module 185a includes the obtaining module 187a, the encoder 188a, and the interaction module 189a. The music representation module 185a obtains an audio file, which can be in Moving Picture Experts Group Audio Layer III (MP3) format or Wave Audio Files (WAV), and the like, which is not limited in the embodiments of the application.

[0288] The music classification module 186a can be implemented as a classifier, which can use any existing classifier model, such as support vector machine (SVM), neural network, k-nearest neighbor classification, and the like, which is not limited in the embodiments of the application.

[0289] Specifically, if the client 181a detects that a new music is added to its song library, the client 181a will send a request to the cloud server 183a to request the music classification service in the cloud server. After receiving the request, the cloud server 183a will call the interface of the music classification service to perform the music classification operation. The obtaining module 187a, the encoder 188a, and the interaction module 189a implement the above-mentioned Figure 16The illustrated function obtains a fused representation vector of each audio segment. The classifier obtains the fused representation vector of each audio segment, and then performs category judgment according to the fused representation vector to determine the category corresponding to the music file. The cloud server 182a returns the category corresponding to the music file to the client 181a. The client 181a generates or updates a classification interface according to the category corresponding to the music file. When the user accesses the music application 182a in the client 181a, the music application 182a displays the classification interface in response to the operation of the user.

[0290] In the application scenario shown in (a) of FIG. 18, the representation learning model provided by the embodiments of the present application is trained and tested using the public standard data sets MagnaTagATune and GTZAN. We use two classification indicators, Receiver Operating Characteristic-Area Under roc Curve (ROC-AUC) and precision recall-Area Under roc Curve (PR-AUC), where Receiver Operating Characteristic is the Receiver Operating Characteristic curve, Area Under roc Curve refers to the area under the ROC curve, precision is precision, and recall is recall. The higher these indicators are, the better the classification effect is. The experimental results of the representation learning model provided by the embodiments of the present application and other existing schemes (such as PyTorch officially implemented music representation contrast learning: CLMR, BYOL-A, Multi-Format, COLA) using standard data sets MagnaTagATune or GTZAN under different encoders (SampleCNN, four-layer fully convolutional network: FCN-4, seven-layer fully convolutional network: FCN-7) are compared in Tables 1 and 2, respectively.

[0291] Table 1 Comparison of overall performance of music classification model (MagnaTagATune)

[0292]

[0293] Table 2 Comparison of overall performance of music classification model (GTZAN)

[0294]

[0295] From the above Tables 1 and 2, it can be concluded that the representation learning method of the present application applied to music classification can effectively improve the effect of music classification, that is, the quality of the representation vector extracted using the learning representation model provided by the embodiments of the present application is high.

[0296] As shown in (b) of FIG. 18, the difference between (b) of FIG. 18 and (a) of FIG. 18 is that the cloud server 183b is deployed with a song cover version retrieval service 184b. The song cover version retrieval service 184b includes a music representation module 185b and a cover version identification module 186b. The cover version identification module 186b obtains the representation vector of the audio file from the music representation module 185b, and then identifies using the representation vector, and retrieves the song file with the highest similarity to the audio file from the song library.

[0297] The cover version identification module 186b can use any existing audio identification device, mainly relying on an audio identification method to identify the song. For example, using a fingerprint feature technology, determining the fingerprint feature of the audio to be identified, and matching the song with the fingerprint feature of the audio to be identified from the audio library as the song in the audio.

[0298] Specifically, the representation learning model can be used in advance to extract vectors for each song in the song library and stored in a vector index library (such as a Faiss engine). The user accesses the music application 182b in the client 181b, and inputs a cover retrieval instruction to the music application 182b. The music application 182b responds to the cover retrieval instruction, sends a request to the cloud server 183b through the client 181b to request a song cover version retrieval service in the cloud server. After receiving the request, the cloud server calls the interface of the song cover version retrieval service to perform the song cover version retrieval operation. The obtaining module 187a, the encoder 188a, and the interaction module 189a realize the functions shown above, and obtain the representation vector fused by each audio segment. Figure 16 The cover version identification module 186b obtains the representation vector fused by each audio segment, and then uses a k-nearest neighbor retrieval Faiss engine to retrieve similar songs (for example, the top 100 songs with the highest similarity) from the song library according to the cosine similarity between vectors to obtain a retrieval result. Then, according to the retrieval result, the literal repetition degree is calculated using the song name information of the input song and the song name information in the song library, and a secondary filtering is performed using a set threshold, and the song list remaining after the filtering is returned.

[0299] In some other embodiments, the obtaining module 187b or the encoder 188b is deployed to the client 181b. As shown in (c) of FIG. 18, the difference between (c) of FIG. 18 and (b) of FIG. 18 is that the obtaining module 187c is deployed to the client 181c. After the client 181c obtains the music file or the recording uploaded by the user, the music file is segmented using the obtaining module to obtain audio segments of various scales, and the client 181c only needs to upload the audio segments of various scales to the cloud server 183c, without the need to upload the entire music file to the cloud server 183c, thereby reducing the amount of data transmitted and improving the efficiency.

[0300] In some embodiments, the acquisition module 187c and the encoder 188c are both deployed in the client 181c, and then the client 181c acquires the music file uploaded by the user, uses the acquisition module 187c to segment the music file to obtain audio segments of various scales. The encoder 188c extracts features from the audio segments of various scales to obtain the representation vectors corresponding to the audio segments of various scales. The client 181c only needs to upload the representation vectors corresponding to the audio segments of various scales to the cloud server 183c, which further reduces the amount of data transmitted and improves efficiency.

[0301] In the application scenarios shown in (b) and (c) of FIG. 18, detailed tests are performed using public data and business data. Tests are performed on the public data sets SHS100K-SUB and Covers80 data, and we use three accuracy indicators, MAP, Precision@10, and MR1. The higher these indicators are, the better the cover song recognition effect is. The experimental results of the representation learning method of the present application and existing methods (such as Ki-Net, TPP-Net, FCN, CQT-Net, BYOL-A, Multi-Format, and COLA) are compared as shown in Table 3 below.

[0302] Table 3

[0303]

[0304] As can be seen from Table 3 above, the representation learning method of the present application applied to the cover version retrieval scenario has achieved significant improvement in effect, that is, the quality of the representation vector extracted using the learning representation model provided by the embodiments of the present application is high.

[0305] The difference between (d) of FIG. 18 and (a) of FIG. 18 is that the cloud server 183d is deployed with a similar song recommendation service 184d. The similar song recommendation service 184d includes a music representation module 185d and a recommendation module 186d. The recommendation module 186d acquires the representation vector of the audio file from the music representation module 185d, then uses the representation vector for identification, and retrieves the song file with the highest similarity to the audio file currently being listened to from the song library.

[0306] The similar song recommendation module 186d can use any existing audio identification device, mainly relying on the audio identification method to identify the song. For example, using the fingerprint feature technology, the fingerprint feature of the audio to be identified is determined, and the song matching the fingerprint feature of the audio to be identified is matched from the audio library as the song in the audio.

[0307] Specifically, the feature extraction of all songs in the song library can be performed in advance using the representation learning model to obtain the representation vectors of the songs and store them in the vector index library. The user accesses the music application 182d in the client 181d, the music application 182d detects the song being listened to by the user, and the music application 182d sends a request to the cloud server 183d through the client 181d to request the similar song recommendation service in the cloud server 183d. After receiving the request, the server calls the interface of the similar song recommendation service 184d to perform the similar song recommendation operation. The recommendation module 186d obtains the fused representation vectors of the audio segments, and then uses a k-nearest neighbor search engine to retrieve similar songs (for example, the top 100 songs with the highest similarity) from the song library according to the cosine similarity between the vectors to obtain a retrieval result. Then, according to the retrieval result, the song language tags, versions (whether studio versions), and style tags are used for secondary filtering to retain songs with consistent language and style and high quality, and finally the top 30 song lists with the highest similarity are selected for return.

[0308] It can be understood that the deployment of each module of the representation learning model can be based on actual conditions, for example, in the song recognition scenario, the acquisition module or the acquisition module and the encoder can be deployed to the client.

[0309] Please refer to Figure 19 , Figure 19 A structural schematic diagram of a training device provided by an embodiment of the present application.

[0310] The training device 190 includes an acquisition module 191, an encoder 192, an interaction module 193, a construction module 194, and an optimization module 195.

[0311] The acquisition module 191 is configured to acquire a data set of data to be learned, wherein the data set includes at least one sub-set, the sub-set includes K different scales of data segments, K is an integer greater than or equal to 2, and in the sub-set, the i-th scale data segment is a part of the (i+1)-th scale data segment, i is less than or equal to K-1; the number of data segments of each scale is one or more.

[0312] The encoder 192 is configured to receive the data set and perform feature extraction on the data segments based on parameters of the encoder to obtain representation vectors corresponding to the data segments of each scale, respectively.

[0313] The interaction module 193 is configured to receive the representation vectors and perform information interaction on the representation vectors corresponding to the data segments of adjacent scales in the sub-set based on parameters of the interaction module to obtain fused representation vectors corresponding to the data segments of each scale, respectively.

[0314] The construction module 194 is configured to construct a target function according to the fused representation vector.

[0315] The optimization module 195 is configured to optimize the target function to adjust parameters of the encoder and parameters of the interaction module, so that the encoder and the interaction module learn the representation vector of the to-be-learned data.

[0316] It should be noted that the information interaction and execution process between the modules / units in the training device are the same as those of the training device in the embodiments of the present application Figures 1 to 12 The corresponding method embodiments are based on the same concept, and specific contents can be referred to the description of the method embodiments in the foregoing embodiments of the present application, which will not be described here.

[0317] Please refer to Figure 20 , Figure 20 A structural schematic diagram of the training device provided in the embodiments of the present application.

[0318] The training device 2000 can be deployed with Figure 19 The training device 190 described in the corresponding embodiments is used to implement Figure 19 the functions of the corresponding training device or Figures 1 to 12 the corresponding method embodiments. Specifically, the training device 2000 is implemented by one or more servers, and the training device 2000 can have great differences due to different configurations or performances, and can include one or more central processing units (CPUs) 2022 (for example, one or more processors) and memories 2032, one or more storage media 2030 (for example, one or more mass storage devices) storing application programs 2042 or data 2044. Among them, the memory 2032 and the storage medium 2030 can be temporary storage or persistent storage. The programs stored in the storage medium 2030 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the training device. Further, the central processing unit 2022 can be configured to communicate with the storage medium 2030 and execute a series of instruction operations in the storage medium 2030 on the training device 2000.

[0319] The training device 2000 can also include one or more power supplies 2026, one or more wired or wireless network interfaces 2050, one or more input / output interfaces 2058, and / or one or more operating systems 2041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0320] In one embodiment of this application, the central processing unit 2022 is used to implement... Figure 19 The function of the training device in the corresponding embodiment or Figures 1 to 12 The corresponding method implementations are as follows. It should be noted that this applies to CPU 2022 execution. Figures 1 to 12 For details on the specific implementation of the training device's functions and the beneficial effects described in the corresponding embodiments, please refer to [the relevant documentation / reference]. Figures 11 to 12 The descriptions in the corresponding method embodiments will not be repeated here.

[0321] In another embodiment of this application, the central processing unit 2022 is used to implement... Figure 19 The corresponding embodiment describes the function of the training device. It should be noted that the central processing unit 2022 executes... Figure 19 For details on the specific implementation of the training device's functions and the beneficial effects described in the corresponding embodiments, please refer to [the relevant documentation / reference]. Figure 19 The descriptions in the corresponding method embodiments will not be repeated here.

[0322] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the above-described actions. Figures 1 to 12 The steps performed by the training device in the corresponding embodiment, or the steps described above. Figures 13 to 15 The steps performed by the execution device in the corresponding embodiment, or the steps described above, are executed as follows. Figure 16 The steps performed by the execution device in the corresponding embodiment, or the steps described above, are executed as follows. Figure 17 The steps performed by the training device in the corresponding embodiment.

[0323] This application also provides a computer program product that, when run on a computer, causes the computer to perform the above-described actions. Figures 1 to 12 The steps performed by the training device in the corresponding embodiment, or the steps described above. Figures 13 to 15 The steps performed by the execution device in the corresponding embodiment, or the steps described above, are executed as follows. Figure 16 The steps performed by the execution device in the corresponding embodiment, or the steps described above, are executed as follows. Figure 17 The steps performed by the training device in the corresponding embodiment.

Claims

1. A representation learning method, characterized by, The method comprises: acquiring a data set of to-be-learned data, wherein the data set comprises at least one sub-set, the sub-set comprises data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is a part of the data segment of the (i+1)-th scale, i is less than or equal to K-1; the number of data segments of each scale is one or more; the to-be-learned data is audio data, text data or video data; inputting the data set into an encoder, and performing feature extraction on the data segments based on parameters of the encoder to obtain representation vectors corresponding to the data segments of various scales respectively; inputting the representation vectors into an interaction module, and performing information interaction between a fused representation vector of the data segment of the i-th scale output by the interaction module and a representation vector of the data segment of the (i+1)-th scale output by the encoder based on parameters of the interaction module to obtain fused representation vectors corresponding to the data segments of various scales respectively, wherein the fused representation vector of the data segment of the i-th scale output by the interaction module is a representation vector obtained after information interaction in the (n-1)-th time, wherein n=i, and n is an integer greater than or equal to 1; constructing an objective function according to the fused representation vectors; optimizing the objective function to adjust the parameters of the encoder and the parameters of the interaction module, so that the encoder and the interaction module learn the representation vectors of the to-be-learned data.

2. The method of claim 1, wherein, The information interaction between the fused representation vector of the data segment of the i-th scale output by the interaction module and the representation vector of the data segment of the (i+1)-th scale output by the encoder based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively comprises: determining that the scales to be processed in the n-th time are the i-th scale and the (i+1)-th scale, assigning an initial value of n as 1, and performing the following operations in a loop until n=K-1; performing information interaction between the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the sub-set in the n-th time based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

3. The method of claim 2, wherein, When n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after information interaction in the (n-1)-th time.

4. The method of claim 2 or 3, wherein, The information interaction between the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the n-th time based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively comprises: taking the representation vector corresponding to the data segment of the i-th scale as a first representation vector, and taking the representation vector corresponding to the data segment of the (i+1)-th scale as a second representation vector; mapping the first representation vector into a first query vector, and mapping the second representation vector into a first key vector and a first value vector respectively based on parameters of the interaction module; calculating an attention score between the first query vector and the first key vector through an attention mechanism to obtain a first attention score; obtaining a fused representation vector corresponding to the i-th scale data segment according to the first value vector, the first attention score and the first representation vector; mapping the second representation vector into a second query vector, and mapping the fused representation vector corresponding to the i-th scale data segment into a second key vector and a second value vector respectively based on parameters of the interaction module; calculating an attention score between the second query vector and the second key vector through the attention mechanism to obtain a second attention score; obtaining a fused representation vector corresponding to the i+1-th scale data segment according to the second value vector, the second attention score and the second representation vector.

5. The method according to any one of claims 1 to 3, wherein The method comprises: obtaining a data set of to-be-learned data; segmenting the to-be-learned data to obtain data segments; segmenting the data segment with the smallest scale until the number of scale categories in the sub-set reaches the preset number of scale categories.

6. The method according to any one of claims 1 to 3, wherein The method comprises: obtaining an overall representation vector of other to-be-learned data, wherein the other to-be-learned data and the to-be-learned data are in the same batch, and the overall representation vector is a representation vector corresponding to a data segment with the largest scale in the other to-be-learned data; calculating the similarity between each fused representation vector and the overall representation vector; for each scale, filtering out the fused representation vector corresponding to the lowest similarity; constructing a target function according to the filtered fused representation vector.

7. The method of claim 6, wherein, The method comprises: mapping the filtered fused representation vector as a positive sample; mapping the overall representation vector corresponding to the other to-be-learned data in the same batch as a negative sample; based on contrast learning, taking the positive sample and the negative sample as sub-elements of a target function to construct the target function.

8. The method of claim 7, wherein, The method comprises: taking the positive samples corresponding to two scales adjacent in scale as a positive sample pair; calculating a first similarity between the positive sample corresponding to the largest scale and the negative sample, and a second similarity of the positive sample pair through the target function; optimizing the target function according to the first similarity and the second similarity to update the parameters of the encoder and the parameters of the interaction module.

9. A representation learning method, characterized by, The method comprises: obtaining a data set of to-be-learned data, wherein the data set comprises at least one sub-set, the sub-set comprises data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is a part of the data segment of the (i+1)-th scale, i is less than or equal to K-1, the number of data segments of each scale is one or more, and the to-be-learned data is audio data, text data or video data; inputting the data set into a trained encoder, and performing feature extraction on the data segments based on parameters of the encoder to obtain representation vectors corresponding to the data segments of various scales respectively; inputting the representation vectors into a trained interaction module, and performing information interaction between a fused representation vector of the data segment of the i-th scale output by the interaction module and a representation vector of the data segment of the (i+1)-th scale output by the encoder based on parameters of the interaction module to obtain fused representation vectors corresponding to the data segments of various scales respectively, wherein the fused representation vector of the data segment of the i-th scale output by the interaction module is a representation vector obtained after information interaction in the (n-1)-th time, wherein n is an integer greater than or equal to 1, and n=i; and learning the representation vectors of the to-be-learned data is completed.

10. The method of claim 9, wherein, The information interaction between the fused representation vector of the data segment of the i-th scale output by the interaction module and the representation vector of the data segment of the (i+1)-th scale output by the encoder based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively comprises: determining that the scales to be processed in the n-th time are the i-th scale and the (i+1)-th scale, assigning an initial value of n as 1, and performing the following operations in a loop until n=K-1; performing information interaction between the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the sub-set in the n-th time based on the parameters of the trained interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

11. The method of claim 10, wherein, When n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after information interaction in the (n-1)-th time.

12. The method of claim 10 or 11, wherein, The information interaction between the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the n-th time based on the parameters of the trained interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively comprises: taking the representation vector corresponding to the data segment of the i-th scale as a first representation vector and the representation vector corresponding to the data segment of the (i+1)-th scale as a second representation vector; mapping the first representation vector into a first query vector and mapping the second representation vector into a first key value vector and a first value vector based on the parameters of the trained interaction module; and performing information interaction between the first query vector and the first key value vector based on the parameters of the trained interaction module to obtain a first value vector corresponding to the data segment of the i-th scale. calculating, by the attention mechanism, an attention score of the first query vector and the first key-value vector, to obtain a first attention score; obtaining a fused representation vector corresponding to the data segment of the i-th scale according to the first value vector, the first attention score and the first representation vector; mapping, based on the parameters of the trained interaction module, the second representation vector into a second query vector, and mapping the fused representation vector corresponding to the data segment of the i-th scale into a second key-value vector and a second value vector respectively; calculating, by the attention mechanism, an attention score of the second query vector and the second key-value vector, to obtain a second attention score; obtaining a fused representation vector corresponding to the data segment of the i+1-th scale according to the second value vector, the second attention score and the second representation vector.

13. The method of any one of claims 9 to 11, wherein, The data set of the data to be learned includes: acquiring the data to be learned and a preset scale category number; segmenting the data to be learned to obtain data segments; segmenting the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

14. A training device, characterized by The training device includes: an acquisition module, configured to acquire a data set of data to be learned, wherein the data set includes at least one sub-set, the sub-set includes data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segment of the i-th scale is part of the data segment of the i+1-th scale, i is less than or equal to K-1; the number of data segments of each scale is one or more; the data to be learned is audio data, text data or video data; an encoder, configured to receive the data set, and perform feature extraction on the data segments based on parameters of the encoder, to obtain representation vectors corresponding to data segments of various scales respectively; an interaction module, configured to receive the representation vectors, and perform information interaction between a fused representation vector of the data segment of the i-th scale output by the interaction module and a representation vector of the data segment of the i+1-th scale output by the encoder based on parameters of the interaction module, to obtain fused representation vectors corresponding to data segments of various scales respectively, wherein the fused representation vector of the data segment of the i-th scale output by the interaction module is a representation vector obtained after information interaction of the data segment of the i-th scale in the n-1th time, wherein n=i, and n is an integer greater than or equal to 1; a construction module, configured to construct an objective function according to the fused representation vectors; an optimization module, configured to optimize the objective function, to adjust the parameters of the encoder and the parameters of the interaction module, so that the encoder and the interaction module learn the representation vectors of the data to be learned.

15. The apparatus of claim 14 wherein, The interaction module is specifically configured to: receive the representation vectors; determine that the scales to be processed in the n-th time are the i-th scale and the i+1-th scale, and assign an initial value of n as 1, and perform the following operations in a loop until n=K-1; Based on the parameters of the interaction module, the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set are interacted for the n-th time, and the fused representation vectors corresponding to the data segments of various scales are obtained respectively.

16. The apparatus of claim 15 wherein, When n is an integer greater than or equal to 2, the representation vector corresponding to the data segment of the i-th scale is the fused representation vector obtained after the data segment of the i-th scale is interacted for the n-1-th time.

17. The apparatus of claim 15 or 16, wherein, The specific process of the step of based on the parameters of the interaction module, the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set are interacted for the n-th time, and the fused representation vectors corresponding to the data segments of various scales are obtained respectively includes: The representation vector corresponding to the data segment of the i-th scale is taken as a first representation vector, and the representation vector corresponding to the data segment of the i+1-th scale is taken as a second representation vector; Based on the parameters of the interaction module, the first representation vector is mapped into a first query vector, and the second representation vector is mapped into a first key-value vector and a first value vector respectively; The attention score of the first query vector and the first key-value vector is calculated through the attention mechanism, and a first attention score is obtained; The fused representation vector corresponding to the data segment of the i-th scale is obtained according to the first value vector, the first attention score and the first representation vector; Based on the parameters of the interaction module, the second representation vector is mapped into a second query vector, and the fused representation vector corresponding to the data segment of the i-th scale is mapped into a second key-value vector and a second value vector respectively; The attention score of the second query vector and the second key-value vector is calculated through the attention mechanism, and a second attention score is obtained; The fused representation vector corresponding to the data segment of the i+1-th scale is obtained according to the second value vector, the second attention score and the second representation vector.

18. The apparatus of any one of claims 14 to 16, wherein, The acquisition module is specifically configured to: acquire the to-be-learned data and a preset scale category number; segment the to-be-learned data to obtain data segments; segment the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

19. The apparatus of any one of claims 14 to 16, wherein, The construction module is specifically configured to: acquire an overall representation vector of other to-be-learned data, wherein the other to-be-learned data and the to-be-learned data are in the same batch, and the overall representation vector is a representation vector corresponding to a data segment with the largest scale in the other to-be-learned data; calculate the similarity between each fused representation vector and the overall representation vector; for each scale, filter out the fused representation vector corresponding to the lowest similarity; construct a target function according to the filtered fused representation vector.

20. The apparatus of claim 19 wherein, The step of constructing a target function according to the filtered fused representation vector includes: mapping the filtered fused representation vector as a positive sample; mapping the overall representation vector corresponding to the other to-be-learned data in the same batch as a negative sample; Based on contrastive learning, the positive sample and the negative sample are taken as sub-elements of a target function to construct the target function.

21. The apparatus of claim 20 wherein, The optimization module is specifically configured to: take the positive samples corresponding to two scales adjacent in scale as a positive sample pair; calculate, by the target function, a first similarity between the positive sample and the negative sample corresponding to the largest scale and a second similarity of the positive sample pair; optimize the target function according to the first similarity and the second similarity to update the parameters of the encoder and the parameters of the interaction module.

22. An execution device, comprising: The execution device comprises: an acquisition module configured to acquire a data set of to-be-learned data, wherein the data set comprises at least one sub-set, the sub-set comprises data segments of K scales, K is an integer greater than or equal to 2, in the sub-set, the data segments of the i-th scale are part of the data segments of the (i+1)-th scale, i is less than or equal to K-1, the number of data segments of each scale is one or more, and the to-be-learned data is audio data, text data or video data; an encoder configured to receive the data set and perform feature extraction on the data segments based on parameters of the encoder to obtain representation vectors corresponding to the data segments of various scales respectively, wherein the encoder is trained; an interaction module configured to receive the representation vectors and perform information interaction on the representation vectors corresponding to the data segments of scales adjacent in scale in the sub-set based on parameters of the interaction module to obtain fused representation vectors corresponding to the data segments of various scales respectively, including performing information interaction on the fused representation vectors of the data segments of the i-th scale output by the interaction module and the representation vectors of the data segments of the (i+1)-th scale output by the encoder based on the parameters of the interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively, wherein the fused representation vectors of the data segments of the i-th scale output by the interaction module are the representation vectors obtained after information interaction of the data segments of the i-th scale in the (n-1)-th time, wherein n=i, n is an integer greater than or equal to 1, and the interaction module is trained.

23. The apparatus of claim 22 wherein, The interaction module is specifically configured to: receive the representation vectors; determine that the scales to be processed in the n-th time are the i-th scale and the (i+1)-th scale, and assign an initial value of 1 to n, and perform the following operations in a loop until n=K-1; perform information interaction on the representation vectors corresponding to the data segments of the i-th scale and the (i+1)-th scale in the sub-set in the n-th time based on the parameters of the trained interaction module to obtain the fused representation vectors corresponding to the data segments of various scales respectively.

24. The apparatus of claim 23 wherein, When n is an integer greater than or equal to 2, the representation vectors corresponding to the data segments of the i-th scale are the fused representation vectors obtained after information interaction of the data segments of the i-th scale in the (n-1)-th time.

25. The apparatus of claim 23 or 24, wherein, The parameter of the trained interaction module is used to perform information interaction between the representation vectors corresponding to the data segments of the i-th scale and the i+1-th scale in the sub-set for the n-th time, respectively obtaining the fused representation vectors corresponding to the data segments of various scales, which comprises: The representation vector corresponding to the data segment of the i-th scale is taken as a first representation vector, and the representation vector corresponding to the data segment of the i+1-th scale is taken as a second representation vector; The first representation vector is mapped into a first query vector based on the parameter of the trained interaction module, and the second representation vector is respectively mapped into a first key-value vector and a first value vector; The attention score of the first query vector and the first key-value vector is calculated through an attention mechanism, and a first attention score is obtained; The fused representation vector corresponding to the data segment of the i-th scale is obtained according to the first value vector, the first attention score and the first representation vector; The second representation vector is mapped into a second query vector based on the parameter of the trained interaction module, and the fused representation vector corresponding to the data segment of the i-th scale is respectively mapped into a second key-value vector and a second value vector; The attention score of the second query vector and the second key-value vector is calculated through the attention mechanism, and a second attention score is obtained; The fused representation vector corresponding to the data segment of the i+1-th scale is obtained according to the second value vector, the second attention score and the second representation vector.

26. The apparatus of any one of claims 22 to 24, wherein, The acquisition module comprises: Acquiring to-be-learned data and a preset scale category number; Segmenting the to-be-learned data to obtain data segments; Segmenting the data segment with the smallest scale until the scale category number in the sub-set reaches the preset scale category number.

27. A computer device, comprising: A memory and a processor are included, wherein the memory is used to store computer readable instructions; the processor is used to read the computer readable instructions and implement the method of any one of claims 1 to 8 or the method of any one of claims 9 to 13.

28. A computer storage medium, comprising, Computer readable instructions are stored, and the computer readable instructions implement the method of any one of claims 1 to 8 or the method of any one of claims 9 to 13 when executed by a processor.

29. A computer program product, characterised in that, The computer program product includes computer readable instructions, which implement the method of any one of claims 1 to 8 or the method of any one of claims 9 to 13 when executed by a processor.

Images