Media retrieval with semi-supervised contrastive learning

Abstract

Methods and apparatuses for media retrieval are disclosed. In some examples, a media retrieval method includes receiving a set of input media data of a first modality and obtaining, with a first modality encoder, a first embedding representing the set of input media data in a latent space. The method further includes selecting a subset of second embeddings based on a similarity measure configured to quantify pairwise similarities between different embeddings in the latent space. Each of the second embeddings represents a respective candidate set of media data of a different second modality in the latent space. The second embeddings are obtained using a second modality encoder. The method further includes identifying one or more sets of output media data of the different second modality in the respective candidate sets of media data based on the selected subset of second embeddings. The first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

Description

[0001] 1. Cross-references to related applications This application claims the benefit of Indian Provisional Patent Application No. 202411029608, filed on April 12, 2024, entitled “A SYSTEM FOR CONTROLLABLEMUSIC-VIDEO RETRIEVAL USING SEMI-SUPERVISED CONTRASTIVE LEARNING”. 2. Technical Field Various example implementations relate to media retrieval, and more specifically, but not exclusively, to using machine learning methods to find suitable music for a given video or suitable video for a given music. 3. Background Technology The synergy between visuals and music is crucial for compelling storytelling. However, finding music that matches the style / genre / emotion of a video, or vice versa, can be challenging and time-consuming. Therefore, a system that can automatically match music to video (and vice versa) is desired. Summary of the Invention

[0004] Various embodiments provide a media retrieval system that can automatically match music with videos and vice versa. In some examples, the media retrieval system includes an audio encoder and a video encoder, which are jointly trained using semi-supervised contrastive learning and further using a modal symmetric contrastive loss function. In operation, the media retrieval system projects an input media dataset (e.g., video) into a latent space, identifies the latent vector in a database that is most similar to the projection, and retrieves an output media dataset (e.g., audio) corresponding to the identified latent vector from the media library. In some examples, users of the media retrieval system can customize the media retrieval process to ensure that the media retrieval process is well-suited to their specific creative goals and optimize the results based on a selected balance between self-supervised learning components and label-supervised learning components based on information obtained via semi-supervised contrastive learning.

[0005] In one example, an apparatus for media retrieval includes: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured, together with the at least one processor, to cause the apparatus to at least: receive an input media dataset of a first modality; obtain a first embedding representing the input media dataset in a latent space using a first modality encoder; select a subset of second embeddings based on a similarity metric configured to quantify pairwise similarity between different embeddings in the latent space, each of the second embeddings representing a corresponding candidate media dataset of a different second modality in the latent space, wherein the second embeddings are obtained using a second modality encoder; and identify one or more output media datasets of the different second modality in the corresponding candidate media datasets based on the selected subset of the second embeddings, wherein the first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

[0006] In another example, a media retrieval method includes: receiving an input media dataset of a first modality; obtaining a first embedding representing the input media dataset in a latent space using a first modality encoder; selecting a subset of second embeddings based on a similarity metric configured to quantify pairwise similarity between different embeddings in the latent space, each of the second embeddings representing a corresponding candidate media dataset of a different second modality in the latent space, wherein the second embeddings are obtained using a second modality encoder; and identifying one or more output media datasets of the different second modality in the corresponding candidate media datasets based on the selected subset of the second embeddings, wherein the first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

[0007] According to yet another example, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the methods described above. Attached Figure Description

[0008] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent by way of example, in conjunction with the following detailed description and accompanying drawings, in which: Figures 1A to 1B This is a block diagram illustrating the functionality of music-video retrieval based on some examples.

[0009] Figure 2 This is a block diagram illustrating the training process of a music-video retrieval system based on some examples.

[0010] Figure 3This is a block diagram illustrating the process of generating a music embedding database for video-to-music retrieval, based on some examples.

[0011] Figure 4 The illustration shows, based on some examples, the configuration for using... Figure 3 The flowchart illustrates the process of embedding music generated during the process of retrieval into the database and then into the music retrieval process.

[0012] Figure 5 This is a block diagram illustrating the process of generating a video embedding database for music-to-video retrieval, based on some examples.

[0013] Figure 6 The illustration shows, based on some examples, the configuration for using... Figure 5 The flowchart illustrates the process of embedding music from the database into the generated video and then into the video retrieval process.

[0014] Figure 7 This is a flowchart illustrating media retrieval methods based on some examples.

[0015] Figure 8 It is a block diagram illustrating at least some of the computing devices used to implement the above-described systems, methods, and processes, according to some examples. Detailed Implementation

[0016] Content creators can use music to enhance their videos, from film scores to background music in vlogs and social media content. In some examples, the process of choosing music for a given video can be conceptualized as including the following steps: (1) the content creator decides which music best matches their video, and (2) the content creator searches for music that matches their decided preferences and expectations. For step (1), the content creator typically needs a deep understanding of music (genre, style, etc.) and some expertise and / or talent in identifying which music might be suitable for various types of visual content. Step (2) involves searching for music that matches the decision made in step (1).

[0017] In some examples, music search engines operate by matching language queries against music metadata. Metadata typically includes relevant details such as artist names, album titles, and / or track names. In some additional examples, music search engines may allow searches based on certain musical characteristics, such as genre or mood. However, such engines may lack support for highly specific and / or rather, fairly general queries. Music search engine users may often be limited to using a predetermined set of descriptors, such as "jazz" (genre) and "happy" (mood), rather than more nuanced musical descriptors, such as "a cheerful and lively Latin jazz track featuring saxophone and bass." In some additional examples, music retrieval systems may tend to prioritize metadata over the auditory attributes of the music itself. Therefore, even if content creators know fairly precisely what kind of music they expect for a given video, finding the appropriate music using the aforementioned search tools can still be challenging and / or time-consuming. Furthermore, when search results are returned, content creators may still need to carefully listen to a relatively large number of retrieved tracks to identify the best match.

[0018] At least some of the problems described above in the prior art can be advantageously addressed using the various embodiments disclosed herein. For example, one embodiment provides a system capable of automatically recommending matching music for a given video or recommending matching videos for a given music. In this document, these processes are collectively referred to as the term "music-video retrieval," which should be interpreted as encompassing both video-to-music retrieval use cases and music-to-video retrieval use cases. In some examples, a music-video retrieval system can be viewed as effectively combining steps (1) and (2) above into a single step, as perceived by the user. That is, the music-video retrieval system internally determines which music best matches a given video, then finds and presents one or more recommended music tracks to the user, which in some cases are listed in ranking order.

[0019] Figures 1A to 1B This is a block diagram illustrating the functionality of music-video retrieval based on some examples. More specifically, Figure 1A The illustration shows the video-to-music search function 110 in the music-video search functionality. Figure 1B Similarly, the music-video retrieval function 120 in the music-video retrieval functionality is illustrated. In some examples, retrieval functions 110 and 120 are both implemented as different modalities of the same music-video retrieval unit.

[0020] like Figure 1AAs shown, the video-to-music retrieval function 110 operates to retrieve a music set 112 from a music track set 104 in response to a video query 102. In some examples, set 112 includes multiple music tracks ordered according to their estimated relevance to the video input to the video-to-music retrieval function 110 as part of the video query 102. In some examples, set 104 includes a relatively large number of music tracks pre-mapped to the feature space used by the video-to-music retrieval function 110.

[0021] like Figure 1B As shown, the music-to-video retrieval function 120 operates to retrieve a video set 122 from a video track set 108 in response to a music query 106. In some examples, set 122 includes multiple videos sorted in order of their estimated relevance to the music tracks input into the music-to-video retrieval function 120 as part of the music query 106. In some examples, set 108 includes a relatively large number of video clips pre-mapped to the feature space used by the music-to-video retrieval function 120.

[0022] In some examples, retrieval functions 110 and 120 are implemented using a music-video retrieval framework based on both artistic correspondences and temporal alignment between music tracks and video tracks. In one example, this music-video retrieval framework is based on a large-scale self-supervised learning method that does not rely on human annotation and relies primarily on the intrinsic correspondences between musical and visual elements in the music video. However, in some examples, the music-video retrieval framework can benefit from human annotations, which can sometimes provide useful high-level information that might not be available from a purely self-supervised objective. The music-video retrieval framework can also benefit from the ability to impose explicit user-defined control over the retrieval process.

[0023] At least some of the problems described above can be advantageously solved using the various embodiments disclosed herein. At least some of the disclosed embodiments have some or all of the following features: 1. Semi-supervised contrastive learning for music videos.

[0024] 1.1. In some examples, music-video retrieval systems use a multi-task learning approach to combine both self-supervised and supervised training objectives.

[0025] 1.1.1. For self-supervised learning, take advantage of the natural co-occurrence between the music audio and video in music videos.

[0026] 1.1.2. For supervised learning, manually labeled music tags are used.

[0027] 1.2. Self-supervised contrastive learning and supervised contrastive learning are used to learn the joint embedding space between music audio and video.

[0028] 1.2.1. The contrastive loss function is modally symmetric, in which the directional audio-to-video loss component and the video-to-audio loss component are added with equal weights.

[0029] 2. A method that allows for explicit user control over the retrieval process.

[0030] 2.1. A model architecture that enables fine-grained control over the degree of attention paid to self-supervised information and label information during the retrieval process during inference.

[0031] 2.2. This approach is considered advantageous because the model only needs to be trained once, but the retrieval process can be supported with various configurations through user-defined inputs.

[0032] Figure 2 This is a block diagram illustrating the training process of a music-video retrieval system 200 based on some examples. For example... Figure 2 As shown, system 200 has a dual-branch architecture for separate processing of audio and video. The audio branch of system 200 includes an audio encoder 210. The video branch of system 200 includes a video encoder 250. The audio encoder 210 and video encoder 250 are jointly trained using multiple training video clips 202, each training video clip including its own audio track 204 and its own video track 206. The audio track 204 and video track 206, after being extracted from the video clips 202, are fed into the audio encoder 210 and video encoder 250, respectively, as shown. Figure 2 As shown. Multiple loss functions, calculated based on various corresponding embeddings generated in the audio encoder 210 and the video encoder 250, are used to adjust the parameters of various trainable elements of the system 200 (e.g., as described in more detail below) until the training stopping criterion is met.

[0033] Given the audio track 204 of the selected video clip 202 And video track 206 Pre-trained audio representation model 212 and pre-trained video representation model 252 Perform separate operations to extract the audio fundamental feature vector 214 and video basic feature vector 254 Each audio track 204 is represented by a single corresponding basic feature vector 214 obtained by temporal aggregation of the corresponding sequence features. Similarly, each video track 206 is represented by a single corresponding basic feature vector 254 obtained by temporal aggregation of the corresponding sequence features.

[0034] In some examples, the pre-trained audio representation model 212 is the MERT model. The abbreviation "MERT" stands for "Music Understanding Model with Large-Scale Self-Supervised Training". The MERT model is an open-source music audio representation model with a transformer architecture, which is described in more detail, for example, in the following publication: Yizhi Li, Ruibin Yuan, Ge Zhang, et al., "MERT: Acousic Music Understanding Model with Large-Scale Self-Supervised Training", arXiv:2306.00107v5, which is incorporated herein by reference in its entirety. MERT is pre-trained through large-scale self-supervised learning and produces music representations that demonstrate state-of-the-art performance on a wide range of music understanding tasks. In other examples, other pre-trained audio representation models can be similarly used to implement model 212.

[0035] In some examples, model 212 is configured to ingest 24 kHz raw audio and output seventy-five 1024-dimensional feature vectors per second. To generate the fundamental audio feature vector 214, model 212 operates to extract the output of twenty-five transformer encoder layers, resulting in features of shape (8, 1024). In this way, model 212 generates a feature sequence of shape (75, 8, 1024) for each second of audio. Model 212 then operates to perform a global temporal average on the entire feature sequence, resulting in a tensor of shape (8, 1024) for each audio track 204. Model 212 further operates to compute a learnable weighted average across the transformer layer dimensions, resulting in a 1024-dimensional feature representation 214. Figure 2 During the training process illustrated, model 212 is frozen (remains locked / unchanged), except for its learnable weighted average component.

[0036] In some examples, the pre-trained video representation model 252 is the visual component (ViT-B / 32 architecture) of a contrastive language-image pre-trained (CLIP) model. The CLIP model is described, for example, in: A. Radford, JW Kim, C. Hallacy et al., “Learning transferable visual models from natural language supervision,” 38th International Conference on Machine Learning (ICML), 2021, which is incorporated herein by reference in its entirety. Model 252 operates by extracting CLIP image features for each video frame and then temporally averaging the extracted features to produce a single 512-dimensional feature vector 254 for each video. In other examples, other pre-trained video representation models can be similarly used to implement model 252. Figure 2 During the training process illustrated, model 252 is frozen.

[0037] In the following description, symbols are used. To represent the audio of system 200 ( Branches and videos Both branches. Those skilled in the art will readily understand how the provided description can be applied to the relevant components of the audio encoder 210 and the video encoder 250.

[0038] Each of the extracted basic features 214 and 254 is processed by the corresponding basic network in basic networks 216 and 256. One objective of the base networks 216 and 256 in the corresponding processing path is to learn a general audio / video representation shared between downstream self-supervised and supervised tasks. Subsequently, the outputs 218 and 258 of the base networks 216 and 256 are passed through corresponding task-specific head networks. and More specifically, the audio encoder 210 includes a head network 220 connected to receive the output 218 of the base network 216. ) and 230 ( The video encoder 250 similarly includes a header network 260 connected to receive the output 258 of the base network 256. ) and 270 ( Head networks 220, 230, 260, and 270 are used to learn embeddings specific to self-supervised and supervised tasks, respectively. and Self-supervised cross-modal contrastive loss In the embedded { and On the operation. Supervised cross-modal contrastive loss. In the embedded { and Operations on}. In this way, They are encouraged to express self-monitoring information, and Encouraged to express supervisory information. Compare the loss functions. and The relevant details will be described below. Based on the corresponding inputs in inputs 218 and 258, head networks 220, 230, 260 and 270 generate task-specific embeddings 222, respectively. ), 232 ( ), 262 ( ), 272 ( ).

[0039] Task-Specific Projection Networks and Used to embed task-specific elements in 222, 232, 262, and 272 ( and Projections can be combined into different embedding spaces. More specifically, projection network 224 ( ) in task-specific embedding 222 ( The operation on ) is performed to generate the corresponding projection 226 ( Projection network 234 ( ) embedding 232 in a task-specific manner The operation on ) is performed to generate the corresponding projection 236 ( Projection network 264 ( ) in task-specific embedding 262 ( The operation on ) is performed to generate the corresponding projection 266 ( Projection network 274 ( ) in task-specific embedding 272 ( The operation on ) generates the corresponding projection 276 ( ).

[0040] Projections 226, 236, 266, and 276 are weighted and linearly combined in pairs to generate the corresponding combined projection 240. ) and 280 ( For example, as follows: (1) in, It's a combination of weights. In some examples, The value is the same in both the audio and video branches of System 200. In some other examples, the audio branch is configured to use the first combined weight. Furthermore, the video branches are configured to use different second combined weights. During the training process, This is a hyperparameter value that can be set to any selected value within the range of 0 to 1. In some examples, it can be used... = 0.5. Used during inference (and retrieval). The (multiple) values ​​may not necessarily be the same as those used during training, as will be explained in more detail below.

[0041] In some examples, System 200 is trained using cross-modal semi-supervised contrastive learning between audio and video. As used herein, the term "semi-supervised contrastive learning" refers to a combination of self-supervised contrastive learning and supervised (e.g., labeled supervised) contrastive learning. Self-supervised contrastive learning (SSCL) is a machine learning technique that uses unlabeled data to train a model to distinguish between similar and dissimilar data instances (without relying on explicit labels) to learn meaningful representations. SSCL falls under the category of self-supervised learning, where the model learns from the data itself by creating its own "supervisory signal" or "excuse task." Contrastive learning typically aims to train machine learning (ML) models to bring similar data instances closer together in the latent space and dissimilar data instances further apart. Supervised contrastive learning (SupCon) extends self-supervised contrastive learning to a fully supervised setting, using labels to guide the model in learning representations that bring similar data instances closer together and push dissimilar data instances further apart in the latent space.

[0042] The following symbols will be used in this article: Representing modes samples Embedding; Representing modes samples Tags; This represents all indices in the training batch; and This indicates the temperature hyperparameter.

[0043] For SSCL, a cross-modal version of the self-supervised InfoNCE loss is used. Generally, the Information-Noise Contrast Estimation (InfoNCE) loss maximizes consistency between positive samples and minimizes consistency between negative samples in the learned representation space. The cross-modal version of the self-supervised InfoNCE loss can be described as follows. Given a batch Each music video is broken down into corresponding audio files. Track 204 and video ( Track 206) Calculate two different cross-modal self-supervised InfoNCE losses: audio-to-video loss. and video to audio loss In one example, audio-to-video loss The expression is as follows: (2) Similar expressions can be used for video-to-audio loss. Audio-to-video loss and video to audio loss The operation "pulls in" audio and video embeddings from the same music video and "pulls out" audio and video embeddings from different music videos. To create a modal-symmetric loss function, the following is used: and Combine with equal weights: (3) For supervised contrastive learning, the SupCon loss is adapted to cross-modal settings as follows. Given a batch A label Music audio tracks and A label The video was used to calculate two different cross-modal SupCon losses: audio-to-video loss. and video to audio loss In one example, audio-to-video loss The expression is as follows: (4) in, Anchor point sample Positive samples The set of indices is defined as follows: (5) Similar expressions can be used for video-to-audio loss. The cross-modal SupCon loss operation "brings" audio and video embeddings with the same label closer together and "pulls" audio and video embeddings with different labels further apart. To maintain the modality-symmetric loss function, [the following is used:] and Combine with equal weights: (6) like Figure 2 As shown, two different loss functions and Embedded in task-specific and Above operation: (7a) (7b) therefore, Trained to primarily contain self-supervised information, while It is trained to primarily contain supervisory information.

[0044] like Figure 2 As shown, two different loss functions and In combined projection (embedded) 240, 280 The above operations. In some examples, these loss functions are as follows: (8a) (8b) Therefore, system 200 is trained to make The group comes from Self-supervised information and from Supervisory information.

[0045] In some examples, system 200 is trained using a multi-task learning method designed to balance the aforementioned multiple tasks / objectives. Under this method, the total loss function to be optimized during the training process... It is constructed as a weighted sum of the various individual loss functions described above. In one example, the total loss function can be expressed as follows: (9) Among them, weight and Used to control the balance between emphasizing task-specific embeddings and combined embeddings; and weights and Used to control the balance between self-supervised training objectives and supervised training objectives.

[0046] After the audio encoder 210 and video encoder 250 have been trained as described above, encoders 210 and 250 can be used to generate video and audio (music) embeddings (represented as...). and This is for retrieval purposes. In some examples, users can set and / or change the parameters used in encoders 210 and 250. To controllably adjust the embedding vector 240 ( ) and 280 ( Emphasis on different types of information within a given context. For example, when... When set to 0, embedding and It emphasizes the self-supervised information learned by System 200 during the training process, thereby reducing the influence of labeled supervision content. On the other hand, when When set to 1, embed and It emphasizes the content supervised by labels while reducing the influence of self-supervised information. This is achieved by allowing selection within a range of 0 to 1. Any value of allows users to effectively fine-tune the balance between self-supervised and tag-supervised information during the retrieval process. This controllability endows the retrieval mechanism with a "customizable" attribute.

[0047] Figure 3 This is a block diagram illustrating process 300 for generating a music embedding database 308 for video-to-music retrieval, based on some examples. Process 300 employs methods already referenced above. Figure 2 The audio encoder 210 is trained as described. The resulting music embedding database 308 can be used for video-to-music retrieval, for example, as referenced below. Figure 4 More detailed description.

[0048] In some examples, process 300 includes selection. The value is then used as a configuration parameter for the audio encoder 210. Recall that, as mentioned earlier, The value controls the relative weighting between self-supervised information and label-supervised information in the embedding vector. It is used for selection in process 300. The various options for the value include, but are not limited to: (i) user selection; (ii) automatic or algorithmic selection; and (iii) default value, such as the same value used during the training process.

[0049] In some examples, the music embedding database 308 is generated as follows. Each music track 304 from the music library 302 is processed using an audio encoder 210 to generate a corresponding embedding (latent space vector) 306. Embedding 306 is similar to the one referenced above. Figure 2 The described projection 240 ( Then, the multiple embeddings 306, which correspond to the multiple music tracks 304 from the music library 302 and are generated in this way, are sorted and organized in a suitable searchable format to create a music embedding database 308.

[0050] Figure 4 This is a block diagram illustrating a video-to-music retrieval process 400 configured to use music embedding database 308, based on some examples. Process 400 employs methods already referenced above. Figure 2 The video encoder 250 is trained as described. In response to video query 402, process 400 generates the top-ranked music in the music library 302. K A list of 412 music tracks. In various examples, the quantity... K This is a parameter selected by the user, which can be set to, for example...K = 1, 5, or 10. Using list 412, the actual corresponding music track 304 can then be retrieved from the music library 302 (see also...). Figure 3 ).

[0051] Given the video track of video query 402, video encoder 250 operates to generate the corresponding video embedding 404. Video embedding 404 is similar to projection 280 (…). The sorting module 410 operates to compare the video embedding 404 with the music embeddings 306 in the music embedding database 308 using a suitable similarity metric, and calculates a similarity score between each of the video embedding 404 and the music embeddings 306. In some examples, the similarity metric used is the cosine similarity metric. In other examples, other suitable similarity metrics may also be used. Then, it will be compared with the previous... K The music tracks 304 corresponding to the most similar embeddings 306 are listed in list 412 in order of ranking and retrieved from music library 302.

[0052] Figure 5 This is a block diagram illustrating process 500 for generating a video embedding database 508 for music-to-video retrieval, based on some examples. Process 500 uses methods already referenced above. Figure 2 The video encoder 250 is trained as described. The resulting video embedding database 508 can be used for music-to-video retrieval, for example, as referenced below. Figure 6 More detailed description.

[0053] In some examples, process 500 includes selection. The value is then used as a configuration parameter for the video encoder 250. As mentioned earlier, The value controls the relative weighting between self-supervised information and label-supervised information in the embedding vector. It is used for selection in process 500. The various options for the value include, but are not limited to: (i) user selection; (ii) automatic or algorithmic selection; and (iii) default value, such as the same value used during the training process.

[0054] In some examples, the video embedding database 508 is generated as follows. Each video track (segment) 504 from the video database 502 is processed using the video encoder 250 to generate a corresponding embedding (latent spatial vector) 506. Embedding 306 is similar to the reference above. Figure 2 The described projection 280 ( Then, the multiple embeddings 506, which correspond to the multiple video tracks 504 from the video library 502 and are generated in this way, are sorted and organized in a suitable searchable format to create a video embedding database 508.

[0055] Figure 6 This is a block diagram illustrating a music-to-video retrieval process 600 configured to use video embedding database 508, based on some examples. Process 600 employs methods already referenced above. Figure 2 The audio encoder 210 is trained as described. In response to music query 602, process 600 generates the top-ranked videos in the video library 502. K A list of 612 bit video tracks. In various examples, the number... K This is a parameter selected by the user, which can be set to, for example... K = 1, 5, or 10. Using list 612, the actual corresponding video track 504 can then be retrieved from video library 502 (see also...). Figure 5 ).

[0056] Given the music track of audio query 602, audio encoder 210 operates to generate the corresponding audio embedding 604. Audio embedding 604 is similar to projection 240 (…). The sorting module 610 operates to compare the audio embedding 604 with the audio embedding 506 in the video embedding database 508 using a suitable similarity metric, and calculates a similarity score between each of the audio embeddings 604 and 506. In some examples, the similarity metric used is a cosine similarity metric. In other examples, other suitable similarity metrics may also be used. Then, it will be compared with the previous... K The video tracks 604 corresponding to the most similar embeddings 506 are listed in list 612 in order of ranking and retrieved from video library 502.

[0057] The value of is a crucial factor controlling the balance between self-supervised information and label-supervised information in the embedding. Therefore, for Controlling the value enables control over the cross-modal retrieval process. For example, when When set to 0, process 400 emphasizes self-supervised similarity, thereby retrieving music track 304 that most closely matches the patterns independently learned by the model during training. Conversely, when When set to 1, process 400 primarily focuses on label-supervised similarity, thereby retrieving music track 304 that is closely aligned with the labeled data. Similar observations apply to process 600. By selecting a value between 0 and 1... The system allows the retrieval process to benefit from both self-supervised and label-supervised learning paradigms. This flexibility beneficially provides users with the ability to fine-tune the retrieval process 400-600 to meet their specific needs, thus adapting to different situations where one type of information may be more valuable than another. In some examples, this controllability provides users with the ability to customize the retrieval process, ensuring it aligns well with their specific objectives and optimizing results based on a chosen balance between self-supervised and label-supervised learning.

[0058] Figure 7 This is a flowchart illustrating media retrieval method 700 based on some examples. See below for further reference. Figures 2 to 7 To describe method 700.

[0059] Box 702 of method 700 includes setting weight parameters. The value (for example, see) Figures 3 to 6 Each of the middle characters is marked as " (The input of ""). In some examples, the operation of box 702 includes receiving weight parameters. The first value. In some examples, the first value received is provided via user input.

[0060] Box 704 of method 700 includes receiving the input media dataset of the first modality (e.g., Figure 4 402 or Figure 6 (602). The first modality is selected from the group consisting of an audio (e.g., music) modality and a video modality. In some examples, the input media dataset is an audio track. In other examples, the input media dataset is a video track.

[0061] Box 706 of method 700 includes obtaining a first embedding representing the input media dataset in the latent space using a first modality encoder (e.g., Figure 4 404 or Figure 6 (604). In some examples, the first modal encoder is an audio encoder 210. In other examples, the first modal encoder is a video encoder 250.

[0062] In some examples, weight parameters The relative emphasis of a first component and a second component of a first embedding is controlled by a first modal encoder (e.g., one of 210, 250). The first component corresponds to information learned by the first modal encoder via self-supervised learning. The second component corresponds to information learned by the first modal encoder via label-supervised learning. In some examples, weight parameters... It is restricted to the numerical range [0, 1].

[0063] Box 708 of method 700 includes selecting a subset of second embeddings based on a similarity measure configured to quantify pairwise similarity between different embeddings in the latent space (e.g., Figure 4 412 or Figure 6 (612). Each of the second embeddings in the second embedding represents a corresponding candidate media dataset for a different second modality in the latent space (e.g., Figure 3 306 or Figure 6 (506). The second embedding is obtained using a second modal encoder. In some examples, the second modal encoder is an audio encoder 210. In other examples, the second modal encoder is a video encoder 250. The first modality and different second modalities are selected from a group consisting of the audio modality and the video modality. In some examples, the similarity measure is a cosine measure.

[0064] In some examples, weight parameters The relative emphasis of the first and second components of the second embedding is further controlled by the second modality encoder. The first component of the second embedding corresponds to information learned by the second modality encoder via self-supervised learning. The second component of the second embedding corresponds to information learned by the second modality encoder via label-supervised learning. In some examples, the first and second modality encoders use weight parameters. The second value, different from the first value, is used for joint training. In some examples, the first embedding is computed as a weighted sum of the first and second components. The corresponding weighting coefficients applied to the first and second components are based on the weight parameters. It is determined by the first value.

[0065] In some examples, the operation of box 708 includes receiving a second embedded element of size [missing information]. K A subset of, in which, K It is a positive integer less than twenty. In some examples, K = 1.

[0066] In some examples, the first modality encoder and the second modality encoder are jointly trained using a modality-symmetric contrastive loss function (e.g., see equations (2) through (9)). In some examples, the modality-symmetric contrastive loss function is a weighted sum of a first loss function operating on task-specific embeddings and a second loss function operating on latent space embeddings (e.g., see equation (9)). The first loss function is a weighted sum of a cross-modal self-supervised learning loss operating on task-specific embeddings and a cross-modal label-supervised learning loss operating on task-specific embeddings. The second loss function is a weighted sum of a cross-modal self-supervised learning loss operating on latent space embeddings and a cross-modal label-supervised learning loss operating on latent space embeddings. In some examples, the values ​​of at least a subset of the weighting coefficients used in the weighted sum can be selected based on the training objective.

[0067] Box 710 of method 700 includes identifying one or more output media datasets of different second modalities in the respective candidate media datasets based on a subset of the second embeddings selected in box 708. In some examples, each of the one or more output media datasets is a video track. In other examples, each of the one or more output media datasets is an audio track.

[0068] Box 712 of method 700 includes retrieving one or more output media datasets identified in box 710 from a media library. In some examples, the media library is music library 302 ( Figure 3 In some other examples, the media library is the video library 502 (). Figure 5 ).

[0069] Figure 8 The diagram illustrates a block diagram of a computing device 800 according to some examples, one or more instances of which can be used to implement at least some of the systems, methods and processes described above. Figure 8 The computing device 800 is illustrated as having multiple components, but any one or more of these components may be omitted or repeated as appropriate for application and setup. In some embodiments, some or all of the components included in the computing device 800 may be attached to one or more motherboards and encapsulated in a housing. In some embodiments, some of these components may be manufactured onto a single system-on-a-chip (SoC) (e.g., the SoC may include one or more electronic processing devices 802 and one or more storage devices 804). Additionally, in various embodiments, the computing device 800 may not include... Figure 8One or more of the components shown may be included, but may include interface circuitry for coupling to the one or more components using any suitable interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface. For example, computing device 800 may not include display device 810, but may include display device interface circuitry (e.g., connector and driver circuitry) to which an external display device 810 may be coupled.

[0070] Computing device 800 includes processing device 802 (e.g., one or more processing devices). As used herein, the terms "electronic processor device" and "processing device" are interchangeable to refer to any device or part of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. In various embodiments, processing device 802 may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), server processors, or any other suitable processing devices.

[0071] The computing device 800 also includes a storage device 804 (e.g., one or more storage devices). In various embodiments, the storage device 804 may include one or more memory devices, such as random access memory (RAM) devices (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive bridged RAM (CBRAM) devices), hard disk drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices, etc. In some embodiments, the storage device 804 may include memory that shares a die with the processing device 802. In such embodiments, the memory may be used as cache memory and may include, for example, embedded dynamic random access memory (eDRAM) or spin-transfer torque magnetic random access memory (STT-MRAM). In some embodiments, the storage device 804 may include a non-transitory computer-readable medium having instructions thereon that, when executed by one or more processing devices (e.g., processing device 802), cause the computing device 800 to perform any suitable method or portion thereof disclosed herein.

[0072] The computing device 800 further includes an interface device 806 (e.g., one or more interface devices 806). In various embodiments, the interface device 806 may include one or more communication chips, connectors, and / or other hardware and software to manage communication between the computing device 800 and other computing devices. For example, the interface device 806 may include circuitry for managing wireless communication to enable data transmission to and from the computing device 800. The term “wireless” and its derivatives can be used to describe circuitry, devices, systems, methods, techniques, communication channels, etc., that can transmit data via modulated electromagnetic radiation over a non-solid medium. This term does not imply that the associated devices do not contain any wires, although in some embodiments they may not contain wires. The circuitry included in the interface device 806 for managing wireless communication may implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards (including Wi-Fi (IEEE 802.11 series, IEEE 802.16 standards), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as “3GPP2”), etc.). In some embodiments, the circuitry included in the interface device 806 for managing wireless communications may operate according to a Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, the circuitry included in the interface device 806 for managing wireless communications may operate according to GSM Evolved Enhanced Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included in the interface device 806 for managing wireless communications may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications System (DECT), Evolved Data Optimization (EV-DO) and its derivative protocols, and any other wireless protocol designated as 3G, 4G, 5G, etc. In some embodiments, the interface device 806 may include one or more antennas (e.g., one or more antenna arrays) configured to receive and / or transmit wireless signals.

[0073] In some embodiments, interface device 806 may include circuitry for managing wired communications (such as electrical communication protocols, optical communication protocols, or any other suitable communication protocols). For example, interface device 806 may include circuitry supporting communications based on Ethernet technology. In some embodiments, interface device 806 may support both wireless and wired communications, and / or may support multiple wired communication protocols and / or multiple wireless communication protocols. For example, a first set of circuitry for interface device 806 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and a second set of circuitry for interface device 806 may be dedicated to longer-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some other embodiments, a first set of circuitry for interface device 806 may be dedicated to wireless communications, and a second set of circuitry for interface device 806 may be dedicated to wired communications.

[0074] The computing device 800 also includes a battery / power circuit 808. In various embodiments, the battery / power circuit 808 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 800 to an energy source (e.g., AC line power) separate from the computing device 800.

[0075] The computing device 800 also includes a display device 810 (e.g., one or more separate display devices). In various embodiments, the display device 810 may include any visual indicator, such as a head-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0076] The computing device 800 also includes additional input / output (I / O) devices 812. In various embodiments, I / O devices 812 may include one or more data / signal transmission interfaces, audio I / O devices (e.g., microphones or microphone arrays, speakers, headsets, earphones, sirens, etc.), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, etc.), image capture devices (e.g., one or more cameras), human-machine interface devices (e.g., keyboards, cursor control devices such as mice, styluses, trackballs, or touchpads), etc.

[0077] According to specific embodiments, various components of interface device 806 and / or I / O device 812 can be configured to output suitable control signals, receive suitable control / telemetry signals, and receive and transmit data streams. In some examples, interface device 806 and / or I / O device 812 includes one or more analog-to-digital converters (ADCs) for converting received analog signals into a digital form suitable for operations performed by processing device 802 and / or storage device 804. In some additional examples, interface device 806 and / or I / O device 812 includes one or more digital-to-analog converters (DACs) for converting digital signals provided by processing device 802 and / or storage device 804 into an analog form suitable for transmission over a communication channel.

[0078] According to the example embodiments disclosed above, for example, in the summary of the invention and / or referring to Figures 1 to 12, Figure 8 A media retrieval apparatus is provided, comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured, together with the at least one processor, to cause the apparatus to at least: receive an input media dataset of a first modality; obtain a first embedding representing the input media dataset in a latent space using a first modality encoder; select a subset of second embeddings based on a similarity metric configured to quantify pairwise similarity between different embeddings in the latent space, each of the second embeddings representing a corresponding candidate media dataset of a different second modality in the latent space, wherein the second embeddings are obtained using a second modality encoder; and identify one or more output media datasets of different second modalities in the corresponding candidate media datasets based on the selected subset of second embeddings, wherein the first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

[0079] According to another example embodiment disclosed above, for example, in the summary of the invention and / or referring to Figures 1 to 12, Figure 8 A media retrieval method is provided, comprising: receiving an input media dataset of a first modality; obtaining a first embedding representing the input media dataset in a latent space using a first modality encoder; selecting a subset of second embeddings based on a similarity metric configured to quantify pairwise similarity between different embeddings in the latent space, each of the second embeddings representing a corresponding candidate media dataset of a different second modality in the latent space, wherein the second embeddings are obtained using a second modality encoder; and identifying one or more output media datasets of different second modalities in the corresponding candidate media datasets based on the selected subset of second embeddings, wherein the first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

[0080] In some embodiments of the above method, the first mode and different second modes are selected from the group consisting of audio modes and video modes.

[0081] In some embodiments of any of the methods described above, the input media dataset is an audio track; and each of one or more output media datasets is a video track.

[0082] In some embodiments of any of the methods described above, the input media dataset is a video track; and each of one or more output media datasets is an audio track.

[0083] In some embodiments of any of the methods described above, the method further includes retrieving one or more identified output media datasets from a media library.

[0084] In some embodiments of any of the methods described above, the selection includes receiving a second embedded element of size [missing information]. K A subset of, in which, K It is a positive integer less than twenty.

[0085] In some embodiments of any of the methods described above, the method further includes receiving a first value of a weight parameter, wherein the weight parameter controls the relative emphasis of a first component and a second component applied by the first modal encoder to the first embedding, the first component corresponding to information learned by the first modal encoder via self-supervised learning, and the second component corresponding to information learned by the first modal encoder via label-supervised learning, wherein the weight parameter is restricted to a numerical range of [0, 1].

[0086] In some embodiments of any of the methods described above, the weight parameters further control the relative emphasis of the first and second components of the second embedding applied by the second modality encoder, the first component of the second embedding corresponding to information learned by the second modality encoder via self-supervised learning, and the second component of the second embedding corresponding to information learned by the second modality encoder via label-supervised learning.

[0087] In some embodiments of any of the methods described above, the first modal encoder and the second modal encoder are jointly trained using second values ​​of the weight parameters that are different from the first values.

[0088] In some embodiments of any of the methods described above, the first value received is provided via user input.

[0089] In some embodiments of any of the methods described above, obtaining the first embedding involves calculating a weighted sum of a first component and a second component; and wherein the corresponding weighting coefficients applied to the first component and the second component are determined based on a first value.

[0090] In some embodiments of any of the methods described above, the first modal encoder and the second modal encoder are jointly trained using a modal symmetric contrastive loss function.

[0091] In some embodiments of any of the methods described above, the modal symmetric contrastive loss function is a weighted sum of a first loss function operating on a task-specific embedding and a second loss function operating on a latent space embedding.

[0092] In some embodiments of any of the methods described above, the first loss function is a weighted sum of the cross-modal self-supervised learning loss operating on task-specific embeddings and the cross-modal label-supervised learning loss operating on task-specific embeddings.

[0093] In some embodiments of any of the methods described above, the second loss function is a weighted sum of the cross-modal self-supervised learning loss operating on the latent spatial embedding and the cross-modal label-supervised learning loss operating on the latent spatial embedding.

[0094] In some embodiments of any of the methods described above, the values ​​of at least a subset of the weighting coefficients used in the weighted sum can be selected based on the training objective.

[0095] In some embodiments of any of the methods described above, the similarity measure is a cosine measure.

[0096] Some embodiments provide a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including any of the methods described above.

[0097] Regarding the processes, systems, methods, heuristics, etc., described herein, it should be understood that although the steps of such processes, etc., have been described as being performed in a specific ordered order, such processes can be practiced using the described steps performed in a different order than that described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0098] Accordingly, it should be understood that the above description is intended to be illustrative rather than restrictive. Many embodiments and applications beyond the examples provided will become apparent upon reading the above description. The scope should not be determined by reference to the above description, but rather by reference to the appended claims and the full scope of their equivalents. It is anticipated and desired that the techniques discussed herein will evolve in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In conclusion, it should be understood that modifications and changes are possible with this application.

[0099] All terms used in the claims are intended to be given the broadest reasonable interpretation and common meaning as understood by one skilled in the art described herein, unless expressly indicated otherwise herein. In particular, the use of singular articles such as “a,” “the,” and “the” should be understood to refer to one or more of the indicated elements, unless the claims expressly limit this to the contrary.

[0100] An abstract of this disclosure is provided to allow the reader to quickly determine the nature of this technical disclosure. This abstract is submitted on the premise that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen in the foregoing detailed description, various features are combined together in various embodiments for the purpose of making this disclosure a coherent whole. The method of this disclosure should not be construed as reflecting an intention to incorporate more features than expressly recited in each claim in the claimed embodiments. Rather, as reflected in the appended claims, the inventive subject matter lies in fewer than all the features of a single disclosed embodiment. Therefore, the appended claims are thus incorporated into the detailed description, each claim being an independent subject matter claimed.

[0101] While this disclosure includes references to illustrative embodiments, this specification is not intended to be limiting. Various modifications to the described embodiments, as well as other embodiments within the scope of this disclosure, will be apparent to those skilled in the art to which this disclosure pertains, and are considered to be within the principles and scope of this disclosure, for example, as expressed in the following claims.

[0102] Some embodiments can be implemented as circuit-based processes, including possible implementations on a single integrated circuit.

[0103] Some embodiments may be embodied in the form of methods and apparatus for practicing these methods. Some embodiments may also be embodied in the form of program code recorded in a tangible medium, such as a magnetic recording medium, optical recording medium, solid-state storage, floppy disk, CD-ROM, hard disk drive, or any other non-transitory machine-readable storage medium, wherein when the program code is loaded into and executed by a machine (such as a computer), the machine becomes an apparatus for practicing the patented invention(s). Some embodiments may also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium (including being loaded into and / or executed by the machine), wherein when the program code is loaded into and executed by a machine (such as a computer or processor), the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments are combined with the processor to provide a unique device that operates similarly to a particular logic circuit.

[0104] Unless otherwise expressly stated, each value and range should be interpreted as approximate as if preceded by the words “about” or “approximately”.

[0105] The use of figure numbers and / or reference numerals in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate the interpretation of the claims. Such use should not be construed as necessarily limiting the scope of these claims to the embodiments shown in the corresponding figures.

[0106] Although the elements in the following method claims (if any) are recited in a particular order with corresponding markings, these elements are not necessarily intended to be limited to being recited in a particular order unless the recitation of the claims otherwise implies a particular order for implementing some or all of these elements.

[0107] References to “one embodiment” or “an embodiment” herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may include at least one embodiment of this disclosure. In this specification, the phrase “in one embodiment” appearing in various places does not necessarily refer to the same embodiment, nor is it necessarily a separate or alternative embodiment that is mutually exclusive with other embodiments. The same applies to the term “implementation.”

[0108] Unless otherwise specified herein, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to refer to objects among multiple similar objects merely indicates different instances of such similar objects being mentioned, and is not intended to imply that such similar objects must be in a corresponding order or sequence in time, space, ranking, or any other way.

[0109] Unless otherwise specified herein, the conjunction “if” may also be interpreted, in addition to its simple meaning, as meaning “when…”, “at…”, “in response to determination”, or “in response to detection,” depending on the specific context. For example, the phrases “if it is determined…” or “if [the stated condition] is detected” may be interpreted as meaning “after it is determined…”, “in response to determining…”, “after [the stated condition or event] is detected”, or “in response to [the stated condition or event] being detected.”

[0110] For the purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or subsequently developed that allows energy to be transferred between two or more elements, and envision the insertion of one or more additional elements, although this is not required. Conversely, the terms “direct coupling,” “direct connection,” etc., imply the absence of such additional elements.

[0111] As used in this document with respect to components and standards, the term "compatible" means that the component communicates with other components in a manner wholly or partially specified in the standard, and is recognized by other components as capable of communicating with other components in a manner specified in the standard. Compatible components do not need to operate internally in a manner specified in the standard.

[0112] The functions of the various elements shown in the figure, including any functional blocks labeled “processor” and / or “controller”, can be provided by using dedicated hardware and hardware capable of executing the software, associated with appropriate software. When provided by a processor, the functionality can be provided by a single dedicated processor, a single shared processor, or multiple separate processors (some of which may be shared). Furthermore, the explicit use of the terms “processor” or “controller” should not be construed as exclusively referring to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage devices. Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the figure are conceptual only. Their functionality can be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, the specific technology of which may be chosen by the implementer, as understood more specifically from the context.

[0113] As used herein, the terms “circuit” and “circuitry” may refer to one or more of the following: (a) a purely hardware circuit implementation (such as implementations in purely analog and / or digital circuits); (b) a combination of hardware circuitry and software, such as (if applicable): (i) a combination of analog and / or digital hardware circuitry with software / firmware, and (ii) any portion of a hardware processor with software (including digital signal processors), software, and memory, which work together to enable a device such as a mobile phone or server to perform various functions; and (c) a hardware circuitry and / or processor, such as a microprocessor or a portion thereof, which requires software (e.g., firmware) to operate, but may be absent when operation is not required. This definition of circuitry applies to all uses of the term in this application (including all claims). As a further example, as used herein, the term “circuit” also covers implementations of only one hardware circuit or one or more processors, or a portion thereof, and its accompanying software and / or firmware. The term "circuit" may also cover, for example, baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices or other computing or networking devices, where applicable to certain elements of the claims.

[0114] Those skilled in the art will recognize that any block diagram herein represents a conceptual view of an illustrative circuit embodying the principles of this disclosure. Similarly, it will be appreciated that any flow chart, flow diagram, state transition diagram, pseudocode, etc., represents various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0115] The "Summary" in this specification is intended to introduce some exemplary embodiments, and additional embodiments are described in the "Detailed Description" and / or with reference to one or more accompanying drawings. The "Summary" is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. A media retrieval method, comprising: Receive the input media dataset of the first modality; A first embedding representing the input media dataset in the latent space is obtained using a first modality encoder; A subset of second embeddings is selected based on a similarity metric configured to quantify pairwise similarity between different embeddings in the latent space, each of the second embeddings representing a corresponding candidate media dataset for a different second modality in the latent space, wherein the second embeddings are obtained using a second modality encoder; and Based on the selected subset of the second embedding, one or more output media datasets of the different second modalities are identified in the corresponding candidate media datasets. The first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

2. The method as described in claim 1, wherein, The first mode and the different second modes are selected from a group consisting of audio modes and video modes.

3. The method as described in claim 2, in, The input media dataset is an audio track; and Each of the one or more output media datasets is a video track.

4. The method as described in claim 2, in, The input media dataset is a video track; and Each of the one or more output media datasets is an audio track.

5. The method of claim 1, further comprising retrieving one or more identified output media datasets from a media library.

6. The method of claim 1, wherein, The selection includes receiving the second embedded, which is of size [missing information]. K A subset of, in which, K It is a positive integer less than twenty.

7. The method of claim 6, wherein, K = 1。 8. The method of claim 1, further comprising receiving a first value of the weighting parameter, in, The weight parameters control the relative emphasis of a first component and a second component of the first embedding applied by the first modal encoder, the first component corresponding to information learned by the first modal encoder via self-supervised learning, and the second component corresponding to information learned by the first modal encoder via label-supervised learning, wherein the weight parameters are restricted to a numerical range of [0, 1].

9. The method of claim 8, wherein, The weight parameters further control the relative emphasis of the first and second components applied by the second modal encoder to the second embedding, wherein the first component of the second embedding corresponds to information learned by the second modal encoder via self-supervised learning, and the second component of the second embedding corresponds to information learned by the second modal encoder via label-supervised learning.

10. The method of claim 9, wherein, The first modal encoder and the second modal encoder are jointly trained using a second value of the weight parameters that is different from the first value.

11. The method of claim 8, wherein, The first value received is provided via user input.

12. The method of claim 8, in, The acquisition includes calculating the first embedding as a weighted sum of the first component and the second component; and The corresponding weighting coefficients applied to the first component and the second component are determined based on the first value.

13. The method of claim 1, wherein, The first modal encoder and the second modal encoder are jointly trained using a modal symmetric contrastive loss function.

14. The method of claim 13, wherein, The modal symmetric contrastive loss function is a weighted sum of a first loss function operating on task-specific embeddings and a second loss function operating on latent space embeddings.

15. The method of claim 14, wherein, The first loss function is a weighted sum of the cross-modal self-supervised learning loss operating on the task-specific embedding and the cross-modal label-supervised learning loss operating on the task-specific embedding.

16. The method of claim 15, wherein, The second loss function is a weighted sum of the cross-modal self-supervised learning loss operating on the latent spatial embedding and the cross-modal label-supervised learning loss operating on the latent spatial embedding.

17. The method of claim 16, wherein, The values ​​of at least a subset of the weighting coefficients used in the weighted sum can be selected based on the training objective.

18. The method of claim 1, wherein, The similarity measure is a cosine measure.

19. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method of claim 1.

20. An apparatus for media retrieval, the apparatus comprising: At least one processor; as well as At least one memory containing program code, The at least one memory and the program code are configured, together with the at least one processor, to cause the device to perform at least the following operations: Receive the input media dataset of the first modality; A first embedding representing the input media dataset in the latent space is obtained using a first modality encoder; A subset of second embeddings is selected based on a similarity metric configured to quantify pairwise similarity between different embeddings in the latent space, each of the second embeddings representing a corresponding candidate media dataset for a different second modality in the latent space, wherein the second embeddings are obtained using a second modality encoder; and Based on the selected subset of the second embedding, one or more output media datasets of the different second modalities are identified in the corresponding candidate media datasets. The first modality encoder and the second modality encoder are jointly trained using semi-supervised contrastive learning.

Images