Search method, device, equipment and readable storage medium

Abstract

The application discloses a retrieval method, device and equipment and a readable storage medium. The method comprises the following steps: obtaining first feature information of to-be-retrieved content, and performing retrieval in a first feature library to obtain a first result set; when entries in the first result set are first-type entries, obtaining second feature information of the entries; performing retrieval in a second feature library containing second-type entry features according to the second feature information to obtain a second result set; and outputting a matched result sequence according to the first result set and the second result set. The application can output a matched list containing specific entries such as original content, improve the retrieval experience of users, and ensure the exposure rate of original content in a platform.

Description

Technical Field

[0001] This application relates to the field of content retrieval technology, specifically to a retrieval method, apparatus, device, and readable storage medium. Background Technology

[0002] Content retrieval is now widely used in various aspects of users' lives, such as song recognition on music platforms or video search on video platforms. In some scenarios, platforms often need to output a list of results in a certain order. Taking song recognition as an example, after finding several matching songs, it is often desirable to return the original songs to the user first, and then return non-original songs (such as covers, adaptations, and derivative works) later. Similarly, for other search content, such as videos, images, and documents, there is often a need to return original content to the user first, and then return non-original content, such as remakes, derivative videos, edited documents, or images later.

[0003] However, current search methods mostly output matching results based on matching judgment thresholds. This can lead to situations where non-original content slightly exceeds the judgment threshold, while the corresponding original content does not exceed the judgment threshold. As a result, the output matching results do not contain original content, and original content cannot be prioritized, which affects the user's search experience. Summary of the Invention

[0004] This application provides a retrieval method, apparatus, device, and readable storage medium, aiming to solve the technical problem that non-original content is matched and identified, while original content is not successfully matched and identified, resulting in the inability to output original and non-original content in sequence.

[0005] Firstly, this application provides a retrieval method, including: The first feature information of the content to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library contains features of the first type of entries and features of the second type of entries. When an entry in the first result set is a first type of entry, the second feature information of the entry is obtained; Based on the second feature information of the entry, a second feature database is searched to obtain a second result set. The second feature database contains features of the second type of entry. Based on the first result set and the second result set, output the matching result sequence.

[0006] In one embodiment of this application, the step of searching in the first feature database based on the first feature information to obtain the first result set includes: Obtain a matching score that indicates the similarity between the first feature information and each feature information in the first feature library; Feature information in the first feature library whose matching score exceeds a preset score threshold is identified as target feature information; Output the first result set based on the target feature information.

[0007] In one embodiment of this application, the content to be retrieved includes an audio segment, the feature information includes multiple energy extreme points in the audio spectrogram, and the matching score is determined based on the number of matching energy extreme points between the first feature information and each feature information in the first feature library at the corresponding time. Alternatively, the feature information may include feature vectors output by a neural network model, and the matching score may be determined based on the vector similarity between the first feature information and each feature information in the first feature library.

[0008] In one embodiment of this application, obtaining the second feature information of the entry includes: Determine the score difference between the matching score corresponding to the entry and the preset score threshold; When the score difference of the entry is less than a preset difference threshold, the second feature information of the entry is obtained.

[0009] In one embodiment of this application, obtaining the second feature information of the entry includes: Determine whether a second type of entry corresponding to the stated entry exists in the first result set; When there is no second type entry corresponding to the entry in the first result set, the second feature information of the entry is obtained.

[0010] In one embodiment of this application, obtaining the second feature information of the entry includes: Extract the attribute information of the content to be retrieved; When the attribute information of the content to be retrieved meets the preset conditions, the second feature information of the entry is obtained.

[0011] In one embodiment of this application, the method further includes: Based on the first feature information, determine the first timestamp in the entry that matches the content to be retrieved; The process of obtaining the second feature information of the entry includes: Obtain the second feature information corresponding to the first timestamp of the entry; The second result set is obtained by searching the second feature database based on the second feature information of the entry, including: Based on the second feature information of the entry, a search is performed in the second feature database to obtain a second result set and a second timestamp of each entry in the second result set that matches the second feature information.

[0012] In one embodiment of this application, the method further includes: Based on the first timestamp and the second timestamp, a corresponding playback identifier is generated for each item in the result sequence. The playback identifier is used to control playback from the timestamp corresponding to each item in the result sequence.

[0013] In one embodiment of this application, the step of outputting a matching result sequence based on the first result set and the second result set includes: The entries in the first result set and the second result set are deduplicated and sorted to output a matching result sequence, wherein the sorting priority of the second type of entries in the matching result sequence is higher than that of the first type of entries.

[0014] Secondly, this application also provides a retrieval device, comprising: The acquisition module is used to acquire the first feature information of the content to be retrieved, and to perform a retrieval in the first feature library based on the first feature information to obtain a first result set. The first feature library contains fingerprints of the first type of entries and fingerprints of the second type of entries. The extraction module is used to obtain the second feature information of the entry when the entry in the first result set is a first type of entry; The retrieval module is used to perform a retrieval in a second feature library based on the second feature information of the entry, and obtain a second result set, wherein the second feature library contains features of the second type of entries; The output module is used to output a sequence of matching results based on the first result set and the second result set.

[0015] Thirdly, this application also provides a computer device, the computer device comprising: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the retrieval method as described in any of the preceding claims.

[0016] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the retrieval method as described in any of the preceding claims.

[0017] The retrieval method provided in this application first obtains the first fingerprint information of the content to be retrieved, then searches in a first feature library containing first-class and second-class entries based on the first feature information to obtain a first result set. Then, when an entry in the first result set is a first-class entry, the second feature information of that entry is obtained, and further searches are performed in a second feature library containing only second-class entries to obtain a second result set. Finally, the matching result sequence is output based on the first and second result sets in the order of second-class entries first and then first-class entries. This ensures that the output result sequence contains both first-class and second-class entries, that is, the output result sequence contains both original and non-original content, improving the user's retrieval experience and ensuring the exposure rate of popular original content on the platform. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating the effect of determining the boundary of the content to be retrieved, provided in an embodiment of this application; Figure 2 A flowchart illustrating the steps of a retrieval method provided in an embodiment of this application; Figure 3 This is a schematic flowchart illustrating the steps of retrieving information from a feature database based on feature information, provided in an embodiment of this application. Figure 4 This application provides a schematic flowchart of a step for extracting the timestamp of the content to be detected, as shown in the embodiment of the present application. Figure 5 A flowchart illustrating the implementation of a complete song retrieval method provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of a retrieval device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0022] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0023] To clearly understand the retrieval method, apparatus, device, and readable storage medium provided in the embodiments of this application, the relevant background of the retrieval method will first be explained. Specifically, the retrieval method provided in this application is generally applied to the retrieval of content on platforms, such as music platforms and video platforms. For ease of understanding, the following embodiments of this application will mainly use the song recognition scenario on a music platform as an example for explanation. Specifically, song recognition refers to a user recording the surrounding music using a microphone and matching the recording with a pre-built music library. After identifying a specific song, the song details can be displayed on the results page for the user to listen to or add to their favorites. Because song recognition allows users to easily find songs of interest through music, it has become a mainstream function of various music software and music platforms. There are various possible methods for matching and searching within a music library, but the mainstream underlying technology currently relies on audio fingerprinting. This primarily involves using signal processing or AI (artificial intelligence) modules to extract basic features representing the audio from the audio or song, converting them into fingerprint information in the form of numerical values, numerical vectors, or sequences of numerical vectors. Then, by comparing the similarity between the fingerprint of the audio to be searched and the fingerprints of various songs in the music library, a matching score is calculated. Songs with scores exceeding a threshold are then returned as search results, sorted in descending order of matching score. Furthermore, based on actual product needs, timestamp information matching the audio in the search results can be provided, allowing users to directly play songs from the corresponding timestamp. Of course, similar to song recognition, the retrieval of videos, images, or other content items also largely involves building a feature library and then retrieving content items based on the similarity of feature information. This will not be repeated in the embodiments of this application.

[0024] With the booming development of various industries and the increasing accessibility of creative content, the databases of various platforms, such as music platforms, typically contain a large amount of original content and corresponding derivative works. For example, music platforms (music software) usually include original songs and non-original songs that are covers, adaptations, or other derivative works based on the original songs. To improve user experience and further increase the exposure of popular original content, compared to simply sorting the matching results in descending order by matching score, various platforms prefer to prioritize original content and place non-original content that has been retrieved later. For example, the output results may prioritize original songs before showing covers or adaptations. Therefore, after the search is completed, the sorting module can adjust the order of the items in the result list based on specific business knowledge, such as specific original content identifiers. For example, in the music library, songs may carry identifiers indicating their type (such as original or non-original tags). Therefore, in some implementation schemes, taking song recognition as an example, song recognition typically performs the following steps: 1. Perform fingerprint music recognition retrieval and obtain a list of music recognition results (sorted in descending order of fingerprint matching score). 2. For the songs in the list, move the songs marked "Original Singer" to the front of the results list and move the songs without the "Original Singer" mark to the back. 3. Return the list of results after adjusting the order, as the final sorting result.

[0025] Similarly, for content retrieval in other fields, there are usually similar steps, namely, performing feature retrieval and adjusting the order of the output retrieval content according to the tags of the retrieval content.

[0026] However, in practical applications, especially those with some noise interference, the method of identifying matching content items based on the decision threshold is prone to the following phenomenon, which causes the above sorting strategy to fail: The matching score between the content to be retrieved (for ease of description, we will use query and query vector as examples below) and a non-original content X is T+δ1, where T is the decision threshold and δ1 is a very small positive number. This score means that the similarity between the content to be retrieved and the non-original content X slightly exceeds the decision threshold. In this case, the non-original content X will be included in the search results list. Meanwhile, under normal circumstances, the matching degree between original content and non-original content is high. Therefore, the score of the original content A corresponding to the non-original content X is usually close to the score of the original content A corresponding to the non-original content X. However, when the score of the original content A corresponding to the non-original content X is T-δ2, where T is the decision threshold and δ2 is a very small positive number, this also means that the matching score of the original content A corresponding to the content to be retrieved is just below the decision threshold. In this case, the original content A will not be included in the search results list. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 This diagram illustrates the effect of determining the boundary of the content to be retrieved, as provided in an embodiment of this application. As shown, the distance between the content to be retrieved (query) and the non-original content (X) (corresponding to higher similarity and a higher matching score) is relatively close, and the matching score exceeds the determination threshold. However, the distance between the content to be retrieved and the original content (A) just exceeds the edge of the determination threshold. In this situation, since the original content is not included in the search results list during the retrieval process, it cannot be properly ranked first in the subsequent sorting module, and the non-original content will be ranked last. Taking audio retrieval as an example, the content to be retrieved is the audio segment to be retrieved, and the original content (A) is the original song, or a song in the music library that carries the "original singer tag," while the non-original content (X) is a non-original song, such as a cover song, an adapted song, or a song in the music library that does not carry the "original singer tag."

[0027] To at least partially address the aforementioned technical issues, related technologies propose establishing content item relationships offline within the database in advance. Multiple versions of the same content are grouped into a single "group" relationship. During retrieval, for each retrieved result, the corresponding group information is read, and based on this information, potential original content (or content with high popularity) is recalled to expand the search results. However, this approach still presents significant problems in terms of backend maintenance and user experience. For example, a dedicated backend database for maintaining "group" relationships is required. Establishing "group" relationships between content items offline and maintaining them online introduces a massive computational workload. For instance, offline determination of "group" relationships between pairs of content items in the database requires billions of judgments, assuming hundreds of thousands of original and hundreds of thousands of non-original content items. Furthermore, during online maintenance, establishing group relationships between newly added non-original content and all original content in the database takes time. During this period, pre-established group relationships cannot be used, and shortening this waiting time requires significant additional computational resources. Furthermore, the above method can only provide the simple relationship that non-original content and original content belong to the "same group". For some content retrieval involving time series, such as music audio or video, since the time correspondence between non-original content and original content is unknown, it is impossible to provide additional timestamp information to help users directly play from the corresponding timestamp to determine whether the search results are correct.

[0028] Therefore, based on the above scenario, this application provides a retrieval method. By retrieving the first type of entries, it further matches a result set containing the second type of entries from a feature library containing the second type of entries, and then outputs a sequence of matching results by combining the two result sets sequentially. This improves the user's retrieval experience, especially for retrieval content involving time information, and can also assist users in judging the accuracy of the retrieval results by playing the corresponding timestamp. Specifically, the following will describe this in conjunction with specific embodiments, particularly using song recognition as an example in some embodiments.

[0029] like Figure 2 As shown, Figure 2This is a flowchart illustrating the steps of a retrieval method provided in an embodiment of this application. Specifically, this method typically runs on a multimedia platform, such as a music platform, video platform, or image platform server. Typically, the server communicates with the platform's database to process user requests sent through the client, such as playing music, playing videos, or displaying images. The database usually contains detailed information about various content items within the platform, such as original and non-original songs on a music platform, original and derivative videos on a video platform, etc. In particular, the database typically includes a feature database, also known as a fingerprint database, to store feature information for various content items, such as audio features and image features. Specifically, the feature information can typically be represented as feature vectors indicating the characteristics of each content item. Different methods can be used to process different content items, as will be explained in subsequent embodiments. Specifically, the introduction of a feature database can be used to implement various retrieval functions such as song recognition and image retrieval. Specifically, the retrieval method includes steps S210-S240: S210, obtain the first feature information of the content to be searched, and perform a search in the first feature database based on the first feature information to obtain the first result set.

[0030] In one embodiment of this application, the content to be searched typically refers to the initial search content that the user needs to search. In different search scenarios, the content to be searched can take different forms. Specifically, for ease of understanding, taking song recognition as an example, in one possible scenario, when a user hears a song they are interested in but do not know its specific name, they can open the music software client program and click the song recognition function module to enter the song recognition page. At this time, the user can control the terminal device to continuously collect audio from the current environment by touching the recording control and continuously send song recognition requests to the server to attempt song recognition. The audio fragments collected by the terminal device in the current environment can be understood as the content to be searched. For ease of description, the following will use the audio fragments to be searched as an example. Of course, the aforementioned solution is merely one possible retrieval solution and should not be construed as a limitation on the technical solution of this application. In fact, content retrieval required in other scenarios is within the scope of protection claimed by this application. For example, when a user needs to retrieve videos or images, they can also do so by entering corresponding keyword description text or uploading a simple diagram. This application does not restrict the source of the specific content to be retrieved. Any initial retrieval content that needs to retrieve matching content entries from the database can be as mentioned in this application.

[0031] Building upon the aforementioned foundation, once the server obtains the content to be searched for matching, specific feature extraction techniques are needed to extract the feature information of that content. For example, taking the aforementioned audio segment as an example, it is often necessary to extract the audio features of that audio segment. Specifically, this application does not limit the specific implementation scheme for extracting feature information. For instance, it can be done through signal processing methods or based on a trained AI model to extract the first feature information, which represents the basic features of the search content, in the form of numerical values, numerical vectors, or sequences of numerical vectors. Of course, it should be noted that in the embodiments of this application, the method of processing the content to be detected to extract its first feature information usually requires the same feature extraction method used when constructing a feature library to process the content of each entry in the database to extract its features.

[0032] After obtaining the first feature information of the content to be retrieved through the aforementioned method, a search is performed based on this first feature information in a first feature database (fingerprint database), which contains features of the first type of entries and features of the second type of entries, i.e., the full feature database. Typically, this yields the first result set. Specifically, in some embodiments of this application, the first type of entries are usually non-original content, i.e., entries that do not carry an "originality identifier," while the second type of entries are usually original content, i.e., entries that carry an "originality identifier." For example, taking music platforms as an example, the first category of entries is non-original songs, such as cover songs or adapted songs, while the second category of entries is original songs. Similarly, for video platforms, the first category of entries is non-original videos, such as videos created by optimizing, beautifying, or adapting original videos using AI assistance or other means, while the second category of entries is original videos uploaded by users. For image platforms, the first category of entries is non-original images, such as images created by optimizing, beautifying, or adapting original images using AI assistance or other means, while the second category of entries is original images uploaded by users, such as photos taken by users or artistic drawings, etc.

[0033] Specifically, based on the actual search and matching results, the first result set can typically contain both the first type of entries and the second type of entries.

[0034] To facilitate understanding, in one embodiment, taking audio retrieval as an example, the first feature information of the content to be retrieved is obtained, and a search is performed in a first feature library based on the first feature information to obtain a first result set. The first feature library contains features of a first type of entry and features of a second type of entry, specifically including: The first feature information of the audio segment to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library includes features of non-original songs and features of original songs.

[0035] Similarly, taking video retrieval as an example, the first feature information of the content to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library contains features of the first type of entries and features of the second type of entries, specifically including: The first feature information of the video segment to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library includes features of non-original videos and features of original videos.

[0036] Furthermore, in one embodiment of this application, the retrieval is performed in the first feature database based on the first feature information. Typically, it is based on the similarity of the feature information. Entries whose matching score with the first feature information of the content to be retrieved exceeds a preset threshold (i.e., the similarity exceeds a certain threshold) are used as retrieval entries in the first result set. It is in such a scenario that the content to be retrieved is subject to various noise interferences, such as interference from the collected content or interference from the feature information extracted by AI, which may cause the matching score of a certain first-type entry to exceed the preset threshold, while the matching score of its corresponding second-type entry does not exceed the preset threshold.

[0037] Specifically, to facilitate understanding of the above content, in one embodiment of this application, the content to be retrieved is an audio segment, and the feature information of the audio segment is constructed based on the energy extrema points in the audio spectrogram of the audio segment, i.e., an audio fingerprint method. The following will explain the method of constructing an audio fingerprint through energy extrema points. Specifically, it typically includes the following steps: (1) Perform a Short Time Fourier Transform (STFT) on the audio signal to convert the one-dimensional time-domain audio signal into a two-dimensional spectrogram. The horizontal axis of the spectrogram is the time axis, in units of audio frames, and the vertical axis is the frequency axis, in units of frequency band numbers. At this point, the value of a point p(t,f) in the spectrogram represents the signal strength of frequency band f at time t. (2) After completing the short-time Fourier transform, find the local peak in the two-dimensional spectrogram. That is, if the value of a point in the spectrogram is greater than that of all the points around it (e.g., a rectangular area around it), then the point is a local peak, which is also an energy extreme point. (3) Construct fingerprints based on all peak points found in the spectrogram. For example, in one possible implementation, the time frame t and frequency band number f of the peak point can be directly used as the fingerprint corresponding to the current peak point, thereby constructing the fingerprint features of all peak points in the audio segment. Of course, the above scheme only uses time frame information and frequency band information as an example to construct the fingerprint features of the audio segment. In fact, based on actual needs, the audio fingerprint can also contain more feature information, such as signal strength, etc. This application embodiment does not limit the method of constructing fingerprint features in the audio segment. Of course, in addition to the audio segments mentioned above, similar methods can be used for images, videos, or other content, or features of each content item can be directly extracted through a trained neural network model, such as image features, video features, etc. This application embodiment will not repeat the description here.

[0038] After the feature extraction of the audio signal is completed, if it is the step of establishing a music recognition fingerprint database, the fingerprint can be further stored as an index in the form of an inverted index to facilitate subsequent fast retrieval. If it is the feature extraction of the audio segment to be retrieved, the fingerprint features extracted at this time are the audio fingerprints that need to be matched in the fingerprint database later, which is the first feature information in this application.

[0039] For details, please refer to Figure 3 , Figure 3 This application provides a flowchart illustrating a step for retrieving information from a feature database based on feature information, specifically including steps S310 to S330: S310, obtain a matching score that indicates the similarity between the first feature information and each feature information in the first feature library.

[0040] In the embodiments of this application, feature information typically exists in the form of feature vectors. Therefore, a matching score indicating the similarity between two feature vectors can usually be calculated based on a formula for calculating vector similarity, such as cosine or Euclidean distance. Specifically, for ease of understanding, the following will use music retrieval as an example.

[0041] Specifically, taking audio fingerprints, which are audio feature information including multiple energy extreme points in the audio spectrogram, as an example, the matching score is determined based on the first feature information, that is, the number of matching energy extreme points between the first audio fingerprint and each audio fingerprint in the first feature library at the corresponding time. For example, in one embodiment of this application, by matching the time frame t and frequency band number f of each energy extreme point in the obtained first audio fingerprint with the time frame t′ and frequency band number f′ of each song's energy extreme point in the first fingerprint database, matching information of the energy extreme points of the first audio fingerprint and each audio fingerprint in the first fingerprint database at the corresponding time can be obtained. Here, the matching information may include the number of matches. Of course, based on actual needs, considering that the time frame or frequency band number of the energy extreme point may have a certain error within the allowable range due to interference of the audio segment, the matching information may also include other more information, such as the degree of matching, etc. This embodiment of the application does not limit the method of matching audio fingerprints to obtain the matching information of the corresponding energy extreme points. Any matching information that can be used to indicate the similarity between the first audio fingerprint and each audio fingerprint in the first fingerprint database is within the scope of protection claimed in this application.

[0042] Alternatively, in another embodiment of this application, taking the audio fingerprint as an example, which includes the audio feature vector output by the neural network model, for example, the audio fingerprint can be a 256-dimensional or 512-dimensional feature vector extracted by a trained audio feature extraction model. Then, the matching score used to indicate the matching similarity can be determined by the vector similarity between the first audio fingerprint and each audio fingerprint in the first fingerprint database. For example, in one possible implementation, it can be calculated based on the cosine angle. Of course, based on actual needs, it can also be obtained based on other schemes for calculating vector similarity, such as Euclidean distance. This application embodiment does not limit this.

[0043] S320, the feature information in the first feature library whose matching score exceeds a preset score threshold is determined as the target feature information.

[0044] S330, output the first result set based on the target feature information.

[0045] Building upon the aforementioned foundation, based on the matching information used to indicate the similarity between the first feature information and each feature information in the first feature library, and according to the system's preset scoring rules (such as a specific matching score calculation formula), for example, by weighted summation of each matching information based on a weighted score calculation method, the matching score between the first feature information and each feature in the first feature library can be obtained. Specifically, the matching score calculation formula can be set based on the actual needs of each platform. This application embodiment does not limit the specific method of calculating the matching score; the matching score can be used to represent the similarity between feature information in numerical form.

[0046] Specifically, taking audio retrieval as an example, by calculating the similarity between audio fingerprints, audio features in the first feature library whose matching scores exceed a preset score threshold can be identified as target audio fingerprints, thereby outputting a first result set of searched songs that contain the target audio fingerprints. At this time, the searched songs in the first result set can be considered as songs that are similar to the audio segment to be searched. These songs can include original songs or non-original songs (cover songs).

[0047] Furthermore, in one embodiment of this application, for a special type of time-series content item, such as audio or video, while retrieving from the first feature database based on the first feature information to obtain the first result set, the timestamps of each item in the first result set and the content to be retrieved can be further determined based on the time frame t information in the feature information. Typically, a timestamp can represent the start time of the match between the retrieved item and the content to be retrieved. For example, taking song retrieval as an example, if a music segment starting at 1 minute and 30 seconds of a song matches the audio segment to be retrieved, then a timestamp of 1 minute and 30 seconds can be marked accordingly. At this time, generating playback controls for the songs in the result set based on this timestamp information allows the user to directly start playing the song from the start time matching the audio segment to be retrieved, or from a preset interval before the start time, by touching the playback controls, thus facilitating the user's confirmation of the accuracy of the song recognition result. Of course, based on actual needs, the timestamp can further include the end time of the segment in the song that matches the audio segment to be retrieved; this embodiment of the application does not limit this. Furthermore, generating timestamps can effectively assist in subsequent retrieval in the original song library, allowing the simultaneous acquisition of the original song library's timestamps. Similarly, for video retrieval, the corresponding timestamp of the retrieved video can be marked, and the retrieved video can be played based on the corresponding timestamp. The specific implementation scheme will be described in detail in subsequent specific embodiments.

[0048] S220, when an entry in the first result set is a first type of entry, the second feature information of the entry is obtained.

[0049] In order to avoid the second type of entries, i.e., original content, or content carrying an "original identifier", not being included in the first result set after retrieving the first result set from the first feature library containing the features of the first type of entries and the features of the second type of entries, i.e., the full feature library, in order to avoid the second type of entries, i.e., original content, or content carrying an "original identifier", being included in the first result set due to the aforementioned situation, in one embodiment of this application, when an entry in the first result set is detected, such as a song, video or image retrieved that is a first type of entry, i.e., non-original content, a subsequent secondary search process will be performed, as detailed below.

[0050] Specifically, identifying whether an entry in the first result set is a first-category or second-category entry can typically be based on specific business knowledge. For example, in one possible implementation, an entry can be classified as a second-category entry (i.e., original content) based on whether it carries the "original content" identifier. If an entry does not carry the original identifier, it is considered a first-category entry (i.e., non-original content). Of course, the above scheme is merely an example using the original identifier. In fact, based on a similar approach, the non-original content identifier can also determine whether each entry in the first result set is a first-category or second-category entry. Specifically, taking song retrieval as an example, when an entry in the first result set is a first-category entry, the second feature information of the entry is obtained, including: When the searched song in the first result set is a first-category song (a non-original song), the second feature information of the searched song is obtained.

[0051] Furthermore, in one embodiment of this application, the server typically determines sequentially whether each entry in the first result set is a second-type entry, thereby determining whether subsequent processes need to be executed. If the current entry is a second-type entry, the processing of the current entry ends directly, and the judgment of the next entry is executed. If the current entry is non-original content, i.e., a first-type entry, the subsequent processing process will begin until all entries in the first result set are finally processed. Specifically, to facilitate understanding of the processing steps for first-type entries, in this embodiment of the application, music retrieval is used as an example. In this case, when the first searched song in the first result set is a non-original song, the second feature information of the first searched song will be further obtained for secondary retrieval. Similarly, for image (video) retrieval, when the first searched image (video) in the first result set is a non-original image (video), the second feature information of the first searched image (video) will be further obtained for secondary retrieval.

[0052] Specifically, in the embodiments of this application, obtaining the second feature information of an entry can generally be similar to obtaining the first feature information of the content to be retrieved, and this application embodiment does not limit this. In this case, in a specific application scenario, since the features of the entries in the first result set are usually included in the first feature library, the second feature information of the entries in the first result set can also be obtained directly from the first feature library without additional feature extraction processing. In particular, based on the timestamps that correspond to the matching of each entry in the first result set with the content to be retrieved, determined by the aforementioned first feature information, the second feature information here can generally also be feature information extracted based on the timestamps. That is to say, obtaining the second feature information of the entry includes: Obtain the second feature information corresponding to the first timestamp of the entry, wherein the first timestamp is specifically determined in the following manner: The first timestamp that matches the content to be retrieved in the entry is determined based on the first feature information.

[0053] Furthermore, considering that in practical application scenarios, the content to be retrieved is often affected by various factors, causing non-original content to just exceed the judgment threshold of the matching score, while original content just does not exceed the judgment threshold of the matching score, in order to further ensure the accurate judgment of such entries and avoid introducing additional processing steps for some non-original content (first type of entries) that already have corresponding original content (second type of entries) in the first result set, in some embodiments of this application, it is also possible to determine whether it is necessary to perform feature extraction on the first type of entries for secondary retrieval based on some set conditions. Specifically, for the sake of understanding the above content, the following will be described in conjunction with specific embodiments.

[0054] In some embodiments of this application, a feasible prerequisite for obtaining the second feature information of the entry is that there is no second type entry corresponding to the entry (first type entry) in the first result set. That is, obtaining the second feature information of the entry includes: Determine whether a second type of entry corresponding to the stated entry exists in the first result set; When there is no second type entry corresponding to the entry in the first result set, the second feature information of the entry is obtained.

[0055] Specifically, in one embodiment of this application, determining whether a second type of song corresponding to a first type of entry exists in the first result set can typically be achieved by using the original and / or non-original identifiers (such as cover songs) of each entry. The association between original and non-original content can usually be pre-built. Specifically, in related embodiments of this application, since it is only necessary to determine whether a corresponding second type of entry (original content) exists for a first type of entry (non-original content) in the first result set, the comparison process can be greatly simplified. Of course, besides building the association between original and non-original content, identification can also be based on whether the matching score of each entry in the first result set exceeds a higher score threshold (used to indicate whether two retrieved entries are original and non-original versions of the same content). This embodiment of the application does not limit the method for determining whether original content corresponding to non-original content exists in the first result set. Furthermore, only when a second type of entry corresponding to a first type of entry does not exist in the first result set will the second feature information of the first type of entry be further extracted for subsequent secondary retrieval. If a second-category entry corresponding to a first-category entry exists in the first result set, the subsequent processing of the first-category entry will be skipped, and the processing of the next entry in the first result set will continue.

[0056] Specifically, to facilitate understanding of the above, taking music retrieval as an example, when it is determined that the first result set does not contain an original song corresponding to a non-original song, the second feature information of the non-original song can be obtained. That is, determining whether the first result set contains a second type of entry corresponding to the entry; when the first result set does not contain a second type of entry corresponding to the entry, obtaining the second feature information of the entry includes: Determine whether there is an original song corresponding to a non-original song in the first result set; if there is no original song corresponding to the non-original song in the first result set, obtain the second feature information of the non-original song.

[0057] Furthermore, in other embodiments of this application, considering that typically the matching score of the first type of entry (non-original content) and the content to be retrieved just exceeds the decision threshold of the matching score, while the matching score of the corresponding second type of entry (original content) and the content to be retrieved just does not exceed the decision threshold of the matching score, resulting in the absence of original content for a certain non-original content in the first result set, therefore, when filtering out the required retrieval entries based on the matching score, the score difference between the matching score of each entry in the first result set and the preset score threshold can be further determined. This score difference is also δ1 (or δ2) mentioned in the aforementioned related embodiments. In this scenario, when the score difference is small enough, it is easy for the corresponding original content not to be included in the first result set. Therefore, in the embodiments of this application, the matching score obtained in the retrieval process can be reused to determine whether a subsequent secondary retrieval process is needed for the non-original content in the first result set without introducing additional overhead. That is, in other embodiments of this application, obtaining the second feature information of the entry includes: Determine the score difference between the matching score corresponding to the entry and the preset score threshold; When the score difference of the entry is less than a preset difference threshold, the second feature information of the entry is obtained. Specifically, taking music retrieval as an example, when the score difference between the matching score of the retrieved non-original song and the preset score threshold is small, for example, when it is less than a specific difference threshold, it indicates that the original song corresponding to the non-original song may not be included in the first result set because the matching score with the audio segment to be retrieved just does not exceed the preset matching score threshold. At this time, the second feature information of the non-original song can be obtained.

[0058] Furthermore, in another embodiment of this application, considering that in specific application scenarios, the content to be retrieved may be subject to various noise interferences, making it difficult to accurately extract its corresponding feature information, thus resulting in the presence of first-type entries in the first result set without correctly including their corresponding second-type entries, it is further possible to extract the attribute information of the content to be retrieved through analysis. That is, obtaining the second feature information of the entry includes: Extract the attribute information of the content to be retrieved; When the attribute information of the content to be retrieved meets the preset conditions, the second feature information of the entry is obtained.

[0059] In this context, the attribute information can correspond to different content depending on the type of search content. Typically, this attribute information affects the accuracy of feature extraction from the searched content. For example, in some embodiments, the attribute information can be noise components within the searched content. For instance, in music search, the attribute information could be noise components in the audio clip being searched, while in image search, it could be noise components in the image being searched, such as the number of noise points.

[0060] At this time, the fact that the attribute information of the content to be retrieved meets the preset conditions usually means that the attribute information of the content to be retrieved affects the accuracy of feature extraction of the content to be retrieved. For example, in one embodiment, when there is interference from the external environment, that is, when the audio segment to be retrieved includes a certain degree of noise, it may be considered that due to the large external noise interference, there is a greater probability that the first feature information cannot be accurately obtained, thus failing to accurately include the first result set of songs of the second type of entries. That is to say, in another embodiment of this application, obtaining the second feature information of the entry includes: Extract the noise components from the audio segment to be retrieved; When the proportion of noise components in the audio segment to be retrieved is higher than a preset proportion threshold, the second fingerprint features of non-original songs in the first result set are extracted.

[0061] Specifically, the noise components of the content to be retrieved, such as the noise components of an audio segment, can be extracted through relevant signal processing methods. For example, in one possible implementation, it can be determined based on the signal-to-noise ratio of the audio segment to be retrieved. Of course, based on actual needs, it is also feasible to indicate the proportion of noise components of the audio segment to be retrieved based on other indicators of the audio segment. This application does not limit this.

[0062] Of course, the solutions provided above are only some possible prerequisites for secondary retrieval of the first type of items in the first result set. In fact, based on the needs of actual scenarios, the prerequisites provided above can be further combined, such as by combining conditions of and or and, to more accurately realize the subsequent processing of specific first type of items. For example, if there is no second type of item corresponding to the first type of item in the first result set, or if the proportion of noise components in the content to be retrieved is higher than a preset proportion threshold and the difference between the matching score of the first type of item in the first result set and the score threshold of the content to be retrieved is less than a preset difference threshold, then feature extraction will be performed on the first type of item for subsequent secondary retrieval.

[0063] Of course, the solutions provided above are only some possible implementation schemes. In fact, more ways can be set to determine whether to process the first type of entries. For example, when the first result set does not contain any second type of entries, feature extraction can be performed on each first type of entry in the first result set for secondary retrieval. This application embodiment does not limit this.

[0064] S230, based on the second feature information of the entry, a search is performed in the second feature library to obtain a second result set, wherein the second feature library contains features of the second type of entry.

[0065] After obtaining specific non-original content, i.e., the second feature information of the first type of item, through the solution provided by any of the foregoing embodiments, in the embodiments of this application, a second search will be performed again in the second feature library containing the features of the second type of item, i.e., original content, based on the second feature information, thereby obtaining the second result set.

[0066] It should be noted that, in one embodiment of this application, the second feature library is typically a fingerprint library containing only features of the second type of entries. For example, taking song retrieval as an example, the second feature library here is also the original singer fingerprint library, which contains only the fingerprint features of original songs. In other words, the second feature library is usually a sub-database of the first feature library, and together with the feature library containing only non-original content, it forms the aforementioned first feature library. In particular, with the explosive growth of the industry and the increasing democratization of the creative threshold, the number of features in the feature library of original content is usually much lower than the number of features in the feature library of non-original content. For example, taking music platforms as an example, the number of features in the feature library of original songs is usually in the hundreds of thousands, while the number of features in the feature library of non-original songs, such as various cover or adapted songs, is usually in the hundreds of millions.

[0067] Specifically, similar to the scheme of searching and matching in the first feature library based on the first feature information, searching in the second feature library based on the second feature information can also be determined based on the matching score information between the feature information. For details, please refer to the relevant description above. The embodiments of this application will not be repeated here.

[0068] Specifically, all search entries in the second result set are of the second type. For example, in music retrieval, all search songs in the second result set are original versions. Considering that each first type entry usually corresponds to one second type entry, unlike filtering search entries by a decision threshold, in the embodiments of this application, the entry in the second feature library that is closest to the second feature information can be used as the search entry in the second result set. Of course, based on actual needs, the second result set can also include multiple search entries, and the embodiments of this application do not limit this.

[0069] S240, based on the first result set and the second result set, output the matching result sequence.

[0070] In the aforementioned method, after obtaining the first result set and the second result set containing only the second type of entries, the songs from the first and second result sets are combined, and after deduplication, a result sequence matching the content to be searched is output again in the order of the songs from the second type of entries and the songs from the first type of entries. That is, in one embodiment of this application, outputting the matching result sequence based on the first result set and the second result set includes: The entries in the first result set and the second result set are deduplicated and sorted to output a matching result sequence, wherein the sorting priority of the second type of entries in the matching result sequence is higher than that of the first type of entries.

[0071] Specifically, in one embodiment of this application, the output sequence of matching results can be output in the order of the second type of entries in the first result set, the second type of entries in the second result set, and the first type of entries in the first result set, that is, the second type of entries first, then the first type of entries, so as to ensure that users can obtain original content first, thereby increasing the exposure rate of popular original content on the platform.

[0072] Specifically, for ease of understanding, taking song retrieval as an example, the entries in the first result set and the second result set are deduplicated and sorted to output a sequence of matching results. In this sequence, the sorting priority of the second type of entries is higher than that of the first type of entries. Specifically: The searched songs in the first result set and the second result set are deduplicated and sorted to output a matching result sequence, wherein the original songs in the matching result sequence have a higher sorting priority than the non-original songs.

[0073] In addition, in some embodiments of this application, besides deduplicating and sorting the first result set and the second result set, the second type of entries in the second result set can be filtered again based on actual needs. For example, the first feature information of the content to be retrieved can be matched again with the feature information of the second type of entries (original content) in the second result set to determine whether the matching score is above a relatively low score threshold. This application does not limit this.

[0074] The retrieval method provided in this application first obtains the first fingerprint information of the content to be retrieved, then searches in a first feature library containing first-class and second-class entries based on the first feature information to obtain a first result set. Then, when an entry in the first result set is a first-class entry, the second feature information of that entry is obtained, and further searches are performed in a second feature library containing only second-class entries to obtain a second result set. Finally, the matching result sequence is output based on the first and second result sets in the order of second-class entries first and then first-class entries. This ensures that the output result sequence contains both first-class and second-class entries, that is, the output result sequence contains both original and non-original content, improving the user's retrieval experience and ensuring the exposure rate of popular original content on the platform.

[0075] Furthermore, in some embodiments of this application, for a type of multimedia content involving time series, such as music audio or video, the solution provided in this application can also simultaneously output timestamps in the first result set and the second result set that match the content to be searched, thereby allowing the user to start playing the multimedia content from the corresponding timestamp, making it easier for the user to determine the search results. For details, please refer to [link to relevant documentation]. Figure 4 , Figure 4 This application provides a flowchart illustrating the steps for extracting a timestamp, specifically related to... Figure 2 Compared to the flowchart shown, the main steps include the following: S410, determine the first timestamp in the first entry that matches the content to be retrieved based on the first feature information.

[0076] In one embodiment of this application, to facilitate understanding of the above content, taking audio as an example, after retrieving the corresponding first result set from the first feature library based on the first feature information, the first timestamp of each searched song in the first result set that matches the audio segment to be searched can be further obtained.

[0077] S420, obtain the second feature information corresponding to the first timestamp of the entry.

[0078] Building upon the aforementioned foundation, the second feature information of the first type of entries, such as non-original songs within the first timestamp, can be further extracted as a query vector in the second feature library, thereby preserving the timestamp information that matches the audio segment to be retrieved.

[0079] Specifically, in one embodiment of this application, the length of the second feature information is usually the same as the length of the first feature information. For example, to extract the first feature information of an audio segment with a duration of d seconds, it is usually necessary to start from the first timestamp of the non-original song and extract the second feature information of the music segment with a duration of d seconds (if the music duration allows it, otherwise it needs to be truncated at the end of the music).

[0080] S430, based on the second feature information of the entry, a search is performed in the second feature database to obtain a second result set and a second timestamp of each entry in the second result set that matches the second feature information.

[0081] In one embodiment of this application, during the retrieval process in the second feature database based on the second feature information, in addition to obtaining the second result set, since the second feature information is associated with the feature information of non-original songs within the corresponding first timestamp, the second timestamp of each song in the second result set that matches the second feature information can also be obtained based on the currently extracted second feature information. It can be understood that the second timestamp of each original song in the second result set also matches the audio segment to be retrieved.

[0082] Building upon the aforementioned foundation, in the subsequent process of outputting the matching song sequence for the audio segment to be retrieved, a corresponding playback identifier can be generated for each song in the matching song sequence based on the first and second timestamps. This playback identifier can be used to control the playback of the song from its corresponding timestamp, facilitating user identification. In other words, the method further includes: Based on the first timestamp and the second timestamp, a corresponding playback identifier is generated for each item in the result sequence. The playback identifier is used to control playback from the timestamp corresponding to each item in the result sequence.

[0083] Compared to the implementation scheme of constructing a relationship group between original content and non-original content, the solution provided by the embodiments of this application can, when outputting original content, especially a type of original content with time sequence information, such as audio or video, also determine the original content that matches the content to be searched, such as the timestamp information of the original song, thereby further improving the user's content search experience.

[0084] To clearly understand the solutions provided in the embodiments of this application, in some embodiments of this application, an audio retrieval method, i.e., song recognition, will be used as an example to describe a complete song retrieval process. Please refer to... Figure 5 , Figure 5This is a flowchart illustrating the implementation of a complete song retrieval method provided in this application embodiment. This scheme primarily uses song retrieval as an example. However, based on actual needs, the song retrieval method provided in this application can also be used to retrieve other multimedia content, such as videos or images. Specifically, it includes the following steps: (1) Full database fingerprint retrieval For the current recording segment, assuming a duration of d seconds, extract the fingerprint (i.e., the corresponding audio feature query) from it, and then search the entire fingerprint database. Here, the entire fingerprint database refers to all songs included in the fingerprint database, usually the entire music library, with a volume of tens of millions to hundreds of millions of songs.

[0085] After the search is complete, for each song retrieved, the following can be obtained: Unique identifier information for a song, such as its ID; The query is matched with the song's fingerprint and scored. The query corresponds to the timestamp position in the song; For example, if a song X is matched, the match score is 0.8 and the timestamp is t1.

[0086] If the currently matched song X is the original version, the process ends; otherwise, proceed to the next step.

[0087] (2) Export the fingerprint of the corresponding position of the hit song Export song X in [t1, min(t1+d, d)] X The fingerprint within the time interval is treated as a new query. Where d X Let be the total duration of song X. The "min" parameter is for boundary protection, meaning that the first d seconds of song X starting at position t1 are treated as a single recording to extract audio features as a new query. Specifically, since the fingerprint of song X already exists in the fingerprint database, the method provided in this application can directly export the fingerprint corresponding to the song's position, thus eliminating the need for additional fingerprint extraction calculations.

[0088] (3) Searching the original singer fingerprint database based on the new query: For the new query obtained in the previous step, search the original singer fingerprint database. The original singer fingerprint database refers to the fingerprint database composed of all original songs. The original singer fingerprint database and the non-original singer fingerprint database (composed of non-original songs) together can form the complete fingerprint database.

[0089] If no songs are found, the search ends.

[0090] If a song A with timestamp t2 is found, it means that position t1 of song X matches position t2 of song A. This also means that position 0 of the original query matches position t2 of song A. Finally, (X, t1) and (A, t2) are returned together as the two songs matched by the original query.

[0091] (4) Final sorting After the above operations, the original song A is included in the search results list, avoiding the situation where the original query could only retrieve song X but not the original song A. Finally, the search results list is reordered, for example, as described above: original songs are placed at the beginning of the results list, and non-original songs are placed at the end. This ensures that the original song A is displayed to the user first, and includes the timestamp information t2.

[0092] In addition, it should be noted that the song retrieval method provided in this application does not require additional high computational costs. For ease of understanding, it will be explained in detail below.

[0093] 1. When searching for cover songs in the original singer fingerprint database, since it is not necessary to search the entire fingerprint database, assuming the original singer fingerprint database has M songs and the entire fingerprint database has N songs, the search ratio is M / N. 2. For the original query, it will only proceed to this stage if it finds a song in the music library. Generally, during a single song recognition session, multiple queries will be sent to attempt song recognition until the song is successfully recognized or the timer expires. Assuming that on average, it takes K attempts to successfully recognize a song once, this means the probability of entering this stage is 1 / K. In summary, the additional computational cost provided in this application is equivalent to that of conventional music recognition:

[0094] Typically, M is much smaller than N, especially when only top-tier original songs are considered, for example, M = 500,000 and N = 100 million. K is generally between 3 and 5, for example, K = 3. Substituting into the above formula, we get: 500,000 / 100 million / 3 ≈ 0.002 In other words, the additional computational cost of the technical solution proposed in this application is only two-thousandths of that of routine song recognition, which is negligible. Furthermore, the technical solution provided in this application does not require offline maintenance of any "same-song group" relationships. While reusing existing song recognition processes as much as possible, it can also provide timestamp information, improving the user experience.

[0095] Based on the retrieval method provided in this application, this application also provides a retrieval device; for details, please refer to [link to details]. Figure 6 , Figure 6 A schematic diagram of a retrieval device provided in this application embodiment includes: The acquisition module 610 is used to acquire the first feature information of the content to be retrieved, and to perform a retrieval in the first feature library based on the first feature information to obtain a first result set. The first feature library contains fingerprints of the first type of entries and fingerprints of the second type of entries. Extraction module 620 is used to obtain the second feature information of an entry when the entry in the first result set is a first type of entry; The retrieval module 630 is used to perform a retrieval in a second feature library based on the second feature information of the entry to obtain a second result set, wherein the second feature library contains features of the second type of entries; The output module 640 is used to output a matching result sequence based on the first result set and the second result set.

[0096] In one embodiment of this application, the extraction module 620 is further configured to obtain a matching score indicating the matching similarity between the first feature information and each feature information in the first feature library; Feature information in the first feature library whose matching score exceeds a preset score threshold is identified as target feature information; Output the first result set based on the target feature information.

[0097] In one embodiment of this application, the extraction module 620 is further configured to determine the score difference between the matching score corresponding to the entry and the preset score threshold; When the score difference of the entry is less than a preset difference threshold, the second feature information of the entry is obtained.

[0098] In one embodiment of this application, the extraction module 620 is further configured to determine whether a second type of entry corresponding to the entry exists in the first result set; When there is no second type entry corresponding to the entry in the first result set, the second feature information of the entry is obtained.

[0099] In one embodiment of this application, the extraction module 620 is further configured to extract attribute information of the content to be retrieved; When the attribute information of the content to be retrieved meets the preset conditions, the second feature information of the entry is obtained.

[0100] In one embodiment of this application, the extraction module 620 is further configured to determine a first timestamp in the entry that matches the content to be retrieved based on the first feature information; and obtain second feature information corresponding to the first timestamp of the entry; The retrieval module 630 is further configured to perform a retrieval in a second feature database based on the second feature information of the entry, to obtain a second result set and a second timestamp of each entry in the second result set that matches the second feature information.

[0101] In one embodiment of this application, the output module 640 is further configured to generate a corresponding playback identifier for each entry in the result sequence based on the first timestamp and the second timestamp, the playback identifier being used to control playback from the timestamp corresponding to each entry in the result sequence.

[0102] In one embodiment of this application, the output module 640 is further configured to sort the entries of the first result set and the second result set after deduplication, and output a matching result sequence, wherein the sorting priority of the second type of entries in the matching result sequence is higher than the sorting priority of the first type of entries.

[0103] The retrieval device provided in this application, after obtaining the first fingerprint information of the content to be retrieved, first searches in a first feature library containing first-class and second-class entries based on the first feature information to obtain a first result set. Then, when an entry in the first result set is a first-class entry, the second feature information of that entry is obtained, and further searches are performed in a second feature library containing only second-class entries to obtain a second result set. Then, based on the first result set and the second result set, a matching result sequence is output in the order of second-class entries first and then first-class entries. This ensures that the output result sequence contains both first-class and second-class entries, that is, the output result sequence contains both original and non-original content, thereby improving the user's retrieval experience and ensuring the exposure rate of popular original content on the platform.

[0104] In some embodiments of this application, the retrieval method can be implemented as a computer program, which can be implemented in, for example... Figure 7 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the retrieval device, for example, Figure 6 The diagram shows an acquisition module 610, an extraction module 620, a retrieval module 630, and an output module 640. The computer program comprised of these modules causes the processor to execute the steps of the retrieval methods in the various embodiments of this application described in this specification.

[0105] For example, Figure 7 The computer equipment shown can be used as follows Figure 6The acquisition module 610 in the retrieval device shown executes step S210. The computer device can execute step S220 via the extraction module 620. The computer device can execute step S230 via the retrieval module 630. The computer device can execute step S240 via the output module 640. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements a retrieval method.

[0106] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0107] In some embodiments of this application, a computer device is provided, including one or more processors; memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to perform the following steps: The first feature information of the content to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library contains features of the first type of entries and features of the second type of entries. When an entry in the first result set is a first-type entry, the second feature information of the entry is obtained; Based on the second feature information of the entry, a second feature library is searched to obtain a second result set. The second feature library contains features of the second type of entry. Based on the first result set and the second result set, output the matching result sequence.

[0108] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to perform the following steps: The first feature information of the content to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library contains features of the first type of entries and features of the second type of entries. When an entry in the first result set is a first-type entry, the second feature information of the entry is obtained; Based on the second feature information of the entry, a second feature library is searched to obtain a second result set. The second feature library contains features of the second type of entry. Based on the first result set and the second result set, output the matching result sequence.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0110] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The song retrieval method, apparatus, device, and storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A retrieval method, characterized in that, include: The first feature information of the content to be retrieved is obtained, and a search is performed in the first feature library based on the first feature information to obtain a first result set. The first feature library contains features of the first type of entries and features of the second type of entries. When an entry in the first result set is a first-type entry, the second feature information of the entry is obtained; Based on the second feature information of the entry, a second feature library is searched to obtain a second result set. The second feature library contains features of the second type of entry. Based on the first result set and the second result set, output the matching result sequence.

2. The method according to claim 1, characterized in that, The step of retrieving data from the first feature database based on the first feature information to obtain a first result set includes: Obtain a matching score that indicates the similarity between the first feature information and each feature information in the first feature library; Feature information in the first feature library whose matching score exceeds a preset score threshold is identified as target feature information; Output the first result set based on the target feature information.

3. The method according to claim 2, characterized in that, The content to be retrieved includes audio segments, the feature information includes multiple energy extreme points in the audio spectrogram, and the matching score is determined based on the number of matching energy extreme points between the first feature information and each feature information in the first feature library at the corresponding time. Alternatively, the feature information may include feature vectors output by a neural network model, and the matching score may be determined based on the vector similarity between the first feature information and each feature information in the first feature library.

4. The method according to claim 2, characterized in that, The process of obtaining the second feature information of the entry includes: Determine the score difference between the matching score corresponding to the entry and the preset score threshold; When the score difference of the entry is less than a preset difference threshold, the second feature information of the entry is obtained.

5. The method according to claim 1, characterized in that, The process of obtaining the second feature information of the entry includes: Determine whether a second type of entry corresponding to the stated entry exists in the first result set; When there is no second type entry corresponding to the entry in the first result set, the second feature information of the entry is obtained.

6. The method according to claim 1, characterized in that, The method further includes: Based on the first feature information, determine the first timestamp in the entry that matches the content to be retrieved; The process of obtaining the second feature information of the entry includes: Obtain the second feature information corresponding to the first timestamp of the entry; The second result set is obtained by searching the second feature database based on the second feature information of the entry, including: Based on the second feature information of the entry, a search is performed in the second feature database to obtain a second result set and a second timestamp of each entry in the second result set that matches the second feature information.

7. The method according to any one of claims 1 to 6, characterized in that, The step of outputting a matching result sequence based on the first result set and the second result set includes: The entries in the first result set and the second result set are deduplicated and sorted to output a matching result sequence, wherein the sorting priority of the second type of entries in the matching result sequence is higher than that of the first type of entries.

8. A retrieval device, characterized in that, include: The acquisition module is used to acquire the first feature information of the content to be retrieved, and to perform a retrieval in the first feature library based on the first feature information to obtain a first result set. The first feature library contains fingerprints of the first type of entries and fingerprints of the second type of entries. The extraction module is used to obtain the second feature information of the entry when the entry in the first result set is a first type of entry; The retrieval module is used to perform a retrieval in a second feature library based on the second feature information of the entry, and obtain a second result set, wherein the second feature library contains features of the second type of entries; The output module is used to output a sequence of matching results based on the first result set and the second result set.

9. A computer device, characterized in that, The computer device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the retrieval method as described in any one of claims 1 to 7.

Images