An audio information processing method and device

Abstract

This invention provides an audio information processing method and apparatus. The method includes: retrieving a first query result corresponding to the query audio from an audio-visual database based on entity objects extracted from a user-initiated query audio and search conditions; selecting target text segments with a relevance score greater than or equal to a first threshold from a pre-constructed text vector library based on the query audio, and using the audio-visual information extracted from the target text segments as a second query result; constructing a user preference vector, and determining the audio-visual query result corresponding to the query audio from the set of the first and second query results based on the preference vector. This implementation improves the accuracy and efficiency of voice-based audio-visual content retrieval on devices such as smart TVs by combining query results returned under both precise retrieval and fuzzy recall modes and obtaining the final query result based on user preferences.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to an audio information processing method and apparatus. Background Technology

[0002] In applications such as smart TVs or set-top boxes, users typically use voice search to find desired audio-visual content. However, because user voice input is generally colloquial, it may contain omissions or unclear references. Forcibly parsing such requests results in significant network latency, leading to low efficiency in retrieving search results. Furthermore, existing voice search methods often rely on internet search architectures, which cannot accurately handle long-tail content in the audio-visual field, resulting in low accuracy of the final retrieved search results, failing to meet user expectations. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide at least one audio information processing method, apparatus, electronic device, and storage medium, which can improve the accuracy and efficiency of audio-visual works retrieval based on voice on devices such as smart TVs by combining the query results returned under both precise retrieval and fuzzy recall modes and obtaining the final query result according to user preferences.

[0004] In a first aspect, embodiments of the present invention provide an audio information processing method, comprising: Based on the entity objects extracted from the user-initiated audio query and the search conditions, the first query result corresponding to the audio query is retrieved from the audio-visual database. Based on the query audio, target text segments with a relevance score greater than or equal to the query audio are selected from a pre-built text vector library, and the audio-visual information extracted from the target text segments is used as the second query result. Construct a user preference vector, and based on the preference vector, determine the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result.

[0005] Optionally, before retrieving the first query result corresponding to the query audio from the audio-visual database based on the entity object extracted from the user-initiated query audio and the search conditions, the process further includes: Collect user-input voice data, perform voice activity detection on the voice data, and extract valid segments from the voice data as query audio; Determine the type of instruction to query the audio; In response to the query audio being a complex command type, the query audio is converted into text format to extract entity objects and search conditions from the query audio, and to determine the confidence level of the entity objects and search conditions; whereby the confidence level is used to represent the clarity of the description of entity objects and search conditions.

[0006] Optionally, based on the entity objects extracted from the user-initiated audio query and the search criteria, a first query result corresponding to the query audio is retrieved from the audio-visual database, including: In response to the confidence level of the entity object and the search conditions being greater than or equal to a pre-set second threshold, a query vector is constructed based on the entity object and the search conditions, and the first query result corresponding to the query audio is retrieved from the audio-visual database based on the query vector; In response to the fact that the confidence level of the entity object and the search conditions is less than or equal to a pre-set third threshold and less than a second threshold, a query vector is constructed based on the entity object and the search conditions, and a fuzzy vector is constructed based on the entity object and the search conditions, so as to retrieve the first query result corresponding to the query audio from the audio-visual database according to the query vector and the fuzzy vector.

[0007] Optionally, target text segments with a relevance score greater than or equal to a first threshold are selected from a pre-built text vector library based on the query audio, and the audio-visual information extracted from the target text segments is used as the second query result, including: The query audio is converted into query text, and a similarity search is performed on the query text in a pre-built text vector library to obtain candidate text fragments corresponding to the query text; the text vector library is constructed based on the plot synopsis of the film and television works, the evaluation text of the film and television works, and the subtitle text of the film and television works. The candidate text fragments are input into a pre-trained re-ranking model, which calculates the relevance score between the candidate text fragments and the query text, and outputs the candidate text fragments whose relevance score with the query text is greater than or equal to a first threshold as the target text fragments. Extract audio-visual information from the target text fragment as the second query result; the audio-visual information includes the title of the audio-visual work and the names of the creators of the audio-visual work.

[0008] Optionally, construct the user's preference vector, including: Obtain the user's interaction history within the current session and identify the audio-visual works in the interaction history; The feature vectors of the audio-visual works in the interaction records are extracted in multiple dimensions, and the feature vectors in multiple dimensions are concatenated to obtain the audio-visual work vector corresponding to each audio-visual work; among them, the multiple dimensions include actor dimension, director dimension, genre dimension and era dimension. Based on the user's behavior type towards each audio-visual work, determine the weight parameters of the audio-visual work vector corresponding to each audio-visual work, and construct the user's preference vector based on the weight parameters of the audio-visual work vector corresponding to each audio-visual work.

[0009] Optionally, based on the preference vector, the audio-visual query results corresponding to the query audio are determined from the set of the first query results and the second query results, including: The first query result and the second query result are merged and deduplicated to obtain the audio and video query results, and then sorted according to the preference vector. The sorted audio and video query results are returned to the user's terminal device as the response data for the queried audio.

[0010] Secondly, embodiments of the present invention provide an audio information processing apparatus, comprising: The retrieval module is used to retrieve the first query result corresponding to the query audio from the audio-visual database based on the entity objects extracted from the user-initiated query audio and the retrieval conditions. The filtering module is used to filter target text segments from a pre-built text vector library based on the query audio, and to use the audio-visual information extracted from the target text segments as the second query result. The determination module is used to construct the user's preference vector, and based on the preference vector, determine the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result.

[0011] Thirdly, embodiments of the present invention also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the first aspect or any optional implementation of the first aspect are performed.

[0012] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the first aspect or any optional implementation thereof.

[0013] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0014] Any of the above aspects or any implementation thereof can perform a structured search from an audio-visual database based on entity objects and search conditions extracted from the user-initiated query audio, to obtain a first query result. Since the entity objects and search conditions accurately represent the user's explicitly expressed query needs, the first query result can achieve precise recall of target audio-visual works. Simultaneously, target text segments with a relevance score greater than or equal to a first threshold are selected from a pre-constructed text vector library based on the query audio, and the audio-visual information extracted from the target text segments is used as a second query result. Since the text vector library uses semantic vector representation to store audio-visual related knowledge, the second query result can overcome keyword matching limitations and perform associated recall of user colloquial expressions, vague descriptions, alternative references, and incomplete query content, thereby supplementing relevant audio-visual works that may have been missed in the precise retrieval. Then, a user preference vector can be constructed; wherein, the user preference vector can reflect the user's interests and preferences in multiple dimensions such as actor, director, genre, and era. Furthermore, by calculating the matching degree between the audio-visual work vectors corresponding to candidate audio-visual works and the user's preference vector, the candidate audio-visual work set is sorted, and the audio-visual query results corresponding to the query audio are determined. By combining the precise retrieval capability based on entity objects and retrieval conditions with the semantic recall capability based on text vector databases, the accuracy of precise retrieval is maintained while also improving the ability to mine potential targets in fuzzy recall scenarios. By combining the user's preference vector to personalize and sort the obtained query results, the final returned audio-visual query results are more in line with user interests, thereby improving the accuracy and efficiency of audio-visual work queries based on voice on smart TVs and other devices.

[0015] The beneficial effects of the aforementioned audio information processing device, electronic device, and storage medium are described in the description of the aforementioned audio information processing method, and will not be repeated here. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to the present invention and, together with the specification, serve to explain the technical solutions of the present invention. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart of an audio information processing method provided by an embodiment of the present invention is shown; Figure 2A flowchart illustrating an audio information processing method provided by an embodiment of the present invention is shown. Figure 3 A schematic diagram of an audio information processing device provided in an embodiment of the present invention is shown; Figure 4 An exemplary system architecture in which embodiments of the present invention can be applied is shown; Figure 5 A schematic diagram of the structure of a computer system used to implement embodiments of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0020] The above problems and solutions are the result of the inventor's practice and careful research. The discovery process of the above problems and the solutions proposed for the above problems should be considered as the inventor's contribution to the invention.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] To facilitate understanding of this embodiment, a detailed description of the audio information processing method disclosed in this invention will be provided first. The execution entity of the audio information processing method provided in this invention is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this audio information processing method can be implemented by a processor calling computer-readable instructions stored in memory.

[0023] See Figure 1 The diagram shows a flowchart of an audio information processing method provided in an embodiment of the present invention. The method includes steps S101 to S103, wherein: S101: Based on the entity objects extracted from the user-initiated query audio and the search conditions, retrieve the first query result corresponding to the query audio from the audio-visual database.

[0024] In this embodiment of the invention, taking searching for desired audio-visual works via voice input on a smart TV as an example, the user can input the query audio by pressing and holding the voice input button on the remote control; or the user can input the query audio via voice after summoning the smart TV assistant. This embodiment of the invention does not specifically limit the method by which the user inputs the query audio, and in practical applications, the input can be made according to the device's functions and the user's needs. It should be noted that the method by which the user initiates a query request for audio-visual works is not limited to voice input. For users or devices that cannot use voice input, a query request can also be initiated via text input on any device capable of querying audio-visual works. The subsequent processing steps are similar to those for querying via voice input. The following embodiments are only used as an example of querying audio-visual works on a smart TV using voice input and do not constitute an improper limitation of the invention.

[0025] In this embodiment of the invention, before retrieving the first query result corresponding to the query audio from the audio-visual database based on the entity objects and search conditions extracted from the user-initiated query audio, the method further includes: collecting user-inputted voice data, performing voice activity detection on the voice data to extract valid segments from the voice data as query audio; determining the instruction type of the query audio; and, in response to the query audio being a complex instruction type, converting the query audio into text form to extract the entity objects and search conditions from the query audio, and determining the confidence level of the entity objects and search conditions; wherein, the confidence level is used to represent the clarity of the description of the entity objects and search conditions.

[0026] In specific implementation, such as Figure 2 As shown, in a smart TV voice search scenario, the user's input voice data can be collected first through the microphone array on the user's end. Since the raw voice data typically contains environmental noise, user pauses, and invalid silence segments, Voice Activity Detection (VAD) needs to be performed on the voice data. By detecting the start and end positions of the speech, valid segments containing the user's effective expression are extracted from the continuous speech stream, and these valid segments are identified as the query audio. For example, when a user presses the voice button on the remote control and says, "I want to watch Stephen Chow's comedy movies," VAD can automatically remove silence and background noise before and after the speech, retaining only the actual speech content, thereby reducing the amount of data processed subsequently and improving recognition accuracy.

[0027] like Figure 2As shown, after obtaining the query audio, the command type can be determined first. Specifically, a command classification model can be constructed based on features such as sentence length, number of entities, semantic domain, and action type, and then matched with a pre-configured high-frequency command template library. Sentence length reflects the complexity of the user's expression; the number of entities reflects whether the query involves specific audio-visual objects; the semantic domain distinguishes between system control operations and audio-visual retrieval operations; and the action type identifies different operation intentions such as play, pause, fast forward, and search. For example, when the query audio contains content such as "pause playback," "turn up the volume," or "next episode," its character count is small, it does not contain entity objects, and the action words match the predefined system control template. In this case, it can be determined as a simple command type, and the terminal device can directly call the local control module to execute the corresponding operation without uploading it to the execution entity of this embodiment for processing, thereby reducing response latency and network resource consumption. Furthermore, the possibility of incorrect instruction classification can be reduced by calculating the confidence level of the classification results. For example, local execution can only be allowed when the confidence level of the classification result is greater than a preset threshold and a predefined template is successfully matched. If the classification result is uncertain, it is uniformly processed as a complex instruction and uploaded to the execution entity of this embodiment to avoid erroneous execution. For query audio containing multiple conditional constraints or semantic reasoning requirements, such as "Find me Stephen Chow's comedy movies from the 1990s" or "Are there any high-scoring suspense movies suitable for watching this weekend," it can be identified as a complex instruction type. Then, the query audio can be converted into text form, and the recognized text can be input into a pre-trained large language model for intent parsing to output the parsing result of the query audio. The parsing result includes intent type, entity object, search conditions, and reconstructed query statement; among them, entity object can be used to represent core search objects such as actors, directors, and work titles, and search conditions can be used to represent filtering constraints such as genre, era, and rating. At the same time, corresponding confidence levels can be generated for entity objects and search conditions. The confidence level here can be used to represent the clarity of the description and the reliability of the identification of the entity object and the search conditions. For example, when a user explicitly says "Actor A's comedy movie", the confidence level of "Actor A" can reach 0.98. However, when a user says "The movie made by the person who played Spider-Man", since it may correspond to multiple actors, the confidence level of the relevant entity object is relatively low.

[0028] For example, a user inputs "I want to watch high-scoring comedy movies from the 1990s starring actor A, not horror movies" via voice input. After speech recognition, the corresponding query text is obtained. The query text can then be input into a pre-trained large language model for intent parsing. The large language model identifies and extracts the user's intent, entity objects, and search conditions from the query text, obtaining the parsing results. The intent type can be represented by `intent_type`. When the user has a clear search goal, it can be identified as "exact_search"; when the user only expresses interests without specifying a specific object, it can be identified as "fuzzy_recommend"; when the user wants to obtain knowledge-based Q&A content, it can be identified as "QA". In the above example, since the user wants to find audio-visual works that meet specific conditions, the intent type can be identified as "exact_search". Entities extracted from the query text can be represented by "entities". For example, from "Actor A's highly rated comedy movies from the 1990s", the entity "Actor A" is identified and classified as an actor, with a confidence score of 0.98, indicating that the system's identification of this entity is highly reliable. Search criteria extracted from the query text can be represented by "filters". For example, the genre criterion is identified as "comedy", the era criterion as "1990s", and the exclude criterion as "horror", thus forming structured search criteria. The rewritten query, obtained after semantic completion and query reconstruction of the original query text, can be represented by "rewritten_query". For example, generating "Actor A's highly rated comedy movies from the 1990s" transforms the originally colloquial expression into a standardized search statement suitable for machine execution. Through the above processing, the original query text "I want to watch high-scoring comedy movies from the 1990s starring actor A, not horror movies" can be converted into structured content in the following format: "intent_type: "exact_search"; entities: [{"name": "actor A", "type": "actor", "confidence": 0.98}]; filters: {"genre": "comedy", "era": "1990s", "exclude_genre": "horror"}; rewritten_query: "Actor A 1990s comedy movies high-scoring"". It should be noted that the above target format is only illustrative of one implementation method in this invention and does not constitute an improper limitation of the invention. In practical applications, the target format can be set according to actual needs and circumstances, and this invention does not impose specific limitations in this regard.

[0029] In this embodiment of the invention, when performing a retrieval based on entity objects extracted from a user-initiated query audio and retrieval conditions, not all entity objects and retrieval conditions are directly matched in a uniform manner. Instead, the retrieval strategy is dynamically adjusted according to the confidence level corresponding to the entity objects and retrieval conditions. Specifically, based on the entity objects extracted from the user-initiated query audio and retrieval conditions, a first query result corresponding to the query audio is retrieved from the audio-visual database. This includes: in response to the confidence level of the entity objects and retrieval conditions being greater than or equal to a pre-set second threshold, constructing a query vector based on the entity objects and retrieval conditions, and retrieving the first query result corresponding to the query audio from the audio-visual database based on the query vector; and in response to the confidence level of the entity objects and retrieval conditions being less than or equal to a pre-set third threshold and less than the second threshold, constructing a query vector based on the entity objects and retrieval conditions, and constructing a fuzzy vector based on the entity objects and retrieval conditions, so as to retrieve the first query result corresponding to the query audio from the audio-visual database based on the query vector and the fuzzy vector.

[0030] In practical implementation, when the confidence level of the entity object and the search criteria is greater than or equal to a pre-set second threshold, it indicates that the system can accurately understand the user's query intent. At this point, a query vector can be constructed based on the entity object and the search criteria, and a precise search can be performed from the audio-visual database using the query vector. For example, if a user queries "I want to watch a high-scoring comedy movie from the 1990s with actor A," after intent parsing, the entity object "actor A" with a confidence level of 0.98 is obtained, along with the search criteria "comedy," "1990s," and "high score." Since the confidence levels of each field are all higher than the second threshold, a query vector can be directly constructed based on the entity object and the search criteria. This query vector can then be used to search the audio-visual database for audio-visual works that simultaneously meet the requirements of actor, genre, era, and rating, thus obtaining a highly accurate first query result. In this process, the high-confidence entity object and the search criteria act as strong constraints in the search, effectively narrowing the search scope and improving search efficiency and result accuracy.

[0031] When the confidence level of the entity object and the search criteria is less than the second threshold but greater than or equal to the pre-set third threshold, it indicates that the system has identified the corresponding entity object or search criteria, but the identification result has a certain degree of uncertainty. In this case, in addition to constructing a query vector based on the entity object and search criteria, a fuzzy vector can be further constructed based on the entity object and search criteria, and the retrieval can be performed by combining the query vector and the fuzzy vector. The query vector can be used to retain the core search intent in the user's original expression, while the fuzzy vector can be used to represent extended information that is semantically similar, literally similar, or has recognition errors to the entity object or search criteria. For example, if a user issues a voice query "Find me a movie starring Zhang San," the voice recognition result may identify "Zhang San" as "Zhang San." Although the intent parsing module identifies the corresponding actor entity, its confidence level is only 0.78. In this case, a query vector can be constructed based on "Zhang San," and a fuzzy vector can be constructed based on the actor entity, encoding semantic information related to "Zhang San," "B-series movie actor," etc., into the fuzzy vector. When performing a search, the query vector can be used to match audio-visual works related to the original entity object, while the fuzzy vector can be used to supplement and recall results that may have been missed due to recognition errors, thereby avoiding search failures caused by issues such as users' non-standard pronunciation or speech recognition errors.

[0032] In this embodiment of the invention, information such as title matching degree, work rating, and popularity can also be used as scoring factors for sorting. During the retrieval process, the audio-visual database can first complete the accurate recall of the first query results based on the query vector, and then use the built-in scoring algorithm to score the audio-visual works included in the first query results. For example, when multiple audio-visual works meet the conditions such as "Actor A", "comedy", and "1990s", audio-visual works with higher ratings, higher popularity, or higher title matching degree with the query content can receive higher ratings and be prioritized at the top. Thus, while ensuring the accuracy of the retrieval results, the audio-visual works in the first query results can be initially sorted.

[0033] Furthermore, when the confidence level of an entity object and a search condition falls below the third threshold, it indicates that the system cannot reliably determine the true meaning of the corresponding entity object or search condition. Continuing to use exact matching in this case can easily result in zero results or a large number of incorrect results. Therefore, the constraint strength of entity objects and search conditions on the search process can be reduced by incorporating the corresponding text description as semantic extension information into a fuzzy vector and using the fuzzy vector to perform semantic-level retrieval. For example, if a user queries "the absurdist comedy that the actor from the B-series movies later starred in," the system may identify the entity clue "actor from the B-series movies," but it cannot accurately determine whether the user is referring to actor B, actor C, or actor D who starred in the B-series movies. In this case, the confidence level of the corresponding entity object might only be 0.65. Therefore, this entity object can no longer be used as a strict search condition. Instead, a fuzzy vector can be constructed based on descriptions such as "actor from the B-series movies" and "absurdist comedy," and related works can be retrieved from the audiovisual database using semantic similarity, thereby expanding the recall scope and improving adaptability to fuzzy query scenarios. This not only maintains accurate and efficient retrieval capabilities in high-confidence scenarios but also enhances recall and fault tolerance in medium- and low-confidence scenarios, thereby achieving a balance between search accuracy and recall robustness and improving the overall performance of smart TV voice and video retrieval. It should be noted that the thresholds mentioned above can be set according to actual needs and circumstances, and this embodiment of the invention does not impose specific limitations on them.

[0034] S102: Based on the query audio, select target text segments from the pre-built text vector library that have a relevance score greater than or equal to the first threshold, and use the audio-visual information extracted from the target text segments as the second query result.

[0035] In this embodiment of the invention, to improve the retrieval capability of long-tail audio-visual content in smart TV scenarios, a text vector library can be pre-constructed. Specifically, unstructured text data related to audio-visual works can be collected as the original data source. This unstructured text data can include movie synopses, professional film reviews, user comments, and subtitle text, etc. Then, the data can be cleaned by denoising, format standardization, and duplicate content filtering. The cleaned text is then sliced ​​according to a preset length to obtain multiple text fragments, each retaining relatively complete semantic information. Further, each text fragment can be input into a text vector generation model for feature extraction to obtain text vectors corresponding to each text fragment; this embodiment of the invention does not specifically limit the text vector generation model. Finally, the text fragments and their corresponding text vectors can be associated and stored in a vector database to form a text vector library. This transforms a large amount of unstructured audio-visual knowledge into searchable vector representations, providing a data foundation for subsequent retrieval and recall, thereby improving the retrieval capability for niche films, long-tail content, and ambiguous descriptions.

[0036] In this embodiment of the invention, target text segments with a relevance score greater than or equal to a first threshold are selected from a pre-constructed text vector library based on the query audio, and the audio-visual information extracted from the target text segments is used as the second query result. This includes: converting the query audio into query text; performing a similarity search on the pre-constructed text vector library based on the query text to obtain candidate text segments corresponding to the query text; wherein the text vector library is constructed based on film and television work synopsis, film and television work review text, and film and television work subtitle text; inputting the candidate text segments into a pre-trained re-ranking model, so that the re-ranking model calculates the relevance score between the candidate text segments and the query text, and outputs candidate text segments with a relevance score greater than or equal to the first threshold as target text segments; and extracting audio-visual information from the target text segments as the second query result; the audio-visual information includes the title of the audio-visual work and the names of the creators of the audio-visual work.

[0037] In specific implementations, the query text may first be converted into a vector format to perform similarity matching in a text vector library, so as to screen out candidate text segments that are semantically closest to the query text. For example, when a user utters the query audio "that movie about a teenager surviving at sea with a tiger", although the name of the film and television work does not appear directly in the query text, candidate text segments containing plot descriptions such as "a teenager drifting and surviving at sea with a tiger" can still be recalled from the text vector library through vector similarity retrieval. The obtained candidate text segments may then be input into a pre-trained reranking model, which further calculates the correlation score between the candidate text segments and the query text. In this way, a more refined correlation assessment is performed by combining the semantic association between the query text and the candidate text segments, so as to improve the accuracy of retrieval results. Thereafter, candidate text segments whose correlation score is greater than or equal to a first threshold may be determined as target text segments, and text segments with a low degree of association with the query intent are filtered out. For example, when the correlation score between a certain candidate text segment and the query text reaches 0.85 and the first threshold is set to 0.8, the candidate text segment is retained as a target text segment. Further, audio and video information may be extracted from the target text segments as a second query result, where the audio and video information may include the name of the audio and video work and the names of the creators of the audio and video work. For example, from the target text segment "Film B tells the story of a teenager surviving together with a Bengal tiger at sea, directed by Director C", the audio and video work name "Film B" and the creator name "Director C" can be extracted and used as the second query result.

[0038] S103: Construct a preference vector of the user, and determine an audio and video query result corresponding to the query audio from the set of the first query result and the second query result based on the preference vector.

[0039] Smart televisions are usually household shared devices, with mixed user profiles. In the prior art, simple weighting is mostly performed based on single-dimensional historical records, which cannot effectively separate and quantify the multi-dimensional preferences of different family members. For example, the mother in the family likes to watch Hong Kong films from the 1990s but hates horror films, while the child in the family likes to watch horror films but hates romance films; in the prior art, the retrieval results of audio and video works are usually inaccurate due to mixed user profiles. In addition, since most models for managing user profiles are trained in offline batches, they cannot respond in real time to short-term changes in user interests, which results in the system being unable to perceive changes in user interests in a timely manner, for example, the user likes to watch variety shows on workdays and prefers to watch movies on rest days.

[0040] In this embodiment of the invention, a user preference vector can be constructed based on the user's interaction records within the current session, thereby separating user profiles for different family roles and timely sensing changes in user interests. Specifically, constructing the user preference vector includes: obtaining the user's interaction records within the current session and identifying the audio-visual works in the interaction records; extracting feature vectors of the audio-visual works in the interaction records under multiple dimensions, concatenating the feature vectors under multiple dimensions to obtain audio-visual work vectors corresponding to each audio-visual work; wherein, the multiple dimensions include actor dimension, director dimension, genre dimension, and era dimension; determining the weight parameters of the audio-visual work vectors corresponding to each audio-visual work based on the user's behavior type towards each audio-visual work, and constructing the user preference vector based on the weight parameters of the audio-visual work vectors corresponding to each audio-visual work.

[0041] In practical implementation, to accurately represent a user's interests and preferences in the current usage scenario, a user preference vector can be constructed based on the user's interaction records generated within the current session period. The current session period can refer to the complete period from the last time the device was turned on to the present, or it can be a pre-set period length, such as the last 3 hours. Specifically, the user's interaction records generated within the current session period can be obtained first, and the corresponding audio-visual works can be identified. These interaction records can include behaviors such as searching, playing, watching in their entirety, and saving. Then, feature vectors under multiple dimensions can be extracted from each audio-visual work, and these feature vectors under multiple dimensions can be concatenated to obtain the audio-visual work vector corresponding to each audio-visual work. Among these, the actor dimension can be used to represent the actor information involved in the audio-visual work, the director dimension can be used to represent the director information, the genre dimension can be used to represent content type information such as comedy, action, and suspense, and the era dimension can be used to represent the time characteristics or creation time characteristics of the audio-visual work. For example, when a user watches a 1990s comedy film starring actor A, the corresponding actor feature vector, director feature vector, genre feature vector, and era feature vector can be extracted separately, and then concatenated to form the corresponding audiovisual work vector. Furthermore, weight parameters can be determined based on the user's behavior type towards each audiovisual work; different behavior types correspond to different weight parameters to reflect the degree of influence of different behaviors on user interests and preferences. For example, search behavior usually indicates that the user has initial interest in the relevant content, so it can be assigned a lower weight parameter; watching the entire film indicates that the user has a strong interest in the relevant content, so it can be assigned a higher weight parameter; saving the film usually reflects that the user has a continued willingness to pay attention, so it can be assigned an even higher weight parameter. Then, the audiovisual work vectors and corresponding weight parameters can be used to accumulate and update the user's historical interest information, thereby generating the user's preference vector. Since the user's preference vector integrates the feature information of multiple audiovisual works in the current session period across the actor, director, genre, and era dimensions, it can comprehensively reflect the user's recent interests and preferences. For example, when a user watches multiple comedy films starring actor A consecutively, the feature weights corresponding to the actor and genre dimensions in their preference vector will gradually increase, enabling the system to identify the user's preference for actor A and comedy content. It should be noted that the dimensions used to construct the user preference vector are merely illustrative examples of feasible implementation methods in this invention and do not constitute an undue limitation on the invention. In practical applications, these dimensions can be arbitrarily combined or added / removed according to actual needs and circumstances. For example, information such as region, language, production company, and role dimensions can also be introduced to construct the user vector. This invention does not impose specific limitations in this regard, but rather focuses on achieving its intended function.

[0042] In this embodiment of the invention, determining the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result based on the preference vector includes: merging and deduplicating the first query result and the second query result to obtain the audio-visual query result, and sorting the audio-visual query result according to the preference vector; returning the sorted audio-visual query result as the response data of the query audio to the user's terminal device.

[0043] In specific implementation, such as Figure 2 As shown, after obtaining the first and second query results, the two types of query results can be merged to form an initial candidate set. Since the first query result focuses on structured precise matching, while the second query result focuses on semantic expansion and recall, there may be duplicate audio-visual works or results from the same source at different granularities. To avoid duplication interfering with the subsequent ranking process, deduplication can be performed on the merged candidate set. Duplicate audio-visual works are merged into a single candidate based on the unique identifier or name of the audio-visual work. On this basis, a user preference vector constructed based on the current session period can be introduced. By calculating the matching degree between the audio-visual work vector and the user preference vector, the correlation strength between each audio-visual work and the user's interests is obtained. This matching degree is then used as the ranking criterion to re-rank the candidate set, prioritizing audio-visual works that better match the user's recent interests in the ranking results. For example, when a user frequently watches comedy films related to actor A during the current session period, their preference vector will show strong weights in both the actor and genre dimensions. Therefore, comedy films and television shows related to actor A can be ranked higher in the ranking process. Finally, the sorted audio-visual query results can be returned to the user's terminal device as response data for the query audio, enabling the user to quickly obtain a list of audio-visual works that match their voice query intent and personalized preferences on the smart TV interface, thereby improving the overall search experience and response efficiency.

[0044] According to a second aspect of the embodiments of the present invention, such as Figure 3 As shown, an audio information processing device 300 is provided, comprising: The retrieval module 301 is used to retrieve the first query result corresponding to the query audio from the audio-visual database based on the entity object extracted from the query audio initiated by the user and the retrieval conditions. The filtering module 302 is used to filter out target text segments with a relevance score greater than or equal to a first threshold from a pre-built text vector library based on the query audio, and use the audio-visual information extracted from the target text segments as the second query result. The determination module 303 is used to construct the user's preference vector and determine the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result based on the preference vector.

[0045] Optionally, the retrieval module 301 is also used for: Collect user-input voice data, perform voice activity detection on the voice data, and extract valid segments from the voice data as query audio; Determine the type of instruction to query the audio; In response to the query audio being a complex command type, the query audio is converted into text format to extract entity objects and search conditions from the query audio, and to determine the confidence level of the entity objects and search conditions; whereby the confidence level is used to represent the clarity of the description of entity objects and search conditions.

[0046] Optionally, the retrieval module 301 is specifically used for: In response to the confidence level of the entity object and the search conditions being greater than or equal to a pre-set second threshold, a query vector is constructed based on the entity object and the search conditions, and the first query result corresponding to the query audio is retrieved from the audio-visual database based on the query vector; In response to the fact that the confidence level of the entity object and the search conditions is less than or equal to a pre-set third threshold and less than a second threshold, a query vector is constructed based on the entity object and the search conditions, and a fuzzy vector is constructed based on the entity object and the search conditions, so as to retrieve the first query result corresponding to the query audio from the audio-visual database according to the query vector and the fuzzy vector.

[0047] Optionally, the filtering module 302 is specifically used for: The query audio is converted into query text, and a similarity search is performed on the query text in a pre-built text vector library to obtain candidate text fragments corresponding to the query text; the text vector library is constructed based on the plot synopsis of the film and television works, the evaluation text of the film and television works, and the subtitle text of the film and television works. The candidate text fragments are input into a pre-trained re-ranking model, which calculates the relevance score between the candidate text fragments and the query text, and outputs the candidate text fragments whose relevance score with the query text is greater than or equal to a first threshold as the target text fragments. Extract audio-visual information from the target text fragment as the second query result; the audio-visual information includes the title of the audio-visual work and the names of the creators of the audio-visual work.

[0048] Optionally, module 303 is specifically used for: Obtain the user's interaction history within the current session and identify the audio-visual works in the interaction history; The feature vectors of the audio-visual works in the interaction records are extracted in multiple dimensions, and the feature vectors in multiple dimensions are concatenated to obtain the audio-visual work vector corresponding to each audio-visual work; among them, the multiple dimensions include actor dimension, director dimension, genre dimension and era dimension. Based on the user's behavior type towards each audio-visual work, determine the weight parameters of the audio-visual work vector corresponding to each audio-visual work, and construct the user's preference vector based on the weight parameters of the audio-visual work vector corresponding to each audio-visual work.

[0049] Optionally, the determining module 303 is also used for: The first query result and the second query result are merged and deduplicated to obtain the audio and video query results, and then sorted according to the preference vector. The sorted audio and video query results are returned to the user's terminal device as the response data for the queried audio.

[0050] According to a third aspect of the present invention, an electronic device for audio information processing is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0051] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0052] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0053] Figure 4 An exemplary system architecture 400 is shown that can be applied to the audio information processing method or audio information processing apparatus of the present invention.

[0054] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0055] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0056] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0057] Server 405 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 401, 402, and 403 (for example only). The backend management server can process received audio information processing requests and return the processing results (for example only) to the terminal devices.

[0058] It should be noted that the audio information processing method provided in the embodiments of the present invention is generally executed by server 405, and correspondingly, the audio information processing device is generally disposed in server 405. The audio information processing method provided in the embodiments of the present invention can also be executed by terminal devices 401, 402, and 403, and correspondingly, the audio information processing device can be disposed in terminal devices 401, 402, and 403.

[0059] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0060] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing a terminal device of the present invention. Figure 5 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0061] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0062] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0063] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0064] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0065] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0066] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor. For example, a processor may include a retrieval module, a filtering module, and a determination module. The names of these modules do not necessarily limit the module itself. For instance, the retrieval module can be described as "a module that retrieves a first query result corresponding to the query audio from an audio-visual database based on entity objects extracted from the user-initiated query audio and search conditions."

[0067] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: based on entity objects extracted from a user-initiated query audio and search conditions, a first query result corresponding to the query audio is retrieved from an audio-visual database; target text segments with a relevance score greater than or equal to a first threshold to the query audio are selected from a pre-constructed text vector library according to the query audio, and audio-visual information extracted from the target text segments is used as a second query result; a user preference vector is constructed, and an audio-visual query result corresponding to the query audio is determined from the set of the first query result and the second query result based on the preference vector.

[0068] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An audio information processing method, characterized in that, include: Based on the entity objects extracted from the user-initiated query audio and the search conditions, a first query result corresponding to the query audio is retrieved from the audio-visual database; Based on the query audio, target text segments with a relevance score greater than or equal to a first threshold are selected from a pre-built text vector library, and the audio-visual information extracted from the target text segments is used as the second query result; Construct the user's preference vector, and based on the preference vector, determine the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result.

2. The method according to claim 1, characterized in that, Before retrieving the first query result corresponding to the query audio from the audio-visual database based on the entity object extracted from the user-initiated query audio and the search conditions, the process also includes: Collect user-input voice data, perform voice activity detection on the voice data, and extract valid segments from the voice data as query audio; Determine the type of instruction for the queried audio; In response to the query audio being a complex instruction type, the query audio is converted into text format to extract entity objects and search conditions from the query audio, and the confidence level of the entity objects and search conditions is determined; wherein, the confidence level is used to represent the clarity of the description of the entity objects and search conditions.

3. The method according to claim 2, characterized in that, Based on the entity objects extracted from the user-initiated audio query and the search criteria, a first query result corresponding to the audio query is retrieved from the audio-visual database, including: In response to the confidence level of the entity object and the search conditions being greater than or equal to a preset second threshold, a query vector is constructed based on the entity object and the search conditions, and a first query result corresponding to the query audio is retrieved from the audio-visual database based on the query vector; In response to the fact that the confidence level of the entity object and the search conditions is less than or equal to a preset third threshold and less than the second threshold, a query vector is constructed based on the entity object and the search conditions, and a fuzzy vector is constructed based on the entity object and the search conditions, so as to retrieve the first query result corresponding to the query audio from the audio-visual database according to the query vector and the fuzzy vector.

4. The method according to claim 1, characterized in that, Based on the query audio, target text segments with a relevance score greater than or equal to a first threshold are selected from a pre-built text vector library, and the audio-visual information extracted from the target text segments is used as the second query result, including: The query audio is converted into query text, and a similarity search is performed on the query text in a pre-built text vector library to obtain candidate text fragments corresponding to the query text; wherein, the text vector library is constructed based on the plot synopsis of the film and television work, the evaluation text of the film and television work, and the subtitle text of the film and television work; The candidate text fragments are input into a pre-trained re-ranking model, which calculates the relevance score between the candidate text fragments and the query text, and outputs candidate text fragments with a relevance score greater than or equal to a first threshold as target text fragments. The audio-visual information is extracted from the target text fragment as the second query result; the audio-visual information includes the name of the audio-visual work and the name of the creator of the audio-visual work.

5. The method according to any one of claims 1-4, characterized in that, Constructing the user's preference vector includes: Obtain the user's interaction records during the current session and identify the audio-visual works in the interaction records; The feature vectors of the audio-visual works in the interaction records are extracted in multiple dimensions, and the feature vectors in multiple dimensions are concatenated to obtain the audio-visual work vector corresponding to each audio-visual work; wherein, the multiple dimensions include actor dimension, director dimension, genre dimension and era dimension. Based on the user's behavior type towards each audio-visual work, the weight parameters of the audio-visual work vector corresponding to each audio-visual work are determined, and the user's preference vector is constructed based on the weight parameters of the audio-visual work vector corresponding to each audio-visual work.

6. The method according to claim 1, characterized in that, Based on the preference vector, determine the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result, including: The first query result and the second query result are merged and deduplicated to obtain the audio-visual query result, and the audio-visual query result is sorted according to the preference vector; The sorted audio and video query results are returned to the user's terminal device as the response data for the queried audio.

7. An audio information processing device, characterized in that, include: The retrieval module is used to retrieve a first query result corresponding to the query audio from the audio-visual database based on the entity objects extracted from the query audio initiated by the user and the retrieval conditions. The filtering module is used to filter target text segments from a pre-built text vector library based on the query audio, and to use the audio-visual information extracted from the target text segments as the second query result. The determination module is used to construct the user's preference vector and determine the audio-visual query result corresponding to the query audio from the set of the first query result and the second query result based on the preference vector.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images