Video content retrieval method and smart TV

Abstract

The present application provides a video content retrieval method and a smart TV, belonging to the field of smart TV technology. The method includes: obtaining text information, key frame images, and original voice signals of each video content in the video library; vectorizing the text information, key frame images, and original voice signals of each video content to obtain corresponding text vectors, image vectors, and audio vectors, and generating content vectors of each video content to store in a vector database; obtaining user input information, vectorizing the input information to obtain a target retrieval vector; performing vectorized retrieval on the vector database according to the target retrieval vector to obtain at least one target content vector, and using the video content corresponding to each target content vector as a retrieval result. The present application can achieve the effect of reducing the accuracy requirements for user input information, and combining the actual context of user input information to improve the accuracy of video content retrieval results.

Description

Technical Field

[0001] The present application relates to the technical field of smart televisions, and in particular to a video content retrieval method and a smart television. Background Art

[0002] In recent years, with the continuous improvement of living standards, smart TVs have become popular due to their diverse functions. Among them, smart TVs can break away from the single infrared remote control technology of traditional TVs and provide users with video content retrieval functions, through which users can more conveniently obtain corresponding video content.

[0003] In the related art, video content retrieval on smart TVs is mainly based on matching the text information input by the user with the keywords in the video content database to achieve video retrieval on the smart TV. One retrieval method is to extract keywords from the text content input by the user, and match and retrieve the extracted text keywords with the video keywords in the video database, and recommend corresponding video content to the user based on the matching results of the text keywords and the video keywords; the other is to convert the text content input by the user into a text vector, and recommend corresponding video content to the user based on the comparison results of the text vector and the video vector in the video database.

[0004] However, when performing video content retrieval based on relevant technologies, there are high requirements for the accuracy of the text content input by the user, and excessive reliance on the label data of the video content and failure to combine the actual context. This leads to low accuracy of video content retrieval results and poor user experience. Summary of the invention

[0005] The purpose of this application is to provide a video content retrieval method and a smart TV, which can reduce the accuracy requirements for user input information and combine the actual context of the user input information to improve the accuracy of video content retrieval results.

[0006] The embodiment of the present application is implemented as follows:

[0007] In a first aspect of an embodiment of the present application, a video content retrieval method is provided, the method comprising:

[0008] Obtain text information, key frame images and original voice signals of each video content in the video library;

[0009] Performing vectorization processing on the text information, key frame images and original voice signals of each video content to obtain the text vector, image vector and audio vector corresponding to each video content, generating the content vector of each video content according to the text vector, image vector and audio vector corresponding to each video content, and storing the content vector of each video content in the vector database;

[0010] Obtaining user input information, the input information includes: one or more of text, voice and picture;

[0011] Vectorize the input information to obtain the target retrieval vector;

[0012] A vectorized search is performed on a vector database according to a target search vector to obtain at least one target content vector, and the video content corresponding to each target content vector is used as a search result.

[0013] As a possible implementation method, the text information, key frame images and original voice signals of each video content are vectorized to obtain text vectors, image vectors and audio vectors corresponding to each video content, including:

[0014] Generate text vectors corresponding to the video content based on the semantic features and temporal features of the text information;

[0015] Generate an image vector corresponding to the video content based on the preset visual model and key frame images;

[0016] Generate an audio vector corresponding to the video content based on the preset deep learning model, the preset audio model and the original speech signal.

[0017] As a possible implementation method, a text vector corresponding to the video content is generated according to the semantic features and temporal features of the text information, including:

[0018] Map the keywords in the text information into dense vectors, which are used to represent the semantic features of the text information;

[0019] Encode the text information according to a preset learning model to obtain a text vector representation corresponding to each video content, where the text vector representation is used to represent the deep semantic features of the text information;

[0020] Capture the temporal characteristics of text information based on the text sequence encoding model;

[0021] According to the dense vector of the text information of each video content, the text vector representation and the temporal features, the text vector corresponding to the video content is generated.

[0022] As a possible implementation method, the image vector corresponding to the video content is generated according to the preset visual model and key frame images, including:

[0023] Perform feature extraction on the key frame image to obtain local features and global features of the key frame image;

[0024] Based on the preset visual model, deep semantic encoding is performed on the key frame images to obtain semantic encoding results;

[0025] Perform image segmentation and target detection on key frame images to determine key area features of key frame images;

[0026] Generate image vectors corresponding to the video content based on local features, global features, semantic encoding results, and key area features.

[0027] As a possible implementation method, an audio vector corresponding to the video content is generated according to a preset deep learning model, a preset audio model and an original speech signal, including:

[0028] Converting the original speech signal of each video content into a feature vector, wherein the feature vector contains frequency domain information of the original speech signal;

[0029] Perform data processing on the feature vector based on the preset deep learning model to obtain high-level abstract features of the original speech signal;

[0030] Generate audio vectors of video content based on feature vectors, high-level abstract features, and preset audio models.

[0031] As a possible implementation method, a content vector of each video content is generated according to a text vector, a picture vector, and an audio vector corresponding to each video content, including:

[0032] Based on the preset multimodal fusion model, the text vector, picture vector and audio vector corresponding to the video content are aligned to obtain the content vector of the video content, and the text vector, picture vector and audio vector are in the same feature space.

[0033] As a possible implementation method, the input information is vectorized to obtain a target retrieval vector, including:

[0034] If the input information is text, the text is used as the target text;

[0035] If the input information is voice, the voice is converted into target text;

[0036] Vectorize the target text to obtain a text vector;

[0037] According to the target text, a target image matching the target text is retrieved from the image database, and the target image is vectorized to obtain an image vector;

[0038] Generate a target retrieval vector based on the text vector and the image vector.

[0039] As a possible implementation method, the target text is vectorized to obtain a text vector, including:

[0040] Input the target text into the large language model to obtain the context and sentiment label of the target text;

[0041] Generate text vectors based on the context and sentiment labels of the target text.

[0042] As a possible implementation method, the input information is vectorized to obtain a target retrieval vector, including:

[0043] If the input information is speech, the enhanced text recalled by the large language model is obtained based on the speech;

[0044] According to the voice, a candidate picture matching the voice is retrieved from the picture database;

[0045] Generate a target retrieval vector based on speech, enhanced text, and selected images.

[0046] A second aspect of an embodiment of the present application provides a smart TV, which is used to implement the steps of the video content retrieval method described in the first aspect above.

[0047] According to a third aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video content retrieval method described in the first aspect is implemented.

[0048] The beneficial effects of the embodiments of the present application include:

[0049] A video content retrieval method provided in an embodiment of the present application obtains text information, key frame images and original voice signals corresponding to each video content in a video library, and vectorizes the text information, key frame images and original voice signals corresponding to each video content to obtain text vectors, picture vectors and audio vectors corresponding to each video content; vector fusion is performed on the text vectors, picture vectors and audio vectors corresponding to each video content to obtain content vectors of each video content in the same vector space, and the content vectors corresponding to each video content are stored in a vector database; input information of one or more modalities such as text, pictures and voice input by a user through a human-computer interaction interface of a smart TV is obtained, and the user's input information is vectorized to obtain a target retrieval vector corresponding to the input information; vector recall is performed on each content vector in the vector database according to the target retrieval vector to obtain at least one target content vector matching the target retrieval vector, and the video content corresponding to the target content vector is read from the video library as a retrieval result, and loaded into the display interface of the smart TV. When the user's input information is vectorized, it is also enriched and corrected through a large language model to obtain valid information and actual context corresponding to the user's input information, so that video content retrieval can be performed in combination with the actual context represented by the user's input information. In this way, the accuracy requirements for the user's input information can be reduced, and the actual context of the user's input information can be combined to improve the accuracy of the video content retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a first video content retrieval method provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of a second video content retrieval method provided in an embodiment of the present application;

[0053] Figure 3 A flowchart of a third video content retrieval method provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of a fourth video content retrieval method provided in an embodiment of the present application;

[0055] Figure 5A flowchart of a fifth video content retrieval method provided in an embodiment of the present application;

[0056] Figure 6 A system schematic diagram of a multimodal fusion model provided in an embodiment of the present application;

[0057] Figure 7 A flowchart of a sixth video content retrieval method provided in an embodiment of the present application;

[0058] Figure 8 A flowchart of a seventh video content retrieval method provided in an embodiment of the present application;

[0059] Fig. 9 A flowchart of a text enhancement method provided in an embodiment of the present application;

[0060] Fig.10 A flowchart of an eighth video content retrieval method provided in an embodiment of the present application;

[0061] Fig.11 A flow chart of a vector search method provided in an embodiment of the present application;

[0062] Fig.12 A flow chart of a vector recall method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0065] At present, video content retrieval for smart TVs is mainly based on matching the text information input by the user with keywords in the video content database to realize the video retrieval function on the smart TV side. Among them, one video retrieval method is to extract keywords from the text content input by the user, and recommend corresponding video content based on the matching results between the extracted keywords and the keywords in the video content database; the other is to convert the text content input by the user into a text vector, and recommend corresponding video content based on the comparison results between the text vector and the video vector in the video content database. However, this solution has high requirements for the accuracy of the text content input by the user, and is overly dependent on the label data of the video content and cannot be combined with the actual context. This results in low accuracy of video content retrieval results on the smart TV side and poor user experience.

[0066] To this end, the embodiment of the present application provides a video content retrieval method, which obtains the text information, key frame images and original voice signals of each video content in the video database, and vectorizes the text information, key frame images and original voice signals of each video content to obtain the text vector, picture vector and audio vector corresponding to each video content; according to the vector fusion results of the text vector, picture vector and audio vector of each video content, the content vector corresponding to each video content is obtained and stored in the vector database; at the same time, the user's input information is obtained, and the user's input information is vectorized to obtain a target retrieval vector, and the vector database is vectorized to retrieve the target retrieval vector to obtain at least one target content vector, and the video content corresponding to the retrieved target content vector is displayed to the user as a retrieval result. In this way, the video content retrieval function of the smart TV terminal can be realized from three dimensions such as text, image and audio, which can reduce the accuracy requirements for the user's input information, and combine the actual context of the user's input information to improve the accuracy of the video content retrieval results.

[0067] The video content retrieval method provided in the embodiment of the present application is explained in detail below with reference to the accompanying drawings.

[0068] It is worth noting that the video content retrieval method provided in the embodiment of the present application is mainly applied to smart TVs, but the video content retrieval method can also be applied to computer devices, user terminal devices, cloud servers, etc., and the present application does not make specific limitations on this.

[0069] Figure 1 A flowchart of a video content retrieval method provided in this application, see Figure 1 , the embodiment of the present application provides a video content retrieval method, comprising:

[0070] S101, obtaining text information, key frame images and original voice signals of each video content in a video library.

[0071] Optionally, the video library refers to a database for storing a large amount of video content. The video library can be set in a smart TV, or in a third-party device cloud network server, a third-party server, an operator server, etc. This application does not make any specific limitations on this.

[0072] Optionally, each video content in the video library is loaded to obtain text information, key frame images and original voice signals corresponding to each video content. The video content refers to various types of videos such as TV series, MV recordings, and amateur videos stored in the video library. A TV series contains at least one video content, that is, each episode of a TV series is a video content corresponding to the TV series.

[0073] Optionally, text information refers to information extracted from text-related information such as introduction information, subtitle information, barrage information, lyrics information, etc. contained in each video content; each video content is composed of multiple frames of continuous pictures, and the key frame pictures are extracted from the multiple frames of pictures contained in each video content. The key frame pictures are decisive pictures among the multiple frames of pictures contained in each video content. The key frame pictures can highlight the main content or main shots of the video content. The key frame pictures are often the starting picture or ending picture of the video content. This application does not make specific limitations on this; the original voice signal refers to the original audio signal of the video content, and the original audio signal can be a voice signal such as character dialogue, background music, inner monologue, etc. in the video content.

[0074] In addition, the text information contained in each video content in the video library needs to be optimized by a large language model. The text information after optimization by the large language model is richer in content. The text information after optimization by the large language model can extract the context information corresponding to each video content. The context information includes: region, current era, text sentiment tags, text similarity tags and other information. This application does not make specific limitations on this.

[0075] S102. Perform vectorization processing on the text information, key frame images and original voice signals of each video content to obtain the text vector, image vector and audio vector corresponding to each video content, generate the content vector of each video content according to the text vector, image vector and audio vector corresponding to each video content, and store the content vector of each video content in the vector database.

[0076] Optionally, the text vector is used to indicate the vector obtained after the text information corresponding to the video content is converted by a vector conversion tool, the picture vector refers to the vector obtained after the key frame picture corresponding to the video content is converted by a vector conversion tool, and the audio vector refers to the vector obtained after the original audio signal contained in the video content is converted by a vector conversion tool.

[0077] Optionally, the content vector refers to a vector obtained by fusion of a text vector, an image vector and an audio vector corresponding to the video content. The content vector is specifically used to characterize the fusion features of the text vector, the image vector and the audio vector in the same vector space.

[0078] Optionally, the vector database refers to a database used to store vector data. After fusing the text vectors, picture vectors and audio vectors corresponding to each video content, the content vectors corresponding to each video content are obtained, and the content vectors corresponding to each video content are written into the vector database.

[0079] It is worth mentioning that vectorizing the text information, key frame images and original voice signals corresponding to the video content can greatly simplify the retrieval operation of the video content in the video library, improve the retrieval operation speed, and optimize the memory pressure of the video library.

[0080] S103: Obtain input information from the user, where the input information includes one or more of text, voice, and picture.

[0081] Optionally, the user can input information through the search bar provided by the human-computer interaction interface of the smart TV. The search bar provided by the human-computer interaction interface includes: a text box input bar, a microphone icon, a picture scanning bar, etc. The user can enter text in the text box input bar, the user can also input voice through the microphone icon, the user can also input pictures through the picture scanning bar, and the user can select any one or more of the text box input bar, microphone icon, and picture scanning bar to input input information in different modes.

[0082] Optionally, the text in the input information refers to the text search formula entered by the user through the text box input bar, and the text search formula may be information such as TV series name, actor name, TV series type, movie name, variety show name, singer name, era, etc.; the voice in the input information refers to the voice signal entered by the user after clicking the microphone icon, and the voice in the input information may be classic lines, part of the TV series theme song, movie character names, actor names, TV series names, etc.; the picture in the input information refers to the picture scanned by the user through the picture scanning bar, and the picture in the input information may be a screenshot of the video content by the user, or a key frame picture corresponding to the video content downloaded by the user from other applications via the network, and this application does not make any specific limitations on this.

[0083] Optionally, the user's input information is multimodal input, that is, the user's input information can be text input, audio input, or picture input. The user's input information may include only one of text, voice, and picture, or the user's input information may include any two or more of text, voice, and picture. This application does not make any specific limitations on this.

[0084] It is worth noting that when the user's input information is text input, it is also necessary to optimize the user's input information through a large language model to obtain the actual context represented by the user's input text, and then perform video content retrieval based on the actual context represented by the user's input information.

[0085] S104: Perform vectorization processing on the input information to obtain a target retrieval vector.

[0086] Optionally, data preprocessing is performed on one or more of the text, voice, and picture input by the user through the human-computer interaction interface of the smart TV to obtain valid input information, and the valid input information is vectorized to obtain a target retrieval vector. The target retrieval vector refers to a vector obtained by converting valid input information in the user's input information through a vector conversion tool, that is, the target retrieval vector is also used to indicate a valid retrieval vector corresponding to the user's input information.

[0087] S105 . Perform vectorized search on the vector database according to the target search vector to obtain at least one target content vector, and use the video content corresponding to each target content vector as a search result.

[0088] Optionally, a vectorized search is performed on the vector database according to the target search vector, that is, the content vectors corresponding to each video content stored in the vector database are retrieved by the target search vector to obtain at least one target content vector matching the target search vector. The target content vector refers to a content vector matching the characteristic value of the target search vector, and the target content vector may be one or more, which is not specifically limited in this application.

[0089] It is worth noting that if there are multiple target content vectors, the video content corresponding to each target content vector is sorted according to the degree of feature matching between each target content vector and the target search vector, and each video content is presented to the user in sequence according to the sorting results.

[0090] Optionally, the video content is locked according to the target content vector, so that the video content corresponding to the target content vector is used as the final search result, and is presented to the user in sequence according to the matching of the target content vector and the target search vector. The search result is the video content corresponding to the content vector matching the target search vector, and the search result may include only one video content or multiple video contents, which is not specifically limited in this application.

[0091] In an embodiment of the present application, text information, key frame images and original voice signals corresponding to each video content in a video library are obtained, and the text information, key frame images and original voice signals corresponding to each video content are vectorized to obtain text vectors, picture vectors and audio vectors corresponding to each video content; vector fusion is performed on the text vectors, picture vectors and audio vectors corresponding to each video content to obtain content vectors of each video content in the same vector space, and the content vectors corresponding to each video content are stored in a vector database; input information of one or more modalities such as text, pictures and voice input by a user through a human-computer interaction interface of a smart TV is obtained, and the user's input information is vectorized to obtain a target retrieval vector corresponding to the input information; vector recall is performed on each content vector in the vector database according to the target retrieval vector to obtain at least one target content vector matching the target retrieval vector, and the video content corresponding to the target content vector is read from the video library as a retrieval result, and loaded into the display interface of the smart TV. When the user's input information is vectorized, it is also enriched and corrected through a large language model to obtain valid information and actual context corresponding to the user's input information, so that video content retrieval can be performed in combination with the actual context represented by the user's input information. In this way, the accuracy requirements for the user's input information can be reduced, and the actual context of the user's input information can be combined to improve the accuracy of the video content retrieval results.

[0092] In an optional embodiment, see Figure 2 The operation of step S102 may specifically be:

[0093] S201, generating a text vector corresponding to the video content according to the semantic features and temporal features of the text information.

[0094] Optionally, semantic features are used to indicate the meaning or interpretation expressed by text information. Semantic features are obtained by semantic analysis of keywords and key words in text information. Semantic features refer to the characteristics of keywords or key words in text information in a certain sense. Semantic features are elements obtained by segmenting keywords or key words in text information. Among them, semantic feature analysis can be used to obtain the semantic features shared by keywords with the same structure at the same position, so as to find the same semantic elements and different semantic elements, thereby achieving the ability to distinguish the text information of each video content. It is worth noting that semantic feature analysis is often used to interpret and distinguish text information with the same text information results but completely different meanings.

[0095] Optionally, the timing feature is used to indicate the time sequence of the video content itself, specifically, the playback sequence of multiple frames of images contained in the video content.

[0096] Optionally, the text information may be vectorized according to the semantic features and temporal features of the text information to obtain a text vector corresponding to the video content.

[0097] S202: Generate an image vector corresponding to the video content according to a preset visual model and key frame images.

[0098] Optionally, the preset visual model refers to a model pre-trained based on a large number of multi-frame images of video content. By inputting the key frame images corresponding to the video content into the pre-trained visual model, the image vector corresponding to the video content can be obtained. Among them, the preset visual model can be a ResNet model, etc., which is not specifically limited in this application.

[0099] S203: Generate an audio vector corresponding to the video content according to a preset deep learning model, a preset audio model and an original speech signal.

[0100] Optionally, the preset deep learning model is an existing neural network model. By processing the original speech signal of the video content with the help of the deep learning model, high-level abstract features contained in the original speech signal of the video content can be obtained. The high-level abstract features are specifically used to indicate abstract features such as emotion, context, and timbre contained in the original speech signal.

[0101] Optionally, the preset audio model refers to an audio model pre-trained based on the original speech signals of a large amount of video content. The preset audio model can be used to obtain effective coding information contained in the original speech signals of the video content. The preset audio model can be VGGish, Panns, etc., and this application does not make any specific limitations on this.

[0102] Optionally, the original speech signal of the video content is input into the deep learning model and the audio model to obtain an audio vector corresponding to the video content.

[0103] In an optional embodiment, see Figure 3 The operation of step S201 may specifically be:

[0104] S301 : Map keywords in the text information into dense vectors, where the dense vectors are used to represent semantic features of the text information.

[0105] Optionally, keywords are used to indicate key information that plays a decisive role in text information, and dense vectors are used to characterize semantic features of text information. The dense vectors are specifically used to indicate complete semantic expressions of text information after optimization.

[0106] Optionally, an embedding technology may be used to map keywords contained in the text information into dense vectors to obtain semantic features of the text information of the video content. The embedded technology may be Word2Vec, etc., which is not specifically limited in this application.

[0107] S302: Encode the text information according to a preset learning model to obtain a text vector representation corresponding to each video content, where the text vector representation is used to represent deep semantic features of the text information.

[0108] Optionally, the preset learning model refers to a pre-trained learning model, and the preset learning model may be a Transformer model. The learning model used in this application to encode text information in video content may be a BERT, XLNet or other model, and this application does not make any specific limitations on this.

[0109] Optionally, the text information of the video content is deeply semantically encoded through a preset learning model, so that the semantic information and global context of the text information of each video content can be captured to obtain the text vector representation corresponding to each video content. Among them, the text vector representation is used to characterize the deep semantic features of the text information, and the deep semantic features specifically refer to the global context features corresponding to the text information of the video content. According to the deep semantic features of the text information, the overall meaning and contextual relationship of the text information can be determined.

[0110] It is worth mentioning that the preset learning model can extract deep semantic features of text information to enrich the semantic information of video content and improve the recognition rate of complex scenes and objects.

[0111] S303. Capture the temporal characteristics of text information based on a text sequence encoding model.

[0112] Optionally, the text sequence coding model is a coding model for capturing the temporal features of text information. By inputting the text information of each video content into the text sequence coding model, the temporal features of the text information of each video content can be obtained. The temporal features are used to indicate the playback time nodes of the text information in the video content.

[0113] S304: Generate text vectors corresponding to the video contents according to the dense vectors, text vector representations, and temporal features of the text information of each video content.

[0114] Optionally, a text vector corresponding to each video content is generated based on the semantic features represented by the dense vector of the text information of each video content, the deep semantic features represented by the text vector representation, and the playback timing represented by the timing feature.

[0115] In an optional embodiment, see Figure 4 The operation of step S202 may specifically be:

[0116] S401: Extract features from key frame images to obtain local features and global features of the key frame images.

[0117] Optionally, by using a convolutional neural network to extract features from the key frame images of each video content, local features and global features of the key frame images of each video content can be obtained. Local features refer to some iconic features of the key frame images, such as animal heads, close-ups of people's faces, etc.; global features refer to the overall appearance or overall features of the key frame images, such as close-ups of the whole body of animals, close-ups of the whole body of people, etc. This application does not make any specific limitations on this.

[0118] S402: Perform deep semantic coding on the key frame image based on a preset visual model to obtain a semantic coding result.

[0119] Optionally, the key frame images of each video content are input into a preset visual model, and the visual model performs deep semantic encoding on each input key frame image to obtain a semantic encoding result of each key frame image, and the semantic encoding result is used to represent the semantic information corresponding to the key frame image.

[0120] S403: Perform image segmentation and target detection on the key frame image to determine key area features of the key frame image.

[0121] Optionally, the key frame images of each video content are cut by image segmentation technology, and the key frame images of each video content are detected by target detection technology to determine the position of the key area in the key frame image of each video content and the features disclosed by the key area. Among them, the key area refers to the area with a symbolic or decisive role in the key frame image, and the key area feature is the image feature disclosed by the key area in the key frame image.

[0122] S404: Generate an image vector corresponding to the video content according to the local features, global features, semantic encoding results and key area features.

[0123] Optionally, a picture vector corresponding to the video content is generated according to local features, global features, semantic information and key area features of the key frame pictures of each video content.

[0124] In an optional embodiment, see Figure 5 The operation of step S203 may specifically be:

[0125] S501 : Convert the original speech signal of each video content into a feature vector, where the feature vector contains frequency domain information of the original speech signal.

[0126] Optionally, an audio feature extraction technology is used to convert the original speech signal of each video content into a feature vector, where the feature vector refers to a vector corresponding to the speech feature contained in the original speech signal, and the feature vector contains frequency domain information of the original speech signal, and the frequency domain information is used to indicate the frequency information corresponding to the original speech signal of each video content. The audio feature extraction technology can be any one of MFCC, FBANK and other technologies, and this application does not specifically limit this.

[0127] S502: Process the feature vector based on a preset deep learning model to obtain high-level abstract features of the original speech signal.

[0128] Optionally, a preset deep learning model is used to process the feature vector corresponding to the original speech signal to obtain high-level abstract features corresponding to the original speech signal of the video content. The high-level abstract features are specifically used to indicate abstract features such as emotion, context, and timbre contained in the original speech signal.

[0129] S503: Generate an audio vector of the video content according to the feature vector, the high-level abstract features and the preset audio model.

[0130] Optionally, the original speech signal of each video content is vectorized by preset audio features to achieve effective encoding of the original speech signal, and an audio vector corresponding to the original speech signal of each video content is generated based on the frequency domain information, high-level abstract features and encoding results of the original speech signal of each video content.

[0131] In an optional implementation manner, the operation of step S102 may specifically be:

[0132] Based on the preset multimodal fusion model, the text vector, picture vector and audio vector corresponding to the video content are aligned to obtain the content vector of the video content, and the text vector, picture vector and audio vector are in the same feature space.

[0133] Optionally, the preset multimodal fusion model refers to a model that can perform vector fusion on the multimodal vectors of the video content, and the text vector, picture vector and audio vector of each video content are used together as the multimodal vector of each video content, and the text vector, picture vector and audio vector of each video content are input into the preset multimodal fusion model to obtain the content vector of each video content. Among them, the preset multimodal fusion model is specifically a multimodal fusion technology, such as feature-level fusion, decision-level fusion, etc., which is not specifically limited in this application.

[0134] Optionally, the text vectors, image vectors and audio vectors of each video content are aligned by a preset multimodal fusion model, and adversarial training, multi-task learning and other methods are used to represent the vectors of different modes of each video content in the same feature space. Specifically, being in the same feature space means that the features of each vector extracted in the space of different modes in different scenarios or under different conditions have the same dimension and type.

[0135] Optionally, the feature space refers to the space where the feature vector is located, each feature corresponds to a one-dimensional coordinate in the feature space, and the same feature space where the text vector, picture vector and audio vector of the video content are located is a three-dimensional space.

[0136] In an optional implementation, Figure 6 A system diagram of a multimodal fusion model provided in this application, see Figure 6 , extract the text information, key frame images and original voice signals of each video content in the video library, and vectorize the text information and key frame images of each video content to obtain the text vector and audio vector corresponding to each video content; at the same time, input the text information of each video content into the large language model for semantic enrichment to obtain the actual context and emotion label of the text information of each video content, and vectorize the text information as well as the context and emotion label of each video content to obtain the text vector of each video content; align the text vector, image vector and audio vector corresponding to each video content through the embedded space processing technology to obtain the early fusion result corresponding to each video content, the early fusion result is used to indicate the content vector of each video content, and the content vector of each video content is stored in the vector database.

[0137] In an optional embodiment, see Figure 7 The operation of step S104 may specifically be:

[0138] S701: If the input information is text, use the text as the target text.

[0139] Optionally, when the information input by the user through the search bar provided by the human-computer interaction interface of the smart TV is text data, the text input by the user is used as the target text, and the target text refers to the valid text data in the information input by the user.

[0140] S702: If the input information is voice, convert the voice into target text.

[0141] Optionally, when a user inputs a voice signal via a microphone provided by the human-computer interaction interface of the smart TV, a voice recognition technology is used to convert the voice signal input by the user into text data, and valid text data is extracted from the converted text data as the target text.

[0142] S703: Perform vectorization processing on the target text to obtain a text vector.

[0143] Optionally, a vector conversion tool is used to vectorize the target text to obtain a text vector of the user's input information.

[0144] S704: According to the target text, a target image matching the target text is retrieved from an image database, and the target image is vectorized to obtain an image vector.

[0145] Optionally, the image database refers to a database used to store key frame images of video content. The image database can be deployed in a smart TV or set up in a third-party device cloud network server, a third-party server, an operator server, etc. This application does not make specific limitations on this.

[0146] Optionally, according to the target text, a target image matching the target text is retrieved from an image database, and vectorization is performed on the retrieved target image to obtain an image vector corresponding to the target image.

[0147] S705: Generate a target search vector according to the text vector and the image vector.

[0148] Optionally, a target retrieval vector may be generated based on the text vector and the image vector.

[0149] In an optional embodiment, see Figure 8 The operation of step S703 may specifically be:

[0150] S801: Input the target text into the large language model to obtain the context and sentiment label of the target text.

[0151] Optionally, the target text is input into a large language model, and the generalization ability of the large language model is used to enrich the semantic information of the target text to obtain the context and sentiment labels of the target text, such as region, current era, text sentiment information, text similarity labels, etc. This application does not make specific limitations on this.

[0152] S802: Generate a text vector according to the context and sentiment label of the target text.

[0153] Optionally, the target text is vectorized according to the context and sentiment label of the target text to obtain a text vector of the target text.

[0154] Fig. 9 For a flow chart of a text enhancement method provided in this application, see Fig. 9 , input the text information into the large language model to obtain the context and sentiment label corresponding to the text, and generate the text vector corresponding to the text based on the context and sentiment label of the text.

[0155] In an optional embodiment, see Fig.10 The operation of step S104 may specifically be:

[0156] S1001. If the input information is speech, then the enhanced text recalled by the large language model is obtained according to the speech.

[0157] Optionally, when the information input by the user through the microphone on the human-computer interaction interface provided by the smart TV is voice, the enhanced text recalled by the large language model is obtained according to the voice signal input by the user. The enhanced text refers to the text vector generated by the text optimized by the large language model.

[0158] S1002: According to the voice, search for a candidate picture matching the voice in a picture database.

[0159] Optionally, according to the voice signal input by the user, a candidate picture matching the feature represented by the voice is retrieved from a picture database, where the candidate picture refers to a key frame picture matching the feature represented by the voice input by the user.

[0160] S1003: Generate a target retrieval vector according to the speech, enhanced text, and the selected image.

[0161] Optionally, a target retrieval vector is generated according to the speech input by the user, the enhanced text recalled by the large language model, and the candidate images provided by the image database.

[0162] Fig.11 A flow chart of a vector search method provided in this application, see Fig.11, the text in the user's input information is vectorized to obtain the text vector of the input information; at the same time, the image in the user's input information is vectorized to obtain the image vector of the input information; according to the text vector and image vector in the user's input information, a vectorized search is performed on the vector database to obtain a vector search result.

[0163] Fig.12 For a flow chart of a vector recall method provided in this application, see Fig.12 When the user's input information is a voice signal, the audio features contained in the voice input are extracted; the voice signal is text-enhanced through a large language model to obtain enhanced text; the selected image is retrieved from the image database through the text; the enhanced text, audio features and selected image are vectorized to obtain a target retrieval vector, and the vector database is recalled through the target retrieval vector to obtain a target content vector, and the retrieval result is obtained based on the video content corresponding to the target content vector.

[0164] The following describes a smart TV and a computer-readable storage medium used to execute the video content retrieval method provided in the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0165] The embodiment of the present application also provides a smart TV, which can implement the steps in the above-mentioned method embodiments.

[0166] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0167] Optionally, the present application also provides a program product, such as a computer-readable storage medium, including a program, which is used to execute any of the above-mentioned video content retrieval method embodiments when executed by a processor.

[0168] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0169] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0170] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video content retrieval method, characterized in that: The method comprises: Obtain text information, key frame images and original voice signals of each video content in the video library; Performing vectorization processing on the text information, key frame images and original voice signals of each of the video contents to obtain text vectors, image vectors and audio vectors corresponding to each of the video contents, generating content vectors of each of the video contents according to the text vectors, image vectors and audio vectors corresponding to each of the video contents, and storing the content vectors of each of the video contents in a vector database; Obtaining user input information, where the input information includes: one or more of text, voice, and picture; Performing vectorization processing on the input information to obtain a target retrieval vector; A vectorized search is performed on the vector database according to the target search vector to obtain at least one target content vector, and the video content corresponding to each target content vector is used as a search result.

2. The video content retrieval method according to claim 1, characterized in that: The vectorization processing of the text information, key frame images and original voice signals of each video content to obtain a text vector, an image vector and an audio vector corresponding to each video content includes: Generating a text vector corresponding to the video content according to the semantic features and temporal features of the text information; Generate a picture vector corresponding to the video content according to a preset visual model and the key frame picture; An audio vector corresponding to the video content is generated according to a preset deep learning model, a preset audio model and the original speech signal.

3. The video content retrieval method according to claim 2, characterized in that: Generating a text vector corresponding to the video content according to the semantic features and temporal features of the text information includes: Mapping keywords in the text information into dense vectors, where the dense vectors are used to represent semantic features of the text information; Encoding the text information according to a preset learning model to obtain a text vector representation corresponding to each of the video contents, wherein the text vector representation is used to characterize deep semantic features of the text information; Capturing the temporal characteristics of the text information based on a text sequence encoding model; A text vector corresponding to the video content is generated according to the dense vector, text vector representation and temporal features of the text information of each video content.

4. The video content retrieval method according to claim 2, characterized in that: The step of generating a picture vector corresponding to the video content according to a preset visual model and the key frame picture includes: Performing feature extraction on the key frame image to obtain local features and global features of the key frame image; Performing deep semantic coding on the key frame image based on a preset visual model to obtain a semantic coding result; Performing image segmentation and target detection on the key frame image to determine key area features of the key frame image; A picture vector corresponding to the video content is generated according to the local features, the global features, the semantic encoding results and the key area features.

5. The video content retrieval method according to claim 2, characterized in that: The step of generating an audio vector corresponding to the video content according to a preset deep learning model, a preset audio model, and the original speech signal includes: Converting the original speech signal of each of the video contents into a feature vector, wherein the feature vector includes frequency domain information of the original speech signal; Performing data processing on the feature vector based on a preset deep learning model to obtain high-level abstract features of the original speech signal; An audio vector of the video content is generated according to the feature vector, the high-level abstract feature and a preset audio model.

6. The video content retrieval method according to claim 1, characterized in that: Generating the content vector of each video content according to the text vector, the picture vector and the audio vector corresponding to each video content includes: Based on a preset multimodal fusion model, the text vector, the picture vector and the audio vector corresponding to the video content are vector aligned to obtain the content vector of the video content, and the text vector, the picture vector and the audio vector are in the same feature space.

7. The video content retrieval method according to claim 1, characterized in that: The vectorizing the input information to obtain a target search vector includes: If the input information is text, the text is used as the target text; If the input information is speech, converting the speech into the target text; Performing vectorization processing on the target text to obtain a text vector; According to the target text, a target image matching the target text is retrieved from an image database, and the target image is vectorized to obtain an image vector; The target retrieval vector is generated according to the text vector and the image vector.

8. The video content retrieval method according to claim 7, characterized in that: The vectorization process of the target text to obtain a text vector includes: Inputting the target text into a large language model to obtain the context and sentiment label of the target text; The text vector is generated according to the context and sentiment label of the target text.

9. The video content retrieval method according to claim 1, characterized in that: The vectorizing the input information to obtain a target search vector includes: If the input information is speech, obtaining enhanced text recalled by a large language model according to the speech; According to the voice, searching for a candidate picture matching the voice in a picture database; The target retrieval vector is generated according to the speech, the enhanced text and the selected picture.

10. A smart TV, characterized in that: The smart TV is used to implement the steps of the video content retrieval method described in any one of claims 1-9.

Images