Video content retrieval method, device and terminal based on voice interaction of television system

Abstract

The invention discloses a video content retrieval method and device based on television system voice interaction and a terminal, and relates to the technical field of video processing, and the method comprises the steps: when a video is played for the first time, extracting a picture frame from the video at a preset frequency, converting the picture frame into a multi-dimensional image feature vector and a corresponding video timestamp, and carrying out hierarchical storage in a database, constructing vectorized data containing visual semantic information; obtaining a voice retrieval instruction, performing intention recognition and semantic understanding, extracting a detection keyword, and generating a multi-dimensional retrieval feature vector; calculating a matching degree between the multi-dimensional retrieval feature vector and a multi-dimensional image feature vector of a video picture frame stored in a database, and screening out picture frames of which the similarity is higher than a preset similarity threshold to form a retrieval candidate matching set; and determining matched picture playing. The video content retrieval method is efficient, accurate and high in interactivity, and retrieval experience and operation efficiency of the user in the video watching process are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video content retrieval method, apparatus, smart terminal, and storage medium based on voice interaction in a television system. Background Technology

[0002] With the widespread adoption of smart TVs, users' needs for retrieving video content are becoming increasingly diverse and convenient. Existing TV video retrieval technologies mainly employ the following methods: First, manually operating the remote control to fast forward, rewind, or play frame by frame to find target video content. This method is extremely inefficient, requiring users to expend significant time and effort, and it's difficult to accurately locate the target image. Second, searching through video chapter indexes, but these are usually pre-set by the video uploader or producer, resulting in coarse granularity and failing to meet users' needs for searching specific details (such as a particular scene or character actions). Third, while some smart TVs support voice search, this is mostly for searching different video resources within the video library, rather than searching for content segments of the currently playing video, thus failing to achieve deep interaction with the currently playing video content.

[0003] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a video content retrieval method, device, smart terminal, and storage medium based on voice interaction in a television system. This invention solves the problems of missing voice interaction retrieval for currently playing videos on a television system and insufficient processing of multiple similar screen retrieval in the prior art. It provides an efficient, accurate, and highly interactive video content retrieval method, significantly improving the user's retrieval experience and operational efficiency during video viewing.

[0005] The technical solution of this application is as follows: A video content retrieval method based on voice interaction in a television system, comprising: When the video is played for the first time, the video frames are extracted from the video at a preset frequency, and multi-dimensional feature analysis is performed to convert them into multi-dimensional image feature vectors. The multi-dimensional image feature vectors and the corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information. The system acquires a voice search command, decodes the voice search command into a text command, and combines the current video playback scenario to perform intent recognition and semantic understanding on the text command, extracts detection keywords, and generates a multi-dimensional search feature vector. Calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frames stored in the database, and filter out the frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set; Based on the search candidate matching set, the system receives user operation instructions to select one of the matching screens from the search candidate matching set for playback, or controls the default jump to the first matching screen for playback.

[0006] The video content retrieval method based on voice interaction in a television system, wherein the steps of extracting frame images from the video at a preset frequency when the video is played for the first time, performing multi-dimensional feature analysis to convert them into multi-dimensional image feature vectors, and storing the multi-dimensional image feature vectors and corresponding video timestamps hierarchically in a database to construct vectorized data containing visual semantic information include: When a video is about to play, first check if the current video has historical processing data; If there is no historical data, frame extraction is initiated; a multi-threaded mechanism is used to decode the current video stream in real time based on an open-source multimedia processing framework; and a time-slice sampling mechanism is used to extract frames from the decoded video stream according to a preset time interval. The extracted image frames undergo color space conversion and size normalization preprocessing. The preprocessed image frames are analyzed for multi-dimensional features using computer vision and deep learning algorithms, and the image frames are vectorized to transform the image frame content into multi-dimensional image feature vectors. The multidimensional image feature vectors and their corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information and generate a retrieval index.

[0007] The video content retrieval method based on voice interaction in a television system, wherein the steps of obtaining a voice retrieval command, decoding the voice retrieval command into a text command, and combining the current video playback scenario to perform intent recognition and semantic understanding on the text command, extract detection keywords, and generate a multi-dimensional retrieval feature vector include: Acquire an analog speech signal with search keywords, convert the analog speech signal into a digital audio signal, and obtain a voice search command; Decode the voice search command into a text command; The natural language processing module is used to perform intent recognition and semantic understanding on the decoded text instructions. Combined with the current video playback scenario, detection keywords are extracted and multi-dimensional retrieval feature vectors are generated from the extracted keywords.

[0008] The video content retrieval method based on voice interaction in a television system, wherein, after the step of calculating the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database, and filtering out frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set, the method further includes: Receive and acquire the generated multidimensional retrieval feature vector; The matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database is calculated using the cosine similarity algorithm. By setting a dynamic similarity threshold, frames with a similarity higher than the threshold are filtered out to form a retrieval candidate matching set; If no image is found that matches the conditions of the multidimensional retrieval feature vector, a no-match signal is generated.

[0009] The video content retrieval method based on voice interaction in a television system, wherein the step of calculating the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database, and filtering out frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set, further includes: The cosine similarity algorithm is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database. The similarity calculation formula is as follows:

[0010] Where A is the multidimensional retrieval feature vector and B is the multidimensional image feature vector of the video frame, both of which are 768-dimensional vectors after L2 normalization. Set a dynamic similarity threshold to filter out the top K frames with a similarity higher than the preset similarity threshold, where K is a natural number greater than 1.

[0011] The video content retrieval method based on voice interaction in a television system, wherein, prior to the step of obtaining the voice retrieval command, further includes: Pre-enter and register voiceprint information corresponding to user IDs, and set corresponding user preferences and command priorities for each user ID's voiceprint information, and store them.

[0012] The video content retrieval method based on voice interaction in a television system, wherein the steps of obtaining a voice retrieval command, decoding the voice retrieval command into a text command, and combining the current video playback scenario to perform intent recognition and semantic understanding on the text command, extract detection keywords, and generate a multi-dimensional retrieval feature vector further include: When a voice search command is received, the voiceprint features in the voice search command are analyzed and compared with the registered voiceprint information corresponding to the user ID to identify the user ID that issued the voice search command. The voice search command is decoded into a text command with the user's ID. The identified text commands with user IDs are placed into the command queue in chronological order. Prioritize text commands with user IDs in the command queue according to preset rules; When there are multiple concurrent or nearly concurrent text commands in the command queue, the text command with the highest priority and user ID is selected based on the priority evaluation result and handed over to the natural language processing module for intent recognition and semantic understanding. For text commands with user IDs that have a priority below the highest, control can temporarily suspend intent recognition and semantic understanding, prompt the user to resend the command, or process the command only after the current command has been processed.

[0013] A video content retrieval device based on voice interaction in a television system, wherein the device comprises: The video content processing module is used to extract frame images from the video at a preset frequency when the video is played for the first time, perform multi-dimensional feature analysis, convert them into multi-dimensional image feature vectors, and store the multi-dimensional image feature vectors and corresponding video timestamps in a hierarchical manner in the database to construct vectorized data containing visual semantic information. The voice command acquisition and semantic parsing module is used to acquire voice retrieval commands, decode the voice retrieval commands into text commands, and combine the current video playback scenario to perform intent recognition and semantic understanding on the text commands, extract detection keywords, and generate multi-dimensional retrieval feature vectors. The video content retrieval and matching module is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database, and to filter out the frame with a similarity higher than a preset similarity threshold to form a retrieval candidate matching set. The playback control module is used to receive user operation instructions based on the search candidate matching set, select one of the matching screens from the search candidate matching set for playback, or control the default jump to the first matching screen for playback.

[0014] A smart terminal includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including steps for performing any of the methods described herein.

[0015] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it enables an electronic device to perform the steps of any of the methods described.

[0016] As can be seen from the above, this application provides a video content retrieval method, device, smart terminal, and storage medium based on voice interaction in a television system. This invention achieves efficient video content retrieval in a television system based on voice interaction, significantly improving retrieval efficiency and accuracy compared to traditional retrieval methods. Users can quickly locate target screens without manual operation. Especially when handling the retrieval of multiple similar screens, the similarity threshold setting and progress bar marking clearly display the locations of multiple related screens, facilitating user browsing and selection. Simultaneously, the natural language interaction method lowers the operational threshold, enhances the user's interactive experience with the television system, and meets users' needs for in-depth video content retrieval and personalized viewing. Furthermore, this invention also has the following advantages: 1) It can realize voice interactive retrieval in specific scenarios. Unlike traditional video library retrieval, this invention focuses on voice interactive retrieval of currently playing videos in a TV system, realizing real-time in-depth retrieval of video content and filling the technical gap in this application scenario.

[0017] 2) A multi-similar-screen processing mechanism has been implemented. This invention effectively solves the problem of retrieval and location of multi-similar-screens by setting a similarity threshold and constructing a feature matching algorithm, and provides intuitive markings on the progress bar to facilitate users to quickly browse and switch between them.

[0018] 3) A multi-technology integrated intelligent interactive system has been realized. This invention organically combines technologies such as speech recognition, natural language processing, video analysis and UI display to build a complete intelligent interactive system. It realizes the fully automated processing from voice command input to the presentation of search results, thereby improving the intelligence level of the TV system and the user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the video content retrieval method based on voice interaction in a television system according to Embodiment 1 of the present invention.

[0021] Figure 2This is a flowchart illustrating the video content retrieval method based on voice interaction in a television system according to Embodiment 2 of the present invention.

[0022] Figure 3 The present invention provides a schematic diagram of an embodiment of a video content retrieval device based on voice interaction in a television system.

[0023] Figure 4 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0025] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0026] The main problems with existing television video retrieval technologies are as follows: 1) Cumbersome and inefficient operation: Users need to manually operate the remote control to fast forward, rewind, and play frame by frame to find the target video content. This not only consumes a lot of time and energy, but also makes it difficult to accurately locate specific scenes.

[0027] 2) Low search accuracy and coarse granularity: When searching through video chapter indexes, the chapters are usually pre-set by the video uploader or producer, resulting in coarse granularity that cannot meet users' search needs for specific details (such as a specific scene, character actions, etc.).

[0028] 3) Poor interactivity and inability to engage deeply: Although some smart TVs support voice search, it is mostly for searching different video resources in the video library, rather than searching for content segments of the currently playing video, thus failing to achieve deep interaction with the currently playing video content.

[0029] 4) Insufficient handling of multiple similar images: Existing video content retrieval technologies lack effective similarity calculation and location labeling mechanisms when handling multiple similar images, and cannot accurately display the positions of multiple similar images on the progress bar.

[0030] 5) Difficulty in recognizing and matching fuzzy or complex commands: When users input fuzzy or complex search commands, the system has difficulty in accurately recognizing and matching them.

[0031] These problems result in existing TV video retrieval technologies having drawbacks such as cumbersome operation, low retrieval accuracy, and poor interactivity, failing to provide an efficient, accurate, and highly interactive video content retrieval experience.

[0032] To address the aforementioned technical problems, embodiments of the present invention provide a video content retrieval method based on voice interaction in a television system, as described in the following embodiments.

[0033] Example 1 like Figure 1 As shown in the figure, a video content retrieval method based on voice interaction in a television system according to an embodiment of the present invention includes the following steps: Step S100: When the video is played for the first time, the video frames are extracted from the video at a preset frequency, and multi-dimensional feature analysis is performed to convert them into multi-dimensional image feature vectors. The multi-dimensional image feature vectors and the corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information. The preset frequency in this embodiment refers to a pre-set frame extraction interval, such as extracting 1 frame per second or 1 frame every 0.5 seconds. The specific frequency can be dynamically adjusted according to video clarity and content complexity to balance retrieval accuracy and storage cost. In this embodiment, a frame refers to the basic unit of video, that is, a static image at a certain moment during video playback, similar to a single frame of film.

[0034] The multi-dimensional feature analysis in this embodiment refers to the comprehensive feature extraction of the image frame. The dimensions cover the core information at the visual level, including color features (image hue, brightness distribution), texture features (image detail texture density, direction), shape features (object outlines, shapes in the image), and semantic features (high-level information such as people, scenes, and actions in the image).

[0035] The multidimensional image feature vector in this embodiment refers to converting the parsed multidimensional visual features into a numerical vector that can be recognized and calculated by a computer, such as a set of ordered numbers, to achieve a digital representation of the image information. The video timestamp in this embodiment refers to the specific playback time position of a frame in the video, such as 00:05:23, which represents the frame at 5 minutes and 23 seconds into the video, used to locate the playback node of the frame.

[0036] The hierarchical storage described in this embodiment refers to the layered storage of multi-dimensional image feature vectors and timestamp data based on factors such as the importance of image frames and video playback popularity. For example, core key frames are stored frequently, while ordinary frames are stored on demand, optimizing data retrieval efficiency. The vectorized data (database) in this embodiment refers to a data warehouse specifically used for storing and managing multi-dimensional feature vectors, supporting efficient vector similarity calculation, which differs from traditional databases that store text and images.

[0037] In specific implementation of step S100, when a user plays a video for the first time, the present invention automatically extracts continuous static frames from the video at a pre-set frequency (e.g., 1 frame per second); then, it performs multi-dimensional feature analysis on each frame to comprehensively capture the color, texture, shape, and core semantic information of the image, and transforms this information into a multi-dimensional image feature vector that can be processed by a computer; next, it associates and binds each feature vector with the corresponding video timestamp, that is, the specific playback time of the frame in the video, and stores it in a vectorized database according to a hierarchical storage strategy; finally, through this series of operations, a vectorized data system containing video visual semantic information is constructed, providing a data foundation for subsequent retrieval.

[0038] This step-by-step embodiment can solve the problems of low retrieval accuracy and coarse granularity. This is because the present invention breaks through the coarse granularity limitation of traditional chapter indexes by extracting frame by frame and analyzing multi-dimensional features, and realizes accurate digitization of video details such as people, actions and scenes, providing data support for fine-grained retrieval.

[0039] Furthermore, this invention can improve retrieval efficiency because it adopts vectorized storage and hierarchical management, so that subsequent similarity calculations do not need to rely on the original video footage. Matching can be done quickly through vector calculations alone, avoiding the tedious operation of manual fast forward and rewind, and laying the foundation for efficient retrieval.

[0040] Furthermore, this invention can reduce the computational power consumption of subsequent retrievals because feature extraction and storage are completed during the first playback, and subsequent retrievals do not require repeated video parsing, reducing the computational power consumption of television equipment and ensuring the smoothness of playback and retrieval.

[0041] Step S200: Obtain the voice search command, decode the voice search command into a text command, and combine it with the current video playback scenario to perform intent recognition and semantic understanding on the text command, extract detection keywords, and generate a multi-dimensional search feature vector; The voice search command in this embodiment refers to the user's search request command input via voice, such as finding the scene of the main character drinking tea in a video, or playing the scene of the red car that just appeared. The decoding to text command in this embodiment refers to converting the user's input voice signal into computer-recognizable text content through voice recognition technology, thus completing the text conversion of voice information.

[0042] The current video playback scenario in this embodiment refers to the playback status of the video when the user initiates a search, such as the name of the video being played, the current playback time, and the content of the already played scenes. The intent recognition in this embodiment refers to analyzing the core intent of the user's text command, such as searching for a specific scene, jumping to a target scene, or saving similar scenes, to clarify the user's search purpose.

[0043] The semantic understanding described in this embodiment refers to the in-depth analysis of the semantic logic of text instructions, including core keywords such as protagonist, drinking tea, and red car; keyword associations such as the association between the protagonist and the action of drinking tea; and semantic completion of ambiguous instructions, such as the previous scene corresponding to the scene before the current playback time. The detection keywords in this embodiment refer to extracting core information related to video scene features from text instructions, such as characters, actions, objects, and scenes, as the core basis for retrieval and matching.

[0044] The multidimensional retrieval feature vector described in this embodiment refers to converting the extracted detection keywords into a vector form with the same dimensions as the image frame feature vectors in the database, thereby unifying the format of retrieval requirements and stored data, which facilitates subsequent matching calculations.

[0045] In the specific implementation of step S200, after the user initiates a search request via voice, the system first decodes the voice search command into a text command using voice recognition technology. Then, considering the current video playback context, such as the currently playing video and the current playback progress, it avoids incorrect search scope, such as mistakenly searching for other video resources. Next, it performs intent recognition and semantic understanding on the text command to accurately capture the user's core needs, such as finding specific details in the scene, while extracting detection keywords such as people, actions, and objects. Finally, it transforms these detection keywords into multi-dimensional search feature vectors that match the feature vector dimensions of the scene frames in the database, completing the digital conversion of the search request and preparing for subsequent matching calculations.

[0046] This embodiment of the steps solves the problems of poor interactivity and inability to engage in deep interaction. This is because the invention enables voice retrieval of the currently playing video, breaking the traditional limitation that voice can only search video library resources. It supports deep interaction between the user and the currently playing content, improving operational convenience. Furthermore, this invention also solves the problem of difficulty in recognizing and matching fuzzy or complex commands. Because it combines semantic understanding and intent recognition of the playback scene, it can complete the semantics of fuzzy commands (such as the previous scene or the current scene) and parse the logic of complex commands (such as finding the scene of the main character drinking tea on the balcony in the video), thus improving the accuracy of command recognition.

[0047] This invention also lowers the barrier to entry for users, eliminating the need for manual remote control operation; searches can be initiated solely through voice commands, solving the cumbersome issues of traditional manual fast-forward and rewind operations. This makes it particularly suitable for the elderly and children. More importantly, this invention enables precise matching of search requests with data because it transforms voice requests into standardized multi-dimensional search feature vectors, ensuring consistency with the image frame vector format in the database. This guarantees the accuracy of subsequent matching calculations and improves search precision.

[0048] Step S300: Calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database, and filter out the frames with similarity higher than the preset similarity threshold to form a retrieval candidate matching set; In this embodiment, the matching degree calculation refers to calculating the similarity between the multidimensional retrieval feature vector and the multidimensional image feature vector of each frame stored in the database using vector similarity algorithms such as cosine similarity and Euclidean distance, and using specific numerical values ​​to represent the matching priority.

[0049] The preset similarity threshold mentioned in this embodiment refers to a pre-set matching threshold, such as 0.75, which is used to filter valid matching frames. Frames with a similarity higher than the threshold are considered to meet the user's needs, while those with a similarity lower than the threshold are considered invalid matches.

[0050] The retrieval candidate matching set in this embodiment refers to the set of all screen frames that meet the user's retrieval requirements after matching degree calculation and threshold filtering, including the feature vector, timestamp, matching degree value and other information of each frame.

[0051] In specific implementation of step S300, the multidimensional retrieval feature vector generated in step S200 is matched one by one with the multidimensional image feature vectors of all video frames stored in the vectorized database in step S100. The matching degree between the two is calculated by a vector similarity algorithm. The higher the matching degree value, the more the frame matches the user's retrieval needs. Subsequently, the system of the present invention filters out the frames with matching degrees higher than the preset similarity threshold according to the preset similarity threshold. The relevant information of these frames, including the corresponding timestamp, matching degree, and preview image, is integrated to form a retrieval candidate matching set, providing a basis for the user's subsequent selection or the system's automatic redirection.

[0052] As can be seen, this invention can improve retrieval accuracy and efficiency. It achieves precise matching through vector similarity calculation, avoiding the fuzzy matching problem of traditional retrieval. Furthermore, vector calculation is much faster than original image comparison, significantly reducing retrieval time. This invention also solves the problem of insufficient handling of multiple similar images. It includes the matching degree and timestamp information of all similar images in the candidate matching set, which can be sorted by matching degree. The position of each similar image is marked on a progress bar, facilitating precise user location.

[0053] Furthermore, this invention can reduce invalid search results. By setting a similarity threshold, it can filter out frames that are irrelevant to the user's needs, avoiding a large number of invalid results from interfering with the user's selection and improving the search experience. Moreover, this invention also supports personalized search needs because the matching degree calculation used in this invention can prioritize the presentation of images that best match the user's needs, satisfying the personalized search needs of different users for detailed images, such as some people focusing on human actions and others focusing on scene environment.

[0054] Step S400: Based on the search candidate matching set, receive the user's operation instruction to select one of the matching screens from the search candidate matching set for playback, or control the default jump to the first matching screen for playback.

[0055] In this embodiment, the user operation instructions refer to the operation requirements initiated by the user for the search candidate matching set, such as clicking on a preview image of a certain screen, playing the first matching screen with a voice command, or playing the third screen.

[0056] In this embodiment, selecting one matching frame for playback means that the user selects any frame that meets the requirements from the candidate matching set, and the system jumps to the timestamp position corresponding to that frame to start playback.

[0057] The default jump to the first matching screen in this embodiment means that if the user does not actively select it, the system will sort the screen frames in the candidate matching set from high to low matching degree and automatically jump to the timestamp position corresponding to the first screen frame with the highest matching degree.

[0058] In specific implementation of step S400, after generating the candidate matching set based on step S300, the system of this invention will present the candidate screen frames to the user in the form of preview images, timestamp annotations, and matching degree sorting, such as displaying multiple similar screen preview images at the bottom of the TV screen, and marking the position of each screen with markers on the progress bar; the user can initiate operation commands by clicking with the remote control, using voice commands, etc., to select any screen frame from the candidate matching set; after receiving the operation command, the system will quickly jump to the time point corresponding to the timestamp of the screen frame to play the video; if the user does not make an active selection, the system will default to sorting by matching degree from high to low, and automatically jump to the timestamp position corresponding to the first matching screen frame to play, completing the playback control after retrieval.

[0059] This invention completely solves the problems of cumbersome operation and low efficiency. Users no longer need to manually fast forward, rewind, or search frame by frame; they can quickly locate the target screen simply by selecting from the candidate matching set or waiting for the system's default jump, significantly saving operation time and effort. Furthermore, this invention enhances the interactive experience and flexibility, supporting users to actively select the target screen while providing a system default jump option, catering to the operating habits of different users. For example, users familiar with the operation can choose independently, while users unfamiliar with the operation can rely on the default jump, thus enhancing interactive flexibility.

[0060] Furthermore, this invention enhances the processing capabilities for multiple similar images. The visualization of the candidate matching set and the progress bar annotations allow users to clearly see the position and preview effect of all similar images, solving the problem of inaccurate presentation and selection of multiple similar images in traditional technologies. Moreover, this invention uses precise timestamp-based navigation to ensure that the image frame selected by the user matches the actual playback image, avoiding discrepancies between search results and the target image and improving search reliability.

[0061] In a further embodiment of the present invention, the video content retrieval method based on voice interaction of a television system, wherein step S100 specifically includes: S101. When a video is about to play, first determine whether the current video has historical processing data; In this embodiment of the invention, when a video is about to play, the data storage module of the video content processing module first determines whether there is historical processing data for the current video. If historical data exists, the feature vector index and timestamp of the video are directly retrieved to accelerate the subsequent retrieval process. If no historical data exists, the process proceeds to step S102.

[0062] S102. When there is no historical data, start frame extraction; adopt an open-source multimedia processing framework and use a multi-threaded mechanism to decode the current video stream in real time; and adopt a time-slice sampling mechanism to extract frame images from the decoded video stream according to a preset time interval. In this embodiment of the invention, video frame extraction can obtain continuous frame images from a video played by a video player at a frequency of 1 FPS. Specifically, video stream parsing and frame extraction can be implemented using the FFmpeg open-source framework (an open-source multimedia processing framework), employing the following steps: First, FFmpeg multi-threaded decoding is performed: Utilizing FFmpeg's libavformat (audio / video format processing library) and libavcodec (audio / video codec library), real-time decoding of the video stream is achieved through a multi-threaded mechanism, supporting mainstream video encoding formats such as H.264 and H.265. FFmpeg is a powerful, highly flexible, and widely used open-source multimedia processing framework capable of handling tasks such as audio and video recording, conversion, streaming media transmission, editing, and playback. libavformat is the audio / video format processing library, responsible for handling the muxing and demuxing of multimedia files, i.e., reading and writing various audio and video container formats (such as MP4, MKV, AVI, FLV, TS, HLS, etc.). Libavcodec, the audio / video codec library, provides support for encoding and decoding audio and video data.

[0063] Then, sampling is performed using time-slice sampling: the `av_seek_frame` function precisely locates the video time point, and frames are extracted at a preset frequency of 1 FPS. It also supports passing parameters to set the extraction frequency, ensuring that the extracted frames have accurate timestamp information. The `av_seek_frame` function is a function provided by the libavformat library in FFmpeg, used to locate (jump) to a specified timestamp position in an audio or video file or stream.

[0064] S103. Perform color space conversion and size normalization preprocessing on the extracted image frames; In this embodiment, the frame data preprocessing step specifically involves color space conversion (BGR2RGB) and size normalization (to a uniform 336×336 pixels) of the extracted raw YUV format frame data to adapt to the input requirements of the subsequent feature extraction model.

[0065] S104. Using computer vision and deep learning algorithms, multi-dimensional feature analysis is performed on the preprocessed image frames, and the image frames are vectorized to convert the image frame content into multi-dimensional image feature vectors. This step in the embodiment involves feature vector construction, generating high-dimensional feature vectors for each frame.

[0066] S105. The multidimensional image feature vector and the corresponding video timestamp are hierarchically stored in the database to construct vectorized data containing visual semantic information and generate a retrieval index.

[0067] This step involves data storage, where multidimensional image feature vectors and corresponding timestamps are stored in the database to build a retrieval index.

[0068] Specifically, steps S104 and S105 involve using computer vision technology and deep learning algorithms to perform multi-dimensional feature analysis on the extracted video frames, transforming the image content into high-dimensional feature vectors, and constructing vectorized data containing visual semantic information to provide accurate feature data support for retrieval condition matching.

[0069] Preferably, when analyzing video content, the present invention can employ a multimodal model to achieve a unified vectorized representation of video frames and retrieved text: The model architecture adopts the CLIP (Contrastive Language-Image Pretraining) architecture, using VisionTransformer (ViT-L / 14) as the image encoder and RoBERTa (robust optimized BERT model) as the text encoder. The pretraining corpus contains 1 billion cross-modal data.

[0070] During feature extraction, the image frame is vectorized. Specifically, the preprocessed 336×336 pixel frame image is input into the ViT-L / 14 model, and a 768-dimensional image feature vector is output through the pooling layer.

[0071] In a further embodiment of the present invention, the video content retrieval method based on voice interaction of a television system, wherein step S200 specifically includes: S201. Obtain an analog voice signal with search keywords, convert the analog voice signal into a digital audio signal, and obtain a voice search command; In a specific implementation of this invention, the user's voice signal can be obtained through the voice remote control of a smart TV, and the user's voice signal can be recognized and processed by the TV system's voice processing module.

[0072] The voice remote control is equipped with a high-sensitivity microphone array, which can accurately capture the user's voice input and quickly convert analog voice signals into digital audio signals, which are then stably transmitted to the TV terminal system via Bluetooth communication technology. It also features voice wake-up functionality, allowing it to receive commands from the TV terminal system and enable basic interaction with the user.

[0073] The television system's voice processing module, as the core hub of the system, integrates a voice recognition module and a natural language processing module. The voice recognition module is responsible for efficiently converting the received digital audio signals into text information; the natural language processing module performs in-depth analysis and semantic understanding of the text commands in conjunction with the current context, extracting key search conditions and providing clear basis for subsequent video retrieval.

[0074] In this step, the user can issue a search command via the voice remote control, such as finding a scene of two people riding bicycles. The microphone array of the voice remote control will convert the analog voice signal into a digital audio signal and transmit it to the TV system's voice processing module via Bluetooth.

[0075] S202. Decode the voice search command into a text command; In this embodiment, speech recognition can decode audio signals into text commands through the speech recognition module of the television system's speech processing module.

[0076] S203. The natural language processing module is used to perform intent recognition and semantic understanding on the decoded text instructions, and combined with the current video playback scenario, the detection keywords are extracted and multi-dimensional retrieval feature vectors are generated from the extracted keywords.

[0077] This step in the embodiment is semantic parsing. Specifically, the natural language processing module can perform intent recognition on the text, combine it with the current video playback scenario, extract keywords such as two people, bicycle, and cycling, and generate a high-dimensional retrieval feature vector from the keyword text.

[0078] In practical implementation, this invention vectorizes the retrieved text, that is, it segments the retrieved text output by the natural language processing module (i.e., the detected keywords such as "two people riding bicycles") into words and inputs them into the RoBERTa model to generate a 768-dimensional text feature vector. The RoBERTa model (a robustly optimized BERT model) is a pre-trained language model that systematically optimizes and improves the BERT model.

[0079] In a further embodiment of the present invention, the video content retrieval method based on voice interaction of a television system, wherein step S300 specifically includes: S301, Receive the generated multi-dimensional retrieval feature vector; This step receives the keywords extracted in the previous step, such as two people, bicycle, and cycling, and generates a high-dimensional retrieval feature vector from the keyword text.

[0080] S302. Calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database using the cosine similarity algorithm. In this embodiment of the invention, vector normalization is performed, specifically L2 normalization of the generated multidimensional image feature vector and multidimensional retrieval feature vector to ensure the consistency of the vector space and facilitate similarity calculation. L2 normalization is a vector normalization method that scales a vector so that its L2 norm (Euclidean length) equals 1. The normalized vector lies on a unit hypersphere, preserving its original orientation while eliminating the influence of magnitude.

[0081] In this embodiment of the invention, data storage employs a collaborative architecture combining a vector database and a relational database to achieve efficient vector similarity retrieval. The hybrid storage architecture includes: 1) Vector database: Milvus (vector database) is used as the vector storage engine. It utilizes Milvus's approximate nearest neighbor search algorithm based on HNSW (HierarchicalNavigableSmallWorld) graph to support millisecond-level retrieval of 768-dimensional vectors.

[0082] 2) Relational database: MySQL is used to store video metadata (such as video ID, title, duration) and vector index association information (such as vector ID, timestamp).

[0083] In this embodiment of the invention, the data storage strategy includes: 1) Temporary storage: using an in-memory database to load the vector data of the currently playing video frames to accelerate high-frequency retrieval scenarios. 2) Persistent storage: storing the vector data in separate databases and tables according to video IDs, with each shard containing 1 million vector records to ensure horizontal scalability.

[0084] In this embodiment of the invention, the cosine similarity algorithm is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frames stored in the database, wherein the similarity calculation formula is:

[0085] Where A is the multidimensional retrieval feature vector (text feature vector) and B is the multidimensional image feature vector (frame feature vector) of the video frame, both of which are 768-dimensional vectors after L2 normalization.

[0086] S303. By setting a dynamic similarity threshold, screen frames with similarity higher than the similarity threshold are selected to form a retrieval candidate matching set; In this embodiment, a dynamic threshold can be set, preferably 0.75 in this invention, to quickly filter out frames with similarity higher than the threshold, filter out low-relevance results, and form a candidate matching set.

[0087] A dynamic similarity threshold is set to filter out the top K frames with similarity values ​​higher than the preset threshold, where K is a natural number greater than 1. Specifically, Top-K retrieval can be used: the TOP-K query function of Milvus (a vector database) can be used to quickly obtain the top K frames with the highest similarity. In this embodiment of the invention, K is preferably 10.

[0088] S304. If no image matching the conditions of the multidimensional retrieval feature vector is found, a no-match signal is generated.

[0089] In other words, if no matching image is found in this embodiment of the invention, a no-match result signal is generated.

[0090] In this embodiment of the invention, during rendering and display, based on the retrieved candidate matching set, the system receives user operation instructions to select one matching screen from the candidate matching set for playback, or controls the system to default to playing the first matching screen. Specifically, this invention performs playback control and UI annotation.

[0091] The progress positioning is achieved by the playback control module automatically jumping to the first matching frame (e.g., t10s) based on the search results, triggering the video player to decode the corresponding frame data. This is visualized through the UI; for example, the UI display module marks the matching positions (t10s, t35s) in red on the progress bar, uses blue lines to indicate the association of multiple results, and displays a search result message: "2 related frames found." If no matching results are found, it displays "No related frames found."

[0092] This invention embodiment can also realize multi-result interactive switching, specifically including: command response, such as when the user issues a "next screen" command, the voice processing module parses it and triggers the content retrieval module to call the next matching time point (t35s). Then, playback jump is performed, the playback control module executes the progress jump, and the UI updates the annotations synchronously, forming a closed-loop interaction.

[0093] The present invention will be further described in detail below through specific application examples: like Figure 2 As shown in this specific application embodiment, the video content retrieval method based on voice interaction of a television system includes the following steps: S10, Preprocessing stage; S11, Video is about to start; S12. The video content processing module determines whether there is historical data. The video content processing module of this embodiment consists of a video frame extraction module, a screen content analysis module, a data storage module, and a screen retrieval module. Through collaborative operation, it achieves in-depth processing and efficient retrieval of video content.

[0094] S13. Determine if there is historical data. If yes, proceed to step S14; otherwise, proceed to step S15. S14. Directly retrieve the index; S15. The content frame extraction module extracts frame data; The content frame extraction module in this embodiment adopts a time-slice sampling mechanism, which extracts frames from the currently playing video stream according to a preset time interval. The default sampling frequency is one frame per second (1FPS), which ensures that the acquired frames have temporal continuity and content integrity, providing basic data for subsequent video content analysis.

[0095] S16. Analyze the feature vectors through the content analysis module; The content analysis module in this embodiment, based on computer vision technology and deep learning algorithm system, performs multi-dimensional feature analysis on the extracted video frames, transforms the image content into high-dimensional feature vectors, and constructs vectorized data containing visual semantic information, providing accurate feature data support for retrieval condition matching.

[0096] S17. Store the feature vector in the data storage module and generate index data; The data storage module in this embodiment adopts a hierarchical storage strategy to temporarily store and persistently manage the vectorized data output by the image content analysis module and the corresponding video timestamps. Temporary storage ensures fast data reading and writing and efficient access, meeting the data processing needs of real-time retrieval scenarios; persistent storage enables long-term data storage and fast querying, providing data support for functions such as multi-round retrieval and historical retrieval record management.

[0097] S20. When voice search is required, the voice remote control can be used to obtain the voice search audio signal or switch command. In this specific application embodiment, the system architecture mainly consists of four core parts: a voice remote control, a TV system voice processing module, a video player, and a video content processing module. Among them, the video content processing module and the video player module are responsible for the core processing functions of in-depth analysis, retrieval and matching, and interactive display of video content.

[0098] Voice remote control: Equipped with a high-sensitivity microphone array, it can accurately capture the user's voice input and quickly convert analog voice signals into digital audio signals, which are then stably transmitted to the TV terminal system via Bluetooth communication technology. It also features voice wake-up functionality, allowing it to receive commands from the TV terminal system and achieve basic interaction with the user.

[0099] S21, TV system voice processing module: recognition of text commands enters S22, recognition of command parsing enters S24; The television system voice processing module in this embodiment serves as the core hub of the system, integrating a voice recognition module and a natural language processing module. The voice recognition module is responsible for efficiently converting the received digital audio signals into text information; the natural language processing module performs in-depth analysis and semantic understanding of the text commands in conjunction with the current context, extracting key search conditions to provide clear basis for subsequent video retrieval.

[0100] S22. Semantic parsing, structured retrieval vector; S23, Video content processing module; S24. According to the parsed instructions, enter the content retrieval module; The content retrieval module in this embodiment implements the vector similarity calculation function, quickly filtering out frame images with similarity higher than the threshold to form a candidate matching set.

[0101] S25. Query the matching results from the data storage module; S26. Based on the generated matching set, call the playback control module; S27. The UI display module displays playback position information; S28. Control the video player to play and display according to the progress instructions and annotation instructions.

[0102] The video player in this specific embodiment is mainly responsible for video decoding, smooth playback, and high-quality rendering. It also includes two sub-modules: a playback control module and a UI display module. The playback control module issues commands to precisely implement operations such as playing, pausing, fast forwarding, and rewinding the video. The UI display module marks the position of the searched frames on the video playback progress bar using different colors, icons, or marker styles based on the received search result list. If multiple frames meet the criteria, the UI display module uses lines or special symbols to indicate the correlation between the frame positions in chronological order.

[0103] The following example illustrates a specific application scenario: For instance, in the scenario of "finding images of two people riding bicycles," the following steps are performed: 1) Preprocessing stage: Before the video is played, the data storage module of the video content processing module first determines whether there is historical processing data for the current video. Assuming that there is no historical processing data for the current video, the content frame extraction module will obtain continuous frame images from the video player at a frequency of 1 FPS.

[0104] The content analysis module processes these frames to generate high-dimensional feature vectors. The data storage module stores these feature vectors along with their corresponding timestamps into a database to build a retrieval index.

[0105] 2) Voice Input and Semantic Resolution: The user issues a command via voice remote control: "Find the scene of two people riding bicycles." The voice remote control collects the voice signal, converts it into a digital audio signal, and transmits it to the TV system's voice processing module via Bluetooth.

[0106] The speech recognition module of the TV system's speech processing module decodes the audio signal into text commands. The natural language processing module performs intent recognition and semantic understanding on the text commands, extracts keywords such as "two people," "bicycle," and "cycling," and generates corresponding structured retrieval vectors.

[0107] 3) Retrieval and Matching: The "Content Retrieval Module" of the video content processing module receives the structured retrieval vector. This module uses a cosine similarity algorithm to calculate the matching degree between the retrieval vector and the pre-stored feature vectors of video frames. By setting a dynamic threshold (e.g., the default 0.75), the system quickly filters out frames with a similarity higher than the threshold.

[0108] In this example, the content retrieval module detected the feature "two people riding bicycles" at t10s and t35s of the video, with similarity scores of 0.88 and 0.91, respectively, forming a candidate matching set.

[0109] 4) Playback Display and UI Annotation: Based on the search results, the playback control module jumps to the first matching frame by default, i.e., t10s, and triggers the video player to decode and play the data of that frame. The UI display module marks the two matching positions, t10s and t35s, in red on the video playback progress bar.

[0110] Meanwhile, the UI display module uses a blue line to indicate the correlation between the two matching points and displays a prompt message: "Two related screens found".

[0111] 5) Interactive Switching: After viewing the screen at t10s, if the user wants to view the next matching screen, they can issue the voice command again: "Next". After the voice processing module parses the command, it triggers the content retrieval module to call the next matching time point, i.e., t35s.

[0112] The playback control module jumps the progress bar, switching the video playback position to t35s. The UI display module simultaneously updates the progress bar markers, completing the closed-loop interaction, allowing the user to continue watching or perform other operations.

[0113] In this embodiment of the invention, the UI display module adopts cross-platform UI rendering technology, specifically, different UI rendering schemes can be used for different TV system platforms: For example, in an Android TV system, video playback uses the AndroidMediaPlayerAPI (an interface in the Android platform used to control audio and video playback) to implement video decoding and playback control. UI (user interface) rendering uses native Android View components (such as SeekBar and TextView) to draw progress bars and search result labels.

[0114] The WebOS TV system uses the HTML5 Video tag for video playback, combined with MediaSourceExtensions (MSE) to support streaming media playback. UI rendering utilizes interactive components built on the Vue3 framework, and CSS3 is used to achieve smooth transitions for search result annotations and progress bar annotations.

[0115] The video content retrieval method based on voice interaction in a television system proposed in this invention has broad application prospects in home viewing scenarios. For example, when family members watch a movie or TV series together, children may become curious about a particular character, animal, or object and want to quickly find all the scenes in which that element appears; or, when watching a cooking program, users may want to quickly review the steps of making a dish in order to learn how to cook it. In addition, during live sports broadcasts, users may want to quickly locate a highlight goal or foul.

[0116] Furthermore, in order to avoid conflicts and fuzzy recognition when receiving voice commands from multiple users simultaneously, this invention proposes an optimized scheme for the video content retrieval method based on voice interaction in a television system. This scheme involves adding a voiceprint recognition and user management module and an instruction priority scheduling module to the voice processing module of the television system. The voiceprint recognition and user management module is used for voiceprint registration and storage, real-time voiceprint comparison, and user identity management. Voiceprint registration refers to guiding family members to record their own voice samples upon first use or through the user management interface, generating unique voiceprint features, which are then bound and stored with user IDs (such as "Dad," "Mom," and "Xiaoming"). Real-time voiceprint comparison refers to the process where, when the voice remote control collects voice input and transmits it to the TV system's voice processing module, real-time voiceprint comparison analyzes the voiceprint features in the digital audio signal in parallel and compares them with the registered voiceprints to identify the user who issued the command.

[0117] User identity management refers to the management of registered user information, including user ID, voiceprint features, and optional user preference settings (such as default search scope, display style, etc.).

[0118] The instruction priority scheduling module is mainly used for instruction queuing, priority evaluation, and conflict resolution.

[0119] The instruction queue receives identified text instructions containing user identification information and places them into the queue in chronological order. Priority evaluation assesses the instructions in the queue according to preset rules (e.g., adult instructions have higher priority than child instructions, or the user who most recently successfully executed an instruction has a temporarily higher priority, or the user can define their own priority).

[0120] Conflict resolution refers to the process where, when multiple concurrent or nearly concurrent instructions exist in the instruction queue, the instruction with the highest priority is selected based on priority evaluation and passed to the natural language processing module for parsing. Lower-priority instructions can be temporarily suspended, the user can be prompted to resend the instruction, or the current instruction can be processed first.

[0121] Therefore, based on the above optimization scheme, the embodiments of the present invention further provide an optimization scheme. An embodiment of this optimization scheme provides a video content retrieval method based on voice interaction in a television system, comprising the following steps: S901. Pre-enter and register voiceprint information corresponding to user ID, and set corresponding user preferences and instruction priorities for voiceprint information corresponding to each user ID, and store them.

[0122] Upon first use or in the user management interface, family members are guided to record their own voice samples, generating unique voiceprint features, which are then bound to and stored with user IDs (such as "Dad", "Mom", "Xiaoming").

[0123] S902. When a voice search command is received, the voiceprint features in the voice search command are analyzed and compared with the registered voiceprint information corresponding to the user ID to identify the user ID identity that issued the voice search command. S903. Decode the voice search command into a text command with user ID; For example, when Xiaoming says "Find the puppy" and his mother says "Find the kitten" at the same time, the voice remote control collects the mixed voice signal. In this embodiment of the invention, after receiving the signal through the TV system's voice processing module, the voiceprint recognition and user management module will identify the voiceprint features of Xiaoming and his mother, and parse the two voices as "Xiaoming: Find the puppy" and "Mother: Find the kitten" respectively.

[0124] S904. Place the identified text commands with user IDs into the command queue in chronological order. As mentioned above, the instruction priority scheduling function puts these two instructions with user identities into the queue.

[0125] S905. Prioritize text commands with user IDs in the command queue according to preset rules. S906. When there are multiple concurrent or nearly concurrent text commands in the command queue, the text command with the highest priority and user ID is selected according to the priority evaluation result and handed over to the natural language processing module for intent recognition and semantic understanding. S907. For text commands with user IDs that have a priority below the highest, control to temporarily suspend intent recognition and semantic understanding, prompt the user to resend the command, or process the command only after the current command has been processed.

[0126] As mentioned above, based on the preset priority rules, for example, the default priority of the "Mom" instruction is higher than that of "Xiaoming", the instruction priority scheduling function will send the instruction "Mom: Find the kitten" to the natural language processing module for parsing and retrieval first.

[0127] The system then performs a search for "find the kitten" and displays the results on the UI. Simultaneously, the system can issue a voice prompt to "Xiaoming": "Xiaoming, please wait a moment, Mom's instructions are being processed." Or, it can automatically process Xiaoming's instructions after Mom's instructions are completed.

[0128] As can be seen, by introducing a voiceprint recognition and user management module, the system can accurately identify the specific user issuing the voice command. This solves the problem of voice command confusion among multiple users; even if multiple users speak simultaneously, the system can distinguish between different sound sources.

[0129] The instruction priority scheduling module further addresses the issue of multi-user instruction conflicts. When multiple user instructions are identified, the system no longer processes them blindly but intelligently schedules them according to preset priority rules. For example, if a father and child issue instructions simultaneously, the system can prioritize the father's instruction or prompt the child to speak later. This mechanism avoids identification errors and system chaos caused by concurrent instructions, ensuring the accuracy and efficiency of instruction processing.

[0130] Thus, without introducing complex artificial intelligence algorithms, this invention significantly improves the accuracy, efficiency, and user experience of TV video retrieval in multi-user home scenarios by recognizing user identities and intelligently scheduling commands, and solves specific technical problems of multi-user voice command conflicts and fuzzy recognition.

[0131] Exemplary device like Figure 3 As shown, this embodiment of the invention provides a video content retrieval device based on voice interaction in a television system. The device includes: The video content processing module 310 is used to extract frame images from the video at a preset frequency when the video is played for the first time, perform multi-dimensional feature analysis, convert them into multi-dimensional image feature vectors, and store the multi-dimensional image feature vectors and corresponding video timestamps in a hierarchical manner in the database to construct vectorized data containing visual semantic information. The voice command acquisition and semantic parsing module 320 is used to acquire voice retrieval commands, decode the voice retrieval commands into text commands, and combine the current video playback scenario to perform intent recognition and semantic understanding on the text commands, extract detection keywords, and generate multi-dimensional retrieval feature vectors. The video content retrieval and matching module 330 is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database, and to filter out the frame with a similarity higher than a preset similarity threshold to form a retrieval candidate matching set. The playback control module 340 is used to receive user operation instructions based on the search candidate matching set to select one of the matching screens from the search candidate matching set for playback, or to control the default jump to the first matching screen for playback, as described above.

[0132] Based on the above embodiments, the present invention also provides a smart terminal, which can be a smart TV, and its principle block diagram can be as follows. Figure 4As shown. The intelligent terminal includes a processor, memory, network interface, display screen, and database connected via a system bus.

[0133] The memory stores one or more programs configured to be executed by a processor to implement the video content retrieval method based on voice interaction of a television system as described in the above embodiments.

[0134] Here, "intelligent terminal" refers to a smart computer or similar device with data processing capabilities. The memory can be internal memory, flash memory, hard disk, or cloud storage, used to store program code and various data such as historical video data. The processor can be a central processing unit (CPU), used to execute the algorithmic logic within the program. The program includes a video content retrieval method based on the television system's voice interaction.

[0135] In a further embodiment, a smart terminal of this embodiment includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: When the video is played for the first time, the video frames are extracted from the video at a preset frequency, and multi-dimensional feature analysis is performed to convert them into multi-dimensional image feature vectors. The multi-dimensional image feature vectors and the corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information. The system acquires a voice search command, decodes the voice search command into a text command, and combines the current video playback scenario to perform intent recognition and semantic understanding on the text command, extracts detection keywords, and generates a multi-dimensional search feature vector. Calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frames stored in the database, and filter out the frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set; Based on the search candidate matching set, the system receives user operation instructions to select one of the matching screens from the search candidate matching set for playback, or controls the default jump to the first matching screen for playback, as described above.

[0136] The step of extracting frames from the video at a preset frequency when the video is played for the first time, performing multi-dimensional feature analysis to convert them into multi-dimensional image feature vectors, and storing the multi-dimensional image feature vectors and corresponding video timestamps hierarchically in a database to construct vectorized data containing visual semantic information includes: When a video is about to play, first check if the current video has historical processing data; If there is no historical data, frame extraction is initiated; a multi-threaded mechanism is used to decode the current video stream in real time based on an open-source multimedia processing framework; and a time-slice sampling mechanism is used to extract frames from the decoded video stream according to a preset time interval. The extracted image frames undergo color space conversion and size normalization preprocessing. The preprocessed image frames are analyzed for multi-dimensional features using computer vision and deep learning algorithms, and the image frames are vectorized to transform the image frame content into multi-dimensional image feature vectors. The multidimensional image feature vectors and their corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information and generate a retrieval index.

[0137] The steps of obtaining the voice search command, decoding the voice search command into a text command, and combining the current video playback scenario to perform intent recognition and semantic understanding on the text command, extract detection keywords, and generate a multi-dimensional search feature vector include: Acquire an analog speech signal with search keywords, convert the analog speech signal into a digital audio signal, and obtain a voice search command; Decode the voice search command into a text command; The natural language processing module is used to perform intent recognition and semantic understanding on the decoded text instructions. Combined with the current video playback scenario, detection keywords are extracted and multi-dimensional retrieval feature vectors are generated from the extracted keywords.

[0138] The step of calculating the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database, and filtering out frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set, further includes: Receive and acquire the generated multidimensional retrieval feature vector; The matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database is calculated using the cosine similarity algorithm. By setting a dynamic similarity threshold, frames with a similarity higher than the threshold are filtered out to form a retrieval candidate matching set; If no image is found that matches the conditions of the multidimensional retrieval feature vector, a no-match signal is generated.

[0139] The step of calculating the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database, and filtering out frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set, further includes: The cosine similarity algorithm is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database. The similarity calculation formula is as follows:

[0140] Where A is the multidimensional retrieval feature vector and B is the multidimensional image feature vector of the video frame, both of which are 768-dimensional vectors after L2 normalization. Set a dynamic similarity threshold to filter out the top K frames with a similarity higher than the preset similarity threshold, where K is a natural number greater than 1.

[0141] The video content retrieval method based on voice interaction in a television system, wherein, prior to the step of obtaining the voice retrieval command, further includes: Pre-enter and register voiceprint information corresponding to user IDs, and set corresponding user preferences and command priorities for each user ID's voiceprint information, and store them.

[0142] The steps of obtaining the voice search command, decoding the voice search command into a text command, and performing intent recognition and semantic understanding on the text command in conjunction with the current video playback scenario, extracting detection keywords, and generating a multi-dimensional search feature vector further include: When a voice search command is received, the voiceprint features in the voice search command are analyzed and compared with the registered voiceprint information corresponding to the user ID to identify the user ID that issued the voice search command. The voice search command is decoded into a text command with the user's ID. The identified text commands with user IDs are placed into the command queue in chronological order. Prioritize text commands with user IDs in the command queue according to preset rules; When there are multiple concurrent or nearly concurrent text commands in the command queue, the text command with the highest priority and user ID is selected based on the priority evaluation result and handed over to the natural language processing module for intent recognition and semantic understanding. For text commands with user IDs that have a priority below the highest, control can temporarily suspend intent recognition and semantic understanding, prompt the user to resend the command, or process the command only after the current command has been processed, as described above.

[0143] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, enables an electronic device to perform the steps of any of the methods described above, specifically as described above.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0145] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video content retrieval method based on voice interaction in a television system, characterized in that, include: When the video is played for the first time, the video frames are extracted at a preset frequency and multi-dimensional feature analysis is performed to convert them into multi-dimensional image feature vectors. The multi-dimensional image feature vectors and the corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information. The system acquires a voice search command, decodes the voice search command into a text command, and combines the current video playback scenario to perform intent recognition and semantic understanding on the text command, extracts detection keywords, and generates a multi-dimensional search feature vector. Calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frames stored in the database, and filter out the frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set; Based on the search candidate matching set, the system receives user operation instructions to select one of the matching screens from the search candidate matching set for playback, or controls the default jump to the first matching screen for playback.

2. The video content retrieval method based on voice interaction in a television system according to claim 1, characterized in that, The steps of extracting frames from the video at a preset frequency when the video is first played, performing multi-dimensional feature analysis to convert them into multi-dimensional image feature vectors, and storing the multi-dimensional image feature vectors and corresponding video timestamps hierarchically in a database to construct vectorized data containing visual semantic information include: When a video is about to play, first check if the current video has historical processing data; If there is no historical data, frame extraction is initiated; a multi-threaded mechanism is used to decode the current video stream in real time based on an open-source multimedia processing framework; and a time-slice sampling mechanism is used to extract frames from the decoded video stream according to a preset time interval. The extracted image frames undergo color space conversion and size normalization preprocessing. The preprocessed image frames are analyzed using multi-dimensional features based on computer vision and deep learning algorithms. The image frames are then vectorized, and the image frame content is transformed into multi-dimensional image feature vectors. The multidimensional image feature vectors and their corresponding video timestamps are hierarchically stored in the database to construct vectorized data containing visual semantic information and generate a retrieval index.

3. The video content retrieval method based on voice interaction in a television system according to claim 1, characterized in that, The steps of obtaining the voice search command, decoding the voice search command into a text command, and combining it with the current video playback scenario to perform intent recognition and semantic understanding on the text command, extract detection keywords, and generate a multi-dimensional search feature vector include: Acquire an analog speech signal with search keywords, convert the analog speech signal into a digital audio signal, and obtain a voice search command; Decode the voice search command into a text command; The natural language processing module is used to perform intent recognition and semantic understanding on the decoded text instructions. Combined with the current video playback scenario, detection keywords are extracted and multi-dimensional retrieval feature vectors are generated from the extracted keywords.

4. The video content retrieval method based on voice interaction in a television system according to claim 1, characterized in that, The step of calculating the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frames stored in the database, and filtering out frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set includes: Receive and acquire the generated multidimensional retrieval feature vector; The matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database is calculated using the cosine similarity algorithm. By setting a dynamic similarity threshold, frames with a similarity higher than the threshold are filtered out to form a retrieval candidate matching set; If no image is found that matches the conditions of the multidimensional retrieval feature vector, a no-match signal is generated.

5. The video content retrieval method based on voice interaction in a television system according to claim 1, characterized in that, The step of calculating the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frames stored in the database, and filtering out frames with similarity higher than a preset similarity threshold to form a retrieval candidate matching set, further includes: The cosine similarity algorithm is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vectors of video frames stored in the database. The similarity calculation formula is as follows: ； Where A is the multidimensional retrieval feature vector and B is the multidimensional image feature vector of the video frame, both of which are 768-dimensional vectors after L2 normalization. Set a dynamic similarity threshold to filter out the top K frames with a similarity higher than the preset similarity threshold, where K is a natural number greater than 1.

6. The video content retrieval method based on voice interaction in a television system according to claim 1, characterized in that, The step of obtaining the voice search command is preceded by: Pre-enter and register voiceprint information corresponding to user IDs, and set corresponding user preferences and command priorities for each user ID's voiceprint information, and store them.

7. The video content retrieval method based on voice interaction in a television system according to claim 6, characterized in that, The steps of obtaining the voice search command, decoding the voice search command into a text command, and combining the current video playback scenario to perform intent recognition and semantic understanding on the text command, extract detection keywords, and generate a multi-dimensional search feature vector further include: When a voice search command is received, the voiceprint features in the voice search command are analyzed and compared with the registered voiceprint information corresponding to the user ID to identify the user ID that issued the voice search command. The voice search command is decoded into a text command with the user's ID. The identified text commands with user IDs are placed into the command queue in chronological order. Prioritize text commands with user IDs in the command queue according to preset rules; When there are multiple concurrent or nearly concurrent text commands in the command queue, the text command with the highest priority and user ID is selected based on the priority evaluation result and handed over to the natural language processing module for intent recognition and semantic understanding. For text commands with user IDs that have a priority below the highest, control can temporarily suspend intent recognition and semantic understanding, prompt the user to resend the command, or process the command only after the current command has been processed.

8. A video content retrieval device based on voice interaction in a television system, characterized in that, The device includes: The video content processing module is used to extract frame images from the video at a preset frequency when the video is played for the first time, perform multi-dimensional feature analysis, convert them into multi-dimensional image feature vectors, and store the multi-dimensional image feature vectors and corresponding video timestamps in a hierarchical manner in the database to construct vectorized data containing visual semantic information. The voice command acquisition and semantic parsing module is used to acquire voice retrieval commands, decode the voice retrieval commands into text commands, and combine the current video playback scenario to perform intent recognition and semantic understanding on the text commands, extract detection keywords, and generate multi-dimensional retrieval feature vectors. The video content retrieval and matching module is used to calculate the matching degree between the multidimensional retrieval feature vector and the multidimensional image feature vector of the video frame stored in the database, and to filter out the frame with a similarity higher than a preset similarity threshold to form a retrieval candidate matching set. The playback control module is used to receive user operation instructions based on the search candidate matching set, select one of the matching screens from the search candidate matching set for playback, or control the default jump to the first matching screen for playback.

9. A smart terminal, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include steps for performing the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it enables the electronic device to perform the steps of the method as described in any one of claims 1-7.

Images