Data query method and device and electronic equipment
By identifying video transition points to generate shot and scene description text, and using a question-answering model to provide accurate plot queries, this technology solves the problem of lack of narrative logic and coherence in existing descriptive texts, and improves the user's plot query experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING QIYI CENTURY SCI & TECH CO LTD
- Filing Date
- 2026-01-21
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies for generating video description text lack narrative logic and coherence, resulting in a poor user experience when searching for story content.
By identifying shot transition points in the video, the video is divided into shot segments, generating shot description text, scene description text, and video summary text. Then, a question-answering model is used to generate query answers based on semantic similarity and timestamp information.
It provides more accurate plot search results, improving the user's search experience.
Smart Images
Figure CN121935366A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data query method, apparatus and electronic device. Background Technology
[0002] Video description text is used to describe the plot content of a video, and can take the form of a video synopsis, summary, or exposition of the video content. Related techniques typically involve extracting key video frames or high-quality video clips (highlight segments) from the video footage, and then piecing together these extracted key frames or high-quality video clips to generate the corresponding video description text. However, this method of generating video description text results in a lack of narrative logic and coherence, leading to low-quality description text.
[0003] Therefore, the descriptive text generated based on the above method cannot provide users with a relatively accurate plot query function, thus affecting the user's plot query experience. Summary of the Invention
[0004] This application provides a data query method, apparatus, and electronic device that can provide users with a more accurate plot query function and ensure the user's plot query experience.
[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a data query method is provided, the method comprising: Receive query information from the client; Based on semantic similarity, target descriptive text is determined for the query information in the database; the database includes descriptive text generated based on the first video, the descriptive text including shot description text, scene description text and video summary text, the shot description text is generated based on shot segments in the first video, the shot segments are determined by identifying shot transition points in the first video, the scene description text is generated based on shot description text corresponding to shot segments that constitute the same scene, and the video summary text is generated based on scene description text corresponding to scenes in the first video; The model prompts and the target description text are input into the question-answering model. The model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target description text, and the answer content is returned to the client.
[0006] In an optional implementation, the database further includes timestamp information corresponding to each descriptive text. The timestamp information is used to identify the playback time of the video segment corresponding to the descriptive text in the first video. The step of inputting model prompts and the target descriptive text into the question-answering model, and having the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target descriptive text, includes: The model prompts, the target description text, and its timestamp information are input into the question-answering model. The model prompts drive the question-answering model to generate a timestamped answer based on the target description text and its timestamp information, which serves as the answer to the query information.
[0007] In one optional implementation, the answer content further includes playback link information, which is used to trigger the playback of the first video based on the timestamp.
[0008] In one optional implementation, the shot description text is generated based on at least one of the subtitle fragments, character information, audio fragments, background music, and video footage of the corresponding shot segment. The character information includes at least one of the character name, character tone information, and character expression information identified from the shot segment. The scene description text is generated based on the shot description texts corresponding to shot segments constituting the same scene and the subtitle fragments of the scene.
[0009] In one optional implementation, the scenario is obtained by clustering the video clips based on the visual similarity and temporal proximity between the video frames in each clip.
[0010] In an optional implementation, the database further includes episode identifiers for each descriptive text generated based on the first video. The episode identifiers are used to identify the playback order of the first video within a video sequence, where the video sequence includes multiple videos with a playback order, including the first video. The step of inputting model prompts and the target descriptive text into the question-answering model, and having the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target descriptive text, includes: The model prompts, the target description text, and its episode identifier are input into the question-answering model. The model prompts drive the question-answering model to generate answer content with episode identifiers based on the target description text and its episode identifiers, which serves as the answer content corresponding to the query information.
[0011] In one optional implementation, the description text further includes a main storyline description text corresponding to the video sequence, wherein the main storyline description text is generated based on the video summary text corresponding to each video in the video sequence.
[0012] Based on the above technical solution, the data query method of this application, upon receiving query information, can determine the target descriptive text for the query information from a database including shot description text, scene description text, and video summary text, and generate answer content based on the target descriptive text using a question-answering model. Therefore, this application can provide users with more accurate query results for video description text, improving the user's plot query experience.
[0013] Secondly, a data query device is provided, the device comprising: The first receiving module is used to receive query information from the client; The first determining module is used to determine target descriptive text for the query information in a database based on semantic similarity; the database includes descriptive text generated based on the first video, the descriptive text including shot description text, scene description text and video summary text, the shot description text being generated based on shot segments in the first video, the shot segments being determined by identifying shot transition points in the first video, the scene description text being generated based on shot description text corresponding to shot segments constituting the same scene, and the video summary text being generated based on scene description text corresponding to scenes in the first video; The first generation module is used to input model prompts and the target description text into the question-answering model, and the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target description text, and return the answer content to the client.
[0014] Thirdly, an electronic device is provided, including a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any of the preceding aspects.
[0015] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method as described in any of the preceding aspects.
[0016] The technical effects of the above aspects can be used for reference, and will not be elaborated further here. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart illustrating a data query method provided in an embodiment of this application; Figure 2 A flowchart illustrating a text generation method provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0020] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0021] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.
[0022] The descriptive text of a video is used to describe the plot content of the video. Specifically, it can take the form of a video synopsis, video summary, or video abstract that summarizes, explains, or elaborates on the video content.
[0023] The relevant techniques typically involve extracting key video frames or high-quality video clips (highlight segments) from video footage, and then piecing together the extracted key video frames or high-quality video clips to generate corresponding video description text. However, the above method of generating video description text results in a lack of narrative logic and coherence, leading to low-quality video description text.
[0024] Therefore, the descriptive text generated based on the above method cannot provide users with a relatively accurate plot query function, thus affecting the user's plot query experience.
[0025] To address this, this application provides a data query method. First, it receives query information from a client. Second, it determines target descriptive text for the query information in a database based on semantic similarity. The database includes descriptive text generated from a first video. This descriptive text includes shot description text, scene description text, and video summary text. Shot description text is generated based on shot segments from the first video, which are determined by identifying shot transition points in the first video. Scene description text is generated based on shot description text corresponding to shot segments constituting the same scene. Video summary text is generated based on scene description text corresponding to different scenes in the first video. Then, model prompts and the target descriptive text are input into a question-answering model. The model prompts drive the question-answering model to generate an answer corresponding to the query information based on the target descriptive text, and the answer is returned to the client.
[0026] Upon receiving a query, this application can determine the target descriptive text from a database including shot description text, scene description text, and video summary text, and then generate an answer based on that target descriptive text using a question-answering model. Therefore, this application can provide users with more accurate query results for video descriptive text, improving their experience in searching for storylines.
[0027] See Figure 1 This is a flowchart illustrating a data query method provided in an embodiment of this application. The method can be executed by an electronic device, a chip or module applied in an electronic device, or a data query system. This data query system can be deployed in one electronic device or jointly deployed by multiple electronic devices. Figure 1 As shown, the method includes the following steps: S101: Receive query information from the client.
[0028] The query information can be text or voice descriptions expressed in complete natural language. In the embodiments of this application, the specific content of the query information is not limited. Optionally, the query information may include plot-related query information for the first video, query information related to scene segments or shot segments in the first video, etc.
[0029] The information retrieved can be text or voice input by the user through the client's input window.
[0030] S102: Determine the target description text for the query information in the database based on semantic similarity.
[0031] The database includes descriptive text generated based on the first video. The descriptive text includes shot description text, scene description text, and video summary text. The shot description text is generated based on shot segments in the first video, and the shot segments are determined by identifying shot transition points in the first video. The scene description text is generated based on shot description text corresponding to shot segments that constitute the same scene. The video summary text is generated based on scene description text corresponding to scenes in the first video.
[0032] After receiving the query information, the database can be used to determine the descriptive text that meets the semantic similarity criteria with the query information through semantic similarity matching, and this text will be used as the target descriptive text for the query information.
[0033] Among them, the descriptive text that meets the semantic similarity condition can include the descriptive text with the highest semantic similarity or the descriptive text with semantic similarity not less than a preset similarity threshold.
[0034] In one optional implementation, query information from the client can be converted into a query vector. This query vector is then matched with the feature vectors of each descriptive text in the database to determine the feature vectors that satisfy the similarity criteria with the query vector. The descriptive text corresponding to these feature vectors is then used as the target descriptive text. For example, cosine similarity can be used to calculate the similarity between the converted query vector and the feature vectors corresponding to each descriptive text.
[0035] The method for generating the descriptive text included in the database of this application embodiment will be described in subsequent embodiments and will not be repeated here.
[0036] S103: Input the model prompt words and the target description text into the question answering model. The model prompt words drive the question answering model to generate the answer content corresponding to the query information based on the target description text, and return the answer content to the client.
[0037] In this embodiment, model prompts are obtained, and the model prompts and the target description text corresponding to the query information are input into the question-answering model. The model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target description text. The model prompts may include at least one of user-input prompts and system prompts, and the model prompts are used to describe the specific requirements for the answer content.
[0038] In one scenario, the response to the query includes the text content of the target description, used to answer the user's query.
[0039] In another scenario, each descriptive text in the database has a corresponding timestamp, which is used to identify the playback time of the video segment corresponding to the descriptive text in its respective video (such as the first video).
[0040] Therefore, while matching the target description text from the database, the timestamp information of the target description text is also retrieved from the database. Then, the model prompts, the target description text, and its timestamp information are input into the question-answering model. The model prompts drive the question-answering model to generate timestamped answer content based on the target description text and its timestamp information, serving as the answer content corresponding to the query information. In other words, the answer content for the query information can also include a timestamp, used to represent the playback time of the video segment corresponding to the description text in the answer content within its respective video.
[0041] In another scenario, the response to the query information may include not only the timestamp of the target description text in the response, but also its playback link information, which is used to trigger the playback of the video to which it belongs (such as the first video) based on the timestamp.
[0042] In another scenario, the descriptive texts generated based on the first video in the database can also have episode identifiers. These episode identifiers identify the playback order of the first video within its respective video sequence, which includes multiple videos in a playback order, including the first video. For example, the video sequence could be a television series, and the episode identifiers would identify the episode number of each episode within the series.
[0043] Therefore, while matching the target description text from the database, the TV series identifier of the target description text is also retrieved from the database. Then, the model prompt, the target description text, and its TV series identifier are input into the question-answering model. The model prompt drives the question-answering model to generate answer content with the TV series identifier based on the target description text and its TV series identifier, serving as the answer content corresponding to the query information. In other words, the answer content for the query information can also include the TV series identifier, used to represent the playback order of the video to which the description text in the answer content belongs within its respective video sequence.
[0044] In another scenario, the response to the query may include the text content of the target description and at least one of the following: timestamp, playback link information, and episode identifier. Therefore, some embodiments of this application can provide users with rich and comprehensive response content, enhancing their data query experience.
[0045] To facilitate understanding of the above embodiments, the processes of S101 to S103 will be described below with reference to a specific example.
[0046] For example, suppose the database includes video summary text corresponding to all episode videos in TV series 1, scene description text corresponding to all scenes included in each episode video, and shot description text corresponding to all shot clips in each scene. Users can enter query information on the client, such as entering query information on the artificial intelligence conversation page.
[0047] For example, if the query is "I want to find the scene in TV series 'Yu XX and Cui XX' who pretend to be a married couple and meet for the first time at home," then in response to the user's query, the database will determine that the scene description text corresponding to scene 1, "Yu XX returns home and meets Cui XX, the guerrilla leader who has been arranged by the organization to pretend to be his wife, for the first time. Due to their vastly different lifestyles and personalities, their first meeting is filled with testing and hilarious misunderstandings," meets the similarity criteria of the aforementioned query. Furthermore, the episode video to which scene 1 belongs is "Episode 3," with a timestamp of "00:05:20 - 00:12:45." Therefore, an answer such as "This scene appears in episode 3, starting at approximately 5 minutes and 20 seconds" can be generated.
[0048] Optionally, in some embodiments of this application, in addition to using relevant models to generate descriptive text for the first video, the causal relationship of the plot content in the first video can also be identified, such as the occurrence of plot A leading to the occurrence of plot B. The main plot and side plots can also be automatically identified and output.
[0049] In the data query method provided in this application embodiment, firstly, a query information from a client is received; secondly, a target descriptive text is determined for the query information in a database based on semantic similarity. The database includes descriptive text generated based on a first video. This descriptive text includes shot description text, scene description text, and video summary text. The shot description text is generated based on shot segments in the first video, which are determined by identifying shot transition points in the first video. The scene description text is generated based on shot description text corresponding to shot segments constituting the same scene. The video summary text is generated based on scene description text corresponding to scenes in the first video. Then, model prompts and the target descriptive text are input into a question-answering model. The model prompts drive the question-answering model to generate an answer corresponding to the query information based on the target descriptive text, and the answer is returned to the client.
[0050] Upon receiving a query, this application can determine the target descriptive text from a database including shot description text, scene description text, and video summary text, and then generate an answer based on that target descriptive text using a question-answering model. Therefore, this application can provide users with more accurate query results for video descriptive text, improving their experience in searching for storylines.
[0051] Based on the above embodiments, this application also provides a method for generating descriptive text, referring to... Figure 2 .
[0052] S201: Acquire the first video and identify multiple shot segments by recognizing the shot transition points in the first video.
[0053] In some embodiments of this application, the first video can be any video, such as a single long video like an episode of a TV series or a movie.
[0054] In some embodiments of this application, the first video is divided into multiple shot segments by identifying the transition points of the video frames in the first video. The identification of the transition points of the video frames can refer to the extraction of the transition point features of the video frames in the first video through technical means.
[0055] Optionally, the first video can be divided into multiple shot segments, and a shot segment can include at least one video frame, that is, at least one frame of image.
[0056] Specifically, relevant tools can be used to identify shot transition points in the first video, and then the video can be segmented based on these transition points. For example, the PySceneDetect library can be used to segment the video. PySceneDetect is a video analysis tool library that can automatically detect shot transition locations in the first video based on various physical features such as pixel differences, color histograms, and optical flow using its detectors, thereby segmenting the first video into different shot segments.
[0057] For example, when using pixel difference-based detection of shot transitions, if the pixel differences between adjacent video frames (such as brightness differences, color distribution differences, histogram distances, etc.) exceed a set threshold, a shot transition is determined to have occurred. Of course, other methods can also be used to divide shot segments.
[0058] S202: Generate the shot description text corresponding to each shot segment in the first video.
[0059] In some embodiments of this application, after acquiring the first video and dividing the first video into multiple shot segments, the shot description text corresponding to each shot segment can be acquired.
[0060] Optionally, a shot description text for each shot segment can be generated based on at least one of the following: subtitle segments, character information, audio segments, background music, and video footage.
[0061] The subtitle fragment of the shot segment may be obtained by at least one of the following: the subtitle displayed in its video frame, the subtitle file corresponding to the shot segment, and the audio track corresponding to the shot segment.
[0062] For example, to obtain the subtitle fragment corresponding to a video frame in a shot, optical character recognition (OCR) technology can be used to extract the subtitles displayed in the video frame. For instance, OCR technology can be implemented using engines such as Tesseract or other OCR engines.
[0063] For example, taking the example of obtaining the subtitle segment corresponding to a shot segment based on the subtitle file corresponding to that shot segment, it can be understood that in some embodiments of this application, the subtitle file may include multiple subtitles in text form. Optionally, the subtitle file may also include subtitle sequence numbers, timestamps, etc. The subtitle sequence number can be used to identify the order in which the subtitles are displayed. The timestamp can be used to identify the time range in which the subtitles are displayed.
[0064] For example, the subtitle file can be a .srt file, or it can be a file in other subtitle formats. Optionally, the first video can correspond to one subtitle file. The subtitle segment corresponding to a given shot can be obtained from the subtitle file corresponding to the first video, based on the timestamps (such as start playback time, end playback time, etc.) of the shot segments divided from the first video and the timestamps (such as start display time, end display time, etc.) included in the subtitle file. Of course, in other examples, each shot segment corresponding to the first video can correspond to a separate subtitle file, and the subtitle segment can be directly obtained from the subtitle file corresponding to the shot segment.
[0065] Optionally, the subtitle file corresponding to the first video can be stored in the same directory as the first video and use the same filename identifier, so that when the first video is played, the subtitles in the subtitle file can be automatically loaded into the video frame of the first video. Subtitle files stored in this way can also be called external subtitle files.
[0066] For example, consider obtaining the corresponding subtitle segment based on the audio track of a shot. An audio track is an independently stored audio data stream in a video file, responsible for carrying the sound elements in the video (such as dialogue, background music, sound effects, etc.). Optionally, the first video can correspond to one or more audio tracks. Alternatively, a single shot segment in the first video can correspond to one or more audio tracks.
[0067] Taking a first video corresponding to one or more audio tracks as an example, the audio track corresponding to the first video segment can be obtained from the audio track of the first video to which the first video segment belongs, based on the timestamps of the shot segments (such as start playback time, end playback time, etc.) and the timestamps included in the audio tracks (such as start display time, end display time, etc.). Then, speech recognition can be performed on the audio track to obtain the subtitle segment corresponding to the shot segment. In practical applications, an automatic speech recognition (ASR) model can be used to recognize the audio track corresponding to the shot segment and convert it into text as a subtitle segment. For example, ASR models may include, but are not limited to, Whisper.
[0068] Optionally, the audio track can exist as a separate audio file and be associated with the video (such as the first video, video clips, etc.). Alternatively, the audio track can be embedded in the video file, existing as part of the video.
[0069] Optionally, the character information corresponding to the shot clip can include information related to the characters in the shot clip, specifically, the character information can include the character's name, gender, etc. In practical applications, facial recognition technology can be used to identify the characters in the video frames of the shot clip. For example, facial recognition technology can include, but is not limited to, multi-task cascaded convolutional networks (MTCNN), one-look-once (YOLO) algorithms, etc. Then, facial feature vectors are extracted based on techniques such as additive angular margin loss for deep face recognition (ArcFace), FaceNet, etc., to cluster all faces appearing in the shot clip, thereby obtaining the characters included in the shot clip.
[0070] Optionally, after obtaining the characters included in the shot, the character ID (e.g., face_01, face_02, etc.) can be associated with the character's name in the video (e.g., "Zhang San", "Li Si", etc.) to facilitate the subsequent generation of shot description text for the shot.
[0071] Based on the above embodiments, the character information may further include at least one of character tone information and character facial expression information. The character tone information may be the vocal features of a character in the scene, used to represent the character's emotional attitude. The character facial expression information may include the facial expressions of a character in the scene. The character tone information can be obtained by performing emotion analysis on the voice of a character in the scene using relevant methods; the character facial expression information can be obtained by recognizing facial expressions of a character in the scene using relevant methods.
[0072] Optionally, the background music corresponding to the shot clip can also be used to generate the shot description information for that shot clip. The background music may include the music portion of the audio played synchronously with the shot clip, and this background music is related to the video footage in the shot clip. For example, a tense chase shot is accompanied by fast-paced music. Specifically, the background music corresponding to the shot clip can be obtained by separating the audio from the shot clip. This application does not impose limitations on this aspect.
[0073] By acquiring multi-dimensional information from shot clips, the quality of the subsequently generated shot description text can be further improved.
[0074] The shot description text is used to describe the plot content of the shot segment. Specifically, the shot description text can be text information described in natural language, used to summarize the content of the corresponding shot segment.
[0075] Optionally, multimodal information such as captions, character information, audio clips, background music, and video footage of a shot can be input into the shot description text generation model, which then outputs the shot description text corresponding to that shot.
[0076] As a concrete example, the shot description text generation model can receive cue words such as those shown in Table 1 and the aforementioned multimodal information. The cue words may include instructions to output shot description text corresponding to the shot segment (e.g., "Describe the core content of this shot segment in one sentence"). After receiving the cue words, the shot description text generation model can output shot description text such as those shown in Table 2.
[0077]
[0078] Table 1
[0079] Table 2 Optionally, in addition to generating shot description text corresponding to shot clips, it is also possible to generate description text corresponding to key video frames included in the shot clips.
[0080] Similarly, the segment information corresponding to the shot can be input into the shot description text generation model, which then outputs the description text corresponding to the key video frame. Optionally, various keyframe recognition technologies can be used to identify the key video frame in the shot, and this application embodiment does not limit this. For example, referring to the example in Table 1, the description text corresponding to the key video frame output by the shot description text generation model can be "In a dimly lit warehouse, two people with tense expressions are talking."
[0081] S203: Based on the shot description text corresponding to the shot segments that constitute the same scene and the subtitle segment of the scene, generate the scene description text corresponding to the scene.
[0082] In some embodiments of this application, the scene is obtained by clustering the video clips based on the visual similarity and temporal proximity between the video frames in each clip. That is, a scene includes at least one clip.
[0083] In practical applications, firstly, the transition points in the first video are identified, and then the first video is divided into multiple shot segments. Then, based on this, the multiple shot segments are clustered according to the visual similarity and temporal proximity between the video frames in the multiple shot segments, thereby obtaining at least one scene. Each scene consists of one or more adjacent shot segments, that is, a scene consists of multiple shot segments that are temporally continuous and have high similarity between the video frames.
[0084] Visual similarity can include similarity in multiple dimensions such as objects, actions, and scenes. For example, the visual similarity of a shot can be determined by calculating the cosine similarity of the feature vectors of the shot using a contrastive language-image pretraining (CLIP) model.
[0085] Temporal proximity refers to the degree of continuity between different shot segments on the video timeline. For example, the first, second, and third shots in a video are played consecutively (connected sequentially in time), so their temporal proximity is relatively high; while the first shot and the tenth shot are separated by 7 shots, so their temporal proximity is relatively low.
[0086] It is evident that by clustering shot clips to obtain scenes, it is possible to ensure that shot clips within the same scene are consistent in content and conform to the narrative time logic of the video content.
[0087] After obtaining the multiple scenes corresponding to the first video, corresponding scene description text is generated based on the shot description text corresponding to the shot fragments of each scene. This scene description text is used to describe the plot content of the corresponding scene. One possible implementation is to obtain the shot description text corresponding to each shot segment included in the scene, as well as the scene caption fragment corresponding to the scene. The scene caption fragment is composed of caption fragments corresponding to each shot segment in the scene; that is, the scene caption fragment corresponding to a scene can include the caption fragments corresponding to all shot segments that constitute the scene. For an explanation of the shot description text corresponding to shot segments, please refer to the above text.
[0088] Then, the shot description text corresponding to each shot segment included in the scene, as well as the scene caption segment corresponding to the scene, can be input into the scene description text generation model, and the scene description text generation model can output the scene description text corresponding to the scene.
[0089] As a concrete example, the scene description text generation model can receive prompts such as those shown in Table 3, shot description texts corresponding to the shot segments included in the scene, and scene captions for that scene. Optionally, the prompts may include instructions to output scene description texts for the scene (e.g., "The following are descriptions of several shot segments that constitute the same scene; please combine them into a coherent scene summary of approximately 50 words"). In response to the prompts, the scene description text generation model can output scene description texts such as those shown in Table 4.
[0090]
[0091] Table 3 It is understandable that Table 3 uses scene 01, which includes shot segments 1 to 3, as an example.
[0092]
[0093] Table 4 In some embodiments, after generating the scene description text corresponding to a scene, the scene description text can also be output (e.g., displayed on a screen, or read aloud). For example, when a user is watching an episode of a TV series, the user may drag the playback progress bar to fast forward the video, and correspondingly, the video can switch from the current scene to the next scene. When the user drags the progress bar to the next scene, the scene description text corresponding to that next scene can be displayed so that the user can understand the plot content of the next scene.
[0094] S204: Generate a video summary text for the first video based on the scene description text corresponding to each scene in the first video.
[0095] In one optional implementation, the scene description text corresponding to each scene included in the first video can be input into a summary text generation model, and the summary text generation model can output the video summary text corresponding to the first video. For example, the video summary text can be a complete and clearly narrated synopsis of a single episode of a TV series.
[0096] As a concrete example, the summary text generation model can receive prompts such as those shown in Table 5 and scene description text corresponding to each scene in the first video. Optionally, the prompts may include instructions to output video summary text corresponding to the first video (e.g., "Give a synopsis of approximately 1000 words for this episode based on summaries of all scenes in the following episode, highlighting the main conflicts and plot twists"). The prompts then drive the video summary text model to output video summary text as shown in Table 6.
[0097]
[0098] Table 5
[0099] Table 6 In some embodiments, after generating the video summary text corresponding to the first video, the video summary text can also be output. For example, the video summary text corresponding to the video can be presented in the synopsis of an episode of a TV series. This allows users to quickly understand the plot of each episode of a TV series.
[0100] In another optional implementation, the shot description text corresponding to each shot segment included in the first video can be obtained and input together with the scene description text into the summary text generation model to generate the video summary text of the first video, thereby improving the accuracy of the video summary text generation. For details on obtaining the shot description text corresponding to multiple shot segments and the scene description text, please refer to the above description.
[0101] S205: Based on the video summary text corresponding to each video in the video sequence to which the first video belongs, generate the main storyline description text of the entire series corresponding to the video sequence.
[0102] In some scenarios, it is also possible to generate a main storyline description text for the entire series corresponding to the video sequence to which the first video belongs. Taking a TV series as an example, the video sequence consists of multiple videos, including the first video. In some embodiments, it is also possible to generate a main storyline description text for the entire series corresponding to the video sequence, that is, a description text corresponding to the entire TV series.
[0103] One possible implementation is to obtain the video summary text corresponding to each video in the video sequence to which the first video belongs. For an introduction to the video summary text corresponding to the first video, please refer to the above.
[0104] Then, the video summary text corresponding to each video can be input into the main plot description text generation model, which then outputs the overall main plot description text for the entire series corresponding to the video sequence. The overall main plot description text indicates the plot content of the video sequence. For example, the overall main plot description text corresponding to the video sequence may include a highly compressed storyline of the entire TV series that focuses on the core conflict.
[0105] As a concrete example, the main plot description text generation model can receive prompts such as those shown in Table 7 and video summary text corresponding to each video in the video sequence. Optionally, the prompts can also include instructions to output the main plot description text for the entire series corresponding to the video sequence (e.g., "Here is a summary of all episodes 1-30 of a TV series. Please extract the core main plot of the entire series, ignoring secondary subplots, focusing on the protagonist's growth, goals, and final ending, to form a main plot story of approximately 500 words"). Using this prompt to drive the main plot description text generation model, the model can output the main plot description text for the entire series corresponding to the video sequence.
[0106]
[0107] Table 7 The descriptive text generation method provided in this application identifies shot transition points in a first video, divides it into multiple shot segments, and generates shot description text for each shot segment based on at least one of subtitle segments, character information, audio segments, background music, and video footage. It then generates scene description text based on the shot description texts within the same scene, and finally generates a video summary text for the first video based on the scene description texts of each scene in the first video. As can be seen, this application embodiment, through hierarchical processing and the fusion of multiple dimensions of information (subtitles, character information), ensures the narrative logic and coherence of the descriptive text to a certain extent, thereby improving the quality of the generated descriptive text.
[0108] Furthermore, the descriptive text generation method provided in this application implements hierarchical parsing and step-by-step summarization. Compared with the implementation methods in the prior art, this ensures that the generated descriptive text has narrative logic and coherence. In other words, by summarizing the video clips (shot clips, scene clips) and the plot content of the first video in a bottom-up manner, the generated descriptive text can retain both the specific information of the video (microscopic) and clarify the development of the plot (macroscopic), further ensuring the quality of the generated descriptive text.
[0109] In some scenarios, users need to retrieve video clips of interest from a vast amount of video content. To facilitate quick retrieval, one possible solution is for video platforms to provide a search function. This search function is based on metadata such as title, tags, and cast list. However, metadata like title, tags, and cast list do not reflect the video's plot content, and therefore cannot accurately pinpoint the video clips containing plot elements of interest to the user. This solution is therefore ineffective.
[0110] Another possible solution is to manually write the plot of each episode and an overall synopsis of the entire TV series on various video platforms. However, this method is highly subjective, updates slowly, and cannot cover all video segments. Therefore, it cannot accurately reflect the plot content of the video segments, and consequently, it cannot accurately locate the video segments corresponding to the plot content that users are interested in, nor can it effectively solve the aforementioned technical problems.
[0111] Another possible solution is to generate video summaries by extracting keyframes or highlight segments from the video. However, the generated summaries lack narrative logic and coherence, cannot accurately reflect the plot content of the video segments, cannot accurately locate the video segments corresponding to the plot content that the user is interested in, and cannot effectively solve the aforementioned technical problems.
[0112] Therefore, based on the problems described above, the descriptive text generation method provided in the embodiments of this application can be applied. The video descriptive text generation method provided in the embodiments of this application can be applied to scenarios such as video content retrieval, plot summary generation, and intelligent recommendation (e.g., recommending similar videos based on plot summaries).
[0113] To facilitate understanding of the application scenarios of the descriptive text generation method provided in the embodiments of this application, the following will take the retrieval scenario as an example.
[0114] In some embodiments, after generating descriptive text corresponding to the first video (e.g., shot description text corresponding to a shot clip, scene description text corresponding to a scene, video summary text, etc.), the aforementioned descriptive texts can be stored in a retrieval database so that subsequent responses to user plot query operations can be based on the retrieval database. For example, the retrieval database can be various databases that can be efficiently retrieved, such as Elasticsearch or vector databases.
[0115] Specifically, a first set of associated data and a second set of associated data are constructed. The first set of associated data may include the relationship between the scene description text and the first timestamp information and the character information; the second set of associated data includes the relationship between the shot description text and the second timestamp information and the character information of the shot clip. The first timestamp information is used to identify the playback time of the scene in the first video, and the second timestamp information is used to identify the playback time of the shot clip in the first video.
[0116] For example, Table 8 shows a schematic diagram of the construction structure of the first associated data provided by some embodiments of this application.
[0117]
[0118] Table 8 The segment identifier is used to identify different shot segments or scenes in the first video, and the type is used to identify whether it is a shot segment, scene, or other video segment. Constructing first and second related data can improve the retrieval efficiency of the retrieval database.
[0119] After constructing the first and second associated data, the first and second associated data are stored in a retrieval database. The retrieval database is used to match at least one of scene description text, shot description text, and video summary text with the query information input by the user (i.e., the query information from the client in this application), and generate answer content. The answer content includes at least one of the timestamp information of the search results and the playback link information. The search results include at least one of the scene corresponding to the successfully matched scene description text and the shot segment corresponding to the successfully matched shot description text.
[0120] The query information can be user-inputted information related to the video's plot. The timestamp information included in the response helps users locate the position of successfully matched scene or shot clips, and the playback link information allows users to trigger a jump to the playback page to watch the video.
[0121] Based on the above embodiments, third associated data can also be constructed based on the video summary text of the first video to be stored in the retrieval database for subsequent response to user-inputted query information. The third associated data includes the association between the video summary text of the first video and its episode identifier. The episode identifier identifies the playback position of the first video in its respective video sequence; for example, the episode identifier could be episode 1.
[0122] Specifically, a third set of related data is constructed, and then the third set of related data is stored in a retrieval database. This retrieval database is also used to match video summary text with the query information entered by the user and generate answer content, which also includes the episode identifier of the search results.
[0123] After storing the first, second, and third related data into the retrieval database, some embodiments of this application can perform retrieval based on user-input query information. The specific processing method for performing retrieval in the retrieval database can be understood by referring to the data query method provided in this application, and will not be elaborated further here.
[0124] Figure 3 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application. For example... Figure 3 As shown, the data query device 300 includes: The first receiving module 301 is used to receive query information from the client; The first determining module 302 is used to determine target descriptive text for the query information in a database based on semantic similarity; the database includes descriptive text generated based on the first video, the descriptive text including shot description text, scene description text and video summary text, the shot description text being generated based on shot segments in the first video, the shot segments being determined by identifying shot transition points in the first video, the scene description text being generated based on shot description text corresponding to shot segments constituting the same scene, and the video summary text being generated based on scene description text corresponding to scenes in the first video; The first generation module 303 is used to input model prompts and the target description text into the question-answering model, and the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target description text, and return the answer content to the client.
[0125] In one possible design, the database also includes timestamp information corresponding to each descriptive text. The timestamp information is used to identify the playback time of the video segment corresponding to the descriptive text in the first video. The first generation model is specifically used for: The model prompts, the target description text, and its timestamp information are input into the question-answering model. The model prompts drive the question-answering model to generate a timestamped answer based on the target description text and its timestamp information, which serves as the answer to the query information.
[0126] In one possible design, the response content also includes playback link information, which is used to trigger the playback of the first video based on the timestamp.
[0127] In one possible design, the shot description text is generated based on at least one of the following: subtitles, character information, audio clips, background music, and video footage of the corresponding shot segment. The character information includes at least one of the following: character name, character tone information, and character expression information identified from the shot segment. The scene description text is generated based on the shot description text corresponding to shot segments constituting the same scene and the subtitles of the scene.
[0128] In one possible design, the scenario is obtained by clustering the video clips based on the visual similarity and temporal proximity between the video frames in each clip.
[0129] In one possible design, the database further includes episode identifiers for each descriptive text generated based on the first video. These episode identifiers identify the playback order of the first video within a video sequence, where the video sequence includes multiple videos with a playback order, including the first video. The first generation module is specifically used for: The model prompts, the target description text, and its episode identifier are input into the question-answering model. The model prompts drive the question-answering model to generate answer content with the episode identifier based on the target segment description text and its episode identifier, which serves as the answer content corresponding to the query information.
[0130] In one possible design, the descriptive text may also include a main storyline description text corresponding to the video sequence, wherein the main storyline description text is generated based on the video summary text corresponding to each video in the video sequence.
[0131] The apparatus provided in this application embodiment can execute the method of any of the above method embodiments, and its execution mode and beneficial effects are similar and will not be repeated here.
[0132] It should also be noted that the module division in the data query device described in this application is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules.
[0133] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device or processor to execute all or part of the steps of the methods described in the various embodiments of this application.
[0134] See Figure 4 ,for Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0135] For example, electronic devices may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0136] like Figure 4 As shown, the electronic device includes a memory 121 and a processor 122.
[0137] Memory 121 is used to store programs. In addition to the programs described above, memory 121 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact member data, phone book member data, messages, pictures, videos, etc.
[0138] The memory 121 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0139] The processor 122 is coupled to the memory 121 and executes the program stored in the memory 121 to implement the method of any of the above method embodiments.
[0140] Furthermore, such as Figure 4 As shown, the electronic device may also include other components such as a communication component 123, a power supply component 124, an audio component 125, and a display 126. Figure 4 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 4 The components shown.
[0141] Communication component 123 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 123 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 123 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0142] Power supply component 124 provides power to various components of an electronic device. Power supply component 124 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.
[0143] Audio component 125 is configured to output and / or input audio signals. For example, audio component 125 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in memory 121 or transmitted via communication component 123. In some embodiments, audio component 125 also includes a speaker for outputting audio signals.
[0144] Display 126 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.
[0145] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in any of the above method embodiments.
[0146] This application also proposes a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the above method embodiments.
[0147] In the embodiments of this application, the aforementioned computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO) etc.), optical memory (e.g., CD, DVD, BD, HVD etc.), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD) etc.).
[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.
[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0150] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data query method, characterized in that, The method includes: Receive query information from the client; Based on semantic similarity, target descriptive text is determined for the query information in the database; the database includes descriptive text generated based on the first video, the descriptive text including shot description text, scene description text and video summary text, the shot description text is generated based on shot segments in the first video, the shot segments are determined by identifying shot transition points in the first video, the scene description text is generated based on shot description text corresponding to shot segments that constitute the same scene, and the video summary text is generated based on scene description text corresponding to scenes in the first video; The model prompts and the target description text are input into the question-answering model. The model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target description text, and the answer content is returned to the client.
2. The method according to claim 1, characterized in that, The database also includes timestamp information corresponding to each descriptive text. The timestamp information is used to identify the playback time of the video segment corresponding to the descriptive text in the first video. The step of inputting model prompts and the target descriptive text into the question-answering model, and having the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target descriptive text, includes: The model prompts, the target description text, and its timestamp information are input into the question-answering model. The model prompts drive the question-answering model to generate a timestamped answer based on the target description text and its timestamp information, which serves as the answer to the query information.
3. The method according to claim 2, characterized in that, The response also includes playback link information, which is used to trigger the playback of the first video based on the timestamp.
4. The method according to claim 1, characterized in that, The shot description text is generated based on at least one of the subtitles, character information, audio clips, background music, and video footage of the corresponding shot segment. The character information includes at least one of the character name, character tone information, and character expression information identified from the shot segment. The scene description text is generated based on the shot description text corresponding to the shot segments that constitute the same scene and the subtitles of the scene.
5. The method according to claim 1, characterized in that, The scenario is obtained by clustering the video clips based on the visual similarity and temporal proximity between the video frames in each clip.
6. The method according to claim 1, characterized in that, The database also includes episode identifiers for each descriptive text generated based on the first video. These episode identifiers identify the playback order of the first video within a video sequence, where the video sequence includes multiple videos in a playback order, including the first video. The step of inputting model prompts and the target descriptive text into the question-answering model, whereby the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target descriptive text, includes: The model prompts, the target description text, and its episode identifier are input into the question-answering model. The model prompts drive the question-answering model to generate answer content with episode identifiers based on the target description text and its episode identifiers, which serves as the answer content corresponding to the query information.
7. The method according to claim 6, characterized in that, The descriptive text also includes the main storyline description text corresponding to the video sequence, which is generated based on the video summary text corresponding to each video in the video sequence.
8. A data query device, characterized in that, The device includes: The first receiving module is used to receive query information from the client; The first determining module is used to determine target descriptive text for the query information in a database based on semantic similarity; the database includes descriptive text generated based on the first video, the descriptive text including shot description text, scene description text and video summary text, the shot description text being generated based on shot segments in the first video, the shot segments being determined by identifying shot transition points in the first video, the scene description text being generated based on shot description text corresponding to shot segments constituting the same scene, and the video summary text being generated based on scene description text corresponding to scenes in the first video; The first generation module is used to input model prompts and the target description text into the question-answering model, and the model prompts drive the question-answering model to generate the answer content corresponding to the query information based on the target description text, and return the answer content to the client.
9. An electronic device, characterized in that, include Memory; processor; And computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1-7.