A method, apparatus, computer-readable storage medium, and computer program product for generating abstracts.
By monitoring the time a user leaves the video area and generating and displaying a text summary, the problem of users missing video segments is solved, improving video playback efficiency and viewing experience.
Patent Information
- Application Number
- CN202610976291.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-11-17
AI Technical Summary
Users who briefly leave their seats due to urgent matters while watching videos may miss video segments, resulting in poor video playback efficiency and affecting the viewing experience.
By monitoring the time periods when users leave the video viewing area, the system identifies the missed video segments, generates a text summary, and displays it on the playback interface, allowing users to directly access the missed content upon returning.
It improves the smoothness and efficiency of video playback, eliminating the need for users to repeatedly drag the progress bar to replay missed segments, thus enhancing the viewing experience.
Smart Images

Figure CN123123680A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer-readable storage medium, and computer program product for generating abstracts. Background Technology
[0002] When watching videos on video websites, users often have to leave briefly due to urgent matters. In such situations, users often forget to pause the video, causing them to miss some video segments. After finishing their business, users usually try to reposition and watch the missed segments by repeatedly dragging the progress bar. However, this method severely affects the smoothness of video playback, resulting in poor playback efficiency and negatively impacting the user's viewing experience. Summary of the Invention
[0003] To address the aforementioned technical problems, embodiments of this application aim to provide a summary generation method, apparatus, computer-readable storage medium, and computer program product that can solve the problem in related technologies where users need to repeatedly play video segments when watching videos, resulting in poor video playback efficiency.
[0004] The technical solution of this application is implemented as follows: A method for generating a summary, the method comprising: When playing a target video based on an input playback command, monitor whether a target user leaves the target viewing area corresponding to the target video during the playback process; If it is detected that the target user has left the target viewing area, the target time period when the target user left the target viewing area is determined; The target sub-video is determined from multiple candidate sub-videos of the target video based on the target time period; Obtain first description information of the target sub-video; wherein, the first description information is used to describe the content of the target sub-video; A text summary of the target sub-video is generated based on the first description information and displayed in the target area of the target video's playback interface.
[0005] In the above scheme, monitoring whether a target user leaves the target viewing area corresponding to the target video during the playback of the target video includes: Acquire multiple real-time images of a user watching the target video; wherein the multiple real-time images are captured at different times; A target detection model is used to process each real-time image to obtain the user's position in each real-time image; A target tracking algorithm is used to process each real-time image and the location to obtain the real-time trajectory information of the user while watching the target video; Based on the real-time trajectory information, it is determined whether any of the target users have left the target viewing area during the playback process.
[0006] In the above scheme, before playing the target video, the method further includes: After receiving the playback instruction, the target video is decoded to obtain multiple first images, and the degree of difference between adjacent images in the multiple first images is determined; The target video is divided based on the difference to obtain the multiple candidate sub-videos.
[0007] In the above scheme, determining the target time period when the target user leaves the target viewing area includes: If it is detected that the target user has left the target viewing area, the first time when the target user left the target viewing area and the second time when the target user re-entered the target viewing area are determined based on the real-time trajectory information. The target time period is determined based on the first time and the second time.
[0008] In the above scheme, determining the target time period when the target user leaves the target viewing area includes: The call status of the target user while watching the target video is obtained; wherein, the call status indicates whether the target user is currently on a call; If the call status changes from a first state to a second state, a third time when the call status changes from the first state to the second state is determined, and a fourth time when the call status changes from the second state to the first state is monitored; wherein, the first state indicates that the target user is not making a call; the second state indicates that the target user is making a call; The target time period is determined based on the third time and the fourth time.
[0009] In the above scheme, generating a text summary of the target sub-video based on the first descriptive information and displaying it in the target area of the target video's playback interface includes... If there is a target sub-video, the first descriptive information is processed using a target summary generation model to obtain the text summary; If there are multiple target sub-videos, obtain the target user's preference information for the video's content metadata, and generate the text summary based on the preference information and the first description information of each target sub-video; Determine the target scrolling speed of the text summary, and control the text summary to scroll and display in the target area at the target scrolling speed.
[0010] In the above scheme, generating the text summary based on the preference information and the first descriptive information of each target sub-video includes: Determine the similarity between each piece of first descriptive information and the preference information; Based on the similarity, multiple second description information is determined from multiple first description information, and the multiple second description information is combined to obtain the combined information; The combined information is processed using the target summary generation model to obtain the text summary.
[0011] In the above scheme, determining the target scrolling speed of the text summary includes: Based on the preference information and the text summary, the degree of preference of the target user for the target sub-video is determined; Determine the importance of the target sub-video; The target scrolling speed is determined based on the preference level, the importance level, and the target coefficient.
[0012] In the above scheme, determining the importance of the target sub-video includes: The system acquires the historical playback count and historical slow-motion playback count of the target sub-video, the historical comment count for the target sub-video, and a baseline image of the target video; wherein, the historical slow-motion playback count is the number of times the target sub-video has been played at a target playback speed; the target playback speed is less than the baseline playback speed of the target sub-video; and the baseline image is used to provide key content features of the target video. The popularity of the target sub-video is determined based on the number of historical replays, the number of historical slow-motion playbacks, and the number of historical comments; wherein, the popularity represents the degree of popularity of the target sub-video. A target image is determined from a plurality of second images of the target sub-video, and the degree of matching between the reference image and the target image is determined; The importance level is determined based on the popularity and the matching degree.
[0013] An abstract generation apparatus, the apparatus comprising: The monitoring unit is used to monitor whether a target user leaves the target viewing area corresponding to the target video during the playback of the target video based on the playback command input by the user. The first determining unit is configured to determine the target time period when the target user leaves the target viewing area if it is detected that the target user has left the target viewing area. The second determining unit is used to determine a target sub-video from a plurality of candidate sub-videos of the target video based on the target time period; An acquisition unit is configured to acquire first description information of the target sub-video; wherein the first description information is used to describe the content of the target sub-video; The processing unit is configured to generate a text summary of the target sub-video based on the first description information and display it in the target area of the playback interface of the target video.
[0014] An abstract generation device, the device comprising: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the digest generation program stored in the memory to implement the steps of the above-described digest generation method.
[0015] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the above-described digest generation method.
[0016] A computer program product comprising a computer program that, when executed by a processor, implements the steps of the above-described summary generation method.
[0017] The summary generation method, apparatus, device, computer-readable storage medium, and computer program product provided in this application can, when playing a target video based on an input playback command, monitor whether a target user leaves the target viewing area corresponding to the target video during playback; if a target user leaves the target viewing area, determine the target time period during which the target user left the target viewing area; determine a target sub-video from multiple candidate sub-videos of the target video based on the target time period; obtain first description information of the target sub-video; wherein the first description information is used to describe the content of the target sub-video; generate a text summary of the target sub-video based on the first description information, and display it on the target video playback interface. In this way, the video segments (i.e., target sub-videos) missed by the target user can be determined based on the target time period when the target user leaves the target viewing area. Then, a text summary of the target sub-video is generated and displayed in the target area. In this way, after the target user returns to the target viewing area, they can directly understand the plot description of the missed video segments through the text summary displayed in the target area, without having to repeatedly drag the progress bar to rewatch the missed video segments. This significantly improves the playback smoothness of the target video, thereby improving the video playback efficiency and solving the problem in related technologies where users need to repeatedly play some video segments, resulting in poor video playback efficiency. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating an abstract generation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the system architecture corresponding to an abstract generation method provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the division of a target sub-video in a summary generation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the process of generating a text summary in a summary generation method provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the display of a text summary in a target area in a summary generation method provided in this application embodiment; Figure 6 This is another schematic diagram illustrating the display of a text summary in a target area in a summary generation method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of an abstract generation device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an abstract generation device provided in an embodiment of this application. Detailed Implementation
[0019] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0020] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.
[0021] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.
[0022] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.
[0023] This application provides a method for generating abstracts, which can be applied to an abstract generation device. (Refer to...) Figure 1 As shown, the method may include the following steps: Step 101: When playing the target video based on the user's input playback command, monitor whether the target user leaves the target viewing area corresponding to the target video during the playback process.
[0024] In the embodiments of this application, the abstract generation device may refer to the display device that plays the target video, which may be a television set, a personal computer (PC), etc.; the target video may refer to a movie video, a TV series video, or other popular science videos, etc.
[0025] It should be noted that this application can be applied to single-person viewing scenarios or multi-person viewing scenarios. That is to say, there can be only one user or multiple users watching the target video.
[0026] In this embodiment of the application, the target user can refer to all users who have left the target viewing area among one or more users watching the target video. Specifically, if only one user is watching the target video, then the target user is essentially that single user; if multiple users are watching the target video together, then the target user can be some or all of the multiple users.
[0027] In this embodiment of the application, after receiving a playback instruction input by the user, the target video can be played according to the playback instruction. Then, multiple real-time images of one or more users watching the target video can be acquired, and it can be determined whether the target user has left the target viewing area during the video playback based on the multiple real-time images.
[0028] The playback command can be issued by a user watching the target video via a remote control. Specifically, on the target video's details page, the user can use the remote control to click the "play" button on that page to issue the playback command.
[0029] In this embodiment, the target viewing area can be a pre-defined area for watching video. In one possible implementation, the target viewing area can refer to the capture area of an image acquisition device mounted on a display device. For example, Figure 2 As shown, the image acquisition device can refer to a camera.
[0030] It should be noted that the image acquisition device will immediately begin acquiring images of the user watching the video when the target video starts playing, in order to determine whether the target user has left the target viewing area during the video playback.
[0031] In this embodiment of the application, the reason why the target user leaves the target viewing area may be to answer a phone call, go to the restroom, etc.
[0032] Step 102: If it is detected that a target user has left the target viewing area, determine the target time period during which the target user left the target viewing area.
[0033] In this embodiment, "the target user leaving the target viewing area" can mean that the target user did not appear in the target viewing area (i.e., the collection area), meaning the target user was lost within the target viewing area. Specifically, once it is detected that a target user has been lost within the target viewing area, the time of the user's loss needs to be recorded, and the system continuously monitors whether the user returns to the target viewing area. Once the user is detected to have re-entered the target viewing area, the time of their return is recorded promptly. Thus, the target time period of leaving the target viewing area is determined based on the time of loss and the time of return.
[0034] It should be noted that the target time period refers to the period during which the target user is not present in the target viewing area, including the time the target user leaves and the time they return. For example, the target time period could be 14:00-15:00.
[0035] Step 103: Determine the target sub-video from multiple candidate sub-videos of the target video based on the target time period.
[0036] Among them, multiple candidate sub-videos are obtained by dividing the target video; the target sub-videos can refer to the candidate sub-videos that the target user missed during the target time period when leaving the target viewing area.
[0037] It should be noted that a sub-video is essentially a video segment; that is, multiple candidate sub-videos are multiple video segments of the target video.
[0038] In this embodiment, multiple currently playing first sub-videos can be determined from a plurality of candidate sub-videos. Based on the playback time and target time period of each video frame of each first sub-video, the target sub-video that the target user has not seen, i.e., missed, can be determined from the plurality of candidate sub-videos. It should be noted that there can be only one target sub-video or multiple target sub-videos.
[0039] Step 104: Obtain the first description information of the target sub-video.
[0040] The first descriptive information is used to describe the content of the target sub-video.
[0041] In this embodiment of the application, the first description information is used to describe the plot content of the target sub-video, for example, A helps B into a flower shop, and A asks the flower shop clerk C if they have carnations...
[0042] In one feasible approach, a large language model can be used to process the target sub-video to obtain the first descriptive information of the target sub-video.
[0043] In one feasible approach, a subtitle text extraction model can be used to extract the subtitle text of the target sub-video, and the subtitle text can be processed to obtain the first descriptive information.
[0044] In one feasible approach, the complete descriptive information of the target video can be obtained, and the first descriptive information of the target sub-video can be segmented from the descriptive information based on the identifier of the target sub-video. This process is already achievable with existing technology and will not be elaborated upon here. It should be noted that the target video and the descriptive information are pre-aligned on the timeline.
[0045] Step 105: Generate a text summary of the target sub-video based on the first description information and display it in the target area of the target video playback interface.
[0046] In this embodiment of the application, if there is only one target sub-video, the first description information can be directly processed using the target summary generation model to obtain a text summary of the target sub-video; if there are multiple target sub-videos, the target user's preference information for the video's content metadata can be obtained first, and a text summary can be generated based on the preference information, the first description information, and the target summary generation model.
[0047] In this embodiment, the target area can be a region on the playback interface used to display text summaries. In one possible implementation, the target area can be a region on the playback interface where bullet comments are displayed.
[0048] In this embodiment of the application, the text summary can be controlled to scroll and display in the target area, that is, the text summary is displayed as a bullet screen in the target area.
[0049] It should be noted that, as Figure 2 As shown, steps 101-105 are all executed by the processing module in the summary generation device.
[0050] In this embodiment, for multi-user viewing scenarios, where multiple users watch a video together, if a user has to temporarily leave due to needing to use the restroom, make a phone call, or handle other urgent matters, they often miss some exciting plot points. Upon returning, they need to spend extra time asking other viewers about the missed scenes or have to endure the disjointed plot while continuing to watch the video. This results in a poor viewing experience because users cannot get a smooth connection between the plot and the story. However, in this application, after returning to the target viewing area, users can understand the missed plot points simply by looking at the text summary displayed in the target area, thus obtaining accurate plot connections and significantly improving the user's viewing experience.
[0051] Based on the foregoing embodiments, in other embodiments of this application, the multiple candidate sub-videos may be obtained by dividing the target video before playing the target video. The specific implementation process can be as follows.
[0052] A1, decode the target video to obtain multiple first images, and determine the degree of difference between adjacent images in the multiple first images.
[0053] In the embodiments of this application, the first image may refer to a video frame; the difference may refer to the degree of change of the picture between two adjacent video frames. The greater the difference, the greater the degree of change of the picture between two adjacent video frames.
[0054] Specifically, the target video can first be decoded to obtain a set of video frames I={I1,I2,I3…}, which includes multiple video frames (i.e., multiple first images). Then, the pixel difference between every two adjacent video frames (i.e., every two adjacent first images) can be calculated, i.e., the difference degree. The formula for calculating the difference degree is as shown in the following formula (1).
[0055] Formula (1) in, (x, y) represents the difference (i.e., pixel difference); (x, y) represents pixels. This represents the pixel value of the (x, y) pixel in the t-th first image.
[0056] A2. The target video is divided based on the degree of difference to obtain multiple candidate sub-videos.
[0057] In this embodiment, each difference can be compared with a preset threshold T. If any difference is greater than the threshold, it indicates that the degree of change in the image between adjacent video frames (i.e., adjacent first images) corresponding to that difference is large. In this case, it can be considered that a shot change has occurred between the two video frames (e.g., the shot changes from facing A to facing B). If , it means that a shot switch has occurred between the t-th first image and the (t+1)-th first image.
[0058] Then, all differences greater than the threshold T can be counted, and the first image in time sequence of the two adjacent first images corresponding to each difference that meets the requirements can be used as the segmentation point. Figure 3 Using n1, n2, n3… (in the original text), the target video is segmented to obtain the following result: Figure 3 The multiple candidate sub-videos are shown.
[0059] For example, if the second difference between the second and third first images, the first and sixth differences between the sixth and seventh first images, and the ninth difference between the ninth and tenth first images are all greater than the threshold T, then the second, sixth, and ninth first images can be used as segmentation points to segment the target video, resulting in multiple candidate sub-videos N={N1, N2, N3…}.
[0060] In other embodiments of this application, the step 101 of "monitoring whether a target user leaves the target viewing area corresponding to the target video during the playback of the target video" can be achieved through steps 101a-101d.
[0061] Step 101a: Acquire multiple real-time images of the user watching the target video.
[0062] The multiple real-time images were captured at different times.
[0063] In this embodiment of the application, "user" refers to all users who watch the target video, and there may be only one user or multiple users; the images of users watching the target video at different times can be captured by the image acquisition device set on the display device, that is, real-time images.
[0064] It should be noted that once the target video starts playing, the image acquisition device (i.e., the camera) will begin capturing real-time images of the user.
[0065] In this embodiment, the image acquisition device acquires images at the same speed as the frame rate of the target video; that is, the image acquisition device acquires the same number of images as the number of video frames played per second (i.e., the number of images played per second). Furthermore, the real-time images and the first images have a one-to-one correspondence on the timeline.
[0066] Step 101b: Use an object detection model to process each real-time image to obtain the user's position in each real-time image.
[0067] In this embodiment, the object detection model can refer to the YOLO (You Only Look Once) model. In one possible implementation, the object detection model can be the YOLOv4 model. Specifically, each real-time image can be input as an input parameter into the YOLOv4 model. The YOLOv4 model uses CSPDarknet53 as its backbone network. After receiving the input real-time image, it extracts features from the real-time image through a series of convolutional layers, batch normalization layers, and activation functions. It then extracts a set of anchor boxes A={a1, a2, a3…} for the target to be detected from the features, and defines a set of coordinate data (x,y,w,h) for each anchor box. At the same time, it can define the probability Pobj that the anchor box contains the target and the probability P(c|x,y,w,h) that the target is a person. Finally, it can extract the set of people (i.e., the set of users watching the video) P={p1, p2, p3…} in the real-time image through threshold filtering and non-maximum suppression (NMS), and output the position of each user in each real-time image.
[0068] Where (x,y) represents the coordinate axes of the center point of the anchor frame, and (w,h) represents the width and height of the anchor frame.
[0069] In one feasible approach, after obtaining the position of each user in each real-time image, the position of each user in the first real-time image acquired by the image acquisition device can be obtained, and the sub-image corresponding to each user can be determined based on this position. Then, preprocessing operations can be performed on each face image. Further, a multi-task cascaded convolutional neural network (MTCNN) can be used to process the preprocessed face images to extract the user's facial features. Then, these facial features can be matched with each facial feature in the facial feature set to determine the target facial feature that matches the given facial feature. Finally, [the process can be...] Figure 2 The identity recognition and monitoring module determines the user information corresponding to the target facial features from the user's basic information set, and identifies this user information as the user watching the video. It should be noted that both the facial feature set and the user's basic information set can be pre-stored in a database on the display device (i.e., the summary generation device). Figure 2 The database contains basic user information, and the facial feature set stores the facial features of all users who have ever watched videos through a display device.
[0070] It should be noted that this approach is suitable for scenarios where all users detected by the object detection model have watched the video on the display device. For example, a gathering of multiple acquaintances or friends.
[0071] In another possible approach, after receiving a playback command from the user, a dialog box for collecting user information can pop up on the display device's interface. Information about all users watching the video can then be entered into the dialog box, and the display device will store this information in a database. It should be noted that this method is typically suitable for scenarios with new users watching the video; new users refer to those who have never watched a video on that display device before.
[0072] The user's information may include their name and mobile phone number. Specifically, after obtaining the user's mobile phone number, such as... Figure 2 As shown, the identity recognition and monitoring module in the display device can send a query command to the communication device (e.g., mobile phone) corresponding to the mobile phone number in real time to obtain the call status of the user watching the video.
[0073] It should be noted that if only one person is watching the target video, only that user's call status needs to be obtained; if multiple people are watching the target video together, then the call status of each user needs to be obtained.
[0074] In this embodiment of the application, before playing the target video, it is necessary to pre-install pre-made communication software on the communication devices of all users watching the video. This software can obtain the user's call status. After the communication device receives the query command sent by the display device, it can send the user's real-time call status obtained by the communication software to the display device.
[0075] Step 101c: Use a target tracking algorithm to process each real-time image and position to obtain the real-time trajectory information of the user while watching the target video.
[0076] In this embodiment, the target tracking algorithm can be either the DeepSORT algorithm or the ByteTrack algorithm; real-time trajectory information can be used to characterize whether the user appears in the image and their specific location within the image. In one possible implementation, the real-time trajectory information can be as shown in Table 1 below.
[0077]
[0078] Table 1 The sequence number of the real-time image can be determined according to the acquisition time of the real-time image; the earlier the acquisition time, the earlier the sequence number. The user status indicates whether the user is in the acquired real-time image.
[0079] Step 101d: Based on real-time trajectory information, determine whether any target user has left the target viewing area during playback.
[0080] In this embodiment, if it is determined from real-time trajectory information that a user has not appeared in any of N consecutive images, then it can be determined that the user has indeed left the target viewing area, i.e., the user has been lost within the target viewing area. It should be noted that N is an empirically set value.
[0081] It should be noted that if there is only one user, the target user can only be that user; if there are multiple users, the target user can be one or more of the multiple users.
[0082] In other embodiments of this application, step 102 can be implemented by steps 102a-102b or steps 102c-102e.
[0083] Step 102a: If the target user is detected to have left the target viewing area, determine the first time the target user left the target viewing area and the second time the target user re-entered the target viewing area based on the real-time trajectory information.
[0084] In this embodiment, if it is determined from real-time trajectory information that a user has not appeared in N consecutive images, then this time can be defined as the first time. In essence, the first time is the time when it is determined that a user has not appeared in any of the N consecutive images. It should be noted that the first time is the starting time when the target user leaves the target viewing area. .
[0085] For example, if N is 3, and it is determined from the real-time trajectory information that a user did not appear in a certain real-time image at 15:00, did not appear in the next real-time image at 15:03, and did not appear in the third consecutive real-time image at 15:05, then 15:05 can be determined as the first time.
[0086] In this embodiment, after determining the first time, the real-time trajectory information of the target user can be further detected, and the target user's reappearance in the real-time image can be determined based on this trajectory information. Specifically, if the real-time trajectory information determines that the target user has reappeared in a subsequent real-time image, then the real-time trajectory information can be used to determine whether the target user appears in N consecutive images following that real-time image, and the time when it is determined that the target user appears in the Nth consecutive image following the real-time image is taken as the second time. It should be noted that the second time is essentially the time when the target user re-enters the target viewing area. .
[0087] For example, if N is 3, and the target user reappears in a real-time image at 15:13 according to the real-time trajectory information, does not appear in the next real-time image at 15:16, and does not appear in the third consecutive real-time image at 15:18, then 15:18 can be determined as the second time.
[0088] It should be noted that the first and second times are normal clock times, that is, the objective and evenly passing time shown by the clock on the wall.
[0089] Step 102b: Determine the target time period based on the first time and the second time.
[0090] In this embodiment of the application, the time block between the first time and the second time, i.e. the time period, is the target time period.
[0091] For example, if the first time is 15:05 and the second time is 15:18, then the target time period is 15:05-15:18.
[0092] Step 102c: Obtain the call status of the target user while watching the target video.
[0093] The call status indicates whether the target user is currently making a call.
[0094] In this embodiment of the application, as described in step 101b, the display device will monitor the call status of each user watching the target video in real time through the identity recognition and monitoring module. In other words, the display device will monitor in real time whether each user is currently on a call.
[0095] In this embodiment of the application, after determining that the target user has left the target viewing area, the call status of the target user during the entire process of watching the video can be obtained.
[0096] Step 102d: If the call status changes from the first state to the second state, determine the third time when the call status changes from the first state to the second state, and monitor the fourth time when the call status changes from the second state to the first state.
[0097] The first state indicates that the target user is not making a call; the second state indicates that the target user is making a call.
[0098] In this embodiment of the application, if it is determined that the target user's state changes from the first state to the second state, it means that the target user has changed from not being in a call state to being in a call state, that is, the target user has received a call. In the scenario of watching a movie alone or in groups, it may be necessary to briefly leave the seat to answer the phone and handle the corresponding matters. At this time, the time when the call state changes from the first state (not being in a call state) to the second state (being in a call state) can be directly determined as the time when the target user leaves the target viewing area, that is, the third time.
[0099] Afterwards, you can continue to monitor in real time whether the call status changes from the second state back to the first state. Once it is confirmed that the call status has changed from the second state back to the first state, it means that the target user has ended the call. At this time, it can be assumed that the target user will re-enter, that is, return to the target viewing area to continue watching the target video. Then, the time when the call status changes from the second state to the first state can be directly determined as the fourth time, that is, the fourth time when the target user re-enters the target viewing area.
[0100] Step 102e: Determine the target time period based on the third and fourth times.
[0101] In this embodiment of the application, the time block between the third time and the fourth time, i.e. the time period, is the target time period.
[0102] In other embodiments of this application, step 103 can be implemented in the following ways.
[0103] B1. Determine the multiple first sub-videos that have been played from multiple candidate sub-videos.
[0104] In this embodiment of the application, after determining that the target user has returned (i.e. re-entered) the target viewing area, multiple first sub-videos that are currently being played can be determined from multiple candidate sub-videos.
[0105] B2. Based on the target time period and the playback time of each video frame in each first sub-video, determine the target sub-video from multiple first sub-videos.
[0106] In this embodiment of the application, the target time period may include the start time when the target user leaves the target viewing area. And the time when the target user re-enters the target viewing area, i.e., the deadline when the target user leaves the target viewing area. Then, the first video frame whose playback time is the same as the start time and the second video frame whose playback time is the same as the end time can be determined from the multiple video frames of the multiple first sub-videos.
[0107] In one feasible approach, the first sub-video containing the first video frame can be used as the starting sub-video, the first sub-video containing the second video frame can be used as the ending sub-video, and finally, the starting sub-video, the ending sub-video, and all sub-videos in between can be collectively determined as the target sub-video.
[0108] It should be noted that if the starting sub-video and the ending sub-video are the same sub-video, then there will be only one target sub-video in the end; if the starting sub-video is the second first sub-video and the ending sub-video is the fifth first sub-video, then the target sub-video in the end will include the second first sub-video, the third first sub-video, the fourth first sub-video, and the fifth first sub-video.
[0109] In other embodiments of this application, step 104 can be implemented by steps 104a-104c.
[0110] Step 104a: If there is a target sub-video, use the target summary generation model to process the first descriptive information to obtain a text summary.
[0111] In this embodiment, the target summary generation model can refer to a sequence-to-sequence (Seq2Seq) model. The Seq2Seq model can be trained based on the Transformer architecture, which leverages its powerful global sequence modeling and semantic feature capture capabilities to perform end-to-end feature transformation and prediction generation on temporal and textual sequence data.
[0112] Specifically, the first descriptive information can be directly input into the Seq2Seq model as an input parameter, and the Seq2Seq model will process the input first descriptive information to obtain a text summary of the target sub-video.
[0113] It should be noted that the target summarization model can be trained in the following way.
[0114] C1. Obtain the sample description information and sample text summary of the sample video.
[0115] In this embodiment of the application, the sample video can be a complete video or a video clip; the sample description information is used to describe the content of the sample video, that is, to describe the plot of the sample video; the sample text summary can refer to pre-generated information that can briefly describe the content of the sample video.
[0116] C2. Based on the sample description information and sample text summary, the initial summary generation model is trained to obtain the target summary generation model.
[0117] In this embodiment, sample description information and sample text summary can be input as input parameters into the initial summary generation model. The initial summary generation model then processes the sample description information to generate a text summary of the sample video. Further, a first loss value can be calculated based on the difference between the generated text summary and the sample text summary, and the parameters of the initial summary generation model are updated based on the first loss value. Then, the updated initial summary generation model can be used to continue processing the input sample description information to generate a new text summary, and a second loss value can be calculated based on the difference between the new text summary and the sample text summary. The parameters of the model are then updated again based on the second loss value. This process is repeated until the loss value meets the target condition, at which point the loop stops, and the target summary generation model is obtained.
[0118] Step 104b: If there are multiple target sub-videos, obtain the target user's preference information for the video's content metadata, and generate a text summary based on the preference information and the first description information of each target sub-video.
[0119] In this embodiment, the content metadata may include information such as video name, type, style, actors, directors, roles, and duration. In other words, preference information can characterize which type of video, actor, director, or role the target user prefers. In one feasible implementation, preference information typically includes multiple preference keywords, each representing a preference of the target user.
[0120] For example, preference information may include Chris (director's name), Cage (actor's name), suspense (video type), etc.
[0121] In one feasible way, such as Figure 4 As shown, an Optical Character Recognition (OCR) model based on a Convolutional Neural Network (CNN) can be used to process multiple video frames (i.e. multiple images) of the target sub-video to extract the subtitle text of the target sub-video. Then, for each target sub-video, the subtitle text can be combined to obtain the first descriptive information of each target sub-video.
[0122] In this embodiment of the application, multiple second description information can be determined from multiple first description information based on preference information, and then a text summary can be generated based on the multiple second description information.
[0123] In one feasible approach, the target user's historical viewing list—that is, all the videos the target user has previously watched—can be obtained first, and then analyzed to obtain the target user's preference information. It should be noted that the target user's historical viewing list can be pre-stored in the database of the summary generation device.
[0124] In another possible approach, the target user's preference information can be collected in real time after the playback command is received.
[0125] In this embodiment of the application, step 104b can be implemented through steps 104b1-104b3.
[0126] Step 104b1: Determine the similarity between each first description and preference information.
[0127] In this embodiment of the application, for each first description information, a word segmentation model can be used to segment the information into multiple words, and then the similarity between the multiple words corresponding to each first description information and multiple preference keywords can be calculated according to the following formula (2), thereby obtaining the similarity between each first description information and the preference information. Among them, the preference information includes multiple keywords.
[0128] It should be noted that the higher the similarity, the better the match between the primary description information and the preference information, which indicates that the user has a higher degree of liking for the primary description information.
[0129] Formula (2) in, This represents the similarity between the i-th first descriptive information and the preference information; This represents multiple keywords corresponding to the i-th first descriptive information; This indicates multiple preferred keywords.
[0130] Step 104b2: Determine multiple second description information from multiple first description information based on similarity, and combine the multiple second description information to obtain the combined information.
[0131] In this embodiment, multiple first description information can be sorted in descending order of similarity, and the first M first description information are determined as second description information from the sorted first description information. Then, the multiple second description information can be sorted in ascending order of the playback time of the target sub-video corresponding to each second description information to obtain sorted second description information. The sorted second description information is then combined to obtain combined information. In this way, the multiple second description information can be sorted according to the plot development order of the video, thereby ensuring that the final generated text summary has narrative integrity.
[0132] It should be noted that M can be a pre-set empirical value.
[0133] In the embodiments of this application, the second description information may be only one or multiple.
[0134] Step 104b3: Process the combined information using the target summary generation model to obtain a text summary.
[0135] In this embodiment of the application, the combined information can be directly input as input parameters into the trained target summary generation model. Then, the target summary generation model can process the input combined information to obtain a text summary for the target sub-video.
[0136] In this embodiment of the application, by filtering out the second description information that the user is interested in from multiple first description information based on the user's preference information, and then generating a text summary based on the second description information, the user can fully understand the plot of interest that they missed through the text summary, without having to drag the progress bar to repeatedly play the missed video segments, or disturb other users to understand the missed plot, thereby improving the smoothness of video playback and thus improving the video playback efficiency.
[0137] In the embodiments of this application, step 104c can be executed after either step 104a or step 104b.
[0138] Step 104c: Determine the target scrolling speed of the text summary and control the text summary to scroll and display in the target area at the target scrolling speed.
[0139] In this embodiment, the target area may refer to the area on the playback interface of the target video used to display the text summary. In one possible implementation, such as... Figure 5 As shown, the target area can be set at the bottom of the playback interface; in another possible way, such as Figure 6 As shown, the target area can be set at the top of the playback interface. In this case, the target area is the area where the bullet comments are displayed.
[0140] In this embodiment of the application, the text summary can be displayed in the same way as the bullet screen, that is, both are displayed by scrolling.
[0141] In this embodiment of the application, the target user’s preference for the target sub-video and the importance of the target sub-video can be determined first. Then, the target scrolling speed of the text summary can be determined based on the preference, importance and target coefficient.
[0142] In this embodiment of the application, step 104c can be implemented through steps 104c1-104c3.
[0143] Step 104c1: Based on preference information and text summary, determine the degree of preference of the target user for the target sub-video.
[0144] In this embodiment of the application, the degree of preference can characterize the interest of the target user in the target sub-video that they missed. The higher the degree of preference, the more interested the user is in the target sub-video, and conversely, the lower the degree of preference, the less interested the user is in the target sub-video.
[0145] In this embodiment of the application, the historical comment information of the target user when watching the movie can be obtained, that is, the comment information published by the target user when watching the movie in the past, and sentiment analysis can be performed on the historical comment information to extract keywords that the user is interested in (e.g., actors, characters, plot tags, etc. that the user is interested in). Then, the extracted keywords that the user is interested in and the preference keywords in the preference information can be deduplicated to obtain the deduplicated keywords. Furthermore, the text summary can be segmented using a word segmentation model to obtain multiple target words. Then, the similarity between the multiple target words and the deduplicated keywords can be calculated according to formula (3), which is the degree of preference of the target user for the target sub-video.
[0146] Formula (3) Where B represents the target user's preference for the target sub-video; This represents the j-th target word; This represents the s-th deduplicated keyword.
[0147] Step 104c2: Determine the importance of the target sub-video.
[0148] In this embodiment, importance can include the popularity of the target sub-video and its plot relevance. Specifically, a reference image of the target video can be obtained first, and a target image can be determined from the target sub-video. Then, the matching degree of the target image with the reference image (i.e., the plot relevance of the target sub-video) and the popularity of the target sub-video can be determined. Based on the matching degree and popularity, the importance of the target sub-video can be determined.
[0149] In this embodiment of the application, step 104c2 can be implemented by steps 104c21-104c24.
[0150] Step 104c21: Obtain the historical playback count and historical slow-motion playback count of the target sub-video, the historical comment count for the target sub-video, and the baseline image of the target video.
[0151] Among them, the historical slow playback count is the number of times the target sub-video was played at the target playback speed; the target playback speed is less than the baseline playback speed of the target sub-video; the baseline image is used to provide key content features of the target video.
[0152] In this embodiment, the number of historical playbacks can refer to the number of times the target sub-video is played repeatedly; the baseline playback speed can refer to the baseline speed of video playback, i.e., 1x speed, and the target playback speed is less than the baseline speed, that is, the target playback speed is less than 1x speed, i.e., 0.5x speed, 0.25x speed, etc. Then, the number of historical slow playbacks can refer to the number of times the target sub-video is played at a speed less than 1x speed; the number of historical comments can refer to the number of times the target sub-video is commented on, that is, the number of comments about the sub-video posted by users who have watched the target sub-video.
[0153] It should be noted that the more times the target sub-video is played in history, the more times it is played in slow motion, and the more times it is commented on, the more popular the target sub-video becomes, which means that the target sub-video is a relatively exciting video clip.
[0154] In this application embodiment, the reference image may refer to the cover image of the target video, which is typically used to provide key information about the video. In one possible implementation, if the target video is a movie / TV series, the reference image may refer to a promotional image for the movie / TV series.
[0155] It should be noted that one target video corresponds to one reference image.
[0156] In this embodiment, the number of historical playbacks, the number of historical slow-motion playbacks, the number of historical comments, and the baseline image can all be obtained from the database of the video playback platform where the target video is located. It should be noted that the user watches the target video precisely through the video playback platform.
[0157] Step 104c22: Determine the popularity of the target sub-video based on the number of historical replays, the number of historical slow-motion playbacks, and the number of historical comments.
[0158] Popularity represents the popularity of the target sub-video.
[0159] In this embodiment of the application, the number of historical revisits, the number of historical slow-motion playbacks, and the number of historical comments can be normalized to obtain the normalized number of revisits, the number of normalized slow-motion playbacks, and the number of normalized comments. Then, the normalized number of revisits, the number of normalized slow-motion playbacks, and the number of normalized comments can be calculated according to the following formula (4) to obtain the popularity of the target sub-video.
[0160] Formula (4) in, This indicates the popularity of the r-th target sub-video; This represents the normalized revisit count of the r-th target sub-video; This represents the normalized slow playback count of the r-th target sub-video; This represents the normalized number of comments for the r-th target sub-video.
[0161] It should be noted that the higher the popularity of a target sub-video, the more popular that target sub-video is with the audience.
[0162] Step 104c23: Determine the target image from multiple second images of the target sub-video, and determine the degree of matching between the reference image and the target image.
[0163] In this embodiment, the target sub-video can be decoded to obtain multiple second images (i.e., multiple video frames), and keyframes (i.e., target images) can be determined from the multiple video frames. It should be noted that one target sub-video corresponds to one keyframe, and if there are multiple target sub-videos, each target sub-video corresponds to one keyframe, that is, each target sub-video corresponds to one target image.
[0164] In one feasible approach, the keyframe can be determined by analyzing the differences between multiple video frames. This process is existing technology and will not be elaborated upon here.
[0165] In this embodiment, after determining the target image corresponding to each target sub-video, the vector similarity between the reference image and each target image can be calculated according to the following formula (5), i.e., the matching degree. The higher the matching degree, the more crucial the plot of the target sub-video corresponding to the target image.
[0166] Formula (5) in, Hr represents the degree of matching between the reference image and the target image of the r-th target sub-video; Hr represents the target image of the r-th target sub-video; keyframe represents the reference image.
[0167] It should be noted that if there are multiple target sub-videos, each target sub-video will correspond to a matching degree.
[0168] Step 104c24: Determine importance based on popularity and matching degree.
[0169] In this embodiment, the initial importance pop of each target sub-video can be determined by directly adding the popularity value and the matching value, i.e., pop = pv + pk. Then, the initial importance of multiple target sub-videos can be summed to obtain the importance of all target sub-videos.
[0170] It should be noted that if there is only one target sub-video, then the initial importance of that target sub-video is the final importance.
[0171] Step 104c3: Determine the target scrolling speed based on preference level, importance level, and target coefficient.
[0172] In this embodiment of the application, the target coefficient can be a fixed value set by the user based on the size of the display device (i.e., the summary generation device) and the user's viewing experience; that is, the target coefficient can be a numerical value.
[0173] Specifically, the sum of the target user's preference for the target sub-video and the importance of the target sub-video can be determined first. Then, the target coefficient n can be divided by this sum to obtain the target scrolling speed of the text summary. The specific calculation formula is shown in the following formula (6).
[0174] Formula (6) Where Speed represents the target scrolling speed; B represents the target user's preference for the target sub-video; n represents the target coefficient; and pop represents the importance of the target sub-video.
[0175] In this embodiment, the scrolling speed of the text summary can be determined based on the plot excitement and popularity of the video clips missed by the target user while leaving the target viewing area. In other words, the more exciting and popular the plot of the target sub-video (i.e. the missed video clip) is, the slower the scrolling speed of the target video summary will be.
[0176] In this application embodiment, if this application is applied to a single-person movie-watching scenario, the method provided by this application can solve the problem in related technologies where it is necessary to repeatedly watch missed video segments by dragging the progress bar, resulting in low video playback smoothness and poor video playback efficiency, thereby improving the overall video playback efficiency and the user's viewing experience.
[0177] In this embodiment of the application, if the application is applied to a multi-person video viewing experience, the method provided by the application can provide users who briefly leave the room with an instant summary of the missed video segments without disturbing other users to understand the missed plot. This achieves the effect of ensuring users' continuous understanding of the video content and improving the overall viewing experience.
[0178] The summary generation method provided in the embodiments of this application can determine the video segment (i.e., the target sub-video) missed by the target user based on the target time period when the target user leaves the target viewing area, and then generate a text summary of the target sub-video and display it in the target area. In this way, after the target user returns to the target viewing area, he / she can directly understand the plot description of the missed video segment through the text summary displayed in the target area, without having to repeatedly drag the progress bar to rewatch the missed video segment. As a result, the playback smoothness of the target video can be significantly improved, thereby improving the video playback efficiency. This solves the problem in related technologies where users need to repeatedly play some video segments when watching videos, resulting in poor video playback efficiency.
[0179] Based on the foregoing embodiments, this application provides a summary generation apparatus that can be applied to... Figure 1 In the summary generation method provided in the corresponding embodiment, refer to Figure 7 As shown, the summary generation device 2 may include: a monitoring unit 21, a first determining unit 22, a second determining unit 23, an acquisition unit 24, and a processing unit 25, wherein: The monitoring unit 21 is used to monitor whether a target user leaves the target viewing area corresponding to the target video during the playback of the target video based on the input playback command. The first determining unit 22 is used to determine the target time period when the target user leaves the target viewing area if it is detected that the target user has left the target viewing area. The second determining unit 23 is used to determine the target sub-video from multiple candidate sub-videos of the target video based on the target time period; The acquisition unit 24 is used to acquire first description information of the target sub-video; wherein, the first description information is used to describe the content of the target sub-video; The processing unit 25 is used to generate a text summary of the target sub-video based on the first description information and display it in the target area of the playback interface of the target video.
[0180] In other embodiments of this application, the monitoring unit 21 is further configured to perform the following steps: Acquire multiple real-time images as the user watches the target video; these multiple real-time images are captured at different times. The target detection model is used to process each real-time image to obtain the user's position in each real-time image; A target tracking algorithm is used to process each real-time image and location to obtain the real-time trajectory information of the user while watching the target video; Based on real-time trajectory information, it can be determined whether any target users left the target viewing area during playback.
[0181] In other embodiments of this application, the monitoring unit 21 is further configured to perform the following steps: The target video is decoded to obtain multiple first images, and the degree of difference between adjacent images in the multiple first images is determined; The target video is divided based on its difference, resulting in multiple candidate sub-videos.
[0182] In other embodiments of this application, the first determining unit 22 is further configured to perform the following steps: If it is detected that the target user has left the target viewing area, the first time when the target user left the target viewing area and the second time when the target user re-entered the target viewing area are determined based on real-time trajectory information. The target time period is determined based on the first and second time periods.
[0183] In other embodiments of this application, the first determining unit 22 is further configured to perform the following steps: Obtain the call status of the target user while watching the target video; where call status indicates whether the target user is currently on a call. If the call status changes from the first state to the second state, determine the third time when the call status changes from the first state to the second state, and monitor the fourth time when the call status changes from the second state to the first state; wherein, the first state indicates that the target user is not making a call; the second state indicates that the target user is making a call. The target time period is determined based on the third and fourth time periods.
[0184] In other embodiments of this application, the processing unit 25 is further configured to perform the following steps: If there is a target sub-video, the first descriptive information is processed using a target summary generation model to obtain a text summary; If there are multiple target sub-videos, obtain the target user's preference information for the video's content metadata, and generate a text summary based on the preference information and the first description information of each target sub-video; Determine the target scrolling speed for the text summary and control the text summary to scroll and display at the target speed in the target area.
[0185] In other embodiments of this application, the processing unit 25 is further configured to perform the following steps: Determine the similarity between each piece of first descriptive information and preference information; Multiple second description information is determined from multiple first description information based on similarity, and the multiple second description information is combined to obtain the combined information; The combined information is processed using a target summarization generation model to obtain a text summary.
[0186] In other embodiments of this application, the processing unit 25 is further configured to perform the following steps: Based on preference information and text summaries, determine the degree of preference of target users for target sub-videos; Determine the importance of the target sub-videos; The target rolling speed is determined based on the degree of preference, importance, and target coefficient.
[0187] In other embodiments of this application, the processing unit 25 is further configured to perform the following steps: The system retrieves the historical playback count and historical slow-motion playback count of the target sub-video, the historical comment count for the target sub-video, and the baseline image of the target video. The historical slow-motion playback count is the number of times the target sub-video was played at the target playback speed. The target playback speed is less than the baseline playback speed of the target sub-video. The baseline image is used to provide key content features of the target video. The popularity of a target sub-video is determined based on the number of historical replays, historical slow-motion playbacks, and historical comments; where popularity represents the degree of popularity of the target sub-video. The target image is determined from multiple second images of the target sub-video, and the degree of matching between the reference image and the target image is determined. Importance is determined based on popularity and relevance.
[0188] It should be noted that a detailed explanation of the steps performed by each unit can be found in [reference needed]. Figure 1 The description of the summary generation method provided in the corresponding embodiments will not be repeated here.
[0189] The summary generation method provided in the embodiments of this application can determine the video segment (i.e., the target sub-video) missed by the target user based on the target time period when the target user leaves the target viewing area, and then generate a text summary of the target sub-video and display it in the target area. In this way, after the target user returns to the target viewing area, he / she can directly understand the plot description of the missed video segment through the text summary displayed in the target area, without having to repeatedly drag the progress bar to rewatch the missed video segment. As a result, the playback smoothness of the target video can be significantly improved, thereby improving the video playback efficiency. This solves the problem in related technologies where users need to repeatedly play some video segments when watching videos, resulting in poor video playback efficiency.
[0190] Based on the foregoing embodiments, embodiments of this application provide a digest generation device, which can be applied to... Figure 1 In the summary generation method provided in the corresponding embodiment, refer to Figure 8 As shown, the summary generation device 3 may include: a processor 31, a memory 32, and a communication bus 33, wherein: Communication bus 33 is used to realize the communication connection between processor 31 and memory 32; Processor 31 is used to execute the digest generation program in memory 32 to perform the following steps: When playing a target video based on an input playback command, monitor whether the target user leaves the target viewing area corresponding to the target video during the playback process; If it is detected that a target user has left the target viewing area, determine the target time period during which the target user left the target viewing area; The target sub-video is determined from multiple candidate sub-videos based on the target time period; Obtain the first description information of the target sub-video; wherein, the first description information is used to describe the content of the target sub-video; A text summary of the target sub-video is generated based on the first descriptive information and displayed in the target area of the target video's playback interface.
[0191] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: Acquire multiple real-time images as the user watches the target video; these multiple real-time images are captured at different times. The target detection model is used to process each real-time image to obtain the user's position in each real-time image; A target tracking algorithm is used to process each real-time image and location to obtain the real-time trajectory information of the user while watching the target video; Based on real-time trajectory information, it can be determined whether any target users left the target viewing area during playback.
[0192] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: The target video is decoded to obtain multiple first images, and the degree of difference between adjacent images in the multiple first images is determined; The target video is divided based on its difference, resulting in multiple candidate sub-videos.
[0193] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: If it is detected that the target user has left the target viewing area, the first time when the target user left the target viewing area and the second time when the target user re-entered the target viewing area are determined based on real-time trajectory information. The target time period is determined based on the first and second time periods.
[0194] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: Obtain the call status of the target user while watching the target video; where call status indicates whether the target user is currently on a call. If the call status changes from the first state to the second state, determine the third time when the call status changes from the first state to the second state, and monitor the fourth time when the call status changes from the second state to the first state; wherein, the first state indicates that the target user is not making a call; the second state indicates that the target user is making a call. The target time period is determined based on the third and fourth time periods.
[0195] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: If there is a target sub-video, the first descriptive information is processed using a target summary generation model to obtain a text summary; If there are multiple target sub-videos, obtain the target user's preference information for the video's content metadata, and generate a text summary based on the preference information and the first description information of each target sub-video; Determine the target scrolling speed for the text summary and control the text summary to scroll and display at the target speed in the target area.
[0196] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: Determine the similarity between each piece of first descriptive information and preference information; Multiple second description information is determined from multiple first description information based on similarity, and the multiple second description information is combined to obtain the combined information; The combined information is processed using a target summarization generation model to obtain a text summary.
[0197] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: Based on preference information and text summaries, determine the degree of preference of target users for target sub-videos; Determine the importance of the target sub-videos; The target rolling speed is determined based on the degree of preference, importance, and target coefficient.
[0198] In other embodiments of this application, processor 31 is used to execute a digest generation program in memory 32 to perform the following steps: The system retrieves the historical playback count and historical slow-motion playback count of the target sub-video, the historical comment count for the target sub-video, and the baseline image of the target video. The historical slow-motion playback count is the number of times the target sub-video was played at the target playback speed. The target playback speed is less than the baseline playback speed of the target sub-video. The baseline image is used to provide key content features of the target video. The popularity of a target sub-video is determined based on the number of historical replays, historical slow-motion playbacks, and historical comments; where popularity represents the degree of popularity of the target sub-video. The target image is determined from multiple second images of the target sub-video, and the degree of matching between the reference image and the target image is determined. Importance is determined based on popularity and relevance.
[0199] It should be noted that a detailed description of the steps performed by processor 31 can be found in [reference needed]. Figure 1 The summary generation method provided in the corresponding embodiments will not be described in detail here.
[0200] The summary generation method provided in the embodiments of this application can determine the video segment (i.e., the target sub-video) missed by the target user based on the target time period when the target user leaves the target viewing area, and then generate a text summary of the target sub-video and display it in the target area. In this way, after the target user returns to the target viewing area, he / she can directly understand the plot description of the missed video segment through the text summary displayed in the target area, without having to repeatedly drag the progress bar to rewatch the missed video segment. As a result, the playback smoothness of the target video can be significantly improved, thereby improving the video playback efficiency. This solves the problem in related technologies where users need to repeatedly play some video segments when watching videos, resulting in poor video playback efficiency.
[0201] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement... Figure 1 The steps of the summary generation method provided in the corresponding embodiment.
[0202] Based on the foregoing embodiments, embodiments of this application provide a computer program product, which includes a computer program that is implemented when executed by processor 31. Figure 1 The steps of the summary generation method provided in the corresponding embodiment.
[0203] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0204] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0206] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0207] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.
Claims
1. A method for generating summaries, characterized in that, The method includes: When playing a target video based on an input playback command, monitor whether a target user leaves the target viewing area corresponding to the target video during the playback process; If it is detected that the target user has left the target viewing area, the target time period during which the target user left the target viewing area is determined; The target sub-video is determined from multiple candidate sub-videos of the target video based on the target time period; Obtain first description information of the target sub-video; wherein, the first description information is used to describe the content of the target sub-video; A text summary of the target sub-video is generated based on the first description information and displayed in the target area of the target video's playback interface.
2. The method according to claim 1, characterized in that, The monitoring of whether a target user leaves the target viewing area corresponding to the target video during the playback of the target video includes: Acquire multiple real-time images of a user watching the target video; wherein the multiple real-time images are acquired at different times; A target detection model is used to process each real-time image to obtain the user's position in each real-time image; A target tracking algorithm is used to process each real-time image and the location to obtain the real-time trajectory information of the user while watching the target video; Based on the real-time trajectory information, determine whether any of the users are the target user who left the target viewing area during the playback process; Accordingly, before playing the target video, the method further includes: The target video is decoded to obtain multiple first images, and the degree of difference between adjacent images in the multiple first images is determined; The target video is divided based on the difference to obtain the multiple candidate sub-videos.
3. The method according to claim 2, characterized in that, Determining the target time period during which the target user leaves the target viewing area includes: If it is detected that the target user has left the target viewing area, the first time when the target user left the target viewing area and the second time when the target user re-entered the target viewing area are determined based on the real-time trajectory information. The target time period is determined based on the first time and the second time.
4. The method according to claim 1, characterized in that, Determining the target time period during which the target user leaves the target viewing area includes: The call status of the target user while watching the target video is obtained; wherein, the call status indicates whether the target user is currently on a call; If the call status changes from a first state to a second state, a third time when the call status changes from the first state to the second state is determined, and a fourth time when the call status changes from the second state to the first state is monitored; wherein, the first state indicates that the target user is not making a call; the second state indicates that the target user is making a call; The target time period is determined based on the third time and the fourth time.
5. The method according to claim 1, characterized in that, The step of generating a text summary of the target sub-video based on the first description information and displaying it in the target area of the target video's playback interface includes: If there is a target sub-video, the first description information is processed using a target summary generation model to obtain the text summary; If there are multiple target sub-videos, obtain the target user's preference information for the video's content metadata, and generate the text summary based on the preference information and the first description information of each target sub-video; Determine the target scrolling speed of the text summary, and control the text summary to scroll and display in the target area at the target scrolling speed.
6. The method according to claim 5, characterized in that, The process of generating the text summary based on the preference information and the first descriptive information of each target sub-video includes: Determine the similarity between each piece of first descriptive information and the preference information; Based on the similarity, multiple second description information is determined from multiple first description information, and the multiple second description information is combined to obtain the combined information; The combined information is processed using the target summary generation model to obtain the text summary; Accordingly, determining the target scrolling speed of the text summary includes: Based on the preference information and the text summary, the degree of preference of the target user for the target sub-video is determined; Determine the importance of the target sub-video; The target scrolling speed is determined based on the preference level, the importance level, and the target coefficient.
7. The method according to claim 6, characterized in that, Determining the importance of the target sub-video includes: The system acquires the historical playback count and historical slow-motion playback count of the target sub-video, the historical comment count for the target sub-video, and a baseline image of the target video; wherein, the historical slow-motion playback count is the number of times the target sub-video has been played at a target playback speed; the target playback speed is less than the baseline playback speed of the target sub-video; and the baseline image is used to provide key content features of the target video. The popularity of the target sub-video is determined based on the number of historical replays, the number of historical slow-motion playbacks, and the number of historical comments; wherein, the popularity represents the degree of popularity of the target sub-video. A target image is determined from a plurality of second images of the target sub-video, and the degree of matching between the reference image and the target image is determined; The importance level is determined based on the popularity and the matching degree.
8. A summary generation device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute a digest generation program in memory to implement the steps of the digest generation method as described in any one of claims 1-7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the digest generation method as described in any one of claims 1-7.
10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the digest generation method according to any one of claims 1-7.
