Method, apparatus, device and program product for video processing

Abstract

Methods, apparatus, devices, and program products for video processing are provided. The method includes obtaining first information associated with a first video; obtaining a first video segment in the first video based on the first information; obtaining second information for generating a second video based on the first video segment and the first information; obtaining the second video based on the first video and the second information; and establishing a first association relationship, the first association relationship indicating an association relationship between the second video and a first play position in the first video.

Description

Technical Field

[0001] The examples in this article generally relate to the field of computer science, and in particular to methods, apparatuses, devices, and program products for video processing. Background Technology

[0002] With the development of multimedia and network communication technologies, video has become an important medium for information dissemination and content consumption. To enrich the presentation of video content and the viewing experience, interactive features such as likes, bullet comments, and bullet screens have been introduced. Summary of the Invention

[0003] In a first aspect of this document, a video processing method is provided. The method includes: acquiring first information associated with a first video; acquiring a first video segment from the first video based on the first information; acquiring second information based on the first video segment and the first information, the second information being used to generate a second video; acquiring the second video based on the first video and the second information; and establishing a first association relationship, the first association relationship indicating the association relationship between the second video and a first playback position, the first playback position being the playback position within the first video.

[0004] In a second aspect of this document, an apparatus for video processing is provided. The apparatus includes: an information acquisition module configured to acquire first information associated with a first video; a segment acquisition module configured to acquire a first video segment from the first video based on the first information; the information acquisition module is further configured to acquire second information for generating a second video based on the first video segment and the first information; a video acquisition module configured to acquire a second video based on the first video and the second information; and an association establishment module configured to establish a first association, the first association indicating an association between the second video and a first playback position, the first playback position being a playback position within the first video.

[0005] In a third aspect of this document, an electronic device is provided. The electronic device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions executable by the at least one processor, which, when the at least one processor executes the instructions, cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this document, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0007] In a fifth aspect of this document, a computer program product is provided. This computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] In this way, a first video segment and second information for generating a second video can be obtained based on first information associated with the first video, and the second video can be obtained based on the first video and the second information. By establishing a first association between the second video and the first playback position, the second video can be temporally associated with the corresponding video content in the first video, and the processing operations required for manually determining video segments, generating information, and associating positions can be reduced, thereby improving video processing efficiency.

[0009] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figure 2 A schematic diagram of the overall processing procedure for video processing is shown; Figure 3A The diagram illustrates example processes for acquiring the first video segment in several scenarios; Figure 3B The diagram illustrates multiple time windows for several scenarios; Figure 4 The diagram illustrates example processes for generating candidate information in several scenarios. Figure 5 A schematic diagram illustrating an example process for obtaining the second information is shown; Figure 6 A schematic diagram illustrating an example process for generating third-party information is shown; Figure 7 A schematic diagram illustrating an example process for acquiring a second video is shown; Figure 8 A flowchart illustrating example processes for video processing is shown, based on several scenarios. Figure 9 Block diagrams of apparatuses for video processing in several scenarios are shown; and Figure 10 A block diagram is shown of an electronic device capable of implementing one or more examples. Detailed Implementation

[0011] The examples in this document will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the examples, any solutions involving the processing of personal information will be processed only on the premise of a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] As mentioned above, some solutions can pre-set fixed entry points in the video playback interface and present preset animations or visual content at predetermined times. These solutions typically require configuring entry points, presentation times, and content resources separately for different videos. The correlation between the presented additional content and the specific video content is limited, and it is also difficult to automatically generate differentiated video content for different videos.

[0017] Some related solutions utilize visual language models to analyze video frames, shot clips, or extracted images to identify objects, scenes, actions, and corresponding story semantics within the video. However, when the analysis is primarily based on visual changes or visual semantics, factors such as shot transitions, the magnitude of actions, and scene changes may influence the judgment of story events, and story nodes embodied by character dialogue, character identities, and context may be difficult to fully represent.

[0018] Other related solutions can generate text, adapt scripts, or produce prompts based on video descriptions, video titles, human prompts, or partial video understanding results. They can also generate new videos based on text, images, or video clips. When these processes are executed separately, the video content, objects, actions, and other information used in different processing stages may lack correlation. This makes it difficult to maintain consistency between the acquired video clips, generated content, and their relationship to the original video, thus affecting the continuity of content between the new video content and the original video.

[0019] Therefore, when generating a second video associated with the first video, there is still room for improvement in how to continuously utilize the information associated with the first video to match the first video segment, the information used to generate the second video, the associated location, and the second video.

[0020] A scheme for video processing is proposed. According to this scheme, server 130 and / or electronic device 110 obtains first information associated with a first video, and obtains a first video segment from the first video based on the first information. Server 130 and / or electronic device 110 obtains second information for generating a second video based on the first video segment and the first information, and obtains the second video based on the first video and the second information. Server 130 and / or electronic device 110 further establishes a first association relationship, indicating the association relationship between the second video and a first playback position, where the first playback position is the playback position within the first video.

[0021] The above scheme allows the acquisition of the first video segment, the acquisition of the second information, the acquisition of the second video, and the establishment of the first association to revolve around interrelated video content. Since the second information is acquired based on the first video segment and the first information, and the second video is associated with the first playback position, the second video can correspond to a playback position that matches its video content. This helps reduce inconsistencies between objects, dialogues, or actions in the second video and the first video, and improves the continuity of the second video and the first video in terms of content and timing.

[0022] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, the example environment 100 may include an electronic device 110, an application 120 running on the electronic device 110, a server 130, a user 140, and an interface 150 presented by the electronic device 110.

[0024] In example environment 100, application 120 may be an application that supports video processing and video playback. Electronic device 110 can receive a first video, present the first video or a second video related to the first video through application 120, and receive operations from user 140.

[0025] When application 120 is active, electronic device 110 can present interface 150 through application 120. Interface 150 may include a playback area for a first video, prompt elements related to a second video, and interactive elements for controlling video playback.

[0026] In some cases, electronic device 110 communicates with server 130 to obtain at least one of a first video segment, second information, a first playback position, third information, and a second video. Electronic device 110 may also send the first video, the first information, the second information, or data corresponding to an operation from user 140 to server 130.

[0027] Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, virtual reality (VR) / augmented reality (AR) devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some cases, electronic device 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0028] Server 130 can be a single physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 in electronic devices 110 that support video interaction.

[0029] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some cases, server 130 and electronic device 110 can exchange signaling information through their communication connection.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0031] The examples described herein can be implemented on electronic device 110. It should be noted that the operations described relative to electronic device 110 can be performed by a related application 120 running on electronic device 110. The interface described relative to electronic device 110 can be provided by application 120; for example, electronic device 110 presents the corresponding interface by running application 120. Furthermore, some operations described relative to electronic device 110 may require the assistance of server 130 to complete.

[0032] In one implementation, electronic device 110 can independently perform the acquisition of first information, acquisition of first video segment, acquisition of second information, acquisition of second video, and establishment of first association relationship. In another implementation, server 130 can independently perform the above processing and send the processing results to electronic device 110. In yet another implementation, electronic device 110 can perform first video playback, data acquisition, and interface presentation, while server 130 can perform model processing and video generation. Electronic device 110 and server 130 exchange intermediate processing results through a communication connection.

[0033] The following description of the example will continue with reference to the accompanying drawings.

[0034] Figure 2 A schematic diagram of the overall processing procedure 200 for video processing in several scenarios is shown. Procedure 200 can be executed by server 130, electronic device 110, or a combination of both. For ease of explanation, the following description primarily uses electronic device 110 as the executing entity. It should be understood that the processing described herein as being performed by electronic device 110 can be performed separately by electronic device 110 and server 130, or jointly by electronic device 110 and server 130; this is not a limitation herein.

[0035] like Figure 2 As shown, the input to process 200 may include a first video 205 and first information 211. The first video 205 may be a source video of the video content to be acquired and associated with it. The first video 205 may include continuous video content, one or more video segments, or video content formed by multiple shots.

[0036] The first information 211 is associated with the first video 205. The first information 211 may include at least one of the information obtained by processing the first video 205 and the information related to the historical playback of the first video 205, and may provide context regarding video content, objects, dialogue, characters, or bullet comments for the acquisition of the first video segment and the acquisition of the second information. Different types of first information 211 may respectively characterize the video content, objects, audio content, characters, or the distribution of bullet comments during the historical playback of the first video 205.

[0037] For example, the first information 211 may include at least one of first text, second text, and bullet screen information. The first text can be obtained by converting speech in the first video 205 to text, and may include text content and the start and / or end times of the corresponding speech in the first video 205. The second text may include character description information in the first video 205, such as the identity of the object, the relationship between objects, or the role of the object in the first video 205.

[0038] The bullet screen information can be obtained based on the bullet screens in the first video 205. Bullet screens can be information generated during the historical playback of the first video 205 and associated with the corresponding playback time in the first video 205, and can include bullet screen text and corresponding video time information. Therefore, bullet screen information can be used to characterize the distribution of multiple bullet screens on the timeline of the first video 205.

[0039] In process 220, electronic device 110 can acquire a first video segment 226 from the first video 205 based on the first information 211. The first video segment 226 may be a portion of video content acquired from the first video 205 to provide video content context for the subsequent acquisition of the second information. For example, the first video segment 226 may correspond to a continuous time range in the first video 205, or it may include multiple portions of video content in the first video 205 that are content-related to each other.

[0040] In some cases, process 220 may determine one or more candidate video segments 223 based on bullet screen information, and obtain the analysis results of each candidate video segment 223 based on at least one of the first text or the second text, thereby selecting a first video segment 226 from the candidate video segments 223. Process 220 may also determine the type 227 of the first video segment 226. The following is combined with... Figure 3A and Figure 3B The process of acquiring the first video segment 226 and its type 227 is further described.

[0041] Figure 3A A schematic diagram of an example process 220 for acquiring a first video segment is shown. Process 220 may receive first information 211.

[0042] The electronic device 110 can determine multiple time windows 222 corresponding to the first video 205 based on the bullet screen information in the first information 211, and acquire multiple candidate video segments 223. The time window can refer to the time range on the time axis of the first video 205. The multiple time windows 222 can be non-overlapping, partially overlapping, or sliding along the time axis of the first video 205 according to a predetermined step size.

[0043] The bullet screen information may include multiple bullet screens and their respective playback times corresponding to the first video 205, or it may include a bullet screen distribution pre-generated based on multiple bullet screens. For example, bullet screens may include bullet screen text, bullet screen generation time, and corresponding video time information.

[0044] For multiple time windows 222, the electronic device 110 can determine the distribution status based on the distribution of multiple bullet comments in the first video 205. The distribution status can indicate the number of bullet comments in each time window, the trend of the number of bullet comments in the multiple time windows 222, or a combination of both. The electronic device 110 can select at least one time window from the multiple time windows 222 based on the distribution status, and obtain at least one candidate video segment 223 based on the selected at least one time window. The candidate video segment 223 may include the video segment corresponding to the corresponding time window.

[0045] For example, electronic device 110 can count the number of valid bullet comments generated within each time unit along the time axis of the first video 205 to obtain the distribution of the number of bullet comments as the playback time changes. In some cases, valid bullet comments can be bullet comments that meet predetermined filtering conditions, such as excluding bullet comments with duplicate content, no actual semantic meaning, or other bullet comments that are not used for subsequent video content analysis. Through the above statistics, a bullet comment density curve reflecting the level of interactive activity corresponding to different playback times can be formed.

[0046] Figure 3B The diagram illustrates multiple time windows for several scenarios. The bullet screen distribution 300 can be indicated by the bullet screen information in the first information 211. As an example, each bar in the bullet screen distribution 300 can represent the number of valid bullet screens obtained within a corresponding time unit of the first video 205.

[0047] like Figure 3B As shown, the electronic device 110 can use a sliding window of preset duration to form multiple time windows 301 to 303 along the time axis of the first video 205, and determine the distribution of bullet comments within each time window. For example, the duration of the sliding window can be 20 seconds, and one time window can cover the 100th to 119th seconds of the first video 205. For this time window, the electronic device 110 can count the total number of valid bullet comments included, and determine the trend of the number of valid bullet comments within this time window or between adjacent time windows.

[0048] It should be understood that the 20-second and 100-119-second intervals are for illustrative purposes only. The duration, step size, and boundaries of the sliding window can be determined based on the duration of the first video 205, the number of bullet comments, bullet comment distribution characteristics, processing resources, or other factors. Multiple time windows 301 to 303 can overlap, thereby enabling continuous analysis of local changes in the bullet comment distribution 300.

[0049] In some cases, the electronic device 110 can select at least one time window from multiple time windows 301 to 303 where the bullet comments are relatively concentrated, based on the number of bullet comments and / or the trend of the number of bullet comments corresponding to each time window. For example, it can select a time window where the number of bullet comments is greater than a predetermined threshold, a time window where the number of bullet comments is within a predetermined proportion of multiple time windows, a time window where the number of bullet comments shows a predetermined upward trend, or a time window determined based on a combination of the above conditions. The selected time window can reflect the candidate time range where user discussion is relatively active in the first video 205.

[0050] Therefore, the electronic device 110 can further obtain the corresponding candidate video segment 223 from the first video 205 based on at least one selected time window, and further analyze the candidate video segment 223 by combining at least one of the first text, the second text and the video content corresponding to the candidate video segment 223 to obtain the first video segment 226.

[0051] For the acquired candidate video segments 223, the electronic device 110 can further analyze the corresponding bullet comments. In some cases, the electronic device 110 can perform content analysis 224 on the corresponding bullet comments, for example, performing semantic clustering to group multiple bullet comments with similar semantics into the same content category. The content category can represent video events, objects, object relationships, actions, video results, or other content semantics that multiple bullet comments are concerned about.

[0052] For example, multiple bullet comments corresponding to the same candidate video segment 223 can express discussions about a certain object, character conflicts, relationship changes, or video results. Through semantic clustering, these bullet comments can be divided into one or more content categories. The electronic device 110 can also determine the time range corresponding to each content category, establishing a temporal association between the content category and the corresponding part of the video content in the first video 205.

[0053] In content analysis 224, electronic device 110 can obtain analysis results based on the bullet comments corresponding to each candidate video segment 223. The analysis results can indicate the matching relationship between the video content of the corresponding candidate video segment 223 and at least one content category, and can also indicate at least one time range corresponding to the content category.

[0054] For example, for a candidate video segment 223, there may be multiple semantically different categories of bullet screen discussions. One content category focuses on changes in object identity, while another focuses on the conflict outcomes between objects. By analyzing the results, it is possible to further distinguish the different video content that is the focus of discussion in the same candidate video segment 223 and their corresponding time ranges in the first video 205.

[0055] Based on the analysis results, the electronic device 110 can select a first video segment 226 from multiple candidate video segments 223. In some cases, the analysis results, at least one of the first or second text, and a portion of the video content of the candidate video segment 223 can be input into a model with video understanding and / or text understanding capabilities. The model can analyze the relationship between the corresponding video content and objects, the association between speech and text, and the context, and determine the matching relationship between the video content of the corresponding candidate video segment 223 and at least one content category.

[0056] For example, the model can determine the matching relationship between the corresponding video content and character conflicts, content reversals, changes in character relationships, the occurrence of important results, enhanced content tension, or other predetermined content categories. Thus, the candidate video segment 223 initially obtained based on the bullet screen distribution can be further constrained by the video semantics represented by the first or second text, thereby obtaining the first video segment 226.

[0057] In some cases, the type 227 of the first video segment 226 may be determined together with the first video segment 226. Type 227 may be a type from a preset set of content categories, used to indicate the content direction when obtaining second information from the first video segment 226. For example, type 227 may include conflict response type, content highlight type, character relationship type, action extension type, content tension enhancement type, or other types derived from the video content of the first video segment 226.

[0058] In some cases, the type 227 of the first video segment 226 does not need to be determined solely based on a single frame, but can be determined by comprehensively considering the object identities, object relationships, voice and text included in the first information 211, as well as the development of events in the first video segment 226. For example, if the first video segment 226 includes a conflict between objects and a conflict outcome has been achieved, a type related to the conflict response can be determined; if the first video segment 226 primarily reflects changes in the relationships between objects, a type related to the relationships between people can be determined.

[0059] In some cases, process 220 may also output reason information for selecting the first video segment 226. This reason information may describe the events, object relationships, audio and text, bullet screen distribution, or the relationship between content category and type 227 in the first video segment 226, for use in subsequent processing or for verifying the selection result of the first video segment 226.

[0060] Through the above processing, the distribution of bullet comments and the video semantics represented by the first or second text can be combined to obtain the first video segment 226 that matches the content of the focused discussion and the predetermined content category from the first video 205, and further provide the acquisition of the second information and the second video with the same video content context as the first video 205.

[0061] Continue to refer to Figure 2 In process 230, electronic device 110 can generate multiple candidate information 232 based on the first video segment 226 and the first information 211. In process 240, electronic device 110 can select second information 244 from the multiple candidate information 232. The multiple candidate information 232 are used to describe the corresponding candidate videos, and the second information 244 is used to generate a second video 275. The candidate video can be additional video content formed based on the video content corresponding to the first video segment 226, for example, further video content can be formed around the objects, events, object relationships, or video results in the first video 205.

[0062] Multiple candidate information items 232 can describe the content to be generated for different candidate videos and provide candidate results for the selection of the second information 244. For example, one candidate information item 232 can describe the participants, interaction methods, visual styles, interactive actions, feedback results, or combinations thereof in the candidate video. Candidate information 232 can be represented using natural language, structured fields, or a combination of both.

[0063] The second information 244 may be information selected from multiple candidate information items 232 to determine the content to be generated for the second video 275. The second information 244 may describe at least one of the participants, interaction methods, visual style, interactive actions, feedback results, and other video content in the second video 275, and may serve as the basis for subsequently determining the first playback position and generating the second video 275. The second information 244 may be represented using natural language prompts, structured descriptions, or a combination thereof.

[0064] The first playback position can be the playback position in the first video 205 used to establish the association with the second video 275. The first playback position can correspond to a playback time point in the first video 205, or it can correspond to a playback time range. In some cases, the first playback position can be determined based on the video content described by the second information 244 and its associated first event. Therefore, different pieces of second information 244 can correspond to different first playback positions.

[0065] Figure 4 A schematic diagram of an example process 230 for generating candidate information in some scenarios is shown.

[0066] like Figure 4As shown, in the candidate information generation process 230, the electronic device 110 can determine the first generation strategy 231 corresponding to the type 227 from multiple generation strategies based on the type 227 of the first video segment 226, and generate multiple candidate information 232 based on the first video segment 226, the first information 211 and the first generation strategy 231.

[0067] A generation strategy can refer to a set of rules, prompts, model configurations, processing flows, or combinations thereof used to constrain the candidate information generation process. For example, each generation strategy may include at least one of the following: content generation direction, generation constraints, sample data, and content boundaries corresponding to the type of the corresponding first video segment. Different types can correspond to different generation strategies, thereby enabling first video segments 226 of different content categories to adopt different information generation methods.

[0068] For example, for types related to character conflict or conflict response, the corresponding generation strategy can make the candidate videos described by the candidate information focus on conflict responses, interactive actions, and their feedback results; for types related to changes in character relationships, the corresponding generation strategy can make the candidate videos described by the candidate information focus on object relationships, content atmosphere, and interactive feedback. Generation strategies can also include content exclusion conditions to restrict candidate video content that does not meet predetermined content requirements. Each generation strategy can be configured or adjusted independently, and new candidate video types can be supported by adding generation strategies corresponding to new types without changing the generation strategies corresponding to existing types.

[0069] In some cases, a candidate information 232 may include at least one of a first description, a second description, a third description, and a fourth description.

[0070] The first descriptive information can be used to describe the interaction method of the candidate video, and this interaction method is associated with a first object. The interaction method can be used to characterize the identity or manner in which the first object participates in the candidate video. For example, the first object can be an object appearing in the first video 205, and the object in the first video 205 continues to participate in the interaction; the first object can also be a predetermined virtual object, and the predetermined virtual object participates in the object or event in the first video 205. As an example, the first object can also correspond to an object represented by a first-person viewing perspective.

[0071] The second descriptive information can be used to describe the visual style corresponding to the candidate video. The visual style can characterize the visual expression of the candidate video. For example, the first visual style can cartoonize the objects in the first video 205 or the interactive content in the candidate video; the second visual style can continue at least one of the following from the first video 205: object proportions, image texture, scene features, and narrative atmosphere. In other examples, different visual styles can also be determined based on a predetermined reference image or other visual conditions.

[0072] The third descriptive information can be used to describe the interaction between the first object and the second object, where the second object appears in the first video 205. For example, the third descriptive information can indicate the object initiating the interaction, the object targeted by the interaction, and the action performed. The fourth descriptive information can be used to describe the feedback result obtained through the corresponding interaction, such as the action response of the second object, state change, change in object relationship, or change in result in the candidate video.

[0073] In some examples, the third and fourth descriptive information can jointly describe "which object performs which action on which object, and what result the corresponding action produces." When generating the second video 275 subsequently, this descriptive information can also be converted into the generation conditions for the object, action, and feedback result in the third information 265.

[0074] In the manner described above, the video content context and its type 227 corresponding to the first video segment 226 can be used to form candidate information 232. Candidate information 232 can further serve as a processing object for comparison and selection among multiple candidate information items, and can provide corresponding video content descriptions for the subsequent acquisition of second information 244 and the generation of the second video 275.

[0075] Figure 5 A schematic diagram of an example process 240 for obtaining second information in some scenarios is shown.

[0076] like Figure 5 As shown, in the second information acquisition process 240, the electronic device 110 can compare and / or filter multiple candidate information 232 based on the content consistency relationship 241 between the candidate information and the first information 211. The content consistency relationship 241 can be used to characterize the degree of relevance between the candidate video described by the candidate information and the first information 211 in terms of object, object relationship, event development, content direction, interactive action, or feedback result.

[0077] For example, when determining the content consistency relationship 241, the judgment can be based on at least one of the following: whether the video content described by the candidate information conflicts with the first information 211; whether the video content described by the candidate information matches the content direction represented by the type 227 of the first video segment 226; whether the candidate information includes interactive actions associated with objects or events in the first video 205; or whether the candidate information includes feedback results that are compatible with the first video 205.

[0078] The electronic device 110 can also determine the interactive effect described by the candidate information 232 and select a second piece of information 244 from multiple candidate information items 232 based on the interactive effect. The interactive effect can be represented in the form of a score, grade, ranking value, multidimensional vector, or other comparable data form. As an example, the content consistency relationship 241 and the interactive effect can also be combined into a quality level 242 to compare multiple candidate information items 232.

[0079] The interactive effect can be associated with at least one of the following: the feedback result described by the candidate information, the degree of influence of the candidate video on the video content of the first video 205, and the degree of matching between the candidate information and the first information 211. For example, the degree of influence can be used to characterize the degree of content enhancement, content transition, or other local content changes produced by the candidate video relative to the first video segment 226; the degree of matching can be used to characterize the degree of association between the objects, actions, and results in the candidate video and the corresponding content in the first video 205.

[0080] In some cases, the multiple candidate information 232 may include first candidate information corresponding to a first visual style and second candidate information corresponding to a second visual style. For example, the electronic device 110 may determine first candidate information with a higher quality level from one or more candidate information adopting the first visual style, and determine second candidate information with a higher quality level from one or more candidate information adopting the second visual style. The electronic device 110 may further compare candidate information corresponding to different visual styles 243 based on quality level 242, and select second information 244 from the first candidate information and the second candidate information.

[0081] When the differences between the quality levels 242 of multiple candidate information satisfy predetermined conditions, the second information 244 can also be determined by combining the default visual style, available processing resources, the picture characteristics of the first video 205, or other selection conditions. For example, when candidate information using different visual styles has similar quality levels, the second visual style that continues the picture quality and narrative atmosphere of the first video 205 can be used as the default visual style to select the corresponding candidate information as the second information 244.

[0082] In some cases, electronic device 110 can directly determine the content consistency relationship 241 and interactive effect based on the text or structured description in candidate information 232, and then generate a second video 275 based on the second information 244 after selecting the second information 244. Thus, the screening and comparison of candidate information can be completed before video generation.

[0083] In other examples, the electronic device 110 may also generate preview data to characterize the expected video effect based on multiple candidate information 232, and further determine the interactive effect of the corresponding candidate information based on the preview data. For example, the preview data may include one or more preview images, low-resolution video, keyframe sequences, or other data with lower generation overhead than the complete second video 275. This document does not limit the specific data format used to evaluate the candidate information 232.

[0084] Through the above processing, based on the video content context provided by the first video segment 226 and the first information 211, multiple candidate information 232 can be formed according to the generation strategy corresponding to the type 227 of the first video segment. Then, second information 244 is selected from the multiple candidate information 232 through content consistency relationships and interactive effects. The second information 244 can constrain the objects, actions, and feedback results in the second video 275, and can also be further used to determine the first playback position associated with the second video 275. Therefore, the generation and selection of candidate information can establish a continuous video content association between the acquisition of the first video segment and the subsequent determination of the playback position and video acquisition, making the finally acquired second video 275 more closely matched with the first video 205 in terms of video content, object relationships, and action development.

[0085] Furthermore, in the example of first screening and evaluating candidate information 232 and then generating a second video 275 for the selected second information 244, the situation of performing full video generation for candidate information that does not meet the content consistency condition or has low interaction effect can be reduced, thereby reducing the computing resources and processing time occupied by invalid video generation, and allowing more video generation resources to be used for the second information 244 that matches the first information 211.

[0086] Continue to refer to Figure 2 In process 250, electronic device 110 can determine a first event from the video content of first video segment 226 based on second information 244, and determine a first playback position based on the first event. The video content of second video 275 is associated with the first event.

[0087] The first event can be an event unit in the first video segment 226, for example, it can be described by one or more objects, actions or dialogues between objects, changes in object state, event results, and a corresponding time range. Since different second information 244 can be formed based on different events in the first video segment 226, the corresponding first playback position can also be different. The first playback position can be a time point in the first video 205, or it can be a time range in the first video 205.

[0088] In some cases, the first playback position may not be earlier than the end position of the first event, thus ensuring that the dialogue, action, object state change, identity change, or event outcome upon which the second information 244 is based has occurred before the first playback position. The first playback position may also not be located within the time frame corresponding to the dialogue in the first video 205, nor within the execution time frame of the predetermined action. For example, the first playback position may be located within a silent interval. A silent interval can be a time frame in the first video 205 where there is no identifiable human voice or continuous lip-reading, and where the predetermined dialogue or predetermined action is not truncated. A silent interval does not require complete silence; it may still include background music, ambient sounds, or non-human semantic sound effects.

[0089] In process 260, electronic device 110 can generate third information 265 based on second information 244, first video 205, and first information 211. Third information 265 can be used to describe the conditions for generating second video 275. In other cases, third information 265 may not be generated separately, but second video 275 may be obtained based on first video 205 and second information 244. Figure 6 A schematic diagram of an example process 260 for generating third information based on certain scenarios is shown. The following is combined with... Figure 6 Further description of the process 260.

[0090] The third information 265 may be natural language information, structured information, control parameters, source material references, or a combination thereof suitable for processing by the video generation model, and is used to convert the candidate video content described by the second information 244 into generation conditions used in the video generation process. The third information 265 may also be generated based on the first information 211, such that at least one of the objects, speech text, character descriptions, and bullet screen information included in the first information 211 can continue to serve as context when the second video 275 is generated.

[0091] In some cases, the electronic device 110 can acquire a first image 261 from the first video 205. The first image 261 may be an image of a third object, and the third object appears in the first video 205. The first image 261 may be acquired from a video frame in the first video 205 where the third object is relatively clear and its appearance meets predetermined conditions, and may retain the outline, hairstyle, clothing, accessories, facial features, or other information that can characterize the appearance of the third object.

[0092] Electronic device 110 can generate a second image 263 based on the first image 261 and the reference image 262. The reference image 262 can be an image used to indicate the visual style or appearance characteristics of a target object, such as a preset style reference image. Electronic device 110 can, for example, input the first image 261 and the reference image 262 into an image generation model, so that the generated second image 263 presents the corresponding visual style while retaining at least some of the identifiable appearance characteristics of the third object. The second image 263 can serve as reference material for the third object when subsequently generating the second video 275.

[0093] For example, reference image 262 can indicate a visual style for cartoonizing objects in the first video 205. In this case, a second image 263 can be generated based on the first image 261 and the cartoonized reference image 262. In other examples, the first image 261 or other images in the first video 205 that include a third object can be directly used as object reference material without performing the style conversion process described above.

[0094] The third information 265 may describe at least one of the following: the time range used to generate the second video 275 in the first video 205; the duration of the second video 275; one or more objects in the second video 275; the interactive actions between objects; the feedback results corresponding to the interactive actions; and the screen constraints of the second video 275. The third information 265 may also include content continuity constraints, which are used to match the object state, action state, scene, shot state, or audio state of the second video 275 with the first video 205 near the first playback position.

[0095] In some cases, the third information 265 can be formed by a combination of natural language description and structured fields. The natural language description can be used to describe the video content of the second video 275, the interactive actions between objects, and the corresponding feedback results. The structured fields can be used to indicate at least one of the following: reference time range, first playback position, duration of the second video 275, object material identifier, visual style, screen constraints, and content continuity constraints.

[0096] Before generating the second video 275, the electronic device 110 can also perform verification on the third information 265. For example, it can determine whether the third information 265 includes the object, interactive action, and feedback result required by the second information 244, whether the referenced object material is available, or whether the reference time range used to generate the second video 275 is compatible with the first playback position and the first event. If the third information 265 does not meet the predetermined conditions, the third information 265 can be supplemented or adjusted, or the corresponding video generation process can be discontinued.

[0097] Through the above processing, the second information 244 can be further converted into third information 265, which includes generation conditions such as video content, objects, actions, feedback results, visual style, and reference video range. These generation conditions continue to be associated with the previously acquired first video segment 226, first event, and first playback position. In cases involving object appearance transformation, appearance associations between objects in the first video 205 and corresponding objects in the second video 275 can also be established through the first image 261, reference image 262, and second image 263.

[0098] Furthermore, in some cases, by generating third-party information 265 before video generation and verifying generation conditions such as object materials, interactive actions, feedback results, reference time ranges, and content continuity constraints, incomplete or mismatched generation conditions can be identified before entering the video generation stage, thereby reducing invalid video generation due to missing materials, mismatched objects, or conflicting generation conditions.

[0099] In process 270, electronic device 110 can acquire second video 275 based on first video 205 and second information 244, and establish a first association relationship, such that the first association relationship indicates the association relationship between second video 275 and first playback position. The first association relationship is used to characterize the correspondence between second video 275 and first playback position. For example, the first association relationship can be represented by mapping entries, index information, playback scheduling information, synthesis configuration information, or other data that enables the second video 275 to establish a correspondence with the first playback position. Establishing the first association relationship does not necessarily require writing or replacing the original video data of first video 205 with second video 275.

[0100] Process 200 can be performed by first acquiring the first video segment 226, then generating and selecting the second information 244, and finally determining the first playback position based on the second information 244. Therefore, the first playback position can be determined based on the video content described by the second information 244 and its associated first event, without needing to be predetermined before acquiring the second information 244.

[0101] In some examples, the electronic device 110 can also obtain third information 265 based on the first video 205 and the second information 244, and obtain the second video 275 based on the third information 265. Optionally, the second information 244 can be further converted into third information 265 suitable for processing by the video generation model. The third information 265 is used to describe the specific generation conditions of the second video 275, and can be generated based on the second information 244 and at least one of the first video 205, the first information 211, the first playback position, and object reference material. For example, the third information 265 can indicate at least one of the following: reference video range, video duration, object material, interactive action, feedback result, visual style, screen constraints, or content continuity constraints. The third information 265 can include natural language prompts, structured fields, control parameters, material references, or combinations thereof.

[0102] Figure 7 A schematic diagram of an example process 270 for acquiring a second video under certain circumstances is shown. Process 270 may receive at least one of a first video 205, a first video segment 226, first information 211, second information 244, and third information 265, and provide the corresponding data to a video generation model 271 to acquire the second video 275.

[0103] In some cases, electronic device 110 may use the first video segment 226 as a reference to generate a second video 275 based on third information 265. Alternatively, electronic device 110 may determine a reference video segment from the first video 205 based on a first playback position and use that reference video segment as a reference to generate the second video 275. The video generation model 271 may employ a reference video generation or video extension method to ensure that the second video 275 continues at least one of the objects, scenes, action states, camera states, and image features from the corresponding reference video content.

[0104] The video generation model 271 can generate objects, interactive actions, feedback results, and corresponding video results in the second video 275 based on the third information 265. If the second information 244 indicates the use of the second image 263, the video generation model 271 can also receive the second image 263 as an object reference; if the second information 244 indicates the continuation of the visual style of the first video 205, the first image 261 or an object image from the first video segment 226 can be used as an object reference.

[0105] In some cases, process 270 may first acquire a second video candidate and examine it based on the second information 244 and the first video 205. For example, it may check whether the second video candidate matches the second information 244, whether the objects in it match the corresponding objects in the first video 205, whether the actions meet the continuity condition, or whether there are any predetermined abnormal scenes. Furthermore, it may also check whether the interactive actions described in the second information 244 have been executed, whether the corresponding feedback results have been presented, whether it includes content unrelated to the first video 205, and whether the state of the scene or object meets predetermined conditions.

[0106] In response to the second video candidate failing the check, the electronic device 110 can adjust the input of the third information 265, reference material, or video generation model 271, and reacquire the second video candidate. When the number of regeneration attempts reaches a predetermined limit and no second video candidate that meets the conditions is still obtained, subsequent processing of the second information 244 can be stopped.

[0107] After obtaining the second video 275 that meets the conditions, a first association can be established. The first association can be represented by a mapping entry, index information, playback scheduling information, or synthesis configuration information, and may include the identifier of the second video 275 and the first playback position. The first association allows the second video 275 to be stored separately from the first video 205 without having to write the second video 275 into the original video data of the first video 205.

[0108] In one interactive example, when playing a first video 205, electronic device 110 may display a first prompt associated with a second video 275 for a period of time related to a first playback position. This period of time may be a time interval starting from the determined first playback position. In response to receiving a trigger operation on the first prompt, electronic device 110 may store the current playback position of the first video 205 and play the second video 275. After the second video 275 finishes playing, playback of the first video 205 can resume from the stored playback position. In other examples, the first association may be used for automatic insertion, video compositing, or other output methods.

[0109] Through the above processing, the first video segment 226 can provide the video state related to the first video 205 for the generation of the second video 275. The third information 265 can further define the objects, actions, feedback results, and screen conditions in the second video 275. Furthermore, the generated video can undergo consistency and continuity checks before being associated with the first video 205. Therefore, the second video 275 can maintain good continuity with the first video 205 in terms of object representation, action development, and screen state, and the second video 275 can be invoked at the first playback position through the first association relationship.

[0110] In some cases, the second information 244 and / or the third information 265 may include a first constraint. The first constraint is used to ensure that the second video 275 provides partial supplementary video content associated with the first playback position, and that the second video 275 does not replace subsequent video content in the first video 205 located after the first playback position, nor does it change the playback order of subsequent video content. The second video 275 may, for example, present partial actions, object feedback, or screen changes not originally included in the first video 205. After the second video 275 ends, the subsequent video content of the first video 205 can still continue playing in its original order.

[0111] In some cases, the duration of the second video 275 can meet preset duration conditions. For example, the duration of the second video 275 can be shorter than the duration of the first video segment 226, or shorter than the duration of subsequent video content after the first playback position. The third information 265 may also include an end state constraint, ensuring that the ending screen, object state, action state, or audio state of the second video 275 matches the corresponding state of the first video 205 at the continue playback position. The second video 275 can provide locally perceptible changes in video content while maintaining the subsequent video content and its direction of the first video 205 unchanged.

[0112] Figure 8 A flowchart of an example process 800 for video processing according to some scenarios is shown. Process 800 can be implemented at electronic device 110 and / or server 130. See below for reference. Figure 8 To describe process 800.

[0113] like Figure 8 As shown, in box 810, electronic device 110 and / or server 130 can obtain first information, which is associated with the first video.

[0114] In box 820, electronic device 110 and / or server 130 can obtain a first video segment from the first video based on the first information.

[0115] In box 830, electronic device 110 and / or server 130 can obtain second information based on the first video segment and the first information, the second information being used to generate a second video.

[0116] In box 840, electronic device 110 and / or server 130 can obtain second video based on first video and second information.

[0117] In box 850, electronic device 110 and / or server 130 can establish a first association relationship, which indicates the association relationship between the second video and the first playback position, which is the playback position in the first video.

[0118] In some cases, the first information may include at least one of the following: first text, obtained by converting speech in the first video to text; second text, including character description information from the first video; and bullet screen information, obtained based on bullet screen comments in the first video. Specific processing examples can be found above. Figure 2 and Figure 3A The example described.

[0119] In some cases, the first information may include bullet screen information, which includes multiple bullet screens. Server 130 and / or electronic device 110 may, based on the distribution of multiple bullet screens in the first video, obtain multiple candidate video segments from the first video; and select the first video segment from the multiple candidate video segments. Specific processing examples can be found above. Figure 3A and Figure 3B The example described.

[0120] In some cases, the first information may also include at least one of the first text or the second text. Server 130 and / or electronic device 110 may, based on at least one of the first text or the second text, obtain analysis results for each of multiple candidate video segments, the analysis results indicating the matching relationship between the video content of the corresponding candidate video segment and at least one content category; and, based on the analysis results, select the first video segment from the multiple candidate video segments. Specific processing examples can be found above. Figure 3A The example described.

[0121] In some scenarios, server 130 and / or electronic device 110 can determine the distribution of multiple bullet comments in a first video, whereby the distribution is the distribution of the multiple bullet comments within multiple time windows, each time window corresponding to multiple video segments in the first video; based on the distribution, select at least one time window from the multiple time windows; and based on the at least one time window, obtain at least one candidate video segment from the first video, where the at least one candidate video segment includes the video segment corresponding to the at least one time window. Specific processing examples can be found above. Figure 3A and Figure 3B The example described.

[0122] In some cases, the distribution can indicate at least one of the following: the number of bullet comments within each of multiple time windows; and the trend of the number of bullet comments across multiple time windows. Specific processing examples can be found above. Figure 3B The example described.

[0123] In some scenarios, server 130 and / or electronic device 110 can generate multiple candidate information items based on the first video segment and the first information, whereby each candidate information item is used to describe a corresponding candidate video; and select second information items from the multiple candidate information items. Specific processing examples can be found above. Figure 4 and Figure 5 The example described.

[0124] In some cases, one of the multiple candidate pieces of information may include at least one of the following: first descriptive information, which describes the interaction method of the candidate video and is associated with a first object; second descriptive information, which describes the visual style corresponding to the candidate video; third descriptive information, which describes the interactive action, which is an action between the first object and a second object that appears in the first video; or fourth descriptive information, which describes the feedback result, which is obtained through the interactive action. Specific processing examples can be found above. Figure 4 The example described.

[0125] In some scenarios, server 130 and / or electronic device 110 can determine a first generation strategy from multiple generation strategies based on the type of the first video segment; and generate multiple candidate information based on the first video segment, the first information, and the first generation strategy. Specific processing examples can be found above. Figure 4 The example described.

[0126] In some scenarios, server 130 and / or electronic device 110 may select second information from multiple candidate information based on at least one of the following: the content consistency relationship between the candidate information and the first information; the interactive effect described by the candidate information. Specific processing examples can be found above. Figure 5 The example described.

[0127] In some scenarios, server 130 and / or electronic device 110 can determine a first event from the video content of a first video segment based on second information, wherein the video content of a second video is associated with the first event; and determine a first playback position based on the first event, wherein the first playback position is no earlier than the end position of the first event. Specific processing examples can be found above. Figure 2 The example described.

[0128] In some scenarios, server 130 and / or electronic device 110 can acquire a first image from a first video, where the first image is an image of a third object appearing in the first video; generate a second image based on the first image and a reference image; and generate a second video based on the second image and a segment of the first video. Specific processing examples can be found above. Figure 6 and Figure 7The example described.

[0129] In some cases, server 130 and / or electronic device 110 may perform semantic clustering on multiple bullet comments corresponding to candidate video segments to determine at least one content category and at least one time range corresponding to the content category, and use the content category, time range, and video content of the corresponding candidate video segment as at least part of the analysis result. Server 130 and / or electronic device 110 may further select a first video segment from multiple candidate video segments based on at least one of the analysis result, a first text, or a second text.

[0130] In some cases, the first playback position may also satisfy at least one of the following conditions: the first playback position is not located within the time range corresponding to the dialogue in the first video; the first playback position is not located within the time range corresponding to the predetermined action in the first video. Specific processing examples can be found above. Figure 2 The example described.

[0131] In some scenarios, server 130 and / or electronic device 110 can generate third information based on the second information, the first video, and the third information, whereby the third information describes the conditions for generating the second video; and obtain the second video based on the first video, the second information, and the third information. Specific processing examples can be found above. Figure 6 and Figure 7 The example described.

[0132] In some cases, the generation conditions may indicate at least one of the following: the time range in the first video used to generate the second video; the duration of the second video; multiple objects in the second video; interactive actions between the multiple objects; feedback results corresponding to the interactive actions; visual constraints on the second video; or content continuity constraints that prevent the second video from altering the content of subsequent videos in the first video. Specific processing examples can be found above. Figure 6 The example described.

[0133] In some cases, server 130 and / or electronic device 110 can use the first video segment as a reference to generate a second video based on third information, such that the second video continues at least one of the objects, scenes, action states, camera states, and screen features from the first video segment. Specific processing examples can be found above. Figure 7 The example described.

[0134] In some scenarios, server 130 and / or electronic device 110 may examine a second video based on the second information and the first video. Examining the second video includes at least one of the following: determining whether the second video matches the second information; determining whether an object in the second video matches an object in the first video; determining whether actions in the second video meet continuity conditions; or determining whether there are abnormal scenes in the second video; and re-acquiring the second video in response to it failing the check. Specific processing examples can be found above. Figure 7 The example described.

[0135] In some cases, the interactive effect can be associated with at least one of the following: the feedback result corresponding to the candidate information; the degree of influence of the candidate video on the video content of the first video; and the degree of matching between the candidate information and the first information. Specific processing examples can be found above. Figure 5 The example described.

[0136] In some cases, content consistency can be determined based on at least one of the following: whether the video content described by the candidate information conflicts with the first information; whether the video content described by the candidate information matches the content direction represented by the type of the first video segment; whether the candidate information includes interactive actions associated with objects or events in the first video; or whether the candidate information includes feedback results that are appropriate for the first video. Specific processing examples can be found above. Figure 5 The example described.

[0137] In some cases, multiple candidate information may include first candidate information and second candidate information. The first candidate information corresponds to a first visual style, and the second candidate information corresponds to a second visual style. The second information is selected from the first and second candidate information based on a quality level. Specific processing examples can be found above. Figure 5 The example described.

[0138] In some cases, the first object may include at least one of the following: an object appearing in the first video; a pre-defined virtual object. Specific processing examples can be found above. Figure 4 The example described.

[0139] Through the above scheme, electronic device 110 and / or server 130 can combine the bullet screen distribution, first text, or second text in the first information to obtain the first video segment, and obtain the second information and the second video based on the first video segment and the first information, thereby solving the difficulty of coordinating the determination of video segments, generated information, second video, and associated positions. The candidate information processing stage can constrain visual style, objects, actions, feedback results, content consistency relationships, and interactive effects, while the video generation stage can utilize third information, reference images, and the first video segment to maintain the continuity of objects and scenes. Therefore, the number of invalid inferences and the amount of intermediate data stored in the model are reduced, and the computational resources and processing time occupied by video processing are correspondingly reduced.

[0140] A corresponding apparatus for implementing the above methods or processes is also provided.

[0141] Figure 9 Block diagrams of an apparatus 900 for video processing in several scenarios are shown. The apparatus 900 may be implemented as or included in electronic device 110 and / or server 130. The various modules / components in the apparatus 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0142] like Figure 9 As shown, the device 900 includes an information acquisition module 910 configured to acquire first information associated with a first video; a segment acquisition module 920 configured to acquire a first video segment from the first video based on the first information; the information acquisition module 910 is further configured to acquire second information based on the first video segment and the first information, the second information being used to generate a second video; a video acquisition module 930 configured to acquire a second video based on the first video and the second information; and an association relationship establishment module 940 configured to establish a first association relationship, the first association relationship indicating the association relationship between the second video and a first playback position, the first playback position being the playback position in the first video.

[0143] In some cases, the first information may include at least one of the following: first text, obtained by converting speech in the first video to text; second text, including character description information in the first video; and bullet screen information, obtained based on bullet screens in the first video.

[0144] In some cases, the segment acquisition module 920 can also be configured to acquire multiple candidate video segments from the first video based on the distribution of multiple bullet comments in the first video; and select the first video segment from the multiple candidate video segments.

[0145] In some cases, the first information may also include at least one of the first text or the second text. The segment acquisition module 920 may also be configured to acquire analysis results of multiple candidate video segments based on at least one of the first text or the second text, wherein the analysis results indicate the matching relationship between the video content of the corresponding candidate video segment and at least one content category; and to select a first video segment from the multiple candidate video segments based on the analysis results.

[0146] In some cases, the segment acquisition module 920 may also be configured to determine the distribution status based on the distribution of multiple bullet comments in the first video, wherein the distribution status is the distribution status of multiple bullet comments in multiple time windows, and the multiple time windows correspond to multiple video segments in the first video; select at least one time window from the multiple time windows based on the distribution status; and acquire at least one candidate video segment from the first video based on the at least one time window, wherein the at least one candidate video segment includes the video segment corresponding to at least one time window.

[0147] In some cases, the distribution can indicate at least one of the following: the number of bullet comments in each of multiple time windows; the trend of the number of bullet comments across multiple time windows.

[0148] In some cases, the information acquisition module 910 can also be configured to generate multiple candidate information based on the first video segment and the first information, wherein the multiple candidate information are used to describe the corresponding candidate video; and to select second information from the multiple candidate information.

[0149] In some cases, one of the candidate pieces of information may include at least one of the following: a first descriptive information, which describes the interaction method of the candidate video and is associated with a first object; a second descriptive information, which describes the visual style of the candidate video; a third descriptive information, which describes the interactive action, which is an action between the first object and a second object, the second object appearing in the first video; or a fourth descriptive information, which describes the feedback result, which is obtained through the interactive action.

[0150] In some cases, the information acquisition module 910 can also be configured to determine a first generation strategy from multiple generation strategies based on the type of the first video segment; and to generate multiple candidate information based on the first video segment, the first information, and the first generation strategy.

[0151] In some cases, the information acquisition module 910 may also be configured to select a second piece of information from multiple candidate information based on at least one of the following: the content consistency relationship between the candidate information and the first information; the interactive effect described by the candidate information.

[0152] In some cases, the association establishment module 940 can also be configured to determine a first event from the video content of the first video segment based on the second information, wherein the video content of the second video is associated with the first event; and to determine a first playback position based on the first event, wherein the first playback position is no earlier than the end position of the first event.

[0153] In some cases, the video acquisition module 930 may also be configured to acquire a first image from a first video, wherein the first image is an image of a third object that appears in the first video; generate a second image based on the first image and a reference image; and generate a second video based on the second image and a first video segment.

[0154] In some cases, the segment acquisition module 920 may also be configured to perform semantic clustering on multiple bullet comments corresponding to candidate video segments, determine at least one content category and at least one time range corresponding to the corresponding content category, and take the content category, time range and video content of the corresponding candidate video segment as at least part of the analysis result; and select a first video segment from multiple candidate video segments based on at least one of the analysis result, a first text or a second text.

[0155] In some cases, the first playback position may also satisfy at least one of the following conditions: the first playback position is not located within the time range corresponding to the dialogue in the first video; the first playback position is not located within the time range corresponding to the predetermined action in the first video.

[0156] In some cases, the information acquisition module 910 can also be configured to generate third information based on the second information, the first video, and the third information, wherein the third information describes the conditions for generating the second video. The video acquisition module 930 can also be configured to acquire the second video based on the first video, the second information, and the third information.

[0157] In some cases, the generation conditions may indicate at least one of the following: the time range in the first video used to generate the second video; the duration of the second video; multiple objects in the second video; interactive actions between the multiple objects; feedback results corresponding to the interactive actions; visual constraints on the second video; or content continuity constraints that prevent the second video from changing the content of subsequent videos in the first video.

[0158] In some cases, the video acquisition module 930 can also be configured to use the first video segment as a reference to generate a second video based on third information, so that the second video continues at least one of the objects, scenes, action states, camera states and screen features in the first video segment.

[0159] In some cases, the video acquisition module 930 may also be configured to check the second video based on the second information and the first video. Checking the second video includes at least one of the following: determining whether the second video matches the second information; determining whether an object in the second video matches an object in the first video; determining whether the action in the second video meets the continuity condition; or determining whether there are abnormal scenes in the second video; and re-acquiring the second video in response to the second video failing the check.

[0160] In some cases, the interactive effect can be associated with at least one of the following: the feedback result corresponding to the candidate information; the degree of influence of the candidate video on the video content of the first video; and the degree of matching between the candidate information and the first information.

[0161] In some cases, content consistency can be determined based on at least one of the following: whether the video content described by the candidate information conflicts with the first information; whether the video content described by the candidate information matches the content direction represented by the type of the first video segment; whether the candidate information includes interactive actions associated with objects or events in the first video; or whether the candidate information includes feedback results that are compatible with the first video.

[0162] In some cases, multiple candidate information may include first candidate information and second candidate information, where the first candidate information corresponds to a first visual style and the second candidate information corresponds to a second visual style. The second information is selected from the first and second candidate information based on the quality level.

[0163] In some cases, the first object may include at least one of the following: an object appearing in the first video; a pre-defined virtual object.

[0164] The modules included in device 900 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 900 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0165] Figure 10 A block diagram of an electronic device 1000 in which one or more examples can be implemented is shown. It should be understood that... Figure 10 The electronic device 1000 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 10 The illustrated electronic device 1000 can be used to implement the electronic device 110 and / or server 130 discussed above.

[0166] like Figure 10 As shown, electronic device 1000 is in the form of a general-purpose electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processing units or processors 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processor 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 1000.

[0167] Electronic device 1000 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 1030 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 1000.

[0168] Electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 10 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1020 may include computer program product 1025 having one or more program modules configured to perform various methods or actions of various examples.

[0169] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 1000 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0170] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 1000, or with any device that enables electronic device 1000 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0171] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0172] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0173] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0174] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0175] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0176] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations described herein.

Claims

1. A video processing method, comprising: Obtain first information, which is associated with the first video; Based on the first information, obtain the first video segment from the first video; Based on the first video segment and the first information, second information is obtained, and the second information is used to generate a second video. Based on the first video and the second information, obtain the second video; as well as Establish a first association relationship, which indicates the association relationship between the second video and the first playback position, where the first playback position is the playback position in the first video.

2. The method according to claim 1, wherein the first information includes at least one of the following: The first text is obtained by converting the speech in the first video into text. The second text includes character description information from the first video; The bullet screen information is obtained based on the bullet screens in the first video.

3. The method according to claim 2, wherein the first information includes the bullet screen information, the bullet screen information includes multiple bullet screens, and obtaining the first video segment in the first video includes: Based on the distribution of the multiple bullet comments in the first video, multiple candidate video segments are obtained from the first video; as well as The first video segment is selected from the plurality of candidate video segments.

4. The method of claim 3, wherein the first information further includes at least one of the first text or the second text, and the step of selecting the first video segment from the plurality of candidate video segments includes: Based on at least one of the first text or the second text, obtain the analysis results of each of the plurality of candidate video segments, wherein the analysis results indicate the matching relationship between the video content of the corresponding candidate video segment and at least one content category; as well as Based on the analysis results, the first video segment is selected from the plurality of candidate video segments.

5. The method according to claim 3, wherein obtaining a plurality of candidate video segments from the first video based on the distribution of the plurality of bullet comments in the first video includes: Based on the distribution of the multiple bullet comments in the first video, the distribution status is determined. The distribution status is the distribution status of the multiple bullet comments in multiple time windows, and the multiple time windows correspond to multiple video segments in the first video. Based on the distribution, at least one time window is selected from the plurality of time windows; as well as Based on the at least one time window, at least one candidate video segment is obtained from the first video, wherein the at least one candidate video segment includes the video segment corresponding to the at least one time window.

6. The method of claim 5, wherein the distribution indicates at least one of the following: The number of bullet comments within each of the multiple time windows. The trend of the number of bullet comments in the multiple time windows.

7. The method of claim 1, wherein obtaining the second information comprises: Based on the first video segment and the first information, multiple candidate information is generated, and the multiple candidate information is used to describe the corresponding candidate video. as well as The second information is selected from the multiple candidate information.

8. The method of claim 7, wherein one of the plurality of candidate information includes at least one of the following: First descriptive information, used to describe the interaction method of the candidate video, the interaction method being associated with a first object. The second description information is used to describe the visual style corresponding to the candidate video; The third descriptive information describes the interactive action, which is an action between the first object and the second object, where the second object appears in the first video. The fourth descriptive information is used to describe the feedback result, which is obtained through the interactive action.

9. The method according to claim 7, wherein, Generating the multiple candidate information includes: Based on the type of the first video segment, a first generation strategy is determined from multiple generation strategies; and Based on the first video segment, the first information, and the first generation strategy, generate the multiple candidate information; and / or Selecting the second information from the plurality of candidate information includes: The second information is selected from the plurality of candidate information based on at least one of the following: The content consistency relationship between the candidate information and the first information. The interactive effects described by the candidate information.

10. The method of claim 1, wherein the first playback position is obtained by: Based on the second information, a first event is determined from the video content of the first video segment, and the video content of the second video is associated with the first event; and The first playback position is determined based on the first event, and the first playback position is no earlier than the end position of the first event.

11. The method of claim 1, wherein acquiring the second video comprises: A first image is obtained from the first video, wherein the first image is an image of a third object that appears in the first video; A second image is generated based on the first image and the reference image; as well as The second video is generated based on the second image and the first video clip.

12. An apparatus for video processing, comprising: The information acquisition module is configured to acquire first information, which is associated with the first video; The segment acquisition module is configured to acquire a first video segment from the first video based on the first information; The information acquisition module is further configured to acquire second information based on the first video segment and the first information, wherein the second information is used to generate a second video; The video acquisition module is configured to acquire the second video based on the first video and the second information; as well as The association establishment module is configured to establish a first association, which indicates the association between the second video and a first playback position, where the first playback position is the playback position in the first video.

13. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions executable by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images