Video processing method and device, electronic equipment and storage medium

Through video processing methods, the high cost and low efficiency of high-light short video clips are solved, and the efficient generation of short video clips in e-commerce live broadcasts is achieved, and the richness of short videos is improved.

CN119031213AActive Publication Date: 2024-11-26BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411273751.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-26
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

The production cost of high-light short video clips in e-commerce live broadcasts is high and the determination efficiency is low, resulting in low material usage rate and ineffective in improving the richness of short videos on the platform.

Method used

By obtaining the target video and dividing its audio, extracting audio features and image information, combining text and interaction information, we can determine whether the video clip contains object details, interaction prompts and interaction information, thereby determining the highlight video clip and multiple highlights Video clips are aggregated and sorted to generate high-quality short video materials.

Benefits of technology

It reduces the production cost of highlight short videos, improves the efficiency of determining highlight video clips, increases the quantity and quality of short videos, and improves the richness of videos on the platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031213A_ABST
    Figure CN119031213A_ABST
Patent Text Reader

Abstract

The invention relates to a video processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target video corresponding to a target virtual space, segmenting the audio in the target video, obtaining a plurality of target sub-audios, and determining a target video segment corresponding to each target sub-audio; performing object feature extraction processing on the target text corresponding to each target sub-audio to obtain target object features; extracting a target object image of the target object and target interaction information corresponding to the target object from the target video clip; determining target object attribute information of the target object; and determining the highlight video clip according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video clip. According to the invention, the highlight video clip in the target video can be quickly and accurately determined.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] In related technologies, the content density of e-commerce live broadcasts varies, and the audience's attention is focused on short video clips containing highlights such as product details, product benefits, and shopping interactions.

[0003] The cost of producing highlight short video materials is high: for short video promotion, customers and merchants need to shoot promotional materials for each product separately, and the cost of material production is high; for short video live broadcast rooms, since the material is effective only during the live broadcast, in order to achieve good delivery effects, independent material production is required for each live broadcast, which places high demands on material production efficiency and results in low material utilization. The promotion and distribution of short videos on the operation side requires a large amount of high-quality materials to increase the richness of short videos on the platform. Summary of the invention

[0004] The present disclosure provides a video processing method, device, electronic device and storage medium to at least solve the problems of high production cost of highlight short video clips and low efficiency of determining highlight video clips in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, including:

[0006] Acquire a target video corresponding to the target virtual space, segment the audio in the target video to obtain a plurality of target sub-audios, and determine a target video segment corresponding to each target sub-audio;

[0007] Performing object feature extraction processing on the target text corresponding to each target sub-audio to obtain target object features; the target object features are features of the target object in the target virtual space;

[0008] Extracting a target object image of the target object and target interaction information corresponding to the target object from the target video clip;

[0009] Determining target object attribute information of the target object based on the target text and the target object image corresponding to the target video clip;

[0010] Determine a target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and determine whether the target video segment includes object detail information, object interaction prompt information and object interaction information of the target object according to the target fusion feature;

[0011] When the target video segment includes at least one of the object detail information, the object interaction prompt information, and the object interaction information, it is determined that the target video segment is a highlight video segment.

[0012] In an exemplary embodiment, the method further comprises:

[0013] Acquire a target virtual resource list corresponding to the target video clip, and extract target object description information of the target object from the target virtual resource list;

[0014] The determining the target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment includes:

[0015] Target object attribute information of the target object is determined based on the target text, the target object image and the target object description information corresponding to the target video clip.

[0016] In an exemplary embodiment, determining a target fusion feature according to the target object feature, the target interaction information, and the target object attribute information corresponding to the target video segment, and judging whether the target video segment includes object detail information, object interaction prompt information, and object interaction information of the target object according to the target fusion feature, includes:

[0017] Performing fusion processing on the target object features, the target interaction information, and the target object attribute information corresponding to the target video segment to obtain the target fusion features of the target video segment;

[0018] Analyzing object detail information according to the target fusion feature to obtain a first judgment result of whether the target video segment includes object detail information of the target object;

[0019] performing object interaction prompt information analysis according to the target fusion feature to obtain a second judgment result of whether the target video segment includes the object interaction prompt information of the target object;

[0020] Object interaction information is analyzed according to the target fusion feature to obtain a third judgment result of whether the target video segment includes the object interaction information of the target object.

[0021] In an exemplary implementation, after determining that the target video segment is a highlight video segment, the method further includes:

[0022] When the target video includes multiple target objects and multiple highlight video segments, obtaining a set of highlight video segments corresponding to the same target object;

[0023] For the highlight video segment set corresponding to each target object, each highlight video segment in the highlight video segment set is aggregated to generate an aggregated video.

[0024] In an exemplary implementation, aggregating the highlight video segments in the highlight video segment set to generate an aggregated video includes:

[0025] Obtaining the duration of each highlight video segment in the highlight video segment set, and sorting the multiple highlight video segments according to the playback time of each highlight video segment in the target video, to obtain a segment sorting result;

[0026] According to the segment sorting result, each of the highlight video segments in the highlight video segment set is divided into a plurality of highlight video segment groups according to a preset aggregation duration; wherein the duration of each of the highlight video segments is less than the preset aggregation duration; and the sum of the durations of the video segments in each of the video segment groups is less than or equal to the preset aggregation duration;

[0027] The highlight video segments in each of the highlight video segment groups are aggregated to obtain an aggregated video corresponding to each of the highlight video segment groups.

[0028] In an exemplary embodiment, the method further comprises:

[0029] Obtaining the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space; the number of users is the sum of the number of viewers of each of the highlight video clips in the aggregated video when they are played in the target virtual space;

[0030] Generating a ranking parameter corresponding to each aggregated video according to the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space;

[0031] Determining a screening video from the multiple aggregated videos according to a ranking parameter corresponding to each aggregated video; the ranking parameter of the screening video is greater than the ranking parameter of the non-screening video;

[0032] Based on the filtered video, an updated video is generated.

[0033] In an exemplary embodiment, generating a ranking parameter corresponding to each aggregated video according to the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space includes:

[0034] Determine a first weight of the number of users corresponding to each aggregated video in the target virtual space;

[0035] Determine a second weight of the amount of interactive information corresponding to each aggregated video in the target virtual space;

[0036] For each aggregated video, calculating the product of the number of users and the first weight to obtain a first weighted number; and calculating the product of the number of interactive information and the second weight to obtain a second weighted number;

[0037] The sum of the first weighted number corresponding to each aggregated video and the first weighted number is calculated to obtain a sorting parameter corresponding to each aggregated video.

[0038] According to a second aspect of an embodiment of the present disclosure, there is provided a video processing device, including:

[0039] A target video segment determination module is configured to execute acquisition of a target video corresponding to a target virtual space, segment the audio in the target video to obtain a plurality of target sub-audios, and determine a target video segment corresponding to each target sub-audio;

[0040] A target object feature determination module is configured to perform object feature extraction processing on a target text corresponding to each target sub-audio to obtain a target object feature; the target object feature is a feature of a target object in the target virtual space;

[0041] A target interaction information extraction module is configured to extract a target object image of the target object and target interaction information corresponding to the target object from the target video clip;

[0042] a target object attribute determination module, configured to determine target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment;

[0043] a judgment module configured to determine a target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and to judge whether the target video segment includes the object detail information, the object interaction prompt information and the object interaction information of the target object according to the target fusion feature;

[0044] The highlight segment determination module is configured to determine that the target video segment is a highlight video segment when the target video segment includes at least one of the object detail information, the object interaction prompt information, and the object interaction information.

[0045] In an exemplary embodiment, the device further comprises:

[0046] A description information acquisition module is configured to acquire a target virtual resource list corresponding to the target video clip, and extract target object description information of the target object from the target virtual resource list;

[0047] The target object attribute determination module is configured to determine the target object attribute information of the target object based on the target text, the target object image and the target object description information corresponding to the target video segment.

[0048] In an exemplary embodiment, the determination module includes:

[0049] a fusion unit configured to perform fusion processing on the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment to obtain the target fusion feature of the target video segment;

[0050] A first judgment unit is configured to perform object detail information analysis according to the target fusion feature to obtain a first judgment result of whether the target video segment includes the object detail information of the target object;

[0051] A second judgment unit is configured to perform object interaction prompt information analysis according to the target fusion feature to obtain a second judgment result of whether the target video segment includes the object interaction prompt information of the target object;

[0052] The third judgment unit is configured to perform object interaction information analysis according to the target fusion feature to obtain a third judgment result of whether the target video segment includes the object interaction information of the target object.

[0053] In an exemplary embodiment, the device further comprises:

[0054] A video set acquisition module is configured to execute, when the target video includes multiple target objects and multiple highlight video segments, acquiring a set of highlight video segments corresponding to the same target object;

[0055] The aggregation module is configured to execute the highlight video segment set corresponding to each target object, aggregate the highlight video segments in the highlight video segment set, and generate an aggregated video.

[0056] In an exemplary embodiment, the aggregation module includes:

[0057] a sorting unit configured to obtain the duration of each highlight video segment in the highlight video segment set, and sort the multiple highlight video segments according to the playback time of each highlight video segment in the target video to obtain a segment sorting result;

[0058] A grouping unit is configured to execute, according to the segment sorting result, sequentially dividing each of the highlight video segments in the highlight video segment set into a plurality of highlight video segment groups according to a preset aggregation duration; wherein the duration of each of the highlight video segments is less than the preset aggregation duration; and the sum of the durations of the video segments in each of the video segment groups is less than or equal to the preset aggregation duration;

[0059] The aggregation unit is configured to aggregate the highlight video segments in each of the highlight video segment groups to obtain an aggregated video corresponding to each of the highlight video segment groups.

[0060] In an exemplary embodiment, the device further comprises:

[0061] A quantity acquisition module is configured to execute acquisition of the number of users and the number of interactive information corresponding to each aggregated video in the target virtual space; the number of users is the sum of the number of viewers of each of the highlight video clips in the aggregated video when the highlight video clips are played in the target virtual space;

[0062] A sorting parameter generating module is configured to generate a sorting parameter corresponding to each aggregated video according to the number of users and the amount of interaction information corresponding to each aggregated video in the target virtual space;

[0063] A screening module is configured to determine a screening video from the plurality of aggregated videos according to a sorting parameter corresponding to each aggregated video; the sorting parameter of the screening video is greater than the sorting parameter of the non-screening video;

[0064] The updated video generation module is configured to generate an updated video based on the filtered video.

[0065] In an exemplary embodiment, the sorting parameter generation module includes:

[0066] A first weight determination unit is configured to determine a first weight of the number of users corresponding to each aggregated video in the target virtual space;

[0067] A second weight determination unit is configured to determine a second weight of the amount of interaction information corresponding to each aggregated video in the target virtual space;

[0068] A weighting unit is configured to calculate, for each aggregated video, a product of the number of users and the first weight to obtain a first weighted number; and calculate a product of the number of interactive information and the second weight to obtain a second weighted number;

[0069] The sorting parameter determination unit is configured to calculate the sum of the first weighted number corresponding to each aggregated video and the first weighted number to obtain the sorting parameter corresponding to each aggregated video.

[0070] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0071] processor;

[0072] a memory for storing instructions executable by the processor;

[0073] The processor is configured to execute the instructions to implement the video processing method as described above.

[0074] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video processing method as described above.

[0075] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the video processing method as described above is implemented.

[0076] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0077] The present invention obtains a target video corresponding to a target virtual space, and divides the audio in the target video to obtain a plurality of target sub-audios, and determines a target video segment corresponding to each target sub-audio; performs object feature extraction processing on a target text corresponding to each target sub-audio to obtain a target object feature; then extracts a target object image of the target object and target interaction information corresponding to the target object from the target video segment; according to the target text and the target object image corresponding to the target video segment, the target object attribute information of the target object can be quickly and accurately determined; thereby, according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, a target fusion feature can be determined, and according to the target fusion feature, it can be judged whether the target video segment includes object detail information, object interaction prompt information and object interaction information of the target object; when the target video segment includes at least one of the object detail information, the object interaction prompt information and the object interaction information, the target video segment is determined to be a highlight video segment; the present invention can quickly and accurately determine the target video segment of the highlight moment, thereby reducing the production cost of the highlight short video.

[0078] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0080] Figure 1 The diagram is a diagram of an application environment of a video processing method according to an exemplary embodiment.

[0081] Figure 2 The figure is a flowchart of a video processing method according to an exemplary embodiment.

[0082] Figure 3 The present invention is a flowchart of a method for performing object feature extraction processing on a target text corresponding to each target sub-audio to obtain target object features according to an exemplary embodiment.

[0083] Figure 4 The figure is a flowchart of a method for training an object feature extraction model according to an exemplary embodiment.

[0084] Figure 5 It is a schematic structural diagram of a first preset model according to an exemplary embodiment.

[0085] Figure 6The figure is a flowchart of a method for training an object attribute recognition model according to an exemplary embodiment.

[0086] Figure 7 It is a schematic diagram showing a recognition method of an object attribute recognition model according to an exemplary embodiment.

[0087] Figure 8 The figure is a flowchart of a method for training a video processing model according to an exemplary embodiment.

[0088] Fig. 9 The figure is a flowchart of a method for generating an updated video according to an exemplary embodiment.

[0089] Fig.10 It is a structural diagram of a short video generation system according to an exemplary embodiment.

[0090] Fig.11 The invention is a block diagram of a video processing device according to an exemplary embodiment.

[0091] Fig.12 It is a block diagram of a server according to an exemplary embodiment.

[0092] Fig.13 The invention is a block diagram of an electronic device for video processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0093] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0094] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0096] In order to quickly and accurately determine highlight video segments in a target video, the present disclosure provides a video processing method, device, electronic device and storage medium.

[0097] See also Figure 1 , which is a diagram of an application environment of a video processing method according to an exemplary embodiment. The application environment may include a server 01 and a client 02.

[0098] Specifically, in the embodiments of this specification, the server 01 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers, and may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, a memory, and the like. Specifically, the server 01 can be used to obtain a target video corresponding to a target virtual space, and segment the audio in the target video to obtain multiple target sub-audios, and determine a target video segment corresponding to each target sub-audio; perform object feature extraction processing on a target text corresponding to each target sub-audio to obtain a target object feature; the target object feature is a feature of a target object in the target virtual space; extract a target object image of the target object and target interaction information corresponding to the target object from the target video segment; determine target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment; determine a target fusion feature based on the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and judge whether the target video segment includes object detail information, object interaction prompt information and object interaction information of the target object based on the target fusion feature; when the target video segment includes at least one of the object detail information, the object interaction prompt information and the object interaction information, determine that the target video segment is a highlight video segment, and send the highlight video segment to the client 02.

[0099] Specifically, in the embodiments of this specification, the client 02 may include physical devices such as smart phones, desktop computers, tablet computers, laptop computers, digital assistants, smart wearable devices, and vehicle-mounted terminals, and may also include software running in physical devices, such as web pages provided by some service providers to users, or applications provided by these service providers to users. Specifically, the client 02 may be used to display highlight video clips.

[0100] Figure 2 is a flowchart of a video processing method according to an exemplary embodiment. Figure 2 As shown, this method can be applied to Figure 1 The server 01 shown includes the following steps.

[0101] In step S201, a target video corresponding to a target virtual space is obtained, and the audio in the target video is segmented to obtain a plurality of target sub-audios, and a target video segment corresponding to each target sub-audio is determined.

[0102] In the embodiments of the present disclosure, the target virtual space may include but is not limited to the live broadcast room in the terminal interface, and the target video may be a live video in the live broadcast process; in some embodiments, segmenting the target video to obtain multiple target sub-audios may include: extracting the target audio in the target video, and using voice activity detection technology (VAD) to segment the target audio to obtain multiple target sub-audios, and the duration of each target sub-audio may be a preset duration, which may be set according to actual conditions; exemplarily, the preset duration may be less than or equal to 10 seconds, and the preset duration may also be 8-10 seconds; after obtaining multiple target sub-audios, the target video may also be segmented according to the duration corresponding to each target sub-audio to obtain a target video segment corresponding to each target sub-audio, that is, multiple target video segments; each target video segment includes a target sub-audio and a target image.

[0103] In step S203, object feature extraction processing is performed on the target text corresponding to each target sub-audio to obtain target object features; the target object features are features of the target object in the target virtual space.

[0104] In the disclosed embodiment, for each target sub-audio, the target sub-audio can be input into a speech recognition model for speech recognition processing to obtain a target text; the speech recognition model may include but is not limited to a Conformer model; the Conformer model is a deep learning model that combines the characteristics of a convolutional neural network (CNN) and a Transformer, and is intended to capture both local and global features. The design of this model structure is intended to balance the advantages of the Transformer in capturing long-range dependencies with the ability of the CNN in extracting fine-grained local feature patterns. The Conformer model extracts local features and global representations through parallel convolution modules and Transformer modules, respectively, and implements feature fusion and alignment of convolution branches and Transformer branches through feature coupling units. Then, object features in the target text are extracted to obtain target object features; the target object in the target virtual space may be a commodity displayed or introduced in the target virtual space; the target object features may include object detail features, object interaction features, and object interaction prompt features of the target object. The object detail features of the target object may be relevant information in the store or shared bike where the target object is located; the object interaction prompt information of the target object may include but is not limited to discount information, coupon information, etc. of the target object; the interaction information of the target object may include but is not limited to the viewer-end user adding the target object to the virtual video list or placing an order for the target object in the target virtual space.

[0105] In step S205, a target object image of the target object and target interaction information corresponding to the target object are extracted from the target video clip.

[0106] In the disclosed embodiment, the target object image may be a video frame containing the target object in the target video segment. Exemplarily, the target object image may be determined by extracting frames from the target video segment at equal intervals. For example, the frames extracted every 20 seconds may be used as the target object image. The target interaction information of the target object may include but is not limited to the viewer-end user adding the target object to the virtual video list or placing an order for the target object in the target virtual space.

[0107] In step S207, target object attribute information of the target object is determined based on the target text and the target object image corresponding to the target video clip.

[0108] In the disclosed embodiment, for a target video clip, the target object attribute information of the target object can be determined according to the corresponding target text and target object image; illustratively, the first attribute information of the target object can be obtained by extracting the object attribute in the target text, the second attribute information of the target object can be obtained by extracting the object attribute in the target object image, and then the first attribute information and the second attribute information can be fused to obtain the target object attribute information. The target text and the target object image can also be subjected to attribute extraction processing by a model to obtain the target object attribute information of the target object.

[0109] In step S209, a target fusion feature is determined based on the target object features, the target interaction information and the target object attribute information corresponding to the target video clip, and based on the target fusion feature, it is determined whether the target video clip includes object detail information, object interaction prompt information and object interaction information of the target object.

[0110] In the disclosed embodiment, the target object features, the target interaction information and the target object attribute information corresponding to the target video clip can be fused to obtain a target fusion feature, and then the target video clip can be further judged based on the target fusion feature whether it includes the object detail information, object interaction prompt information and object interaction information of the target object.

[0111] In step S2011, when the target video segment includes at least one of the object detail information, the object interaction prompt information, and the object interaction information, it is determined that the target video segment is a highlight video segment.

[0112] In the disclosed embodiment, if the target video segment includes one or more of the object detail information, object interaction prompt information, and object interaction information of the target object, the target video segment can be determined to be a video segment of a highlight moment, otherwise, the target video segment can be determined to be a video segment of a non-highlight moment. Thus, the video segment of the highlight moment can be screened out from the multiple target video segments corresponding to the target video.

[0113] The present invention obtains a target video corresponding to a target virtual space, and divides the audio in the target video to obtain a plurality of target sub-audios, and determines a target video segment corresponding to each target sub-audio; performs object feature extraction processing on a target text corresponding to each target sub-audio to obtain a target object feature; then extracts a target object image of the target object and target interaction information corresponding to the target object from the target video segment; according to the target text and the target object image corresponding to the target video segment, the target object attribute information of the target object can be quickly and accurately determined; thereby, according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, a target fusion feature can be determined, and according to the target fusion feature, it can be judged whether the target video segment includes object detail information, object interaction prompt information and object interaction information of the target object; when the target video segment includes at least one of the object detail information, the object interaction prompt information and the object interaction information, the target video segment is determined to be a highlight video segment; the present invention can quickly and accurately determine the target video segment of the highlight moment, thereby reducing the production cost of the highlight short video.

[0114] In the present disclosure, Figure 3 As shown, Figure 3 The present invention is a flow chart of a method for performing object feature extraction processing on a target text corresponding to each target sub-audio to obtain target object features, the method comprising:

[0115] S2031: performing an object feature extraction operation on the target text corresponding to each target sub-audio, wherein the object feature extraction operation includes at least one of a first feature extraction operation, a second feature extraction operation, and a third feature extraction operation; wherein the first feature extraction operation is to extract object detail information in the target text to obtain a first object feature; the second feature extraction operation is to extract object interaction prompt information in the target text to obtain a second object feature; and the third feature extraction operation is to extract object interaction information in the target text to obtain a third object feature;

[0116] S2033: Determine at least one of the first object feature, the second object feature, and the third object feature as the target object feature.

[0117] In the embodiment of the present disclosure, the first object feature is the object detail information in the target text, the second object feature is the object interaction prompt information in the target text, and the third object feature is the object interaction information in the target text; the object interaction prompt information in the target text may include but is not limited to discount information, coupon information, etc. of the target object; if only the first feature extraction operation is performed, the first object feature can be determined as the target object feature, and if only the second feature extraction operation is performed, the second object feature can be determined as the target object feature; if only the third feature extraction operation is performed, the third object feature can be determined as the target object feature; if the first feature extraction operation and the second feature extraction operation are performed at the same time, the first object feature and the second object feature can be determined as the target object features; if the first feature extraction operation, the second feature extraction operation and the third feature extraction operation are performed, the first object feature, the second object feature and the third object feature can be determined as the target object features.

[0118] In the disclosed embodiment, at least one of the first object feature, the second object feature, and the third object feature can be determined as the target object feature based on the result of performing an object feature extraction operation on the target text corresponding to each target sub-audio, thereby improving the diversity of the target object features.

[0119] In the present disclosure, Figure 4 As shown, Figure 4 A training method for an object feature extraction model, the method comprising:

[0120] S401: Acquire a first training text corresponding to a first training video clip; the first training text is annotated with an object feature category label; the object feature category label includes category labels corresponding to the first object feature, the second object feature, and the third object feature, respectively;

[0121] S403: Inputting the first training text into an object feature extraction network of a first preset model to perform object feature extraction processing to obtain training object features;

[0122] S405: Inputting the training object features into the feature classification network of the first preset model for feature classification processing to obtain a training object feature category result corresponding to the first training video clip;

[0123] S407: Training the first preset model according to the difference between the training object feature category result and the object feature category label, and determining the object feature extraction network in the first preset model at the end of the training as the object feature extraction model.

[0124] In the disclosed embodiment, the method for determining the first training video segment is similar to the method for determining the target video segment, and a first training video corresponding to a first training virtual space can be obtained, and the first training virtual space and the target virtual space can be spaces of the same type; then, audio information in the first training video is extracted to obtain a first training audio, and then the first training audio is divided into a plurality of first training sub-audios, and text is extracted from each first training sub-audio to obtain a first training text; and object feature category labels are annotated on the first training text; the object feature category labels include category labels corresponding to the first object feature, the second object feature, and the third object feature, respectively.

[0125] Among them, the first preset model may include but is not limited to models such as roberta and Bayesian classifier. The first preset model includes an object feature extraction network and a feature classification network; the first training text is first input into the object feature extraction network of the first preset model for object feature extraction processing to obtain training object features, and then the training object features are input into the feature classification network for feature classification processing to obtain the training object feature category result corresponding to the first training video clip; thereby, the first training loss data can be determined according to the difference between the training object feature category result and the object feature category label, and the model parameters of the first preset model are adjusted according to the first training loss data until the training end condition is met, and the training end condition may include that the first training loss data is less than the first preset data and / or the first training iteration number reaches the first preset number; then the object feature extraction network in the first preset model at the end of training is determined as the object feature extraction model.

[0126] Exemplarily, the object feature extraction process is performed on the target text corresponding to each target sub-audio to obtain the target object feature, including:

[0127] The target text corresponding to each target sub-audio is input into the object feature extraction model to perform object feature extraction processing to obtain the target object feature.

[0128] In the disclosed embodiment, after the object feature extraction model is obtained through training, the target text corresponding to each target sub-audio may be input into the object feature extraction model for object feature extraction processing to obtain the target object features corresponding to each target sub-audio.

[0129] In the disclosed embodiment, an object feature category label corresponding to the first training text can be constructed, and then the object feature extraction network in the first preset model can be determined as an object feature extraction model through model training, so that the target object features in the target text can be quickly and accurately extracted according to the object feature extraction model.

[0130] In the present disclosure, Figure 5As shown, Figure 5 It is a structural diagram of a first preset model, including an input layer, a RoBERTa layer, a fully connected layer and a softmax; RoBERTa is a pre-trained language model based on BERT, which improves performance by improving optimization functions, training strategies and data preprocessing methods; the input layer is used to input the first training text and convert it into a first training text vector, and the RoBERTa layer is used to extract the first training object feature in the first training text vector; softmax is also called a normalized exponential function. It is a generalization of the binary classification function sigmoid to multi-classification, and its purpose is to present the results of multi-classification in the form of probability. It maps the outputs of multiple neurons to the interval (0, 1). The fully connected layer is used to predict the first object category feature of the first training object feature, and the softmax layer is used to process the first object category feature to obtain the training object feature category result.

[0131] In the embodiment of the present disclosure, the method further includes:

[0132] Acquire a target virtual resource list corresponding to the target video clip, and acquire target object description information of the target object from the target virtual video list;

[0133] The determining the target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment includes:

[0134] Target object attribute information of the target object is determined based on the target text, the target object image and the target object description information corresponding to the target video clip.

[0135] In the disclosed embodiment, when the target virtual space includes multiple target objects, generally one target video clip can correspond to one or more target objects; the target virtual video list can be a shopping cart in the live broadcast room, such as a small yellow car; the server can obtain the target virtual video list corresponding to the target virtual space, or obtain the target virtual video list corresponding to the target video clip, and quickly determine the target object in the target video clip from the target virtual video list, and obtain the target object description information of the target object; the target object description information may include but is not limited to the relevant information characterizing the style, material, and image of the target object. Then, according to the target text corresponding to the target video clip, the target object image, and the target object description information, the target object attribute information of the target object is further determined, and the target object attribute information may include but is not limited to the attribute information such as the image, category, style, and material of the target object.

[0136] In the disclosed embodiment, the target object description information can be queried through the target virtual video list, thereby narrowing the search scope of the target object description information and improving the efficiency of obtaining the target object description information; and when determining the target object attribute information of the target object, not only the target text and the target object image corresponding to the target video clip are considered, but also the target object description information is considered, thereby improving the accuracy of the target object attribute information.

[0137] In the embodiment of the present disclosure, determining the target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and judging whether the target video segment includes the object detail information, the object interaction prompt information and the object interaction information of the target object according to the target fusion feature, includes:

[0138] Performing fusion processing on the target object features, the target interaction information, and the target object attribute information corresponding to the target video segment to obtain the target fusion features of the target video segment;

[0139] Analyzing object detail information according to the target fusion feature to obtain a first judgment result of whether the target video segment includes object detail information of the target object;

[0140] performing object interaction prompt information analysis according to the target fusion feature to obtain a second judgment result of whether the target video segment includes the object interaction prompt information of the target object;

[0141] Object interaction information is analyzed according to the target fusion feature to obtain a third judgment result of whether the target video segment includes the object interaction information of the target object.

[0142] In the disclosed embodiment, the target object features, the target interaction information and the target object attribute information corresponding to the target video clip can be fused. During the fusion process, a model can be used for fusion, or a preset algorithm can be used for fusion to obtain the target fusion features of the target video clip. Thus, through the target fusion features, a first judgment result of whether the target video clip includes the object detail information of the target object, a second judgment result of whether the target video clip includes the object interaction prompt information of the target object, and a third judgment result of whether the target video clip includes the object interaction information of the target object can be quickly and accurately determined.

[0143] In the present disclosure, Figure 6 As shown, Figure 6 A training method for an object attribute recognition model, the method further comprising:

[0144] S601: Acquire a second training text and a training object image corresponding to a second training video segment; the second training video segment is annotated with an object attribute label;

[0145] S603: Inputting the second training text and the training object image into a second preset model for object attribute recognition processing to obtain a training object attribute recognition result corresponding to the second training video clip;

[0146] S605: According to the difference between the training object attribute recognition result and the object attribute label, the second preset model is trained to obtain an object attribute recognition model.

[0147] In the disclosed embodiment, the second training video segment and the first training video segment may be the same or different, and the method for determining the second training video segment is similar to the method for determining the target video segment. A second training video corresponding to the second training virtual space may be obtained, and the second training virtual space and the target virtual space may be spaces of the same type; then, audio information in the second training video is extracted to obtain a second training audio, and the second training audio is then divided into a plurality of second training sub-audios, and text is extracted from each second training sub-audio to obtain a second training text; the training object image may be a video frame containing the second training object in the second training video segment. Exemplarily, the training object image may be determined by extracting frames at equal intervals from the second training video segment; for example, frames extracted every 20 seconds may be used as training object images; The second training video clip is labeled with an object attribute label; the second training text and the training object image are input into the second preset model for object attribute recognition processing to obtain the training object attribute recognition result corresponding to the second training video clip; then, according to the difference between the training object attribute recognition result and the object attribute label, the second training loss data is determined, and the model parameters of the second preset model are adjusted according to the second training loss data until the training end condition is met, and the training end condition may include that the second training loss data is less than the second preset data and / or the second training iteration number reaches the second preset number; the second preset data and the first preset data may be the same or different, and the second preset number may be the same or different from the first preset number, and then the second preset model at the end of the training is determined as the object attribute recognition model. The object attribute recognition model may include but is not limited to the SKU (Stock Keeping Unit) recognition model, which mainly refers to the related models and practices of the smallest stock keeping unit (Stock Keeping Unit) used for tracking and management in the field of inventory management and e-commerce. SKU is the basic unit for measuring inventory in and out, which can be in units of pieces, boxes, pallets, etc. In the logistics management of large chain supermarkets, SKU is a necessary method for tracking the inventory changes of each product. In addition, SKU has also been extended to be the abbreviation of product unified number. Each product has a unique SKU number.

[0148] Exemplarily, determining the target object attribute information of the target object based on the target text and the target object image corresponding to the target video clip includes:

[0149] The target text and the target object image corresponding to the target video clip are input into the object attribute recognition model to perform object attribute recognition processing to obtain target object attribute information of the target object.

[0150] In the disclosed embodiment, the target text and the target object image corresponding to the target video clip may be input into a trained object attribute recognition model for object attribute recognition processing, thereby obtaining target object attribute information of the target object.

[0151] In the embodiment of the present disclosure, an object attribute label corresponding to the second training video clip can be constructed, and then the second preset model at the end of the training can be determined as the object attribute recognition model through model training, so that the target object attribute information of the target object can be quickly and accurately identified based on the object attribute recognition model.

[0152] In the embodiment of the present disclosure, the method includes:

[0153] Acquire training object description information of the training object from the training virtual video list corresponding to the second training video segment;

[0154] The step of inputting the second training text and the training object image into a second preset model for object attribute recognition processing to obtain a training object attribute recognition result corresponding to the second training video clip includes:

[0155] Inputting the second training text, the training object image and the training object description information into a second preset model for object attribute recognition processing to obtain a training object attribute recognition result corresponding to the second training video clip;

[0156] The determining the target object attribute information of the target object based on the target text, the target object image and the target object information corresponding to the target video segment includes:

[0157] The target text, the target object image and the target object information corresponding to the target video clip are input into the object attribute recognition model to perform object attribute recognition processing to obtain the target object attribute information of the target object.

[0158] In the disclosed embodiment, the training virtual video list may be a store or a shopping cart in a second training virtual space, from which the training object description information of the training object can be obtained; then the second training text, the training object image and the training object description information are input into a second preset model for object attribute recognition processing to obtain the training object attribute recognition result corresponding to the second training video clip; thereby combining the second training text, the training object image and the training object description information to train an object attribute recognition model; then the target text, the target object image and the target object information corresponding to the target video clip are input into the object attribute recognition model for object attribute recognition processing, thereby obtaining the target object attribute information of the target object.

[0159] In the disclosed embodiment, the second training text, the training object image and the training object description information are integrated into the object attribute recognition process, thereby improving the accuracy of the training model; and in the model recognition process, three types of information, namely the target text, the target object image and the target object information, are introduced, thereby further enriching the target object attribute information and improving the accuracy of the target object attribute information.

[0160] In the present disclosure, Figure 7 As shown, Figure 7 It is a schematic diagram of a recognition method of an object attribute recognition model, including: obtaining the target text corresponding to the target video clip, for example: nightgown, cotton gauze style, soft and comfortable; and extracting frames from the target video clip to obtain live frames as target object images; obtaining target object description information through the merchant's product library, such as shared bikes, stores, etc., and then inputting the target text, live frames and target object description information into the object attribute recognition model for object attribute recognition processing to obtain the target object attribute information of the target object.

[0161] In the present disclosure, Figure 8 As shown, Figure 8 A training method for a video processing model, comprising:

[0162] S801: Obtain sample object features and sample interaction information corresponding to sample video clips in the sample virtual space; the sample object features are features of sample objects in the sample virtual space; the sample video clips are marked with sample video category labels; the sample video category labels include a first sample category label and a second sample category label; the first sample category label indicates that the sample video clip is a highlight clip including at least one of the object detail information, object interaction prompt information, and object interaction information of the sample object; the second sample category label indicates that the sample video clip does not include the object detail information, object interaction prompt information, and non-highlight clip of the object interaction information of the sample object;

[0163] S803: Determine sample object attribute information of the sample object based on the sample text corresponding to the sample video clip and the sample object image;

[0164] S805: Inputting the sample object features, the sample object attribute information and the sample interaction information corresponding to the sample video clip into a preset model for video processing to obtain a sample video processing result;

[0165] According to the difference between the sample video processing result and the sample video category label, the preset model is trained to obtain a video processing model.

[0166] In the disclosed embodiment, the object interaction information of the sample object may include but is not limited to the viewer-end user adding the sample object to the sample virtual video list or placing an order for the sample object in the sample virtual space. The object interaction prompt information of the sample object may include but is not limited to the discount information, coupon information, etc. of the sample object; the interaction information of the sample object may include but is not limited to the viewer-end user adding the sample object to the virtual video list or placing an order for the sample object in the sample virtual space. If the sample video clip includes at least one of the object detail information, object interaction prompt information, and object interaction information of the sample object, then the sample video clip can be determined to be a video clip of a highlight moment; otherwise, the sample video clip is determined to be a video clip of a non-highlight moment. Video processing can be performed according to the sample object features, the sample object attribute information and the sample interaction information corresponding to the sample video clip to obtain a sample video processing result of the sample video clip; the sample video category labels annotated by the sample video clip include two categories: a first sample category label and a second sample category label; the first sample category label represents that the sample video clip includes at least one of the object detail information, object interaction prompt information and object interaction information of the sample object; that is, the first sample category label represents that the sample video clip is a video clip of a highlight moment, and the second sample category label represents that the sample video clip is not a video clip of a highlight moment; thereby, video clips of highlight moments and video clips of non-highlight moments can be distinguished from multiple sample video clips corresponding to the sample video.

[0167] The sample object attribute information of the sample object can be determined based on the sample text corresponding to the sample video clip and the sample object image; the sample object features, the sample object attribute information and the sample interaction information corresponding to the sample video clip are input into a preset model for video processing to obtain a sample video processing result; the preset model may include but is not limited to a multilayer perceptron (MLP); and then based on the difference between the sample video processing result and the sample video category label, the sample loss data is determined, so as to adjust the model parameters of the preset model according to the sample loss data until the training end condition is met. The training end condition may be that the sample loss data is less than a preset value and / or the number of training iterations reaches a preset number, and the preset model at the end of training is determined as the video processing model.

[0168] In the disclosed embodiment, the preset model can be trained by constructing positive sample video clips of highlight moments and negative sample video clips of non-highlight moments, thereby obtaining a video processing model that can accurately predict whether a video clip belongs to a highlight moment, thereby improving the recognition efficiency and recognition accuracy of highlight moment video clips.

[0169] In the embodiment of the present disclosure, the step of obtaining sample object features and sample interaction information corresponding to sample video clips in the sample virtual space includes:

[0170] Obtaining a sample video corresponding to the sample virtual space;

[0171] Extracting sample audio from the sample video, and segmenting the sample audio in sequence at preset time intervals to obtain a plurality of sample sub-audios; and determining a sample video segment corresponding to each sample sub-audio;

[0172] Perform text recognition processing on each sample sub-audio to obtain a sample text;

[0173] Performing object feature extraction processing on each sample text to obtain the sample object features;

[0174] A sample object image of the sample object and sample interaction information corresponding to the sample object are extracted from the sample video clip.

[0175] In the disclosed embodiment, the sample virtual space and the target virtual space can be the same type of space, and the sample video can be the same or different from the first training video and the second training video; the sample video corresponding to the sample virtual space can be obtained, and the historical live video corresponding to the historical live room can be obtained as the sample video. Then the sample audio in the sample video is extracted, and the sample audio is sequentially segmented according to the preset time interval to obtain a plurality of sample sub-audios; and the sample video segment corresponding to each sample sub-audio is determined; the preset time interval can be set according to the actual situation; for example, the voice activity detection technology (VAD) can be used to segment the sample audio to obtain a plurality of sample sub-audios, and the duration of each sample sub-audio can be a preset duration, and the preset duration can be set according to the actual situation; for example, the preset duration can be less than or equal to 10 seconds, and the preset duration can also be 8-10 seconds; after obtaining a plurality of sample sub-audios, the sample video can also be segmented according to the duration corresponding to each sample sub-audio to obtain a sample video segment corresponding to each sample sub-audio, that is, a plurality of sample video segments; each sample video segment includes a sample sub-audio and a sample image. And the sample interaction information corresponding to the sample object in the sample video clip can be extracted.

[0176] In the disclosed embodiment, the sample video can be parsed to obtain sample text, sample object features, sample object images, and sample interaction information corresponding to the sample objects, thereby facilitating the training of a preset model and obtaining a video processing model with higher accuracy.

[0177] In the embodiment of the present disclosure, the method further includes:

[0178] Acquire sample object description information of the sample object from the sample virtual video list corresponding to the sample video clip;

[0179] The determining, based on the sample text corresponding to the sample video clip and the sample object image, sample object attribute information of the sample object includes:

[0180] Based on the sample text, the sample object image and the sample object description information corresponding to the sample video clip, sample object attribute information of the sample object is determined.

[0181] In the disclosed embodiment, when the sample virtual space includes multiple sample objects, usually one sample video clip can correspond to one or more sample objects; the sample virtual video list can be a shopping cart in the live broadcast room, such as the shared cart; the server can obtain the sample virtual video list corresponding to the sample virtual space, or obtain the sample virtual video list corresponding to the sample video clip, and quickly determine the sample object in the sample video clip from the sample virtual video list, and obtain the sample object description information of the sample object; the sample object description information may include but is not limited to the relevant information characterizing the style, material, and image of the sample object. Then, based on the sample text corresponding to the sample video clip, the sample object image, and the sample object description information, the sample object attribute information of the sample object is further determined, and the sample object attribute information may include but is not limited to the attribute information such as the category, style, and material of the sample object.

[0182] In the disclosed embodiment, sample object information can be queried through a sample virtual video list, thereby narrowing the search scope of the sample object information and improving the efficiency of obtaining the sample object information; and when determining the sample object attribute information of the sample object, not only the sample text and the sample object image corresponding to the sample video clip are considered, but also the sample object attribute information is considered, thereby improving the accuracy of the sample object attribute information.

[0183] In the embodiment of the present disclosure, determining the sample object attribute information of the sample object based on the sample text corresponding to the sample video clip and the sample object image includes:

[0184] The sample text corresponding to the sample video clip and the sample object image are input into the object attribute extraction model to perform object attribute extraction processing to obtain sample object attribute information of the sample object.

[0185] In the disclosed embodiment, after the object attribute extraction model is trained, the sample text corresponding to the sample video clip and the sample object image can be input into the object attribute extraction model for object attribute extraction processing to obtain sample object attribute information of the sample object. The sample object attribute information and the target object attribute information can be the same type of information.

[0186] In the embodiment of the present disclosure, the object feature extraction process is performed on each sample text to obtain the sample object feature, including:

[0187] Performing a sample object feature extraction operation on each sample text, the sample object feature extraction operation including at least one of a first sample feature extraction operation, a second sample feature extraction operation, and a third sample feature extraction operation; wherein the first sample feature extraction operation is to extract object detail information in the sample text to obtain a first sample object feature; the second sample feature extraction operation is to extract object interaction prompt information in the sample text to obtain a second sample object feature; the third sample feature extraction operation is to extract object interaction information in the sample text to obtain a third sample object feature;

[0188] At least one of the first sample object feature, the second sample object feature, and the third sample object feature is determined as the sample object feature.

[0189] In the disclosed embodiment, the first sample object feature is the object detail information in the sample text, the second sample object feature is the object interaction prompt information in the sample text, and the third sample object feature is the object interaction information in the sample text; the object interaction prompt information in the sample text may include but is not limited to discount information, coupon information, etc. of the sample object; if only the first feature extraction operation is performed, the first sample object feature can be determined as the sample object feature, and if only the second feature extraction operation is performed, the second sample object feature can be determined as the sample object feature; if only the third feature extraction operation is performed, the third sample object feature can be determined as the sample object feature; if the first feature extraction operation and the second feature extraction operation are performed at the same time, the first sample object feature and the second sample object feature can be determined as the sample object features; if the first feature extraction operation, the second feature extraction operation and the third feature extraction operation are performed, the first sample object feature, the second sample object feature and the third sample object feature can be determined as the sample object features.

[0190] In the embodiment of the present disclosure, at least one of the first sample object feature, the second sample object feature, and the third sample object feature can be determined as the sample object feature based on the result of performing an object feature extraction operation on the sample text corresponding to each sample sub-audio, thereby improving the diversity of the sample object features.

[0191] In the embodiment of the present disclosure, performing video processing according to the target object features, the target object attribute information, and the target interaction information corresponding to the target video segment to obtain a target video processing result of the target video segment includes:

[0192] The target object features, the target object attribute information and the target interaction information corresponding to the target video clip are input into a video processing model for video processing to obtain a target video processing result of the target video clip.

[0193] In the disclosed embodiment, the target video processing result may be a highlight prediction score, and the highlight prediction score corresponding to the target video segment may be smoothed to obtain a more accurate video processing result.

[0194] In the disclosed embodiment, after the video processing model is trained, the model can be used to quickly and accurately identify video clips of highlight moments, thereby improving the efficiency of determining highlight video clips.

[0195] In the embodiment of the present disclosure, after determining that the target video segment is a highlight video segment, the method further includes:

[0196] When the target video includes multiple target objects and multiple highlight video segments, obtaining a set of highlight video segments corresponding to the same target object;

[0197] For the highlight video segment set corresponding to each target object, each highlight video segment in the highlight video segment set is aggregated to generate an aggregated video.

[0198] In the disclosed embodiment, if there are multiple target objects, a set of highlight video clips of the same target object can be determined from the multiple highlight video clips corresponding to the target video and the individual highlight video clips in the set of highlight video clips can be aggregated to generate an aggregated video and obtain high-quality short video production materials.

[0199] In the disclosed embodiment, a set of highlight video clips corresponding to the same target object can be quickly screened out and aggregated to form an aggregated video, thereby providing production materials for high-quality short videos and avoiding the need to produce short video materials for a single target object.

[0200] In the embodiment of the present disclosure, aggregating the highlight video segments in the highlight video segment set to generate an aggregated video includes:

[0201] Obtaining the duration of each highlight video segment in the highlight video segment set, and sorting the multiple highlight video segments according to the playback time of each highlight video segment in the target video, to obtain a segment sorting result;

[0202] According to the segment sorting result, each of the highlight video segments in the highlight video segment set is divided into a plurality of highlight video segment groups according to a preset aggregation duration; wherein the duration of each of the highlight video segments is less than the preset aggregation duration; and the sum of the durations of the video segments in each of the video segment groups is less than or equal to the preset aggregation duration;

[0203] The highlight video segments in each of the highlight video segment groups are aggregated to obtain an aggregated video corresponding to each of the highlight video segment groups.

[0204] In the disclosed embodiment, a preset aggregation duration corresponding to the aggregated video can be pre-set, the duration of each highlight video segment in the highlight video segment set is obtained, and the multiple highlight video segments are sorted according to the playback time of each highlight video segment in the target video to obtain a segment sorting result; if the sum of the durations of the highlight video segments in the highlight video segment set is greater than the preset aggregation duration, each highlight video segment in the highlight video segment set can be divided into multiple highlight video segment groups according to the preset aggregation duration according to the segment sorting result; the duration of each highlight video segment is less than the preset aggregation duration; and the sum of the durations of the video segments in each video segment group is less than or equal to the preset aggregation duration; then the highlight video segments in each highlight video segment group are aggregated in sequence according to the playback time to obtain an aggregated video corresponding to each highlight video segment group.

[0205] In the disclosed embodiment, when there are a large number of highlight video clips of the same target object and the total duration is long, they can be aggregated to obtain multiple aggregated videos, thereby avoiding the aggregated video from being too long and generating production materials for multiple short videos, thereby increasing the amount of materials.

[0206] In some embodiments, the time intervals between highlight video segments can also be calculated. The playback end time of the current initial video segment and the playback start time of the next adjacent initial video segment can be calculated to obtain the time interval between the two. The segments can be aggregated according to the time interval. For example, multiple initial video segments whose time intervals are less than a preset duration threshold can be aggregated. If the time interval between two initial video segments is greater than or equal to the preset duration threshold, the two initial video segments can be used to construct different aggregated videos.

[0207] In the present disclosure, Fig. 9 As shown, Fig. 9 A flowchart of a method for generating an update video, comprising:

[0208] S901: Obtain the number of users and the number of interactive information corresponding to each aggregated video in the target virtual space; the number of users is the sum of the number of viewers of each of the highlight video clips in the aggregated video when it is played in the target virtual space;

[0209] S903: generating a ranking parameter corresponding to each aggregated video according to the number of users and the amount of interaction information corresponding to each aggregated video in the target virtual space;

[0210] S905: Determine a filtered video from the multiple aggregated videos according to the sorting parameter corresponding to each aggregated video; the sorting parameter of the filtered video is greater than the sorting parameter of the non-filtered video;

[0211] S907: Generate an updated video based on the filtered video.

[0212] In the disclosed embodiment, the number of users may be the current number of viewers in the target virtual space, and the interaction information may include but is not limited to like information, the number of clicks on the shared bike, the number of orders placed on the shared bike, etc.; the sorting parameters corresponding to each aggregated video may be calculated based on the corresponding number of users and the amount of interaction information in the target virtual space; thereby, the filtered videos may be determined from the multiple aggregated videos based on the sorting parameters corresponding to each aggregated video.

[0213] Exemplarily, generating a ranking parameter corresponding to each aggregated video according to the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space includes:

[0214] Determine a first weight of the number of users corresponding to each aggregated video in the target virtual space;

[0215] Determine a second weight of the amount of interactive information corresponding to each aggregated video in the target virtual space;

[0216] For each aggregated video, calculating the product of the number of users and the first weight to obtain a first weighted number; and calculating the product of the number of interactive information and the second weight to obtain a second weighted number;

[0217] The sum of the first weighted number corresponding to each aggregated video and the first weighted number is calculated to obtain a sorting parameter corresponding to each aggregated video.

[0218] Specifically, the sum of the number of users and the amount of interactive information or the weighted sum of the two can be calculated to obtain a sorting parameter, and multiple aggregated videos can be sorted according to the sorting parameter. The target number of aggregated videos with the largest sorting parameter can be determined as the screening videos; wherein the target number can be set according to actual conditions, for example, it can be set to one or ten; thereby, the screened videos can be used as video update materials to generate updated videos.

[0219] In the disclosed embodiment, when there are multiple video clips of highlight moments, the multiple video clips of highlight moments can be aggregated to obtain multiple aggregated highlight videos, and filtered videos with a large number of viewers and a large number of interactions can be further screened out to construct high-quality updated videos, thereby improving the click-through rate of the updated videos.

[0220] In the embodiment of the present disclosure, the step of generating an updated video based on the screened video includes:

[0221] Subtitles, covers, opening credits, ending credits and special effect information are added to the filtered videos to generate updated videos.

[0222] In the disclosed embodiments, the updated video may include but is not limited to short videos, long videos, etc.; the screened video may be re-created by adding subtitles, covers, openings, endings, and special effects information to generate an updated video, thereby further improving the content richness of the updated video.

[0223] In the present disclosure, Fig.10 As shown, Fig.10 The invention is a structural schematic diagram of a short video generation system, which includes a speech recognition model, an object feature extraction model, an object attribute recognition model and a video processing model, wherein the speech recognition model can be a conformer model, the object feature extraction model can be a roberta model, the object attribute recognition model can be a SKU recognition model, and the video processing model can be an MLP model; the segmented live speech can be input into the conformer model to extract text features, and the text features are input into the roberta model to obtain object features; the live screen corresponding to the live speech is framed to obtain image frames, and then the text features and the image frames are input into the SKU model to obtain object attribute information; finally, the object features, the object attribute information and the interaction information in the virtual space are input into the MLP model to identify the video clips of the highlight moments, and generate high-quality short videos according to the video clips of the highlight moments, so as to improve the click-through rate of the short videos.

[0224] Fig.11 is a block diagram of a video processing device according to an exemplary embodiment. Fig.11 , the device comprises:

[0225] The target video segment determination module 1110 is configured to execute acquiring a target video corresponding to a target virtual space, segmenting the audio in the target video to obtain a plurality of target sub-audios, and determining a target video segment corresponding to each target sub-audio;

[0226] The target object feature determination module 1120 is configured to perform object feature extraction processing on the target text corresponding to each target sub-audio to obtain target object features; the target object features are features of the target object in the target virtual space;

[0227] The target interaction information extraction module 1130 is configured to extract the target object image of the target object and the target interaction information corresponding to the target object from the target video segment;

[0228] A target object attribute determination module 1140 is configured to determine target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment;

[0229] The judging module 1150 is configured to determine a target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and judge whether the target video segment includes the object detail information, the object interaction prompt information and the object interaction information of the target object according to the target fusion feature;

[0230] The highlight segment determination module 1160 is configured to determine that the target video segment is a highlight video segment when the target video segment includes at least one of the object detail information, the object interaction prompt information, and the object interaction information.

[0231] In an exemplary embodiment, the device further comprises:

[0232] A description information acquisition module is configured to acquire a target virtual resource list corresponding to the target video clip, and extract target object description information of the target object from the target virtual resource list;

[0233] The target object attribute determination module is configured to determine the target object attribute information of the target object based on the target text, the target object image and the target object description information corresponding to the target video segment.

[0234] In an exemplary embodiment, the determination module includes:

[0235] a fusion unit configured to perform fusion processing on the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment to obtain the target fusion feature of the target video segment;

[0236] A first judgment unit is configured to perform object detail information analysis according to the target fusion feature to obtain a first judgment result of whether the target video segment includes the object detail information of the target object;

[0237] A second judgment unit is configured to perform object interaction prompt information analysis according to the target fusion feature to obtain a second judgment result of whether the target video segment includes the object interaction prompt information of the target object;

[0238] The third judgment unit is configured to perform object interaction information analysis according to the target fusion feature to obtain a third judgment result of whether the target video segment includes the object interaction information of the target object.

[0239] In an exemplary embodiment, the device further comprises:

[0240] A video set acquisition module is configured to execute, when the target video includes multiple target objects and multiple highlight video segments, acquiring a set of highlight video segments corresponding to the same target object;

[0241] The aggregation module is configured to execute the highlight video segment set corresponding to each target object, aggregate the highlight video segments in the highlight video segment set, and generate an aggregated video.

[0242] In an exemplary embodiment, the aggregation module includes:

[0243] a sorting unit configured to obtain the duration of each highlight video segment in the highlight video segment set, and sort the multiple highlight video segments according to the playback time of each highlight video segment in the target video to obtain a segment sorting result;

[0244] A grouping unit is configured to execute, according to the segment sorting result, sequentially dividing each of the highlight video segments in the highlight video segment set into a plurality of highlight video segment groups according to a preset aggregation duration; wherein the duration of each of the highlight video segments is less than the preset aggregation duration; and the sum of the durations of the video segments in each of the video segment groups is less than or equal to the preset aggregation duration;

[0245] The aggregation unit is configured to aggregate the highlight video segments in each of the highlight video segment groups to obtain an aggregated video corresponding to each of the highlight video segment groups.

[0246] In an exemplary embodiment, the device further comprises:

[0247] A quantity acquisition module is configured to execute acquisition of the number of users and the number of interactive information corresponding to each aggregated video in the target virtual space; the number of users is the sum of the number of viewers of each of the highlight video clips in the aggregated video when the highlight video clips are played in the target virtual space;

[0248] A sorting parameter generating module is configured to generate a sorting parameter corresponding to each aggregated video according to the number of users and the amount of interaction information corresponding to each aggregated video in the target virtual space;

[0249] A screening module is configured to determine a screening video from the plurality of aggregated videos according to a sorting parameter corresponding to each aggregated video; the sorting parameter of the screening video is greater than the sorting parameter of the non-screening video;

[0250] The updated video generation module is configured to generate an updated video based on the filtered video.

[0251] In an exemplary embodiment, the sorting parameter generation module includes:

[0252] A first weight determination unit is configured to determine a first weight of the number of users corresponding to each aggregated video in the target virtual space;

[0253] A second weight determination unit is configured to determine a second weight of the amount of interaction information corresponding to each aggregated video in the target virtual space;

[0254] A weighting unit is configured to calculate, for each aggregated video, a product of the number of users and the first weight to obtain a first weighted number; and calculate a product of the number of interactive information and the second weight to obtain a second weighted number;

[0255] The sorting parameter determination unit is configured to calculate the sum of the first weighted number corresponding to each aggregated video and the first weighted number to obtain the sorting parameter corresponding to each aggregated video.

[0256] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0257] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the video processing method provided in any of the above embodiments when executing the instructions stored in the memory.

[0258] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.12 is a block diagram of an electronic device according to an exemplary embodiment. Fig.12 As shown, the server 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1210 (the processor 1210 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1230 for storing data, and one or more storage media 1220 (such as one or more mass storage devices) for storing application programs 1223 or data 1222. Among them, the memory 1230 and the storage medium 1220 can be short-term storage or permanent storage. The program stored in the storage medium 1220 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 1210 can be configured to communicate with the storage medium 1220 and execute a series of instruction operations in the storage medium 1220 on the server 1200. The server 1200 may also include one or more power supplies 1260, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1240, and / or one or more operating systems 1221, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0259] The input / output interface 1240 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 1200. In one example, the input / output interface 1240 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 1240 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0260] It can be understood by those skilled in the art that Fig.12The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.12 More or fewer components as shown, or with Fig.12 Different configurations are shown.

[0261] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1230 including instructions, and the instructions can be executed by the processor 1210 of the device 1200 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0262] In an exemplary embodiment, a computer program product is further provided, including a computer program, and when the computer program is executed by a processor, the video processing method provided in any of the above embodiments is implemented.

[0263] Fig.13 is a block diagram of an electronic device for video processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.13 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0264] Those skilled in the art will understand that Fig.13 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0265] The present invention obtains a target video corresponding to a target virtual space, and divides the audio in the target video to obtain a plurality of target sub-audios, and determines a target video segment corresponding to each target sub-audio; performs object feature extraction processing on a target text corresponding to each target sub-audio to obtain a target object feature; then extracts a target object image of the target object and target interaction information corresponding to the target object from the target video segment; according to the target text and the target object image corresponding to the target video segment, the target object attribute information of the target object can be quickly and accurately determined; thereby, according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, a target fusion feature can be determined, and according to the target fusion feature, it can be judged whether the target video segment includes object detail information, object interaction prompt information and object interaction information of the target object; when the target video segment includes at least one of the object detail information, the object interaction prompt information and the object interaction information, the target video segment is determined to be a highlight video segment; the present invention can quickly and accurately determine the target video segment of the highlight moment, thereby reducing the production cost of the highlight short video.

[0266] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0267] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0268] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: include: Acquire a target video corresponding to the target virtual space, segment the audio in the target video to obtain a plurality of target sub-audios, and determine a target video segment corresponding to each target sub-audio; Performing object feature extraction processing on the target text corresponding to each target sub-audio to obtain target object features; the target object features are features of the target object in the target virtual space; Extracting a target object image of the target object and target interaction information corresponding to the target object from the target video clip; Determining target object attribute information of the target object based on the target text and the target object image corresponding to the target video clip; Determine a target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and determine whether the target video segment includes object detail information, object interaction prompt information and object interaction information of the target object according to the target fusion feature; When the target video segment includes at least one of the object detail information, the object interaction prompt information, and the object interaction information, it is determined that the target video segment is a highlight video segment.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire a target virtual resource list corresponding to the target video clip, and extract target object description information of the target object from the target virtual resource list; The determining the target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment includes: Target object attribute information of the target object is determined based on the target text, the target object image and the target object description information corresponding to the target video clip.

3. The method according to claim 1, characterized in that The determining of the target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and judging whether the target video segment includes the object detail information, the object interaction prompt information and the object interaction information of the target object according to the target fusion feature, comprises: Performing fusion processing on the target object features, the target interaction information, and the target object attribute information corresponding to the target video segment to obtain the target fusion features of the target video segment; Analyzing object detail information according to the target fusion feature to obtain a first judgment result of whether the target video segment includes object detail information of the target object; performing object interaction prompt information analysis according to the target fusion feature to obtain a second judgment result of whether the target video segment includes the object interaction prompt information of the target object; Object interaction information is analyzed according to the target fusion feature to obtain a third judgment result of whether the target video segment includes the object interaction information of the target object.

4. The method according to claim 1, characterized in that: After determining that the target video segment is a highlight video segment, the method further includes: When the target video includes multiple target objects and multiple highlight video segments, obtaining a set of highlight video segments corresponding to the same target object; For the highlight video segment set corresponding to each target object, each highlight video segment in the highlight video segment set is aggregated to generate an aggregated video.

5. The method according to claim 4, characterized in that The step of aggregating the highlight video segments in the highlight video segment set to generate an aggregated video includes: Obtaining the duration of each highlight video segment in the highlight video segment set, and sorting the multiple highlight video segments according to the playback time of each highlight video segment in the target video, to obtain a segment sorting result; According to the segment sorting result, each of the highlight video segments in the highlight video segment set is divided into a plurality of highlight video segment groups according to a preset aggregation duration; wherein the duration of each of the highlight video segments is less than the preset aggregation duration; and the sum of the durations of the video segments in each of the video segment groups is less than or equal to the preset aggregation duration; The highlight video segments in each of the highlight video segment groups are aggregated to obtain an aggregated video corresponding to each of the highlight video segment groups.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space; the number of users is the sum of the number of viewers of each of the highlight video clips in the aggregated video when they are played in the target virtual space; Generating a ranking parameter corresponding to each aggregated video according to the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space; Determining a screening video from the multiple aggregated videos according to a ranking parameter corresponding to each aggregated video; the ranking parameter of the screening video is greater than the ranking parameter of the non-screening video; Based on the filtered video, an updated video is generated.

7. The method according to claim 6, characterized in that The step of generating a ranking parameter corresponding to each aggregated video according to the number of users and the amount of interactive information corresponding to each aggregated video in the target virtual space includes: Determine a first weight of the number of users corresponding to each aggregated video in the target virtual space; Determine a second weight of the amount of interactive information corresponding to each aggregated video in the target virtual space; For each aggregated video, calculating the product of the number of users and the first weight to obtain a first weighted number; and calculating the product of the number of interactive information and the second weight to obtain a second weighted number; The sum of the first weighted number corresponding to each aggregated video and the first weighted number is calculated to obtain a sorting parameter corresponding to each aggregated video.

8. A video processing device, characterized in that: include: A target video segment determination module is configured to execute acquisition of a target video corresponding to a target virtual space, segment the audio in the target video to obtain a plurality of target sub-audios, and determine a target video segment corresponding to each target sub-audio; A target object feature determination module is configured to perform object feature extraction processing on a target text corresponding to each target sub-audio to obtain a target object feature; the target object feature is a feature of a target object in the target virtual space; A target interaction information extraction module is configured to extract a target object image of the target object and target interaction information corresponding to the target object from the target video clip; a target object attribute determination module, configured to determine target object attribute information of the target object based on the target text and the target object image corresponding to the target video segment; a judgment module configured to determine a target fusion feature according to the target object feature, the target interaction information and the target object attribute information corresponding to the target video segment, and to judge whether the target video segment includes the object detail information, the object interaction prompt information and the object interaction information of the target object according to the target fusion feature; The highlight segment determination module is configured to determine that the target video segment is a highlight video segment when the target video segment includes at least one of the object detail information, the object interaction prompt information, and the object interaction information.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the video processing method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Live video interception method and device, commodity information generation method and device, and object information generation method and device

    CN112055225A

  • Video clip integration method and device, electronic equipment and storage medium

    CN113792654A

  • Video generation method and device, electronic equipment and storage medium

    CN114268848A

  • Video generation method and device, electronic equipment and storage medium

    CN114501058A

  • Video editing method and electronic equipment

    CN115052201A