Stream media feature extraction method, multi-modal content representation model establishment method, stream media feature extraction device and electronic equipment

Abstract

Embodiments of the present application provide a stream media feature extraction method, a multi-modal content representation model establishment method, a stream media feature extraction device and an electronic device, relating to the technical field of artificial intelligence. The method extracts the content features of video clips in stream media through a pre-established multi-modal content representation model. In the model establishment process, the multi-modal content representation model not only learns to distinguish the content characteristics of different slices based on content labels, but also forcibly constrains the smooth coherence of the feature changes of adjacent slices. In practical applications, when multiple video clips to be processed in the same multimedia stream are input into the model, the respective content features output not only accurately reflect the semantic content of each clip itself, but also naturally carry the law of smooth evolution of video content over time, thereby avoiding abrupt jumps in the extracted content features caused by model modeling bias or local noise, and further ensuring the application effect of the content features in downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a streaming media feature extraction method, a multimodal content representation model establishment method, a streaming media feature extraction device, and an electronic device. Background Technology

[0002] Video feature extraction from each slice of a multimedia stream refers to dividing a continuous multimedia stream into multiple multimedia stream slices of fixed duration, and generating a feature vector for each multimedia stream slice that represents its visual content, audio information, and semantic meaning. This feature vector is the video feature extraction result for that multimedia stream slice. This video feature extraction result for each slice of a multimedia stream is a crucial foundation for supporting downstream applications such as accurate content retrieval, similarity comparison, and automated analysis; its quality directly determines the actual effectiveness of downstream applications.

[0003] Current mainstream methods for extracting video features from each slice in a multimedia stream typically treat the multimedia stream slices as isolated data units, resulting in fragmented video feature extraction results for each slice in the generated multimedia stream on the timeline. At the same time, due to the use of uniform extraction measurements, the video feature extraction process for each slice in the multimedia stream treats all multimedia stream slices indiscriminately.

[0004] Therefore, improving the accuracy of video feature extraction from each slice in a multimedia stream and avoiding drastic jitter in the extracted video features that does not conform to the natural evolution of the content are urgent technical problems to be solved. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a streaming media feature extraction method, a multimodal content representation model establishment method, a streaming media feature extraction device and an electronic device, which can improve the accuracy and continuity of streaming media feature extraction.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides a streaming media feature extraction method, the method comprising: A pre-established multimodal content representation model is obtained; the multimodal content representation model is established based on multimedia stream slices labeled with content tags, and the parameters of the multimodal content representation model are optimized based on multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; Retrieve multiple video segments to be processed from the same multimedia stream; The multimodal content representation model is used to extract the content features corresponding to each video segment to be processed.

[0007] In an optional implementation, the establishment of the multimodal content representation model includes: Acquire multimodal data corresponding to multiple multimedia stream slices in each batch; Generate content tags corresponding to each multimedia stream slice; The multimodal data behind each mask is input into the multimodal content representation network to obtain the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice; The total loss value is determined based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch. The parameters of the multimodal content representation network are iteratively updated based on the total loss value of each batch to obtain the multimodal content representation model.

[0008] In an optional implementation, determining the total loss value based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch includes: Based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch, generate reconstruction loss value, contrast loss value, and temporal consistency loss value; The weights of the reconstruction loss value and the comparison loss value are determined based on the content tags corresponding to each multimedia stream slice in the same batch. The reconstruction loss value, the comparison loss value, and the temporal consistency loss value are then weighted and summed to obtain the total loss value.

[0009] In an optional implementation, the content tags include activity status tags and audience feedback tags. The step of generating reconstruction loss values, contrast loss values, and temporal consistency loss values ​​based on multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch includes: The reconstruction loss value corresponding to the establishment batch is determined based on the multimodal data and multimodal reconstruction data corresponding to each multimedia stream slice in the same establishment batch; The contrast loss value corresponding to the creation batch is determined based on the activity status label, audience feedback label and fusion feature of each multimedia stream slice in the same creation batch. Two consecutive multimedia stream slices originating from the same original video in the same batch are identified as slice pairs; The temporal consistency loss value corresponding to the batch is determined based on the fusion features of the two multimedia stream slices in each slice pair.

[0010] In an optional implementation, determining the contrast loss value corresponding to the establishment batch based on the activity status label, audience feedback label, and fusion features corresponding to each multimedia stream slice in the same establishment batch includes: Each multimedia stream slice in the same batch is used as the target slice. Other multimedia stream slices that have the same activity status label as the target slice are identified as positive samples of the activity status of the target slice. The activity state loss value of the target slice is determined based on the fusion features corresponding to the positive samples of the activity state, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice. Other multimedia stream slices that have the same audience feedback label as the target slice are identified as positive audience feedback samples of the target slice; The audience feedback loss value of the target slice is determined based on the fusion features corresponding to the positive audience feedback samples, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice. The contrast loss value corresponding to the batch is determined based on the activity state loss value and the audience feedback loss value of all the target slices.

[0011] In an optional implementation, the content tags include audience feedback tags, and determining the weights of the reconstruction loss value and the contrast loss value based on the content tags corresponding to each multimedia stream slice in the same batch includes: If the audience feedback label corresponding to the multimedia stream slice is "audience does not resonate", the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice are set as the first weight and the second weight, respectively. If the audience feedback tag corresponding to the multimedia stream slice is audience resonance, the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice are set as the second weight and the first weight, respectively. The weight of the reconstruction loss value is determined based on the reconstruction weights corresponding to all multimedia stream slices in the same batch. The weight of the contrast loss value is determined based on the contrast loss weights corresponding to all multimedia stream slices in the same batch.

[0012] In an optional implementation, the content tags include activity status tags and audience feedback tags; generating content tags for each multimedia stream slice includes: Obtain the interactive information corresponding to each multimedia stream slice; Based on the multimodal data, an activity status label is generated corresponding to the multimedia stream slice; the activity status label is used to characterize the behavioral intent of the video subject contained in the multimedia stream slice; Based on the interactive information, audience feedback tags are generated corresponding to the multimedia stream slice; the audience feedback tags are used to characterize the video type of audience feedback contained in the multimedia stream slice.

[0013] In an optional implementation, generating audience feedback tags corresponding to the multimedia stream slice based on the interaction information includes: According to the preset statistical step size, the bullet screen index and gift value index are statistically analyzed within each time window based on the interactive information. If both the bullet screen index and the gift value index show peak values ​​higher than a preset threshold, and the difference between the time windows of each peak value is less than a preset time, then a resonance event is determined to have occurred. If no resonance event occurs, the audience feedback label will be set to "audience did not resonate". If a resonance event occurs, the interactive information is semantically classified according to the preset semantic classification rules, and the audience feedback tag is set as the result of the semantic classification.

[0014] Secondly, the present invention provides a method for establishing a multimodal content representation model, the method comprising: Acquire multimodal data corresponding to multiple multimedia stream slices in each batch; Generate content tags corresponding to each multimedia stream slice; The multimodal data behind each mask is input into the multimodal content representation network to obtain the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice; Multiple loss values ​​are determined based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch, and the total loss value is determined based on the multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; The parameters of the multimodal content representation network are iteratively updated based on the total loss value of each batch to obtain the multimodal content representation model.

[0015] Thirdly, the present invention provides a streaming media feature extraction device, the device comprising: The acquisition module is used to acquire a pre-established multimodal content representation model; the multimodal content representation model is based on multimedia stream slices labeled with content tags, and the parameters of the multimodal content representation model are optimized based on multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; and to acquire multiple video segments to be processed in the same multimedia stream. The processing module is used to extract the content features corresponding to each of the video segments to be processed using the multimodal content representation model.

[0016] Fourthly, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the streaming media feature extraction method and / or the multimodal content representation model establishment method described in any of the foregoing embodiments.

[0017] Compared to existing technologies, the streaming media feature extraction method, multimodal content representation model establishment method, streaming media feature extraction device, and electronic device provided in this invention extract content features of video segments in streaming media through a pre-established multimodal content representation model. During model establishment, the multimodal content representation model not only learns to distinguish the content characteristics of different segments based on content tags, but also enforces the smooth and coherent changes in features of adjacent segments. In practical applications, when multiple video segments to be processed from the same multimedia stream are input into the model, the output content features not only accurately reflect the semantic content of each segment, but also naturally carry the smooth evolution of video content over time, thus avoiding abrupt changes in extracted content features due to modeling bias or local noise, thereby ensuring the effectiveness of content features in downstream tasks.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart of a streaming media feature extraction method provided by an embodiment of the present invention is shown.

[0021] Figure 2 This diagram illustrates another flowchart of the streaming media feature extraction method provided in an embodiment of the present invention.

[0022] Figure 3 A block diagram of a multimodal content representation network provided in an embodiment of the present invention is shown.

[0023] Figure 4This diagram illustrates a flowchart of a method for establishing a multimodal content representation model provided in an embodiment of the present invention.

[0024] Figure 5 A block diagram of a streaming media feature extraction device provided in an embodiment of the present invention is shown.

[0025] Figure 6 A block diagram of a multimodal content representation model establishment device provided in an embodiment of the present invention is shown.

[0026] Figure 7 A block diagram of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0029] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0030] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] Please refer to Figure 1 , Figure 1 A schematic flowchart of a streaming media feature extraction method provided by an embodiment of the present invention is shown. The method includes the following steps: Step S200: Obtain a pre-established multimodal content representation model; the multimodal content representation model is established based on multimedia stream slices labeled with content tags, and the parameters of the multimodal content representation model are optimized based on multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fused features corresponding to two consecutive multimedia stream slices in the same original video.

[0032] In this embodiment of the invention, the first step is to obtain a pre-trained multimodal content representation model, i.e., a pre-established multimodal content representation model. This multimodal content representation model is not a general model, but rather one specifically "fed" from a large number of multimedia stream segments with content tags. The content tags are structured descriptions added to each multimedia stream segment (e.g., a live stream clip of about 10 seconds), such as "The streamer is playing game A" or "A viewer just sent a '666' comment." These tags do not require manual frame-by-frame annotation and can be automatically generated using a weakly supervised tag generator.

[0033] Step S210: Obtain multiple video segments to be processed from the same multimedia stream.

[0034] In practical applications, users only need to divide a new multimedia stream (such as a live stream) into multiple video segments to be processed. These video segments belong to the same multimedia stream, that is, different time periods of the same live stream.

[0035] Step S220: Extract the content features corresponding to each video segment to be processed using a multimodal content representation model.

[0036] Next, each video segment to be processed is input into the multimodal content representation model. The video encoder in the multimodal content representation model encodes the video segment to be processed, and the contrastive learning projector in the multimodal content representation model performs feature space mapping on the features of the encoded video segment to extract the content features corresponding to each video segment to be processed.

[0037] It should be understood that multimodal content representation models pay special attention to the semantic continuity of the temporal dimension during the modeling phase. By introducing a temporal consistency loss value, the Euclidean distance between the fused features corresponding to two adjacent multimedia stream slices in the same original video is constrained in the embedding space. This forces the multimodal content representation model to learn the smooth changes in content over time. The more temporally consistent the fused features are, the more the multimodal content representation model's modeling of the dynamic evolution of streaming media content conforms to the real-world propagation logic.

[0038] In summary, the streaming media feature extraction method provided by this invention extracts content features from video segments in streaming media through a pre-established multimodal content representation model. During model building, the multimodal content representation model not only learns to distinguish the content characteristics of different segments based on content tags, but also enforces a smooth and coherent constraint on the feature changes of adjacent segments. In practical applications, when multiple video segments to be processed from the same multimedia stream are input into the model, the output content features not only accurately reflect the semantic content of each segment, but also naturally carry the smooth evolution of video content over time. This avoids abrupt changes in extracted content features due to modeling bias or local noise, thereby ensuring the effectiveness of content features in downstream tasks.

[0039] Alternatively, please refer to Figure 2 Regarding how to pre-establish a multimodal content representation model, the following is a possible implementation method.

[0040] Step S100: Obtain multimodal data corresponding to multiple multimedia stream slices in each batch.

[0041] In this embodiment of the invention, continuous time segments are extracted from a large amount of video data; each such time segment is called a multimedia stream slice. Each multimedia stream slice contains multimodal data within the same time period. The multimodal data includes video frames and corresponding audio content. This can be understood as cutting a live video into frame-by-frame segments every 10 seconds, each segment containing both image and sound.

[0042] In the process of establishing the multimodal content representation model, a batch iterative approach is adopted. The multimedia stream is pre-divided into multiple multimedia stream slices according to time and semantic granularity. Each slice is assigned to a different batch to support the multimodal content representation model in learning cross-modal correlation features in each iteration.

[0043] Step S110: Generate content tags corresponding to each multimedia stream slice.

[0044] In this embodiment of the invention, before establishing a multimodal content representation model, a pre-trained weakly supervised label generator is used to determine the content nature of each slice. This weakly supervised label generator does not require manual annotation of each data point; instead, it is an automated discrimination module trained based on existing small-scale samples. It analyzes each multimedia stream slice and outputs content labels sufficient to reflect the video content.

[0045] Step S120: Input the multimodal data behind each mask into the multimodal content representation network to obtain the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice.

[0046] In this embodiment of the invention, before entering the multimodal content representation network, the multimodal data undergoes masking processing; that is, a portion of the video and audio is randomly masked, similar to blacking out some pixels of an image or muting a segment of sound. This "damaged" data is then fed into a multimodal content representation network, and after a series of processing steps, outputs the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice. The multimodal reconstruction data includes reconstructed video attempting to recover the complete picture and reconstructed audio attempting to restore the original sound.

[0047] As one possible implementation, such as Figure 3 As shown, the multimodal content representation network includes a video encoder, an audio encoder, a video decoder, an audio decoder, and a contrastive learning projector. Figure 2 The intermediate step S120 sub-step includes: First, the video and audio segments in each multimedia stream slice are masked to obtain video masked segments and audio masked segments. By masking, the multimodal content representation network is forced to learn not only to "memorize" the input, but also to infer the complete content based on the remaining information, thereby improving its ability to understand the internal structure of the content.

[0048] Second, the video mask slice and audio mask slice corresponding to each multimedia stream slice are input into the multimodal content representation network.

[0049] Third, a video encoder is used to encode video mask slices to obtain video features; an audio encoder is used to encode audio mask slices to obtain audio features. Video encoding and audio encoding are performed in parallel, each focusing on extracting information from its respective modality.

[0050] Fourth, a video decoder is used to decode the video features to obtain the reconstructed video; an audio decoder is used to decode the audio features to obtain the reconstructed audio. During the decoding process, lightweight video and audio decoders are used to recover the original input content, thereby training the multimodal content representation network to grasp fine-grained structural information within each modality. The reconstructed video and reconstructed audio constitute the multimodal reconstruction data.

[0051] Fifth, video and audio features are fused, and the fused features are mapped to a feature space using a contrastive learning projector to obtain fused features. To enable the multimodal content representation network to understand the relationships between different modalities and establish a unified semantic space, video and audio features are fused. Fusion methods include, but are not limited to, concatenation and cross-attention mechanisms to achieve cross-modal information interaction. Subsequently, the fused features are fed into a contrastive learning projector, mapping high-dimensional features to a unified low-dimensional vector space. The final output is a multimodal representation (i.e., fused features) that contains both intramodal details and cross-modal semantic consistency, improving the multimodal content representation network's ability to understand multimedia content.

[0052] Step S130: Determine the total loss value based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch.

[0053] In this embodiment of the invention, the total loss value is determined by comprehensively considering the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia slice in the current batch. The total loss value is not a single indicator, but rather a weighted combination of multiple loss values.

[0054] Step S140: Iteratively update the parameters of the multimodal content representation network based on the total loss value of each batch to obtain the multimodal content representation model.

[0055] In this embodiment of the invention, the total loss value reflects the overall performance of the multimodal content representation network. The parameters within the multimodal content representation network are adjusted through backpropagation based on the total loss value, making the next prediction more accurate. With thousands upon thousands of such batches of input, the multimodal content representation network gradually learns how to extract stable, general, and semantically meaningful content features from complex multimedia streams (such as live streams), ultimately forming a multimodal content representation model that can be used for various downstream tasks.

[0056] As can be seen, the embodiments of the present invention generate content tags representing the behavioral intentions of the video subject and the video type of the audience feedback for each multimedia stream slice, and reconstruct the original audio and video content based on the masked multimodal data. This enables the model to learn the detailed features of the multimedia content during the model building process, and to distinguish semantic information by combining the content tags. This achieves the goal of using content tags to guide the model to take into account both the understanding of content details and the discrimination of global semantics.

[0057] Based on this, by introducing a temporal consistency loss value, the feature changes between consecutive segments from the same video are constrained to remain smooth. At the same time, the weights of the reconstruction loss value and the contrast loss value are dynamically adjusted according to the content tags, so that the model can adaptively allocate the learning focus on different types of content, realize differentiated learning for different content, and finally obtain a multimodal content representation model that can stably reflect the evolution law of multimedia content.

[0058] Alternatively, one possible implementation for determining the total loss value is provided below. Figure 2 The sub-steps of step S130 may include: Step S131: Generate reconstruction loss value, contrast loss value and temporal consistency loss value based on the multimodal data, multimodal reconstruction data, fusion features and content tags corresponding to each multimedia stream slice in the same batch.

[0059] In this embodiment of the invention, the reconstruction loss value measures the error of the multimodal content representation network in restoring the occluded portion; the smaller the error, the better the multimodal content representation network's understanding of details. The contrast loss value focuses on whether the multimodal content representation network can correctly distinguish content with different semantics, such as bringing together segments belonging to the same "highlight operation" category, while separating "funny scenes" from "ordinary dialogue." The temporal consistency loss value checks whether the fused features at adjacent time points change smoothly, avoiding situations where nearly identical scenes produce huge feature differences.

[0060] Step S132: Determine the weights of the reconstruction loss value and the comparison loss value based on the content tags corresponding to each multimedia stream slice in the same batch, and perform a weighted sum of the reconstruction loss value, the comparison loss value, and the temporal consistency loss value to obtain the total loss value.

[0061] In this embodiment of the invention, by dynamically adjusting the weights of the reconstruction loss and contrast loss based on content tags, the multimodal content representation network learns to adjust its learning strategy according to different video content. For example, when a multimedia stream segment is determined to have a lukewarm audience response, the weight of the reconstruction loss is automatically increased, prompting the multimodal content representation network to focus more on mastering the basic structure of the image and sound; conversely, when a highlight moment is detected, the weight of the contrast loss is increased, guiding the multimodal content representation network to capture more of the global semantic features of such events. This dynamic adjustment mechanism ensures that the multimodal content representation network does not apply its efforts evenly, but rather allocates learning resources in a focused manner.

[0062] As can be seen, this embodiment of the invention determines the total loss value by weighted summing of the reconstruction loss value, the contrast loss value, and the temporal consistency loss value. This allows the multimodal content representation network to simultaneously consider data reconstruction accuracy, semantic discrimination ability, and temporal logical coherence during its construction. The weights of the reconstruction loss value and the contrast loss value are dynamically determined by the content tags, enabling the model to assign differentiated learning intensity to slices with different feedback types, such as audience resonance or lack of audience resonance. This, in turn, improves the practicality and robustness of the features learned by the model in real streaming media scenarios.

[0063] Optionally, content tags include activity status tags and audience feedback tags. Regarding how to determine the reconstruction loss value, contrast loss value, and temporal consistency loss value, a possible implementation method is provided below. The sub-steps of step S131 may include: Step S131-1: Determine the reconstruction loss value corresponding to the creation batch based on the multimodal data and multimodal reconstruction data corresponding to each multimedia stream slice in the same creation batch.

[0064] In this embodiment of the invention, for multimedia stream slices in the same batch, the differences (e.g., pixel differences) between multimodal data and multimodal reconstruction distance are compared. Specifically, the pixel differences between video slices and reconstructed video, and between audio slices and reconstructed audio are compared, and the mean squared error (MSE) is used to calculate the reconstruction loss value corresponding to the batch. This enables the multimodal content representation model to learn to recover fine-grained single-modal content from incomplete input.

[0065] The formula for calculating the reconstruction loss is as follows:

[0066] in, It is the reconstruction loss value; It is the j-th video slice; It is the j-th reconstructed video; It is the j-th audio slice; It is the j-th reconstructed audio; M is the number of multimedia stream slices in the same batch.

[0067] Step S131-2: Determine the contrast loss value corresponding to the creation batch based on the activity status label, audience feedback label and fusion features of each multimedia stream slice in the same creation batch.

[0068] In this embodiment of the invention, based on the activity status labels and audience feedback labels carried by each multimedia stream slice in the same batch, it is determined which slices are semantically similar. Slices with the same labels are regarded as positive samples, and their output fused features are combined to narrow their distance in the feature space through a contrastive learning mechanism; while slices with different labels are regarded as negative samples and are pushed away, thereby enhancing the multimodal content representation model's ability to distinguish global semantics.

[0069] Step S131-3: Two consecutive multimedia stream slices from the same original video in the same batch are identified as slice pairs.

[0070] Step S131-4: Determine the temporal consistency loss value corresponding to the batch based on the fusion features of the two multimedia stream slices in each slice pair.

[0071] In this embodiment of the invention, to preserve the continuous evolution of multimedia content (e.g., live content) over time, consecutive segments originating from the same original video are identified, such as adjacent slices like Clip1_t and Clip1_t+1, resulting in multiple slice pairs. For each slice pair, the L2 distance (the square of the Euclidean distance) between their respective output fused features is calculated, and all L2 distances are averaged to obtain the temporal consistency loss value. This temporal consistency loss value is used to constrain the feature sequence output by the multimodal content representation model to maintain a smooth transition in the temporal dimension, avoiding abrupt jumps.

[0072] The formula for calculating the timing consistency loss is as follows:

[0073] in, This is the time-series consistency loss value; It is the number of slice pairs; and It is the fusion feature corresponding to the two multimedia stream slices in the k-th slice pair.

[0074] As can be seen, by simultaneously calculating the reconstruction loss value, the contrast loss value, and the temporal consistency loss value, the multimodal content representation model can take into account the restoration of content details, the differentiation of global semantics, and the continuity of time during the training process. Through multi-task joint optimization, the overall modeling capability of the multimodal content representation model for multimedia streaming data is improved.

[0075] Optionally, regarding how to calculate the contrast loss value, the following is a possible implementation. The sub-steps of step S131-2 may include: First, the multimedia stream slices in the same batch are used as target slices.

[0076] In this embodiment of the invention, a multi-task contrastive learning mechanism enables the model to distinguish the similarity of content from different semantic dimensions. In practical applications, a small batch (e.g., 256) containing multiple multimedia slices is randomly sampled from a multimedia (e.g., live stream) dataset as a batch.

[0077] Each of these multimedia stream slices carries an activity status label and an audience feedback label pre-generated by a weakly supervised label generator, used to characterize the broadcaster's behavior and audience reactions contained in the slice. For each multimedia stream slice in this batch, it is processed as a target slice in sequence.

[0078] Second, other multimedia stream slices that have the same activity state label as the target slice are identified as positive samples of the target slice's activity state.

[0079] In this embodiment of the invention, positive activity state samples are constructed based on activity state labels. For example, if Clip_i is taken as the target slice, and among all other multimedia stream slices Clip_j (j ≠ i) in the same batch, if the activity state label of Clip_j is exactly the same as that of Clip_i, then Clip_j is taken as a positive activity state sample of Clip_i. Similarly, other multimedia stream slices with different activity state labels from the target slice are identified as negative activity state samples of the target slice.

[0080] For example, suppose the activity_type of the target slice Clip_i is "GAMEPLAY_LOL". If the activity_type of the multimedia stream slice Clip_j is also "GAMEPLAY_LOL", then Clip_j is identified as a positive active sample. If the activity_type of the multimedia stream slice Clip_k is "CHATTING_UI", then Clip_k is identified as a negative active sample.

[0081] It should be understood that when selecting positive samples, a weak data augmentation, such as slight cropping or color perturbation, is performed on the target slice itself to obtain an augmented copy, which serves as the strongest positive sample. This helps to improve the robustness of the multimodal content representation model to input variations.

[0082] Third, the activity state loss value of the target slice is determined based on the fusion features corresponding to the positive samples of the activity state, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice.

[0083] In this embodiment of the invention, based on the fusion features corresponding to the positive active state samples, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice, the active state loss value of the target slice is calculated using the InfoNCE loss function. The InfoNCE loss function is as follows:

[0084] in, It is the InfoNCE loss value; Let be the set of positive samples of target slice i; p is the set of positive samples in the set of positive samples; It is the complete sample set that includes all multimedia stream slices in the same batch; It refers to a sample from the entire sample set; It is a preset temperature coefficient, which controls the sharpness of the similarity distribution. It calculates the dot product similarity between the fusion features of the target slice and the fusion features of the positive sample; It calculates the dot product similarity between the fusion features of the target slice and the fusion features of each multimedia stream slice; exp is an exponential function.

[0085] It should be understood that the goal of the InfoNCE loss function is to maximize the similarity ratio between the target slice and its positive samples, thereby forcing the multimodal content representation model to bring semantically similar content closer together and push away irrelevant content in the feature space.

[0086] Specifically, when calculating the activity state loss value of the target slice, the set of positive samples of the activity state of the target slice is used as the set of positive samples in the InfoNCE loss function, and the multimedia stream slices in the same batch are used as the set of all samples.

[0087] Fourth, other multimedia stream slices that have the same audience feedback labels as the target slice are identified as positive samples of audience feedback for the target slice.

[0088] In this embodiment of the invention, under the "audience resonance" task, if the audience feedback label of the target slice is not "no resonance," other multimedia stream slices with the same audience feedback label in the same batch are identified as positive audience feedback samples, indicating that these slices are similar in atmosphere intensity and type. Similarly, other multimedia stream slices with different audience feedback labels from the target slice are identified as negative audience resonance samples of the target slice.

[0089] It should be understood that for a target segment whose audience feedback is labeled "no resonance", positive samples from other segments are generally not sought for it, because the differences in bland content are too large, and forcibly narrowing them down may mislead the multimodal content representation model.

[0090] For example, suppose the audience feedback label `resonance_type` for target slice Clip_i is "SKILL_PLAY". If the audience feedback label `resonance_type` for multimedia stream slice Clip_j is also "SKILL_PLAY", then multimedia stream slice Clip_j is identified as a positive audience feedback sample. If the audience feedback label `resonance_type` for multimedia stream slice Clip_k is "FUNNY", then multimedia stream slice Clip_k is identified as a negative audience resonance sample. If the audience feedback label `resonance_type` for multimedia stream slice Clip_l is "NONE", where "NONE" indicates no resonance, then multimedia stream slice Clip_l is identified as a negative audience resonance sample.

[0091] Fifth, the audience feedback loss value of the target slice is determined based on the fusion features corresponding to the positive audience feedback samples, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice.

[0092] In this embodiment of the invention, similarly, when calculating the audience feedback loss value of the target slice, the set of positive samples of audience feedback of the target slice is used as the set of positive samples in the InfoNCE loss function, and the multimedia stream slices in the same batch are used as the set of all samples.

[0093] Sixth, determine the corresponding comparative loss value for each batch based on the activity status loss value and audience feedback loss value of all target slices.

[0094] In this embodiment of the invention, the average value of the activity state loss value corresponding to each multimedia stream slice in the same batch is calculated, and the average value of the audience feedback loss value corresponding to each multimedia stream slice in the same batch is calculated. Finally, the average value of the activity state loss value and the average value of the audience feedback loss value are weighted and summed to obtain the comparison loss value corresponding to the batch.

[0095] The formula for calculating the contrast loss value is shown below:

[0096] in, This is the contrast loss value; M is the number of multimedia stream slices in the same batch; It is the activity state loss value corresponding to the i-th multimedia stream slice; It is the audience feedback loss value corresponding to the i-th multimedia stream slice; and It is used to balance the importance of different comparative tasks.

[0097] As can be seen, this embodiment of the invention utilizes multi-dimensional weakly supervised labels (i.e., activity state labels and audience feedback labels) to construct positive and negative sample relationships from multiple perspectives for multimedia stream slices. Through a multi-task contrastive learning mechanism, the multimodal content representation model can distinguish the similarity of content from different semantic dimensions, achieving more refined content semantic modeling. Optionally, the content labels include audience feedback labels. Regarding how to dynamically adjust the weights of the reconstruction loss value and the contrastive loss value, a possible implementation method is provided below. The sub-step of step S132 may include: Step S132-1: If the audience feedback label corresponding to the multimedia stream slice is "audience does not resonate", set the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice as the first weight and the second weight, respectively.

[0098] When building a multimodal content representation model capable of understanding video content, the information value contained in video clips at different times is not the same. Some moments may be mundane, such as the streamer waiting for the game to load or sitting quietly; while at other moments, exciting events may occur, such as a difficult maneuver or a funny scene.

[0099] These "highlight moments" are often accompanied by strong audience interaction, such as a surge of comments and gifts. Therefore, simply applying the same learning strategy to all video clips does not allow multimodal content representation models to optimally utilize the rich information in the data.

[0100] In this embodiment of the invention, if the audience feedback label corresponding to a multimedia stream slice is "audience did not resonate," it indicates that the multimedia stream slice corresponds to relatively bland content. At this time, the image and sound lack significant global semantic features, making it difficult for the multimodal content representation model to extract discriminative high-level concepts. In this case, it is more reasonable for the multimodal content representation model to focus on restoring the original audio and video content itself, that is, to strengthen its ability to understand details. Therefore, the reconstruction weight corresponding to the multimedia stream slice of the "bland moment" is set as the first weight, while the contrast loss weight is set as the second weight, with the first weight being greater than the second weight. For example, the first weight is set to 0.8, and the second weight is set to 0.2.

[0101] Step S132-2: If the audience feedback label corresponding to the multimedia stream slice is audience resonance, set the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice as the second weight and the first weight, respectively.

[0102] Conversely, when the audience feedback label for a multimedia stream slice is "audience resonance," it indicates a key moment with clear semantic significance, such as the end of an intense battle or an unexpected laugh. In this case, rather than pixel-by-pixel reconstruction, it's more important for the multimodal content representation model to learn to identify and categorize the essential characteristics of such events—that is, to determine which category of meaningful behavior they belong to. Therefore, the reconstruction weight corresponding to the "highlight moment" multimedia stream slice is set as the second weight to reduce its influence, while the contrast loss weight is set as the first weight to enhance the dominant role of the contrast loss value.

[0103] Step S132-3: Determine the weight of the reconstruction loss value based on the reconstruction weights of all multimedia stream slices in the same batch.

[0104] Step S132-4: Determine the weight of the contrast loss value based on the contrast loss weights corresponding to all multimedia stream slices in the same batch.

[0105] In this embodiment of the invention, the average reconstruction weights corresponding to all multimedia stream slices in the same batch are determined as the weights of the reconstruction loss value. Similarly, the average contrast loss weights corresponding to all multimedia stream slices in the same batch are determined as the weights of the contrast loss value.

[0106] As can be seen, this embodiment of the invention dynamically adjusts the weights of the reconstruction loss value and the contrastive loss value based on audience feedback tags, enabling the multimodal content representation model to automatically switch its learning focus on data of different natures, achieving differentiated learning for content of different natures. In ordinary times, the multimodal content representation model focuses on mastering basic perceptual abilities; while in high-profile moments, the model shifts to strengthening the understanding and discrimination of advanced semantics. This approach avoids the resource waste caused by traditional fixed weights, making the model building process more intelligent and closely aligned with the characteristics of real content distribution.

[0107] Optionally, content tags include activity status tags and audience feedback tags. Below is one possible implementation for generating content tags using a weakly supervised tag generator. Figure 2 The sub-steps of step S110 may include: Step S111: Obtain the interactive information corresponding to each multimedia stream slice.

[0108] In this embodiment of the invention, interactive information such as bullet comments and donations sent by viewers within the time range covered by each multimedia stream slice is obtained.

[0109] Step S112: Generate activity status labels corresponding to multimedia stream slices based on multimodal data; the activity status labels are used to characterize the behavioral intent of the video subjects contained in the multimedia stream slices.

[0110] In this embodiment of the invention, each multimedia stream slice is input into a pre-trained weakly supervised label generator. This generator acts as an automated content classification assistant, capable of quickly identifying the basic features of the multimedia stream slice. It does not pursue pixel-level precise labeling, but rather grasps the overall content type, providing directional guidance for subsequent training.

[0111] The weakly supervised label generator contains two independently functioning submodules: an activity state detector and an audience resonance detector. The activity state detector is specifically responsible for analyzing video frames within multimedia stream slices to determine the current operational scenario or behavioral state of the main video subject (e.g., the broadcaster). For example, it can identify whether the current frame belongs to a category such as "game battle," "desktop operation," or "live broadcast."

[0112] In this process, the activity state detector needs to be pre-trained. This involves a one-time offline training of a lightweight image classification model based on a small but representative image dataset. The image dataset consists of thousands of manually annotated live stream screenshots, covering various common scenes. Through training, the activity state detector learns to map input video frames to predefined activity types and outputs corresponding activity state labels to characterize the main behaviors contained in the multimedia stream slice.

[0113] Step S113: Generate audience feedback tags corresponding to the multimedia stream slices based on the interactive information; the audience feedback tags are used to characterize the video type of audience feedback contained in the multimedia stream slices.

[0114] In this embodiment of the invention, the audience resonance detector focuses on analyzing interactive information within the same multimedia stream slice, such as bullet comments and gift records, to determine the intensity of the audience's emotional response at that moment. Both work together to complete the content tagging task for each multimedia stream slice, ensuring that the multimodal content representation network receives dual feedback on "what happened (e.g., the broadcaster's behavior)" and "how the audience reacted" during its construction.

[0115] As can be seen, the embodiments of the present invention generate content labels by introducing an activity state detector and an audience resonance detector, so that multimedia stream slices can be automatically labeled with dual-dimensional labels representing behavior and emotional feedback, thereby providing reliable supervision signals for subsequent training of multimodal content representation networks.

[0116] Optionally, regarding how to generate audience feedback tags, the following is a possible implementation. The sub-steps of step S113 may include: Step S113-1: According to the preset statistical step size, statistically analyze the barrage index and gift value index within each time window based on the interactive information.

[0117] In this embodiment of the invention, the interactive information contained in each multimedia stream slice is input into the audience resonance detector. This interactive information mainly includes bullet comments and gift data, and has been precisely aligned with the video content via timestamps.

[0118] Based on this, using a shorter time unit (such as 1 second) as the statistical step size, the bullet screen index and gift value index within each time window are calculated respectively, thus forming multiple data sequences that change over time; among them, the bullet screen index includes, but is not limited to, bullet screen density, bullet screen entropy, and frequency of occurrence of high-value keywords, and the gift value index includes, but is not limited to, gift value flow rate.

[0119] It should be understood that bullet screen density is the number of bullet screens per unit time (e.g., per second). The formula for calculating bullet screen entropy s is:

[0120] in, is the probability of the i-th unique bullet comment appearing, and N is the number of unique bullet comments.

[0121] For example, if there are 10 bullet comments, 9 of which are "666" and 1 is "nice", then the probability of "666" is 0.9 and the probability of "nice" is 0.1. The bullet comment entropy calculated using the formula is approximately 0.469.

[0122] High-value keyword frequency refers to the frequency of occurrence of keywords in a predefined keyword list (such as "awesome," "outrageous," "?", "hahaha," etc.). Gift value flow rate is the total value of gifts sent by viewers per unit of time (e.g., per second).

[0123] Step S113-2: If both the bullet screen index and the gift value index show peak values ​​higher than the preset threshold, and the difference between the time windows of each peak is less than the preset time, then a resonance event is determined to have occurred.

[0124] In this embodiment of the invention, a peak detection algorithm is used to detect the time series corresponding to bullet screen density, bullet screen entropy, frequency of occurrence of high-value keywords, and gift value flow rate, respectively. When a peak is detected in bullet screen density, bullet screen entropy, frequency of occurrence of high-value keywords, and gift value flow rate that are significantly higher than normal levels (i.e., a preset threshold) around the same time period, it is considered that a viewer resonance event may exist. Furthermore, if the difference between the latest time and the earliest time of the peak is less than a preset time, the resonance event is finally confirmed.

[0125] Step S113-3: If no resonance event occurs, set the audience feedback label to "audience did not resonate".

[0126] In this embodiment of the invention, for example, when processing a 10-second live stream segment, it is checked whether a resonance event has occurred. If no resonance event is detected, the audience feedback label for the live stream segment is set to "audience did not resonate," indicating that the overall audience response during this segment was flat and lacked focused feedback.

[0127] In step S113-4, if a resonance event occurs, the interactive information is semantically classified according to the preset semantic classification rules, and the audience feedback tag is set as the result of the semantic classification.

[0128] In this embodiment of the invention, once a resonance event is confirmed, the bullet screen content in the interactive information is classified using a large language model based on preset semantic classification rules. Specifically, the large language model calculates the similarity between the bullet screen content and each semantic category in the preset semantic classification rules, and the semantic category with the highest similarity is used as the audience feedback label. In other words, once a resonance event is confirmed, the corresponding audience feedback label is set as the specific semantic category representing the audience's resonance.

[0129] For example, embedding vector technology can be used to measure the proximity of bullet screen content to different semantic categories, thereby determining which emotional type this resonance leans towards, and accurately reflecting whether the segment evokes surprise, joy, or other types of group emotional responses.

[0130] As can be seen, this embodiment of the invention analyzes interactive information in multimedia stream slices using an audience resonance detector, automatically identifies moments when the audience's emotions are heightened, and generates audience feedback tags reflecting the atmosphere of the event. This process does not require human intervention; instead, it determines whether the audience has generated a collective emotional response at a particular moment based on quantifiable behavioral data and preset rules.

[0131] Please refer to Figure 4 , Figure 4 This diagram illustrates a flowchart of a method for establishing a multimodal content representation model according to an embodiment of the present invention. The method includes the following steps: Step S300: Obtain multimodal data corresponding to multiple multimedia stream slices in each batch.

[0132] Step S310: Generate content tags corresponding to each multimedia stream slice.

[0133] Step S320: Input the multimodal data behind each mask into the multimodal content representation network to obtain the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice.

[0134] Step S330: Determine multiple loss values ​​based on the multimodal data, multimodal reconstruction data, fusion features and content tags corresponding to each multimedia stream slice in the same batch, and determine the total loss value based on the multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video.

[0135] Step S340: Iteratively update the parameters of the multimodal content representation network based on the total loss value of each batch to obtain the multimodal content representation model.

[0136] In the embodiments of the present invention, based on the same inventive concept, the basic principles and technical effects of the multimodal content representation model establishment method provided by the embodiments of the present invention are the same as those of the present invention. Figure 2 The provided embodiments and sub-steps are the same. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0137] In summary, the embodiments of the present invention generate content tags representing the behavioral intentions of the video subject and the video type of the audience feedback for each multimedia stream slice, and reconstruct the original audio and video content based on the masked multimodal data. This enables the model to learn the detailed features of the multimedia content during the model building process, and to distinguish semantic information by combining the content tags. This achieves the goal of using content tags to guide the model to take into account both the understanding of content details and the discrimination of global semantics.

[0138] Based on this, by introducing a temporal consistency loss value, the feature changes between consecutive segments from the same video are constrained to remain smooth. At the same time, the weights of the reconstruction loss value and the contrast loss value are dynamically adjusted according to the content tags, so that the model can adaptively allocate the learning focus on different types of content, realize differentiated learning for different content, and finally obtain a multimodal content representation model that can stably reflect the evolution law of multimedia content.

[0139] Based on the same inventive concept, the basic principle and technical effects of the streaming media feature extraction device provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0140] Please refer to Figure 5 , Figure 5 This is a block diagram of a streaming media feature extraction device 400 provided in an embodiment of the present invention. The streaming media feature extraction device 400 includes an acquisition module 410 and a processing module 420.

[0141] The acquisition module 410 is used to acquire a pre-established multimodal content representation model. The multimodal content representation model is established based on multimedia stream slices labeled with content tags. The parameters of the multimodal content representation model are optimized based on multiple loss values. The temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video. Multiple video segments to be processed in the same multimedia stream are acquired.

[0142] The processing module 420 is used to extract the content features corresponding to each video segment to be processed using a multimodal content representation model.

[0143] In summary, the streaming media feature extraction device provided in this embodiment of the invention extracts content features of video segments in streaming media through a pre-established multimodal content representation model. During model building, the multimodal content representation model not only learns to distinguish the content characteristics of different segments based on content tags, but also enforces a smooth and consistent variation in features between adjacent segments. In practical applications, when multiple video segments to be processed from the same multimedia stream are input into the model, the output content features not only accurately reflect the semantic content of each segment, but also naturally carry the smooth evolution of video content over time. This avoids abrupt changes in extracted content features due to modeling bias or local noise, thereby ensuring the effectiveness of content features in downstream tasks.

[0144] Optionally, the acquisition module 410 is further configured to acquire multimodal data corresponding to multiple multimedia stream slices in each batch. The processing module 420 is further configured to generate content tags corresponding to each multimedia stream slice; input the masked multimodal data into the multimodal content representation network to obtain multimodal reconstruction data and fusion features corresponding to each multimedia stream slice; determine the total loss value based on the multimodal data, multimodal reconstruction data, fusion features and content tags corresponding to each multimedia stream slice in the same batch; and iteratively update the parameters of the multimodal content representation network based on the total loss value of each batch to obtain the multimodal content representation model.

[0145] Optionally, the processing module 420 is specifically used to generate reconstruction loss value, contrast loss value and temporal consistency loss value based on the multimodal data, multimodal reconstruction data, fusion features and content tags corresponding to each multimedia stream slice in the same batch; determine the weight of reconstruction loss value and contrast loss value based on the content tags corresponding to each multimedia stream slice in the same batch; and perform a weighted summation of reconstruction loss value, contrast loss value and temporal consistency loss value to obtain the total loss value.

[0146] Optionally, the content tags include activity status tags and audience feedback tags. The processing module 420 is specifically used to determine the reconstruction loss value corresponding to the creation batch based on the multimodal data and multimodal reconstruction data corresponding to each multimedia stream slice in the same creation batch; to determine the contrast loss value corresponding to the creation batch based on the activity status tags, audience feedback tags, and fusion features corresponding to each multimedia stream slice in the same creation batch; to identify two consecutive multimedia stream slices originating from the same original video in the same creation batch as slice pairs; and to determine the temporal consistency loss value corresponding to the creation batch based on the fusion features corresponding to the two multimedia stream slices in each slice pair.

[0147] Optionally, the processing module 420 is specifically configured to: designate multimedia stream slices within the same creation batch as target slices; identify other multimedia stream slices with the same activity state label as positive activity state samples of the target slice; determine the activity state loss value of the target slice based on the fusion features corresponding to the positive activity state samples, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice; identify other multimedia stream slices with the same audience feedback label as positive audience feedback samples of the target slice; determine the audience feedback loss value of the target slice based on the fusion features corresponding to the positive audience feedback samples, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice; and determine the contrast loss value corresponding to the creation batch based on the activity state loss value and audience feedback loss value of all target slices.

[0148] Optionally, the content tags include audience feedback tags. The processing module 420 is specifically configured to: if the audience feedback tag corresponding to the multimedia stream slice is "audience not resonating," set the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice to a first weight and a second weight, respectively; if the audience feedback tag corresponding to the multimedia stream slice is "audience resonating," set the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice to a second weight and a first weight, respectively; determine the weight of the reconstruction loss value based on the reconstruction weights corresponding to all multimedia stream slices in the same batch; and determine the weight of the contrast loss value based on the contrast loss weights corresponding to all multimedia stream slices in the same batch.

[0149] Optionally, the content tags include activity status tags and audience feedback tags. The processing module 420 is specifically used to acquire interactive information corresponding to each multimedia stream slice; generate activity status tags corresponding to the multimedia stream slice based on multimodal data; the activity status tags are used to characterize the behavioral intent of the video subject contained in the multimedia stream slice; and generate audience feedback tags corresponding to the multimedia stream slice based on the interactive information; the audience feedback tags are used to characterize the video type of audience feedback contained in the multimedia stream slice.

[0150] Optionally, the processing module 420 is specifically used to statistically analyze the barrage index and gift value index within each time window based on the interactive information according to a preset statistical step size; if both the barrage index and the gift value index show peak values ​​higher than a preset threshold, and the difference between the time windows of each peak is less than a preset time, then a resonance event is determined to have occurred; if no resonance event occurs, then the audience feedback label is set to "audience did not resonate"; if a resonance event occurs, then the interactive information is semantically classified according to a preset semantic classification rule, and the audience feedback label is set to the result of the semantic classification.

[0151] Please refer to Figure 6 , Figure 6 This is a block diagram of a multimodal content representation model building device 500 provided in an embodiment of the present invention. The multimodal content representation model building device 500 includes a preprocessing module 510 and a building module 520.

[0152] The preprocessing module 510 is used to acquire multimodal data corresponding to multiple multimedia stream slices in each batch.

[0153] A module 520 is established to generate content tags corresponding to each multimedia stream slice; each masked multimodal data is input into a multimodal content representation network to obtain multimodal reconstruction data and fusion features corresponding to each multimedia stream slice; multiple loss values ​​are determined based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch, and a total loss value is determined based on the multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; the parameters of the multimodal content representation network are iteratively updated based on the total loss value of each batch to obtain a multimodal content representation model.

[0154] Please refer to Figure 7 This is a block diagram illustrating an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 includes, but is not limited to, a personal computer (PC), a handheld computer (PDA), a laptop computer, a tablet computer, and a server. The electronic device 600 includes a memory 610, a processor 620, and a communication module 630. The memory 610, processor 620, and communication module 630 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0155] The memory 610 is used to store programs or data. The memory 610 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0156] The processor 620 is used to read / write data or programs stored in the memory 610 and perform corresponding functions. For example, when a computer program stored in the memory 610 is executed by the processor 620, the streaming media feature extraction method and / or multimodal content representation model establishment method disclosed in the above embodiments can be implemented.

[0157] The communication module 630 is used to establish a communication connection between the electronic device 600 and other communication terminals via a network, and to send and receive data via the network.

[0158] It should be understood that, Figure 7 The structure shown is only a schematic diagram of the electronic device 600. The electronic device 600 may also include components that are larger than... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0159] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor 620, implements the streaming media feature extraction method and / or the multimodal content representation model establishment method disclosed in the above embodiments.

[0160] This invention also provides a program product that, when executed by processor 620, implements the streaming media feature extraction method and / or multimodal content representation model establishment method disclosed in the above embodiments.

[0161] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0162] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0163] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting features from streaming media, characterized in that, The method includes: A pre-established multimodal content representation model is obtained; the multimodal content representation model is established based on multimedia stream slices labeled with content tags, and the parameters of the multimodal content representation model are optimized based on multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; Retrieve multiple video segments to be processed from the same multimedia stream; The multimodal content representation model is used to extract the content features corresponding to each video segment to be processed.

2. The streaming media feature extraction method according to claim 1, characterized in that, The establishment of the multimodal content representation model includes: Acquire multimodal data corresponding to multiple multimedia stream slices in each batch; Generate content tags corresponding to each multimedia stream slice; The multimodal data behind each mask is input into the multimodal content representation network to obtain the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice; The total loss value is determined based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch. The parameters of the multimodal content representation network are iteratively updated based on the total loss value of each batch to obtain the multimodal content representation model.

3. The streaming media feature extraction method according to claim 2, characterized in that, The determination of the total loss value based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch includes: Based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch, generate reconstruction loss value, contrast loss value, and temporal consistency loss value; The weights of the reconstruction loss value and the comparison loss value are determined based on the content tags corresponding to each multimedia stream slice in the same batch. The reconstruction loss value, the comparison loss value, and the temporal consistency loss value are then weighted and summed to obtain the total loss value.

4. The streaming media feature extraction method according to claim 3, characterized in that, The content tags include activity status tags and audience feedback tags. The generation of reconstruction loss values, contrast loss values, and temporal consistency loss values ​​based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch includes: The reconstruction loss value corresponding to the establishment batch is determined based on the multimodal data and multimodal reconstruction data corresponding to each multimedia stream slice in the same establishment batch; The contrast loss value corresponding to the creation batch is determined based on the activity status label, audience feedback label and fusion feature of each multimedia stream slice in the same creation batch. Two consecutive multimedia stream slices originating from the same original video in the same batch are identified as slice pairs; The temporal consistency loss value corresponding to the batch is determined based on the fusion features of the two multimedia stream slices in each slice pair.

5. The streaming media feature extraction method according to claim 4, characterized in that, The step of determining the contrast loss value corresponding to the creation batch based on the activity status label, audience feedback label, and fusion features corresponding to each multimedia stream slice in the same creation batch includes: Each multimedia stream slice in the same batch is used as the target slice. Other multimedia stream slices that have the same activity status label as the target slice are identified as positive samples of the activity status of the target slice. The activity state loss value of the target slice is determined based on the fusion features corresponding to the positive samples of the activity state, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice. Other multimedia stream slices that have the same audience feedback label as the target slice are identified as positive audience feedback samples of the target slice; The audience feedback loss value of the target slice is determined based on the fusion features corresponding to the positive audience feedback samples, the fusion features corresponding to the target slice, and the fusion features corresponding to each multimedia stream slice. The contrast loss value corresponding to the batch is determined based on the activity state loss value and the audience feedback loss value of all the target slices.

6. The streaming media feature extraction method according to any one of claims 3-5, characterized in that, The content tags include audience feedback tags. Determining the weights of the reconstruction loss value and the contrast loss value based on the content tags corresponding to each multimedia stream slice in the same batch includes: If the audience feedback label corresponding to the multimedia stream slice is "audience does not resonate", the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice are set as the first weight and the second weight, respectively. If the audience feedback tag corresponding to the multimedia stream slice is audience resonance, the reconstruction weight and contrast loss weight corresponding to the multimedia stream slice are set as the second weight and the first weight, respectively. The weight of the reconstruction loss value is determined based on the reconstruction weights corresponding to all multimedia stream slices in the same batch. The weight of the contrast loss value is determined based on the contrast loss weights corresponding to all multimedia stream slices in the same batch.

7. The streaming media feature extraction method according to claim 2, characterized in that, The content tags include activity status tags and audience feedback tags; The generation of content tags corresponding to each multimedia stream slice includes: Obtain the interactive information corresponding to each multimedia stream slice; Based on the multimodal data, an activity status label is generated corresponding to the multimedia stream slice; the activity status label is used to characterize the behavioral intent of the video subject contained in the multimedia stream slice; Based on the interactive information, audience feedback tags are generated corresponding to the multimedia stream slice; the audience feedback tags are used to characterize the video type of audience feedback contained in the multimedia stream slice.

8. The streaming media feature extraction method according to claim 7, characterized in that, The step of generating audience feedback tags corresponding to the multimedia stream slice based on the interactive information includes: According to the preset statistical step size, the bullet screen index and gift value index are statistically analyzed within each time window based on the interactive information. If both the bullet screen index and the gift value index show peak values ​​higher than a preset threshold, and the difference between the time windows of each peak value is less than a preset time, then a resonance event is determined to have occurred. If no resonance event occurs, the audience feedback label will be set to "audience did not resonate". If a resonance event occurs, the interactive information is semantically classified according to the preset semantic classification rules, and the audience feedback tag is set as the result of the semantic classification.

9. A method for establishing a multimodal content representation model, characterized in that, The method includes: Acquire multimodal data corresponding to multiple multimedia stream slices in each batch; Generate content tags corresponding to each multimedia stream slice; The multimodal data behind each mask is input into the multimodal content representation network to obtain the multimodal reconstruction data and fusion features corresponding to each multimedia stream slice; Multiple loss values ​​are determined based on the multimodal data, multimodal reconstruction data, fusion features, and content tags corresponding to each multimedia stream slice in the same batch, and the total loss value is determined based on the multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; The parameters of the multimodal content representation network are iteratively updated based on the total loss value of each batch to obtain the multimodal content representation model.

10. A streaming media feature extraction device, characterized in that, The device includes: The acquisition module is used to acquire a pre-established multimodal content representation model; the multimodal content representation model is based on multimedia stream slices labeled with content tags, and the parameters of the multimodal content representation model are optimized based on multiple loss values; the temporal consistency loss value included in the multiple loss values ​​is used to quantify the difference between the fusion features corresponding to two consecutive multimedia stream slices in the same original video; and to acquire multiple video segments to be processed in the same multimedia stream. The processing module is used to extract the content features corresponding to each of the video segments to be processed using the multimodal content representation model.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor, the processor being able to execute the computer program to implement the streaming media feature extraction method of any one of claims 1-8 and / or the multimodal content representation model establishment method of claim 9.

Images