A large model-based video content review system

Abstract

The application relates to the technical field of video content auditing, and discloses a video content auditing system based on a large model, which comprises the following steps: information such as the display order and the decoding order relationship of a coded video stream, a motion vector, a coded block spatial position and a video frame center position is extracted through a data acquisition unit; a local offset calculation unit is used to perform time domain difference operation on the motion intensity, and the time sequence offset caused by compression is quantized; a torque diagram generation unit is used to fuse the local offset, the direction symbol and the space vector torque, and construct a frame-level directed heterogeneous torque diagram; a feature processing unit is used to stack multiple torque diagrams into a three-dimensional feature input convolutional neural network, and output a time anchor point of an event occurrence; finally, an auditing execution unit determines an auditing window according to the time anchor point and a frame rate, intercepts an original video segment, and generates visual information to mark manual auditing.

Description

Technical Field

[0001] This invention relates to the field of video content moderation technology, and more specifically, to a video content moderation system based on a large model. Background Technology

[0002] To improve bandwidth utilization, video compression coding standards (such as H.264 and H.265) introduce Group of Pictures (GOPs) consisting of I-frames, P-frames, and B-frames. B-frames rely on past and future frames for prediction, which can lead to discrepancies between the decoding order and the display order at the decoding end. The decoder must reorder the frames according to timestamps to output them at the correct time. However, during encoding, some motion information (such as the reference vector or skip mode of B-frames) directly utilizes the derivation results of future or adjacent frames, causing motion cues to be ahead or behind the actual physical process.

[0003] In compressed video, many tools further exacerbate the temporal shift of motion information. For example, skip mode may directly use zero motion prediction without recording actual displacement, resulting in consecutive frames lacking effective motion description; loop deblocking filtering introduces smoothing on the reference frame, causing residual information of sudden movements to be scattered into subsequent frames; long GOP or open GOP structures allow cross-boundary referencing, blurring the temporal boundaries of critical movements. Ultimately, this leads to misalignment or trailing of the motion intensity curve in the compressed domain compared to the curve in the original display domain, which is particularly noticeable in high-acceleration instantaneous events such as falls and impacts.

[0004] In practical video content review scenarios, fall detection is considered a key category requiring focused identification. Review systems often rely on convolutional neural networks to classify or locate video clips. However, if the input motion information has already shifted from the actual timeline during the compression stage, the neural network's internal representation will incorrectly align the falling, landing, and stationary phases. This directly leads to misjudgments in the review results, such as misidentifying a rapid squat or bending over as a fall, or delaying the actual point of contact with a fall to an irrelevant scene. This problem not only reduces the accuracy of the review but also increases the burden on manual review, making it the most prominent technical issue in review processes based on compressed video streams. Summary of the Invention

[0005] This invention provides a video content review system based on a large model, which solves the technical problems mentioned in the background section.

[0006] This invention provides a video content review system based on a large model, comprising:

[0007] The data acquisition unit acquires the correspondence between the display order and decoding order of the encoded video stream, and extracts motion vectors, spatial positions of encoded blocks, center positions of video frames, and frame rates.

[0008] The local offset calculation unit performs time-domain difference operations on the motion intensity based on the display order and decoding order to calculate the local offset; wherein, the motion intensity is determined based on the motion vector.

[0009] The torque map generation unit determines the motion direction based on the motion vector and obtains the direction sign by combining the relative relationship between the motion direction and the gravity direction; it calculates the spatial vector moment between the spatial position of the coded block and the center position of the video frame; and it multiplies the local offset, direction sign and spatial vector moment to form a frame-level directed mismatch torque map.

[0010] The feature processing unit stacks frame-level directed mismatch torque maps of multiple consecutive frames in chronological order into three-dimensional feature data, which is used as input to a convolutional neural network; the convolutional neural network outputs time anchor points.

[0011] The review execution unit determines the review window based on the time anchor point, the preset before and after extension duration, and the frame rate; it extracts the original video segment corresponding to the review window, generates visualization information related to the frame-level directed mismatch torque diagram, and marks the visualization information as manual review.

[0012] The beneficial effects of this invention are as follows: By introducing a frame-level directed heterogeneous torque map, it unifies the modeling of temporal misalignment, directional offset, and spatial correlation information caused by differences in encoding and decoding order, GOP structure, and reference frame mechanism in compressed videos. Furthermore, it utilizes a convolutional neural network to extract the spatiotemporal evolution patterns of events at the three-dimensional feature level. This not only significantly improves the accuracy of locating high-acceleration events (such as falls and ground impacts), avoiding common misjudgments and omissions in traditional methods, but also visually highlights deviation areas of motion cues when generating visualization results, providing a clear basis for manual review. Overall, this invention effectively improves the accuracy and reliability of video content review, reduces the burden of manual review, and has broad application value. Attached Figure Description

[0013] Figure 1 This is a module diagram of a video content review system based on a large model according to the present invention. Detailed Implementation

[0014] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0015] like Figure 1 As shown, a video content moderation system based on a large model includes:

[0016] The data acquisition unit acquires the correspondence between the display order and decoding order of the encoded video stream, and extracts motion vectors, spatial positions of encoded blocks, center positions of video frames, and frame rates.

[0017] The local offset calculation unit performs time-domain difference operations on the motion intensity based on the display order and decoding order to calculate the local offset; wherein, the motion intensity is determined based on the motion vector.

[0018] The torque map generation unit determines the motion direction based on the motion vector and obtains the direction sign by combining the relative relationship between the motion direction and the gravity direction; it calculates the spatial vector moment between the spatial position of the coded block and the center position of the video frame; and it multiplies the local offset, direction sign and spatial vector moment to form a frame-level directed mismatch torque map.

[0019] The feature processing unit stacks frame-level directed mismatch torque maps of multiple consecutive frames in chronological order into three-dimensional feature data, which is used as input to a convolutional neural network; the convolutional neural network outputs time anchor points.

[0020] The review execution unit determines the review window based on the time anchor point, the preset before and after extension duration, and the frame rate; it extracts the original video segment corresponding to the review window, generates visualization information related to the frame-level directed mismatch torque diagram, and marks the visualization information as manual review.

[0021] In one embodiment of the present invention, obtaining the correspondence between the display order and decoding order of the encoded video stream, and extracting motion vectors, spatial positions of encoded blocks, center positions of video frames, and frame rates, includes:

[0022] For all video frames contained in the encoded video stream, arrange them in non-descending order according to the display timestamp of each video frame to obtain the frame sequence of display order;

[0023] For all video frames contained in the encoded video stream, sort them in non-descending order according to the decoding timestamp of each video frame to obtain the frame sequence of decoding order;

[0024] Read the frame width and frame height of each video frame;

[0025] For each video frame and all the coded blocks contained in that video frame, read the horizontal motion vector component and the vertical motion vector component of each coded block;

[0026] For each coded block, the motion intensity of that coded block is calculated by taking the square root of the sum of the squares of the horizontal motion vector components and the squares of the vertical motion vector components.

[0027] Read the top-left horizontal coordinate, top-left vertical coordinate, block width, and block height of each encoded block;

[0028] For each coding block, calculate the horizontal and vertical coordinates of the center of the coding block. The horizontal coordinate of the center is the horizontal coordinate of the top left corner of the coding block plus half the width of the block, and the vertical coordinate of the center is the vertical coordinate of the top left corner of the coding block plus half the height of the block.

[0029] For each video frame, based on the frame width and frame height in the frame-level recording unit of the video frame, calculate the center horizontal coordinate and center vertical coordinate of the video frame. The center horizontal coordinate of the video frame is half of the frame width, and the center vertical coordinate of the video frame is half of the frame height.

[0030] Determine the first and last frame display timestamps of all video frames contained in the encoded video stream, and count the total number of frames in all video frames;

[0031] The frame rate is calculated as follows: (total number of frames minus one) divided by (last frame display timestamp minus first frame display timestamp).

[0032] In detail, the horizontal motion vector component (MPV) is the quantized displacement value of a coded block relative to its corresponding block in the reference frame in the horizontal direction (left-right direction of the image) during video coding. The MPV is read from the coded block data of the coded video stream, typically in pixels, and is used to describe the motion trend of the coded block in the horizontal dimension.

[0033] In detail, the vertical motion vector component (VPC) is the quantized displacement value of a coded block relative to its corresponding block in the reference frame in the vertical direction (vertical direction of the image) during video coding. The VPC is read from the coded block data of the coded video stream, typically in pixels, and is used to describe the motion trend of the coded block in the vertical dimension.

[0034] In detail, the motion intensity of a coded block is an indicator that quantifies the amplitude of the block's motion. The motion intensity reflects only the overall intensity of the block's motion and does not involve the direction of motion; a larger value indicates a greater amplitude of motion.

[0035] In detail, the center horizontal coordinates of the coded block are the center position coordinates of the coded block in the horizontal direction of the video frame, used to locate the spatial center of the coded block in the horizontal direction within the frame.

[0036] In detail, the center vertical coordinates of the coded block are the center position coordinates of the coded block in the vertical direction of the video frame, used to locate the spatial center of the coded block in the vertical direction within the frame.

[0037] In detail, the center horizontal coordinates of a video frame are the coordinates of the center position of the video frame in the horizontal direction. They are used to measure the relative position of the coded block in the horizontal direction within the frame, and the coordinate unit is pixels.

[0038] In detail, the center vertical coordinate of a video frame is the coordinate of the center position of the video frame in the vertical direction. It is used to measure the relative position of the coded block in the vertical direction within the frame, and the coordinate unit is pixels.

[0039] In detail, frame rate is an indicator that describes the temporal resolution of a video, referring to the number of frames played per unit of time, usually measured in frames per second.

[0040] In one embodiment of the present invention, based on the display order and decoding order, time-domain difference operations are performed on the motion intensity to calculate the local offset, including:

[0041] For each video frame, determine the preceding and following frames in the frame sequence of the display order. For each video frame and all coded blocks contained in the video frame, calculate the display order difference component of the coded blocks. If the video frame has both preceding and following frames, the display order difference component is: the difference between the motion intensity of the coded block in the following frame and the motion intensity of the coded block in the preceding frame, divided by the difference between the display timestamp of the following frame and the display timestamp of the preceding frame. If the video frame only has following frames, the display order difference component is: the difference between the motion intensity of the coded block in the following frame and the motion intensity of the coded block in the video frame, divided by the difference between the display timestamp of the following frame and the display timestamp of the video frame. If the video frame only has preceding frames, the display order difference component is: the difference between the motion intensity of the coded block in the video frame and the motion intensity of the coded block in the preceding frame, divided by the difference between the display timestamp of the video frame and the display timestamp of the preceding frame.

[0042] For each video frame, determine the preceding and following decoded frames in the frame sequence of the decoding order. For each video frame and all coded blocks contained in the video frame, calculate the decoding order difference component of the coded blocks. If the video frame has both preceding and following decoded frames, the decoding order difference component is: the difference between the motion intensity of the coded block in the following decoded frame and the motion intensity of the coded block in the preceding decoded frame, divided by the difference between the decoding timestamp of the following decoded frame and the decoding timestamp of the preceding decoded frame. If the video frame only has following decoded frames, the decoding order difference component is: the difference between the motion intensity of the coded block in the following decoded frame and the motion intensity of the coded block in the video frame, divided by the difference between the decoding timestamp of the following decoded frame and the decoding timestamp of the video frame. If the video frame only has preceding decoded frames, the decoding order difference component is: the difference between the motion intensity of the coded block in the video frame and the motion intensity of the coded block in the preceding decoded frame, divided by the difference between the decoding timestamp of the video frame and the decoding timestamp of the preceding decoded frame.

[0043] For each video frame and all the coded blocks contained in that video frame, calculate the local offset of that coded block. The calculation method is: the decoding order difference of the coded block minus the display order difference of that coded block.

[0044] In detail, the display order difference component of the coded block is a parameter that quantizes the rate of change of the motion intensity of the coded block with display time in the display order frame sequence. The display order difference component of the coded block reflects the changing trend of the motion intensity of the coded block on the display time axis.

[0045] In detail, the decoding order difference component of the coded block is a parameter that quantizes the rate of change of the motion intensity of the coded block with the decoding time in the decoding order frame sequence. The decoding order difference component of the coded block reflects the changing trend of the motion intensity of the coded block on the display time axis.

[0046] In detail, the local offset of a coded block is a parameter that quantizes the difference in the rate of change of motion intensity of the coded block on the decoding timeline and the display timeline. The role of the local offset of the coded block is to highlight the temporal offset of motion cues caused by video compression (such as B-frame bidirectional reference, GOP structure). When there is no offset, the rate of change of motion intensity is consistent in the decoding and display order, and the local offset approaches zero; when there is an offset, the local offset will show a non-zero value, and the greater the degree of offset, the larger the absolute value of the local offset.

[0047] In one embodiment of the present invention, the direction of motion is determined based on the motion vector, and the direction sign is obtained by combining the relative relationship between the direction of motion and the direction of gravity, including:

[0048] For each video frame and all coded blocks contained in that video frame, obtain the horizontal motion vector component, vertical motion vector component, and motion intensity of that coded block;

[0049] For each coded block, calculate the horizontal and vertical components of the unit motion direction vector of the coded block. The horizontal component of the unit motion direction vector is the horizontal motion vector component of the coded block divided by the sum of the motion intensity of the coded block and the minimum positive number. The vertical component of the unit motion direction vector is the vertical motion vector component of the coded block divided by the sum of the motion intensity of the coded block and the minimum positive number. The value of the minimum positive number is greater than zero and less than one.

[0050] When the motion intensity of the coded block is zero, the unit motion direction vector of the coded block is a vector with both the horizontal and vertical components being zero; when the motion intensity of the coded block is not zero, the unit motion direction vector of the coded block is a vector with a length not exceeding one.

[0051] For each video frame and all the coded blocks contained in that video frame, obtain the vertical component of the unit motion direction vector of that coded block;

[0052] Using a vector with zero horizontal component and one vertical component as the gravity direction vector, calculate the cosine of the angle between the unit motion direction vector of the coded block and the gravity direction vector. The cosine of the angle is equal to the vertical component of the unit motion direction vector of the coded block.

[0053] For each coded block, determine the direction sign. When the cosine of the included angle is greater than or equal to zero, the direction sign is positive one; when the cosine of the included angle is less than zero, the direction sign is negative one.

[0054] In detail, the horizontal component of the unit motion direction vector is a standardized parameter used to quantify the horizontal motion trend of the coded block. The smallest positive value is greater than zero and less than one, its purpose being to avoid division by zero when the motion intensity of the coded block is zero. When the motion intensity of the coded block is zero, this component takes the value of zero; when the motion intensity is not zero, this component is standardized by its ratio to the motion intensity, ensuring that the length of the unit motion direction vector does not exceed one, thus retaining only the horizontal motion direction information of the coded block and eliminating interference from motion amplitude on direction determination.

[0055] In detail, the vertical component of the unit motion direction vector is a parameter used to quantify the vertical motion trend of the coded block after standardization. When the motion intensity of the coded block is zero, this component takes the value of zero; when the motion intensity is not zero, this component is also standardized to ensure that the length of the unit motion direction vector does not exceed one, retaining only the motion direction information of the coded block in the vertical direction.

[0056] In detail, the direction sign is a binary parameter used to determine the relative relationship between the direction of motion of the coded block and the direction of gravity. Its determination process uses a vector with a horizontal component of zero and a vertical component of one as the gravity direction vector. The cosine of the angle between the unit motion direction vector of the coded block and this gravity direction vector is equal to the vertical component of the unit motion direction vector. When the cosine of this angle is greater than or equal to zero, it indicates that the direction of motion of the coded block is in the same direction as the direction of gravity, and the direction sign is positive (1). When the cosine of the angle is less than zero, it indicates that the direction of motion of the coded block is in the opposite direction to the direction of gravity, and the direction sign is negative (1).

[0057] In one embodiment of the present invention, the spatial vector moment between the spatial location of the coding block and the center location of the video frame is calculated; the local offset, direction sign, and spatial vector moment are multiplied to form a frame-level directed misalignment moment map, including:

[0058] For each video frame and all the coded blocks contained in that video frame, obtain the horizontal and vertical coordinates of the center of the coded block, as well as the horizontal and vertical coordinates of the center of the video frame.

[0059] For each coding block, calculate the horizontal and vertical components of the relative position vector of the coding block. The horizontal component of the relative position vector is the center horizontal coordinate of the coding block minus the center horizontal coordinate of the video frame, and the vertical component of the relative position vector is the center vertical coordinate of the coding block minus the center vertical coordinate of the video frame.

[0060] For each coded block, obtain the horizontal and vertical components of the unit motion direction vector of that coded block;

[0061] For each coded block, the spatial vector moment of that coded block is calculated. The spatial vector moment is the horizontal component of the relative position vector multiplied by the vertical component of the unit motion direction vector, minus the vertical component of the relative position vector multiplied by the horizontal component of the unit motion direction vector.

[0062] For each video frame and all the coded blocks contained in that video frame, obtain the local offset, direction sign, and space vector moment of that coded block;

[0063] For each coded block, calculate the torque value of the coded block. The torque value is the local offset of the coded block multiplied by the direction sign and then multiplied by the space vector moment.

[0064] Define a block field indicator function. When a pixel belongs to the region of the coded block in the video frame, the block field indicator function takes a value of one, and when a pixel does not belong to the region of the coded block in the video frame, the block field indicator function takes a value of zero.

[0065] For each video frame, a frame-level directed mismatch torque map is constructed. The value of each pixel in the frame-level directed mismatch torque map is the sum of the product of the torque value of the coding block to which the pixel belongs and the block domain indicator function.

[0066] In detail, the spatial vector moment of a coded block is a parameter that quantifies the relationship between the motion of the coded block and its spatial position within a video frame. The spatial vector moment of a coded block reflects the spatial influence of the block's motion relative to the frame center, and its magnitude is related to the distance of the coded block from the frame center and the direction of motion.

[0067] In detail, the torque value of the encoded block is a core quantitative indicator that integrates the temporal offset, directional attributes, and spatial correlation of the encoded block. It is calculated by multiplying the local offset of the encoded block (reflecting the difference in the rate of change of motion intensity between the decoding and display time axes) by the directional sign (reflecting the relative relationship between motion and the direction of gravity), and then multiplying by the spatial vector moment (reflecting the correlation between motion and spatial position). The role of the torque value of the encoded block is to integrate the degree of temporal offset, directional correlation, and spatial correlation into a single scalar. The larger the absolute value, the higher the degree of directional mismatch caused by compression in the encoded block; the positive or negative value corresponds to the directional attribute of the mismatch.

[0068] In detail, the block domain indicator function is a binary identifier function that establishes the correspondence between coded blocks and video frame pixels. The rule is that when a pixel belongs to the coverage area of ​​a coded block within a video frame, the function takes a value of one; when a pixel does not belong to the coverage area of ​​that coded block, the function takes a value of zero. The function's purpose is to map coded block-level torque values ​​to pixel-level data. Since the coded block is the basic processing unit of video coding (not pixel-level), this function can allocate the torque value of each coded block to all the pixels it covers, ensuring that each pixel in the subsequently constructed frame-level feature map is only associated with the torque information of its own coded block, avoiding cross-block data interference and ensuring the spatial accuracy of the feature map.

[0069] In detail, the frame-level directed mismatch torque map of a video frame is the result of converting the directed mismatch information of all coded blocks within the video frame into a pixel-level spatial feature map. It is constructed by calculating the sum of the products of the torque value of the coded block to which each pixel belongs and the block domain indicator function for each pixel within the video frame (since each pixel belongs to only one coded block, the actual value is the torque value of that coded block). The function of the frame-level directed mismatch torque map of a video frame is to: integrate scattered coded block-level data into continuous spatial features within the frame, intuitively presenting the distribution pattern of mismatch caused by compression within the frame; and form a spatial feature carrier that can be directly input into a convolutional neural network.

[0070] It should be noted that directed mismatch information is a comprehensive information set formed by integrating three dimensions—temporal mismatch, directional attributes, and spatial correlation—to address motion cue deviations caused by video compression. It is used to capture the directional misalignment features between motion cues and real physical processes in compressed videos.

[0071] In one embodiment of the present invention, frame-level directed mismatch torque maps of multiple consecutive frames are stacked in chronological order to form three-dimensional feature data, which is used as input to a convolutional neural network; the convolutional neural network outputs time anchor points, including:

[0072] Select multiple consecutive frames from the frame sequence in the display order as the target frame sequence, and the display timestamps of the target frame sequence are sequentially increased;

[0073] According to the display time order of the target frame sequence, the frame-level directed mismatch torque maps of each frame are stacked into three-dimensional feature data. The value of the position in the three-dimensional feature data, which is determined by the horizontal pixel coordinate, the vertical pixel coordinate, and the frame number, is equal to the value of the pixel corresponding to the horizontal pixel coordinate and the vertical pixel coordinate on the frame-level directed mismatch torque map of the frame corresponding to the frame number in the target frame sequence. The frame number is the arrangement number of the frame in the target frame sequence.

[0074] Set the first convolutional kernel, whose spatial and temporal dimensions are configurable positive integers;

[0075] The three-dimensional feature data is convolved with the first layer convolution kernel, and the result is processed by point-by-point nonlinear transformation to obtain the first hidden feature map.

[0076] One or more convolutional kernels are stacked, and the output of each convolutional kernel is transformed point by point nonlinearly to obtain the hidden feature map of the corresponding layer, until the final hidden feature map is obtained;

[0077] The final hidden feature map is averaged in the horizontal and vertical directions to obtain the time response sequence. The k-th value in the time response sequence is the average value of the final hidden feature map in the horizontal and vertical directions at the position of frame number k.

[0078] Select the frame number corresponding to the position with the largest value in the time response sequence, and use the display timestamp of the frame in the target frame sequence corresponding to that frame number as the time anchor point.

[0079] In detail, since it is necessary to capture the temporal changes of motion cues (such as the continuous process of falling and touching the ground), selecting a sequence of target frames from multiple consecutive frames can ensure that the complete event time segment is covered, while the incremental arrangement based on the display order can ensure that the timing logic is consistent with the actual process.

[0080] In detail, the frame-level directed anomalous torque maps of consecutive frames are stacked into three-dimensional feature data. The purpose is to provide the convolutional neural network with spatial and temporal two-dimensional fused feature input, meeting the network's processing requirements for video spatiotemporal features.

[0081] The frame-level directed mismatch torque map of a single frame only contains spatial dimension information, which can only reflect the spatial distribution of mismatch caused by compression within the frame at a certain moment, and cannot reflect the change of mismatch over time (such as the dynamic migration of mismatch areas during a fall).

[0082] After being stacked in the order of display time, the three-dimensional feature data forms a dimensional structure of horizontal pixels-vertical pixels-frame number, where the frame number corresponds to the time dimension (the increasing frame number means the time progresses), which can completely preserve the spatiotemporal evolution process of heterogeneous information.

[0083] Convolutional neural networks (especially 3D convolutions) need to extract features in the spatiotemporal dimension to capture the dynamic patterns of events (such as the temporal peak of the intensity of the different match when falling and hitting the ground), and three-dimensional feature data provides the carrier of such patterns.

[0084] In detail, the first-layer convolutional kernel is the core operator used for the initial feature extraction of 3D feature data. The first-layer convolutional kernel performs a preliminary abstraction of local spatiotemporal heterogeneous features in the original 3D feature data. For example, if the spatial size is set to 3×3 and the temporal size is set to 3, this convolutional kernel can capture heterogeneous information association patterns within a 3×3 pixel spatial range and a continuous 3-frame temporal range, filtering out local noise in the original data while retaining key spatiotemporal heterogeneous trends.

[0085] In detail, the first hidden feature map is obtained by convolving the 3D feature data with the first convolutional kernel and then processing it through a pointwise nonlinear transformation (such as the ReLU activation function). Its functions are as follows: convolution operations achieve local spatiotemporal feature aggregation of the original 3D feature data, while the pointwise nonlinear transformation introduces nonlinear expressive power into the features, capturing complex spatiotemporal heterogeneous patterns in the original data that cannot be represented by linear operations (such as nonlinear abrupt changes in heterogeneous intensity, corresponding to the instant of falling and hitting the ground); this feature map still retains the 3D structure of horizontal pixels-vertical pixels-frame number, directly reflecting the feature extraction results of the first convolutional kernel, and also serving as the input for subsequent stacked convolutional layers, supporting the gradual deepening of feature abstraction.

[0086] In detail, the final hidden feature map is the result of feature extraction obtained by overlaying one or more convolutional kernels (each layer undergoing convolution operations and pointwise nonlinear transformations) on top of the first-layer hidden feature map. Its function is:

[0087] After multiple rounds of convolution and nonlinear transformation, the feature map has completed a deep abstraction of the original three-dimensional feature data, filtered out a large amount of redundant local noise, and focused on key spatiotemporal heterogeneous features that are strongly correlated with the target event (falling and touching the ground) (such as the spatiotemporal region corresponding to the temporal peak of heterogeneous intensity).

[0088] The feature map retains its horizontal pixel-vertical pixel-frame number dimensional structure, but its feature values ​​are no longer the original heterogeneous data. Instead, they are high-dimensional abstract features after multi-layer filtering and aggregation, providing high-quality feature basis for subsequent generation of time response sequences and location of time anchor points.

[0089] In detail, the time response sequence is a sequence of data containing only the time dimension, obtained after globally averaging the final latent feature map in both the horizontal and vertical directions. Its functions are as follows:

[0090] The core purpose of averaging is to eliminate the differences in spatial dimensions (horizontal and vertical pixels) and extract the global heterogeneous feature intensity of each frame in the final latent feature map. That is, to average all the horizontal and vertical pixel feature values ​​of frame number k in the final latent feature map to obtain the kth value in the sequence, which represents the global heterogeneous feature intensity of the kth frame.

[0091] Each value in the sequence corresponds one-to-one with the frame number of the target frame sequence, reflecting only the trend of the heteromatch feature intensity over time, and can intuitively present the temporal peak of the heteromatch intensity when the target event (such as falling and touching the ground) occurs.

[0092] In detail, the objective of this invention is to locate high-acceleration events (falling and hitting the ground) in compressed videos. Due to the intense motion and instantaneous time of these events, the directed heterogeneous information (a fusion of temporal misalignment, directional correlation, and spatial correlation information) generated by video compression reaches its strongest at the moment of the event. This is reflected in the final hidden feature map, where the global heterogeneous feature strength is the greatest at that moment, corresponding to the maximum value in the time response sequence. The position with the largest value in the time response sequence corresponds to the frame number, which is the frame with the strongest directed heterogeneous information in the target frame sequence. The display timestamp of this frame is the key moment when the target event occurs. Using this display timestamp as a time anchor point, the moment of occurrence of the target event can be locked.

[0093] It should be noted that the pre-training process of the convolutional neural network corresponding to this invention needs to be based on a dataset that covers various video compression scenarios (including different GOP structures, bitrates, and B-frame configurations) and displays timestamps at key moments of high acceleration events (such as falling and touching the ground). First, multiple consecutive frames are selected from the display sequence of each sample video in the dataset, and frame-level directed heterogeneous torque maps of each frame are generated and stacked in chronological order to form three-dimensional feature data, which is used as the pre-training input. The network adopts a 3D convolutional kernel stacking structure with configurable spatial / temporal dimensions. After the first layer of convolutional kernels performs convolution operations on the three-dimensional feature data, a point-by-point nonlinear transformation is performed to obtain the first hidden feature map. The final hidden feature map is obtained by stacking multiple convolutional kernels and performing point-by-point nonlinear transformation on the output of each layer. Then, the final hidden feature map is globally averaged in the horizontal and vertical directions to generate a time response sequence. The pre-training task is set as the time anchor prediction task, that is, the network learns the frame number corresponding to the key moment display timestamp of the output and the sample annotation. The regression loss function (such as mean square error) is used to measure the difference between the frame number predicted by the network and the labeled frame number. The network parameters are iteratively optimized through backpropagation, so that the network can learn the correlation between directed heterogeneous information and key moments of high acceleration events under different compression scenarios, and has the generalization ability to locate time anchors from three-dimensional feature data.

[0094] In one embodiment of the present invention, determining the review window based on a time anchor point, a preset pre- and post-extension duration, and a frame rate includes:

[0095] Obtain the system's preset lead time and follow time;

[0096] The start time is calculated as the larger of the difference between the time anchor point and the preceding delay duration and the display timestamp of the earliest frame in the frame sequence of the display order.

[0097] Calculate the end time, which is the smaller of the sum of the time anchor point and the delay duration, and the display timestamp of the latest frame in the frame sequence of the display order;

[0098] Calculate the forward frame count and the backward frame count. The forward frame count is the result of multiplying the forward delay by the frame rate and rounding up. The backward frame count is the result of multiplying the backward delay by the frame rate and rounding up.

[0099] Among all video frames whose display timestamp is greater than or equal to the start time, the video frame with the smallest display timestamp is selected as the starting frame of the review window;

[0100] Among all video frames whose display timestamp is less than or equal to the end time, select the video frame with the largest display timestamp as the end frame of the review window;

[0101] The time interval of the review window is a closed interval from the start time to the end time;

[0102] The frame set in the review window consists of all video frames whose timestamps fall within the time interval of the review window.

[0103] The frame count in the review window is the number of video frames contained in the frame set of the review window.

[0104] In detail, the system's preset pre-delay and post-delay durations are pre-configured time parameters used to ensure that the review window fully covers the entire stage of a high-acceleration event (fall). The pre-delay duration corresponds to the pre-movement stage before the event occurs (such as changes in body posture before falling), and the post-delay duration corresponds to the result stage after the event occurs (such as the stationary state after hitting the ground).

[0105] In detail, the review window is used to select video clips that contain the entire phase of the high acceleration event.

[0106] In one embodiment of the present invention, the original video segment corresponding to the review window is extracted, and visualization information related to the frame-level directed misalignment torque diagram is generated. The visualization information is then marked as manually reviewed, including:

[0107] Retrieve the frame set and the frame sequence in the display order of the review window;

[0108] From the frame set in the review window, video frames are selected in ascending order of display timestamps corresponding to the frame sequence to form an ordered frame sequence. In the ordered frame sequence, the display timestamp of the previous frame is less than the display timestamp of the next frame.

[0109] The original video clip consists of all the video frames in the ordered frame sequence. The start time of the original video clip is the display timestamp of the first frame in the ordered frame sequence, and the end time of the original video clip is the display timestamp of the last frame in the ordered frame sequence.

[0110] Obtain the frame-level directional mismatch torque map and set the system's preset amplification factor, which is a constant greater than zero.

[0111] Construct a visualization mask. The value of each pixel on the visualization mask is calculated according to the following steps: calculate the magnification factor multiplied by the absolute value of the corresponding pixel value on the frame-level directed aberration torque map to obtain the first intermediate value; take the larger value between the first intermediate value and zero to obtain the second intermediate value; take the smaller value between the second intermediate value and one, and this value is the value of the corresponding pixel on the visualization mask.

[0112] Obtain the red, green, and blue channel values ​​of each video frame in the original video clip. The values ​​of the red, green, and blue channels are all in the range of zero to one. Generate a grayscale image of each video frame. The value of each pixel in the grayscale image is: 0.299 multiplied by the corresponding red channel value, plus 0.587 multiplied by the corresponding green channel value, plus 0.114 multiplied by the corresponding blue channel value.

[0113] Set the system's preset overlay coefficient, which is a constant ranging from zero to one, and generate an overlay image of each video frame; the value of each pixel in the overlay image is: (one minus the overlay coefficient) multiplied by the value of the corresponding pixel in the grayscale image, plus the overlay coefficient multiplied by the value of the corresponding pixel in the visualization mask.

[0114] Arrange the superimposed images corresponding to each video frame in the order of the display time of the ordered frame sequence to form a visual sequence;

[0115] Obtain the defined time interval of the review window and generate annotation units. Each annotation unit contains the original video clip, the visualization sequence, and the time interval of the review window.

[0116] Set an audit mark field, the values ​​of which include manual audit and non-manual audit;

[0117] Set the audit mark field of the annotation unit to manual audit.

[0118] In detail, the frame set in the review window refers to the collection of all video frames whose display timestamps fall within the review window's time interval. Its function is to filter out video frames related to the event to be reviewed (such as falling and hitting the ground) from the complete sequence of displayed frames.

[0119] In detail, the display order frame sequence refers to the frame sequence formed by arranging all video frames contained in the encoded video stream in non-descending order according to the display timestamp (PTS) of each video frame, and is used to provide a time reference.

[0120] In detail, an ordered frame sequence is a sequence of video frames selected from the frame set in the review window in ascending order of the display timestamps corresponding to the frame sequence.

[0121] In detail, the original video clip is a video clip composed of all video frames in an ordered frame sequence. Its start time is the display timestamp of the first frame in the ordered frame sequence, and its end time is the display timestamp of the last frame in the ordered frame sequence. It is used to provide the original visual material for review.

[0122] In detail, the visualization mask is an image constructed based on a frame-level directed mismatch torque map, with pixel values ​​ranging from 0 to 1. Its construction logic is as follows: first, calculate the product of the magnification factor (a constant greater than zero) and the absolute value of the torque map pixel value (the first intermediate value); then, take the larger value between the first intermediate value and zero (the second intermediate value); finally, take the smaller value between the second intermediate value and one as the mask pixel value. The visualization mask is used to convert the directed mismatch information in the torque map into a visually representable grayscale distribution. That is, the higher the degree of mismatch, the closer the mask pixel value is to 1 (brighter), and vice versa.

[0123] In detail, the overlay image is an image obtained by proportionally mixing a grayscale image with a visualization mask. The grayscale image is generated from the red, green, and blue channel values ​​(values ​​0-1) of the original video frame using the formula: 0.299 × red channel value + 0.587 × green channel value + 0.114 × blue channel value. The mixing logic is (1 - overlay coefficient) × grayscale image pixel value + overlay coefficient × visualization mask pixel value (the overlay coefficient is a constant between 0 and 1). The purpose of the overlay image is to balance the original image with the discrepancy information. It preserves the content of the original video frame (facilitating manual scene identification) while highlighting discrepancy areas through mask overlay (indicating motion cue deviations caused by compression), providing a direct basis for manual review.

[0124] In detail, the overlay coefficient is a system-preset constant ranging from 0 to 1. Its core function is to control the blending ratio of the grayscale image and the visualization mask in the overlay image: when the coefficient approaches 1, the visualization mask's display effect is stronger (the mismatched area is more prominent), and the original image is paler; when the coefficient approaches 0, the original image's display effect is stronger, and the mask is paler. By adjusting this coefficient, the clarity of the original image and the recognizability of mismatched information can be balanced, adapting to the visual needs of different review scenarios.

[0125] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A video content review system based on a large model, characterized in that, include: The data acquisition unit acquires the correspondence between the display order and decoding order of the encoded video stream, and extracts motion vectors, spatial positions of encoded blocks, center positions of video frames, and frame rates. The local offset calculation unit performs time-domain difference operations on the motion intensity based on the display order and decoding order to calculate the local offset; wherein, the motion intensity is determined based on the motion vector. The torque map generation unit determines the motion direction based on the motion vector and obtains the direction sign by combining the relative relationship between the motion direction and the gravity direction; it calculates the spatial vector moment between the spatial position of the coded block and the center position of the video frame; and it multiplies the local offset, direction sign and spatial vector moment to form a frame-level directed mismatch torque map. The feature processing unit stacks frame-level directed mismatch torque maps of multiple consecutive frames in chronological order into three-dimensional feature data, which is used as input to a convolutional neural network; the convolutional neural network outputs time anchor points. The review execution unit determines the review window based on the time anchor point, the preset before and after extension duration, and the frame rate; it extracts the original video segment corresponding to the review window, generates visualization information related to the frame-level directed mismatch torque diagram, and marks the visualization information as manual review.

2. The video content review system based on a large model according to claim 1, characterized in that, Obtain the correspondence between the display order and decoding order of the encoded video stream, and extract motion vectors, spatial positions of encoded blocks, center positions of video frames, and frame rates, including: For all video frames contained in the encoded video stream, arrange them in non-descending order according to the display timestamp of each video frame to obtain the frame sequence of display order; For all video frames contained in the encoded video stream, sort them in non-descending order according to the decoding timestamp of each video frame to obtain the frame sequence of decoding order; Read the frame width and frame height of each video frame; For each video frame and all the coded blocks contained in that video frame, read the horizontal motion vector component and the vertical motion vector component of each coded block; For each coded block, the motion intensity of that coded block is calculated by taking the square root of the sum of the squares of the horizontal motion vector components and the squares of the vertical motion vector components. Read the top-left horizontal coordinate, top-left vertical coordinate, block width, and block height of each encoded block; For each coding block, calculate the horizontal and vertical coordinates of the center of the coding block. The horizontal coordinate of the center is the horizontal coordinate of the top left corner of the coding block plus half the width of the block, and the vertical coordinate of the center is the vertical coordinate of the top left corner of the coding block plus half the height of the block. For each video frame, based on the frame width and frame height in the frame-level recording unit of the video frame, calculate the center horizontal coordinate and center vertical coordinate of the video frame. The center horizontal coordinate of the video frame is half of the frame width, and the center vertical coordinate of the video frame is half of the frame height. Determine the first and last frame display timestamps of all video frames contained in the encoded video stream, and count the total number of frames in all video frames; The frame rate is calculated as follows: (total number of frames minus one) divided by (last frame display timestamp minus first frame display timestamp).

3. A video content review system based on a large model according to claim 2, characterized in that, Based on the display order and decoding order, time-domain difference operations are performed on the motion intensity to calculate the local offset, including: For each video frame, determine the preceding and following frames in the frame sequence of the display order. For each video frame and all coded blocks contained in the video frame, calculate the display order difference component of the coded blocks. If the video frame has both preceding and following frames, the display order difference component is: the difference between the motion intensity of the coded block in the following frame and the motion intensity of the coded block in the preceding frame, divided by the difference between the display timestamp of the following frame and the display timestamp of the preceding frame. If the video frame only has following frames, the display order difference component is: the difference between the motion intensity of the coded block in the following frame and the motion intensity of the coded block in the video frame, divided by the difference between the display timestamp of the following frame and the display timestamp of the video frame. If the video frame only has preceding frames, the display order difference component is: the difference between the motion intensity of the coded block in the video frame and the motion intensity of the coded block in the preceding frame, divided by the difference between the display timestamp of the video frame and the display timestamp of the preceding frame. For each video frame, determine the preceding and following decoded frames in the frame sequence of the decoding order. For each video frame and all coded blocks contained in the video frame, calculate the decoding order difference component of the coded blocks. If the video frame has both preceding and following decoded frames, the decoding order difference component is: the difference between the motion intensity of the coded block in the following decoded frame and the motion intensity of the coded block in the preceding decoded frame, divided by the difference between the decoding timestamp of the following decoded frame and the decoding timestamp of the preceding decoded frame. If the video frame only has following decoded frames, the decoding order difference component is: the difference between the motion intensity of the coded block in the following decoded frame and the motion intensity of the coded block in the video frame, divided by the difference between the decoding timestamp of the following decoded frame and the decoding timestamp of the video frame. If the video frame only has preceding decoded frames, the decoding order difference component is: the difference between the motion intensity of the coded block in the video frame and the motion intensity of the coded block in the preceding decoded frame, divided by the difference between the decoding timestamp of the video frame and the decoding timestamp of the preceding decoded frame. For each video frame and all the coded blocks contained in that video frame, calculate the local offset of that coded block. The calculation method is: the decoding order difference of the coded block minus the display order difference of that coded block.

4. A video content review system based on a large model according to claim 3, characterized in that, The direction of motion is determined based on the motion vector, and the direction sign is obtained by combining the relative relationship between the direction of motion and the direction of gravity, including: For each video frame and all coded blocks contained in that video frame, obtain the horizontal motion vector component, vertical motion vector component, and motion intensity of that coded block; For each coded block, calculate the horizontal and vertical components of the unit motion direction vector of the coded block. The horizontal component of the unit motion direction vector is the horizontal motion vector component of the coded block divided by the sum of the motion intensity of the coded block and the minimum positive number. The vertical component of the unit motion direction vector is the vertical motion vector component of the coded block divided by the sum of the motion intensity of the coded block and the minimum positive number. The value of the minimum positive number is greater than zero and less than one. When the motion intensity of the coded block is zero, the unit motion direction vector of the coded block is a vector with both the horizontal and vertical components being zero; when the motion intensity of the coded block is not zero, the unit motion direction vector of the coded block is a vector with a length not exceeding one. For each video frame and all the coded blocks contained in that video frame, obtain the vertical component of the unit motion direction vector of that coded block; Using a vector with zero horizontal component and one vertical component as the gravity direction vector, calculate the cosine of the angle between the unit motion direction vector of the coded block and the gravity direction vector. The cosine of the angle is equal to the vertical component of the unit motion direction vector of the coded block. For each coded block, determine the direction sign. When the cosine of the included angle is greater than or equal to zero, the direction sign is positive one; when the cosine of the included angle is less than zero, the direction sign is negative one.

5. A video content review system based on a large model according to claim 4, characterized in that, Calculate the spatial vector moment between the spatial location of the coded block and the center location of the video frame; multiply the local offset, direction sign, and spatial vector moment to form a frame-level directed misalignment moment map, including: For each video frame and all the coded blocks contained in that video frame, obtain the center horizontal coordinates and center vertical coordinates of the coded block, as well as the center horizontal coordinates and center vertical coordinates of the video frame; For each coding block, calculate the horizontal and vertical components of the relative position vector of the coding block. The horizontal component of the relative position vector is the center horizontal coordinate of the coding block minus the center horizontal coordinate of the video frame, and the vertical component of the relative position vector is the center vertical coordinate of the coding block minus the center vertical coordinate of the video frame. For each coded block, obtain the horizontal and vertical components of the unit motion direction vector of that coded block; For each coded block, the spatial vector moment of that coded block is calculated. The spatial vector moment is the horizontal component of the relative position vector multiplied by the vertical component of the unit motion direction vector, minus the vertical component of the relative position vector multiplied by the horizontal component of the unit motion direction vector. For each video frame and all the coded blocks contained in that video frame, obtain the local offset, direction sign, and spatial vector moment of that coded block; For each coded block, calculate the torque value of the coded block. The torque value is the local offset of the coded block multiplied by the direction sign and then multiplied by the space vector moment. Define a block field indicator function. When a pixel belongs to the region of the coded block in the video frame, the block field indicator function takes a value of one, and when a pixel does not belong to the region of the coded block in the video frame, the block field indicator function takes a value of zero. For each video frame, a frame-level directed mismatch torque map is constructed. The value of each pixel in the frame-level directed mismatch torque map is the sum of the product of the torque value of the coding block to which the pixel belongs and the block domain indicator function.

6. A video content review system based on a large model according to claim 5, characterized in that, Frame-level directed mismatch torque maps from multiple consecutive frames are stacked in chronological order to form three-dimensional feature data, which serves as the input to a convolutional neural network. The convolutional neural network outputs time anchor points, including: Select multiple consecutive frames from the frame sequence in the display order as the target frame sequence, and the display timestamps of the target frame sequence are sequentially increased; According to the display time order of the target frame sequence, the frame-level directed mismatch torque maps of each frame are stacked into three-dimensional feature data. The value of the position in the three-dimensional feature data, which is determined by the horizontal pixel coordinate, the vertical pixel coordinate, and the frame number, is equal to the value of the pixel corresponding to the horizontal pixel coordinate and the vertical pixel coordinate on the frame-level directed mismatch torque map of the frame corresponding to the frame number in the target frame sequence. The frame number is the arrangement number of the frame in the target frame sequence. Set the first convolutional kernel, whose spatial and temporal dimensions are configurable positive integers; The three-dimensional feature data is convolved with the first layer convolution kernel, and the result is processed by point-by-point nonlinear transformation to obtain the first hidden feature map. One or more convolutional kernels are stacked, and the output of each convolutional kernel is transformed point by point nonlinearly to obtain the hidden feature map of the corresponding layer, until the final hidden feature map is obtained; The final hidden feature map is averaged in the horizontal and vertical directions to obtain the time response sequence. The k-th value in the time response sequence is the average value of the final hidden feature map in the horizontal and vertical directions at the position of frame number k. Select the frame number corresponding to the position with the largest value in the time response sequence, and use the display timestamp of the frame in the target frame sequence corresponding to that frame number as the time anchor point.

7. A video content review system based on a large model according to claim 6, characterized in that, The review window is determined based on the time anchor point, the preset before and after extension duration, and the frame rate, including: Obtain the system's preset lead time and follow time; The start time is calculated as the larger of the difference between the time anchor point and the preceding delay duration and the display timestamp of the earliest frame in the frame sequence of the display order. Calculate the end time, which is the smaller of the sum of the time anchor point and the delay duration, and the display timestamp of the latest frame in the frame sequence of the display order; Calculate the forward frame count and the backward frame count. The forward frame count is the result of multiplying the forward delay by the frame rate and rounding up. The backward frame count is the result of multiplying the backward delay by the frame rate and rounding up. Among all video frames whose display timestamp is greater than or equal to the start time, the video frame with the smallest display timestamp is selected as the starting frame of the review window; Among all video frames whose display timestamp is less than or equal to the end time, select the video frame with the largest display timestamp as the end frame of the review window; The time interval of the review window is a closed interval from the start time to the end time; The frame set in the review window consists of all video frames whose timestamps fall within the time interval of the review window. The frame count in the review window is the number of video frames contained in the frame set of the review window.

8. A video content review system based on a large model according to claim 7, characterized in that, Extract the original video segment corresponding to the review window, generate visualization information related to the frame-level directed misalignment torque diagram, and mark the visualization information as manually reviewed, including: Retrieve the frame set and the frame sequence in the display order of the review window; From the frame set in the review window, video frames are selected in ascending order of the display timestamps corresponding to the frame sequence to form an ordered frame sequence. In the ordered frame sequence, the display timestamp of the previous frame is less than the display timestamp of the next frame. The original video clip consists of all the video frames in the ordered frame sequence. The start time of the original video clip is the display timestamp of the first frame in the ordered frame sequence, and the end time of the original video clip is the display timestamp of the last frame in the ordered frame sequence. Obtain the frame-level directional mismatch torque map and set the system's preset amplification factor, which is a constant greater than zero. Construct a visualization mask. The value of each pixel on the visualization mask is calculated according to the following steps: calculate the magnification factor multiplied by the absolute value of the corresponding pixel value on the frame-level directed aberration torque map to obtain the first intermediate value; take the larger value between the first intermediate value and zero to obtain the second intermediate value; take the smaller value between the second intermediate value and one, and this value is the value of the corresponding pixel on the visualization mask. Obtain the red, green, and blue channel values ​​of each video frame in the original video clip. The values ​​of the red, green, and blue channels are all in the range of zero to one. Generate a grayscale image of each video frame. The value of each pixel in the grayscale image is: 0.299 multiplied by the corresponding red channel value, plus 0.587 multiplied by the corresponding green channel value, plus 0.114 multiplied by the corresponding blue channel value. Set the system's preset overlay coefficient, which is a constant ranging from zero to one, and generate an overlay image of each video frame; the value of each pixel in the overlay image is: (one minus the overlay coefficient) multiplied by the value of the corresponding pixel in the grayscale image, plus the overlay coefficient multiplied by the value of the corresponding pixel in the visualization mask. Arrange the superimposed images corresponding to each video frame in the order of the display time of the ordered frame sequence to form a visual sequence; Obtain the defined time interval of the review window and generate annotation units. Each annotation unit contains the original video clip, the visualization sequence, and the time interval of the review window. Set an audit mark field, the values ​​of which include manual audit and non-manual audit; Set the audit mark field of the annotation unit to manual audit.

Images