Model training method, video classification method, electronic device, storage medium and program product

Abstract

The invention discloses a model training method, a video classification method, electronic equipment, a storage medium and a program product, and is applied to the technical field of video processing. The training method comprises the following steps: acquiring a foreground feature set and background features of a first video clip, and training a classification model based on the foreground feature set and the background features; wherein in the training process of the classification model, the foreground feature set and the background feature are subjected to cross fusion to obtain a first fusion feature, and the first fusion feature is used for the classification model to classify the first video clip. Therefore, the game video clips can be finely classified through the trained classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a model training method, a video classification method, an electronic device, a storage medium and a program product. BACKGROUND Sports events, as one of the most influential cultural phenomena in the world, especially the sports events of ball games, attract a large number of audiences with their competitiveness and unpredictability. With the development of network technology and mobile communication technology, the viewing model of sports events has shifted from traditional television viewing to digital network video platform viewing. In order to seize users, major video platforms not only provide live and playback of sports events, but also provide users with highlights of sports events, such as goal moments and key saves of ball game sports events, so as to attract users through the most exciting highlights in the game. Such "highlights" greatly cater to the viewing habits of some users and have become an effective solution to improve user experience.

[0002] In order to extract highlights from game videos, video classification, video extraction and other video research directions have become the focus of video platforms. However, the current highlight recognition scheme for ball games has limited accuracy and cannot meet the needs of users. SUMMARY

[0003] To solve the related technical problems, the embodiments of the present application provide a model training method, a video classification method, an electronic device, a storage medium and a program product.

[0004] The technical scheme of the embodiments of the present application is as follows: The embodiments of the present application provide a model training method, comprising: obtaining a foreground feature set and a background feature of a first video segment, wherein the first video segment is a video segment in a training sample video, and the training video includes a game video of a ball game; the first video segment includes a plurality of first video frames, the foreground feature set includes a first feature, a second feature and a third feature, the first feature is obtained based on a ball in the first video segment, the second feature is obtained based on an athlete in the first video segment, and the third feature is obtained based on the number of frames of the plurality of first video frames; training a classification model based on the foreground feature set and the background feature; wherein the foreground feature set and the background feature are cross-fused to obtain a first fusion feature during the training of the classification model, and the first fusion feature is used for the classification of the first video segment by the classification model.

[0005] In the above scheme, the foreground feature set of the first video segment is obtained, comprising: For any one of the plurality of first video frames, determine the first position information of the sphere and the second position information of the athlete; The first location information, the second location information, and the frame number of the first video frame are embedded respectively to obtain the first feature corresponding to the first location information, the second feature corresponding to the second location information, and the third feature of the first video frame. The foreground feature set is obtained by combining the first feature, second feature, and third feature of each of the multiple first video frames.

[0006] In the above scheme, determining the first position information of the sphere includes: obtaining the first mask of the sphere in the first video frame; determining the minimum bounding rectangle of the sphere based on the first mask; and determining the first position information of the sphere according to the image position of the minimum bounding rectangle in the first video frame. And / or, Determining the athlete's second position information includes: obtaining a second mask of the athlete in the first video frame; and obtaining an image of the athlete from the first video frame based on the second mask. Key point detection is performed on the athlete's image to obtain the second location information, wherein the second location information includes the location information of multiple key points of the athlete.

[0007] In the above scheme, the step of cross-fusing the foreground feature set and the background features to obtain the first fused feature includes: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the first self-attention feature; The classification model is used to cross-fuse the background features and the first self-attention features to obtain the first fused feature.

[0008] In the above scheme, the classification model includes an encoder; the step of performing self-attention processing based on the foreground feature set using the classification model to obtain the first self-attention feature includes: The first feature, the second feature, and the third feature in the foreground feature set are spliced ​​together to obtain the first spliced ​​feature after splicing. The first splicing feature is input into the encoder, and self-attention processing is performed using the encoder to obtain the first self-attention feature output by the encoder.

[0009] In the above scheme, the classification model includes a decoder; the classification model is used to cross-fuse the background features and the first self-attention features to obtain the first fused features, including: The background features and the first self-attention features are input into the decoder, and cross-fusion is performed using the decoder to obtain the first fused feature output by the decoder.

[0010] This application also provides a video classification method, including: Obtain the foreground feature set and background features of the second video segment, wherein the second video segment includes video segments from a ball game competition video, the second video segment includes multiple second video frames, and the foreground feature set of the second video segment includes a fourth feature, a fifth feature, and a sixth feature, wherein the fourth feature is obtained based on the ball in the second video segment, the fifth feature is obtained based on the athlete in the second video segment, and the sixth feature is obtained based on the frame number of the multiple second video frames; Based on the foreground feature set and the background features, the second video segment is classified using a classification model; wherein, during the classification process of the second video segment using the classification model, the foreground feature set and the background features are cross-fused to obtain a second fused feature, and the second fused feature is used by the classification model to classify the second video segment.

[0011] In the above scheme, obtaining the foreground feature set of the second video segment includes: For any one of the plurality of second video frames, determine the third position information of the sphere and the fourth position information of the athlete; The third location information, the fourth location information, and the frame number of the second video frame are embedded respectively to obtain the fourth feature corresponding to the third location information, the fifth feature corresponding to the fourth location information, and the sixth feature of the second video frame. The foreground feature set is obtained by combining the fourth, fifth, and sixth features of each of the multiple second video frames.

[0012] In the above scheme, determining the third position information of the sphere includes: obtaining the fourth mask of the sphere in the second video frame; determining the minimum bounding rectangle of the sphere based on the fourth mask; and determining the second position information of the sphere according to the image position of the minimum bounding rectangle in the second video frame. And / or, Determining the athlete's fourth position information includes: obtaining a fifth mask of the athlete in the second video frame; and obtaining an image of the athlete from the second video frame based on the fifth mask. Key point detection is performed on the athlete's image to obtain the fourth location information, wherein the fourth location information includes the location information of multiple key points of the athlete.

[0013] In the above scheme, the step of cross-fusing the foreground feature set and the background features to obtain the second fused feature includes: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the second self-attention feature; The classification model is used to cross-fuse the background features and the second self-attention features to obtain the second fused feature.

[0014] In the above scheme, the classification model includes an encoder; the step of performing self-attention processing based on the foreground feature set using the classification model to obtain the second self-attention feature includes: The fourth, fifth, and sixth features in the foreground feature set are spliced ​​together to obtain the second spliced ​​feature. The second splicing feature is input into the encoder, and self-attention processing is performed using the encoder to obtain the second self-attention feature output by the encoder.

[0015] In the above scheme, the classification model includes a decoder; the classification model is used to cross-fuse the background features and the second self-attention features to obtain the second fused features, including: The background feature and the second self-attention feature are input into the decoder, and cross-fusion is performed using the decoder to obtain the second fused feature output by the decoder.

[0016] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor; The processor is used to execute the steps of the above-described model training method or the steps of the above-described video classification method when running a computer program.

[0017] This application embodiment also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described model training method or the steps of the above-described video classification method.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described model training method or the steps of the above-described video classification method.

[0019] The model training method, video classification method, electronic device, storage medium, and program product provided in this application obtain foreground feature sets and background features of video clips. This not only yields first and second features corresponding to moving objects such as athletes and balls, but also background features and third features corresponding to the video frame number. The third feature reflects the temporal information of the video frames. Based on the foreground feature set and background features corresponding to the features of moving objects such as athletes and balls and their temporal features, the classification model is trained. This incorporates the temporal information of video frames into the training process, enhancing the classification model's understanding of contextual semantics. Furthermore, the cross-fusion of foreground and background features during training enhances the model's ability to capture the intrinsic relationship between background features and athletes / balls. The trained classification model yields more accurate classification results, enabling refined classification of competition video clips, improving the accuracy of highlight clip recognition, and meeting user needs. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a model training method provided in this application embodiment; Figure 2 A flowchart illustrating an example of a model training method provided in this application embodiment; Figure 3 A flowchart illustrating an example of the feature extraction process provided in this application embodiment; Figure 4 A flowchart illustrating an example of a video classification method provided in this application embodiment; Figure 5 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a video classification device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Highlights of ball sports events typically refer to video clips showcasing high-level athletic performances or dramatic turns in a match. For example, in football, highlights might include goals, crucial saves, and dribbling maneuvers; in badminton, they might include multi-shot rallies, incredible saves, and powerful smashes; and in table tennis, they might include continuous rallies, long-range rallies, and quick passes. These highlights capture the key moments of the sport and are the focus of viewer attention.

[0023] Video platforms will extract highlights from ball sports events and showcase them prominently on their homepages or dedicated channels to attract user clicks, increase platform traffic, and extend user dwell time. Users can watch exciting content from ball sports events simply by viewing these highlights. Therefore, highlight clips have become an effective way to enhance user experience. Sports event videos can be abbreviated as match videos, event videos, etc.

[0024] To meet users' demand for highlights of ball sports, video platforms extract these highlights from match videos using various video classification schemes. For example, in one such scheme, the platform initially filters candidate segments of the match video that may contain exciting events by detecting brightness changes. Then, a machine learning model is used to identify features of these candidate segments, such as the pixel proportions of specific areas like corner kick and penalty kick areas, and the pixel proportions of players and / or referees. These features are then judged to meet preset threshold conditions. If they do, the corresponding video frame is identified as the highlight segment. This video classification scheme primarily focuses on static image features and can identify exciting events in simple ball sports scenes, such as corner kicks and penalty kicks.

[0025] However, this video classification scheme has some shortcomings in practical applications. First, it relies too heavily on static image features, resulting in weak recognition of dynamic features. Using only static image features such as brightness and pixel ratio is insufficient to capture exciting events corresponding to complex movements. Second, this video classification method lacks full utilization of temporal information. Since exciting events in ball sports are dynamic processes, analyzing only static image features is insufficient to effectively capture dynamic features such as the ball's trajectory and player movements. Therefore, this static feature-based video classification scheme struggles to achieve refined classification of exciting events in ball sports, making it difficult to identify key moments in ball sports match videos.

[0026] Based on this, in various embodiments of this application, feature extraction is performed on video clips of ball sports training videos. This not only obtains features of moving objects such as athletes and balls, but also extracts background features and temporal features corresponding to the frame numbers of the video clips. These features, along with background and temporal features, are then fused. The fused features are used to train a classification model, resulting in a trained classification model. This trained model is then used to classify exciting events in match videos, achieving refined classification of these events. Because temporal features of video frames are introduced during the training of the classification model, the model's understanding of contextual semantics is enhanced during video classification. This improves the model's ability to capture dynamic features such as the ball's trajectory and player movements, leading to more accurate classification results and improved recognition accuracy for exciting segments in match videos.

[0027] Furthermore, the solution provided in this application can be applied to various types of electronic devices, such as cloud servers and video processing devices. By executing the video classification scheme, the electronic device can automatically process received competition videos, achieving refined classification of video clips. The electronic device can be used to drive or be integrated into a video platform to classify highlights from competition videos on the platform. The video platform then applies the classification results to personalized recommendations of highlights, thereby improving user experience. The following embodiments all use the electronic device as the execution subject to describe the model training method and video classification method provided in this application.

[0028] The following is combined with Figure 1 The model training method provided in the embodiments of this application will be described.

[0029] like Figure 1 As shown, the model training method includes the following steps: Step 101: The electronic device acquires the foreground feature set and background features of the video segments in the training video; Step 102: The electronic device obtains the input data for the classification model based on the foreground feature set and background features of the video clip; Step 103: The electronic device inputs the acquired input data into the classification model to train the classification model.

[0030] In practical applications, in step 101, the training videos include match videos of ball sports. The electronic device first acquires a video segment (i.e., the first video segment) from the ball training video. For a training video, the electronic device can randomly extract a video segment from the training video and acquire the foreground feature set and background features of that video segment.

[0031] For example, such as Figure 2 As shown, before training the classification model, the electronic device can acquire multiple match videos of ball sports from a computer vision library (such as OpenCV) and store these multiple match videos as training videos in the training video set. The electronic device performs feature extraction on each training video in the training video set to obtain the foreground feature set and background feature set of each training video, and stores the obtained foreground feature set and background feature set of each training video in the video feature set (which can be represented as VAMs).

[0032] The foreground feature set of the training video includes foreground features from multiple video frames. Foreground features are features related to moving objects extracted from video frames. For example, foreground features include ball features, athlete features, and temporal features. The background features of the training video are static scene-related features extracted from video frames. For example, background features include features of static elements such as the court, table, and net.

[0033] For example, the electronic device extracts features from each video frame in the training video to obtain foreground and background features for each video frame. For instance, the electronic device performs target detection on each video frame, detecting the image regions containing the sphere, athlete, table, and net in each frame. Then, it extracts image features from the sphere image region to obtain the sphere features in the foreground features, extracts image features from the athlete image region to obtain the athlete features in the foreground features, and extracts image features from the table and net image regions to obtain the background features. This application does not limit the method of obtaining the foreground feature set and background features. The electronic device combines the foreground features of each video frame into a foreground feature set for the training video and determines the background features of the training video based on the background features of each video frame. For example, the electronic device statistically analyzes the background features that appear most frequently among the background features of multiple video frames and uses this background feature as the unified background feature for training the video. Alternatively, the electronic device performs a weighted average or clustering of the background features of each video frame to obtain the background features of the training video.

[0034] When training the classification model, the electronic device randomly selects a training video from the video feature set VAMs and obtains the foreground feature set (represented as Ams) and background features of that training video. Accordingly, the electronic device randomly selects a start frame number and an end frame number, determines the video segment (i.e., the first video segment) corresponding to the start and end frame numbers in the training video, extracts the foreground feature set (represented as PMs) of that video segment from the foreground feature set Ams of the training video, and uses the background features of that training video as the background features of that video segment.

[0035] The foreground feature set of a video clip includes the first feature, second feature, and third feature of each video frame. The first feature is derived from the sphere in the video clip, i.e., the sphere feature. The second feature is derived from the athlete in the video clip, i.e., the athlete feature. The third feature is derived from the frame number of each video frame, i.e., the temporal feature. The frame number indicates the order of video frames within the video clip and serves as temporal information about the video frames within the clip.

[0036] By randomly selecting training videos and video clips for training, the classification model can learn common features across different competitions, athletes, and competition environments. This reduces the model's dependence on a single video or video clip and improves its generalization ability and robustness.

[0037] It is understandable that the classification model is trained using multiple video clips, with the electronic device processing each clip independently. Therefore, this paper uses a single video clip as an example to introduce the model training method provided in the embodiments of this application.

[0038] In step 101 above, the electronic device acquires the foreground feature set and background features of video segments in the training video, thereby providing a feature basis for the subsequent refined classification of video segments. Specifically, for the foreground feature set of the video segments, in some optional embodiments, for any video frame in the video segment, the electronic device determines the position information of the sphere (i.e., first position information) and the position information of the athlete (i.e., second position information), and then performs embedding processing on the position information of the sphere, the position information of the athlete, and the frame number of the video frame, respectively, to obtain sphere features, athlete features, and temporal features. The electronic device integrates the sphere features, athlete features, and temporal features of each video frame in the video segment to obtain the foreground feature set of the video segment.

[0039] In some optional embodiments, this application also provides a scheme for obtaining the foreground feature set and background features of the training video. The following example uses a table tennis match video as an illustration. Figure 3 The following example illustrates the feature extraction process for the foreground feature set and background features.

[0040] Step 301: The electronic device acquires the i-th video frame of the table tennis training video; The training video is a table tennis match video, which contains multiple video frames. These frames are arranged in chronological order. The electronic device sequentially acquires each video frame from the table tennis match video according to their chronological order. The i-th video frame is any frame in the table tennis match video. i is a positive integer.

[0041] While acquiring the i-th video frame, the electronic device also acquires the frame number of that video frame in the table tennis match video.

[0042] Step 302: The electronic device performs instance segmentation on the i-th video frame to obtain the ping-pong ball mask, athlete mask, and background mask; Electronic devices can use a segmentation model to segment the i-th video frame, obtaining a ping-pong ball mask (also called the first mask), an athlete mask (also called the second mask), and a background mask (also called the third mask) in the i-th video frame. The background mask includes masks of the table and net.

[0043] Segmentation models are pre-trained machine learning models, such as the YOLOv8-seg model. Segmentation models are used to identify and segment various objects in video frames, such as accurately identifying and segmenting objects like ping-pong balls, athletes, tables, and nets.

[0044] A mask is a binary or probabilistic matrix of data used to identify the position of an object within a video frame. The size of the mask corresponds to the video frame, and its outline can be determined. For example, a ping-pong ball mask identifies the position of the ping-pong ball within a video frame, and its outline can be determined. Similarly, an athlete mask identifies the position of the athlete within a video frame, and its outline can be determined. Likewise, masks for the table and net identify the positions of the table and net within a video frame, and their outlines can be determined.

[0045] Step 303: The electronic device extracts the features of the ping-pong ball based on the ping-pong ball mask; The electronic device determines the minimum bounding rectangle of the ping-pong ball based on the ping-pong ball mask obtained in step 302. For example, the electronic device inputs the ping-pong ball mask into a minimum area bounding rectangle algorithm (minAreaRect) to calculate the minimum bounding rectangle of the ping-pong ball. Based on the image position of the minimum bounding rectangle of the ping-pong ball in the i-th video frame, the electronic device determines the position information (i.e., the first position information) of the ping-pong ball. The position information of the ping-pong ball may include the coordinates of the four vertices of the minimum bounding rectangle.

[0046] Determining the position information of a ping-pong ball by using the minimum bounding rectangle defined by the ping-pong ball mask can reduce positioning errors caused by factors such as ball deformation or occlusion. At the same time, the minimum bounding rectangle of the ping-pong ball can also indirectly reflect the motion pattern of the ping-pong ball, thus providing data support for the subsequent refined classification of video segments.

[0047] After obtaining the position information of the ping-pong ball, the electronic device can use this position information as a ping-pong ball feature. Alternatively, the electronic device can perform embedding processing on the ping-pong ball's position information to generate the ping-pong ball feature (i.e., ball feature) for the i-th video frame. The ping-pong ball feature obtained after embedding processing can also be called the ping-pong ball embedded feature.

[0048] Understandably, the positional information of a ping-pong ball can be directly used as input to a classification model. In this case, the classification model may struggle to capture the relationships or local features between multiple points (i.e., coordinates). Through embedding, multiple points in the ping-pong ball's positional information can be mapped to a higher-dimensional space, resulting in ping-pong ball features with more abstract characteristics. This helps the classification model better understand the impact of changes in the ping-pong ball's position on exciting events.

[0049] Step 304: The electronic device extracts athlete features based on the athlete mask; The electronic device crops the i-th video frame based on the athlete mask obtained in step 302, extracts the image region containing the athlete in the i-th video frame, and obtains the athlete's image. Then, the electronic device uses a human pose estimation model (e.g., YOLOv8-pose model) to perform keypoint detection on the athlete image, and obtains the position information of multiple keypoints of the athlete (i.e., the second position information).

[0050] Human pose estimation models are pre-trained machine learning models used to detect key points (such as shoulders, elbows, wrists, hips, etc.) of the human body in images or video frames, such as the YOLOv8-pose model. By using human pose estimation models to detect key points in athlete images, the positional information of multiple key points of the athlete can be obtained, such as the coordinates of multiple key points such as shoulders, elbows, wrists, and hips.

[0051] Keypoint detection can transform athlete data from the overall silhouette (i.e., athlete mask) into the coordinates of keypoints, thereby effectively representing the athlete's posture and movements and providing a valid basis for identifying exciting events that depend on specific postures and movements.

[0052] After obtaining the location information of multiple key points of the athlete, the electronic device can use this information as athlete features. Alternatively, the electronic device can embed the location information of the multiple key points to generate athlete features for the i-th video frame. The athlete features obtained after embedding can also be called athlete embedded features.

[0053] Understandably, the positional information of multiple key points on an athlete can be directly used as input to a classification model. In this case, the model may struggle to capture the relationships or local features between these key points. Embedding allows the positional information of multiple key points to be mapped into a higher-dimensional space, resulting in athlete features with more abstract characteristics. This not only preserves the positional information of each key point but also captures the overall structural features of the athlete's posture. This helps the classification model better understand the impact of human movement or posture changes on key events, providing support for subsequent refined classification.

[0054] Step 305: The electronic device obtains the temporal features based on the frame number corresponding to the i-th video frame; The electronic device embeds the frame number of the i-th video frame to generate the temporal feature of the i-th video frame. The temporal feature obtained after embedding is also called the temporal embedding feature.

[0055] Understandably, frame count is a parameter representing temporal information, and its size can vary within a wide range. If the frame count of a video is directly input into a classification model, the model may struggle to learn from it, resulting in inaccurate classification results. Embedding transforms discrete frame counts into fixed-dimensional feature vectors, thus better representing temporal information. The resulting temporal features provide the classification model with temporal dimension information, helping it better understand the impact of temporal information on key events.

[0056] Step 306: The electronic device constructs a foreground feature set; The electronic device merges the ping-pong ball feature bfi obtained in step 303, the athlete feature kfi obtained in step 304, and the temporal feature ffi obtained in step 305 to form the foreground feature mfi of the i-th video frame.

[0057] For table tennis match videos, electronic devices can put the foreground features (mfi) of all video frames in the table tennis match video into a feature set (i.e., the foreground feature set) to construct the foreground feature set of the table tennis match video.

[0058] Step 307: The electronic device acquires background features; The electronic device constructs a background mask set based on the background masks of multiple video frames in a table tennis match video. For example, the electronic device combines the background masks of multiple video frames to form a background mask set. Alternatively, the electronic device can analyze the trajectory of the table tennis ball based on the table tennis ball masks of multiple video frames in the table tennis match video, and then filter out the valid video frames in progress based on the trajectory of the table tennis ball. For example, it can select video frames of the table tennis ball moving back and forth above the table as valid video frames in progress. The electronic device collects the valid background masks of the valid video frames to form a background mask set.

[0059] An electronic device can determine a target background mask based on multiple background masks in a set of background masks. For example, the electronic device can use the background mask with the highest frequency among multiple background masks as the target background mask. The electronic device then obtains background features based on the target background mask. For instance, the electronic device can input the target background mask into a masked autoencoder (MAE) to obtain the background features bf output by the masked autoencoder.

[0060] Through the above process, the electronic device extracts the foreground and background features of the table tennis match video, providing a reliable feature foundation for the subsequent refined classification of video segments by the trained classification model. The electronic device can acquire the foreground and background features of any video segment in a table tennis match video, providing training data for the classification model.

[0061] It is understandable that the above example, using a table tennis match video as an example, illustrates how to obtain the foreground and background feature sets of training videos. For other types of training videos, such as football or badminton match videos, or for obtaining the foreground and background feature sets of any video segment from the match videos in the video classification methods described below, electronic devices can refer to the methods provided in the above example to obtain the corresponding foreground and background feature sets, which will not be elaborated upon here.

[0062] In step 102, as Figure 2 As shown, the electronic device constructs input data suitable for a classification model based on the foreground features of each video frame in the foreground feature set and the background features of the video clip. For example, the electronic device uses the foreground features of each video frame in the foreground feature set and the background features of the video clip as input data for the classification model. Another example is that the electronic device can concatenate the first, second, and third features of each video frame in the foreground feature set to obtain a concatenated feature (i.e., the first concatenated feature), and then use the concatenated feature and the background features of the video clip as input data for the classification model.

[0063] In step 103, the electronic device trains the classification model based on the foreground and background features of the video clip, that is, it trains the classification model based on the input data of the classification model. During the training process of the classification model, the electronic device cross-fuses the foreground and background features of the video clip to obtain fused features (i.e., the first fused features).

[0064] For example, such as Figure 2 As shown, the electronic device inputs the input data into the classification model, which then performs cross-fusion processing (also known as cross-fusion) on the input data to obtain fused features. These fused features are then subjected to fully connected processing to obtain the classification result output by the classification model. Through the cross-fusion processing of the classification model, the foreground and background features of the video clip can be better integrated, enhancing the classification model's ability to capture the intrinsic relationship between background features and the athlete / ball. After obtaining the classification result output by the classification model, the electronic device calculates the loss value of the classification model based on the classification result and the annotation information of the video clip. The model parameters are then adjusted according to the loss value to obtain the trained classification model.

[0065] The classification result indicates the event category of a video clip. For example, taking a video clip from a table tennis match as an example, the classification result might include one or more of the following exciting event categories: continuous rally, long-range rally, diving save, extremely fast ball speed, and powerful smash (one or more can be understood as at least one). In some alternative implementations, the classification result may include ordinary event categories in addition to exciting event categories. If the classification result for a video clip indicates an ordinary event category, it means that the video clip does not contain exciting events and is not a highlight clip.

[0066] In some optional embodiments, after acquiring the foreground feature set and background features of the current video segment, the electronic device inputs the input data constructed based on the foreground feature set and background features into a classification model. Based on the foreground feature set, the electronic device performs self-attention processing using the classification model to obtain a first self-attention feature. Then, the classification model is used to cross-fuse the background features with the first self-attention feature to obtain a first fused feature.

[0067] The classification model comprises an encoder and a decoder. After acquiring the foreground and background feature sets of the current video segment, the electronic device concatenates the first, second, and third features of each video frame in the foreground feature set along the frame dimension to obtain a concatenated feature (i.e., the first concatenated feature), thus integrating the first, second, and third features into a unified feature vector. Subsequently, the electronic device inputs the concatenated feature into the encoder of the classification model, where the encoder encodes the concatenated feature using a self-attention mechanism to obtain a self-attention feature (i.e., the first self-attention feature). Through the encoder's self-attention mechanism, the spatiotemporal dependencies between the ball's position, the athlete's posture, and temporal information in each video frame of the video segment can be effectively captured. This enhances the classification model's ability to understand dynamic features such as continuous actions and motion trajectories, laying the foundation for subsequent refined classification of video segments in the competition video by the trained classification model.

[0068] After the encoder obtains the self-attention features, the electronic device inputs the background features of the video frame and the self-attention features output by the encoder into the decoder. The decoder then fuses the background information and self-attention features of the video frame through a cross-fusion mechanism to obtain the fused features output by the decoder. The above classification model also includes a fully connected layer. The electronic device inputs the fused features into the fully connected layer of the classification model for fully connected processing to obtain the classification result output by the classification model.

[0069] By employing the cross-attention mechanism of the decoder, the intrinsic relationships between background features and the athlete and the ball can be captured. This includes analyzing changes in the relative position and / or distance between the ball and the table, and between the ball and the net. This provides crucial information for refining the classification model and improving the accuracy of the classification results.

[0070] After obtaining the classification result output by the classification model, such as Figure 2 As shown, the electronic device obtains the annotation information (or label) of the corresponding video segment based on the start and end frame numbers of the selected video segment. This annotation information is used to indicate the real event category of the corresponding video segment, such as indicating the real event categories of continuous rally, long-range rally, diving save, super-fast ball speed, and powerful smash. Based on the classification results obtained by the classification model and the annotation information of the video segment, the electronic device calculates the loss value of the classification model using a preset loss function, and updates various model parameters in the classification model, such as weight parameters and bias parameters, according to the backpropagation algorithm and gradient descent algorithm. This process of event category prediction, loss calculation, backpropagation, and parameter update (i.e., the optimization process of the classification model) is repeated until the loss of the classification model converges or the maximum number of iterations is reached, resulting in the trained classification model.

[0071] By using the training methods described above, the classification model can learn the inherent relationship between the dynamic features of the foreground and the static features of the background in the competition video, thereby achieving refined event classification of video clips.

[0072] To better fuse the foreground and background of video clips, in some optional embodiments, the electronic device generates a query vector in the cross-attention mechanism based on background features, generates a key vector and a value vector in the cross-attention mechanism based on the self-attention feature vector output by the encoder, and then obtains the fused features output by the decoder based on the query vector, key vector, and value vector.

[0073] For example, the electronic device performs a linear transformation on the background features using the transformation matrix Wq corresponding to the query vector to obtain the query vector. Correspondingly, the electronic device performs a linear transformation on the self-attention features output by the encoder using the transformation matrix Wk corresponding to the key vector to obtain the key vector. The electronic device performs a linear transformation on the self-attention features output by the encoder using the transformation matrix Wv corresponding to the value vector to obtain the value vector. Alternatively, the electronic device can directly use the self-attention features as the key and value vectors. Then, the electronic device uses a decoder to perform a cross-attention mechanism on the query vector, key vector, and value vector to fuse the background and foreground information of the video frame, obtaining the fused features output by the decoder.

[0074] In this way, the classification model can adaptively associate relevant background information (such as the table and net) with foreground information in video frames, providing key basis for refined classification and improving the accuracy of classification results.

[0075] To facilitate understanding of the model training method provided in the embodiments of this application, the following uses a table tennis match video as an example to illustrate the model training method provided in the embodiments of this application.

[0076] In the example, the video clip includes n video frames. The first, second, and third features of the n video frames included in the foreground feature set of the video clip can be represented as bfin, ffin, and kfin, respectively, and the background feature can be represented as bf. After the electronic device obtains the foreground feature set and background features of the video clip, it first concatenates the first feature bfin, the second feature ffin, and the third feature kfin of multiple video frames in the frame dimension to form an input sequence (i.e., concatenated features). The input sequence can be represented by the following formula (1): (1) Among them, Input n This represents the concatenated input sequence. ; Represents the set of real numbers; n represents the number of video frames, which is a positive integer; d input The concatenated feature dimensions; concat indicates concatenation processing.

[0077] After obtaining the input sequence, the electronic device inputs the input sequence into the encoder (which can be represented as Encoder) of the classification model, and extracts self-attention features through the encoder's self-attention mechanism. The process of the encoder extracting self-attention features can be expressed as formula (2): (2) Where Hencoder represents the self-attention feature, ; dmodel represents the set of real numbers; n represents the number of video frames, which is a positive integer; dmodel is the feature dimension of the self-attention feature; Encoder represents the encoder.

[0078] Based on this, the electronic device inputs the background features and self-attention features into the decoder of the classification model. The decoder performs a linear transformation on the background features bf to obtain the query vector Qbf. The self-attention features Hencoder are used as the key vector and value vector. The decoder performs a cross-attention mechanism on the query vector, key vector, and value vector to achieve the cross-fusion of background features and self-attention features, and obtains the fused features output by the decoder. The process of the decoder performing cross-fusion to obtain the fused features can be expressed as formula (3): (3) Where Hdecoder represents the fused feature; Decoder represents the decoder; Qbf represents the query vector obtained by linear transformation of the background feature bf; and Hencoder represents the self-attention feature.

[0079] After obtaining the fused features, the electronic device inputs these features into the fully connected layer of the classification model. The fully connected layer then performs activation processing (or fully connected processing) on ​​the fused features, such as using the softmax activation function, to obtain the classification result for the video segment. Based on the classification result and the annotation information of the video segment, the electronic device adjusts the model parameters of the classification model to obtain the trained classification model.

[0080] The model training method provided in this application effectively integrates multi-dimensional information by extracting features of foreground moving targets such as athletes and balls from video clips, as well as the temporal features of video frames, and combining these with background features such as the table tennis table and net. Based on a cross-attention mechanism, the feature fusion method can establish a spatiotemporal relationship between the foreground moving targets and the static background. The trained classification model can capture and distinguish different categories of exciting events. For example, in a table tennis match video, it can identify finely categorized exciting segments such as continuous rallies and long-range rallies. This not only helps meet users' personalized viewing needs for exciting segments but also significantly enhances the viewing experience of sports event videos.

[0081] In this embodiment, the electronic device trains a classification model using training videos to obtain a trained classification model. This embodiment also provides a video classification method implemented using the trained classification model. This method is applied to an electronic device. The electronic device used to train the model and the electronic device used for video classification can be the same electronic device or different electronic devices.

[0082] The following is combined with Figure 4 The following is an exemplary description of the video classification method provided in the embodiments of this application: Step 401: The electronic device acquires the foreground feature set and background features of video segments from a ball game competition video; The electronic device first acquires video segments (i.e., second video segments) from a match video of a ball sport. For a single match video, the electronic device can sequentially extract the corresponding video segments from the match video at preset video frame intervals.

[0083] For example, such as Figure 4As shown, the electronic device determines the start and end frames of a video segment based on a preset video frame interval and the segment's sequence number, and then acquires video segments within the frame range corresponding to those start and end frames from the competition video. For example, using a preset video frame interval of 100 frames, if the electronic device acquires the second video segment, it selects the start frame number of the second video segment in the competition video as 101 and the end frame number as 200, thus acquiring the video segment from start frame 101 to end frame 200 from the competition video. This video segment comprises 100 video frames.

[0084] After acquiring a video segment from the competition video, the electronic device can extract features from that segment to obtain the foreground and background feature sets. The video segment consists of multiple video frames (i.e., the second video frame).

[0085] The foreground feature set of a video clip includes sphere features (the fourth feature), athlete features (the fifth feature), and temporal features (the sixth feature). The sphere feature is derived from the sphere in the video clip, such as by using image features of the sphere within the clip. The athlete feature is derived from the athlete in the video clip, such as by using image features of the athlete within the clip. The temporal feature is derived from the frame number of each video frame in the clip. The frame number indicates the order of video frames within the clip and serves as temporal information about the video frames.

[0086] In some alternative implementations, for any video frame in a video segment, the electronic device acquires a mask of the sphere in the video frame and determines the minimum bounding rectangle of the sphere based on the mask. The electronic device then determines the position information of the sphere based on the image position of the minimum bounding rectangle within the video frame. Similarly, the electronic device acquires a mask of the athlete in the video frame, extracts the athlete's image from the video frame based on the athlete's mask, and then performs keypoint detection on the athlete's image to obtain the athlete's position information.

[0087] After determining the position information of the ball and the athlete, the electronic device embeds the position information of the ball, the position information of the athlete, and the frame number of the video frame to obtain the ball features, athlete features, and temporal features.

[0088] Electronic devices combine the foreground features of individual video frames into a foreground feature set for a video segment, and determine the background features of the video segment based on the background features of each video frame. For example, the electronic device identifies the background feature with the highest frequency among the background features of multiple video frames and uses that background feature as the unified background feature for the video segment. Alternatively, the electronic device performs a weighted average or clustering of the background features of each video frame to obtain the background features of the video segment.

[0089] It is understandable that electronic devices extract features from video segments in the competition video to obtain the foreground feature set and background features of the video segments. For the method and process of obtaining the foreground feature set and background features of the training video mentioned above, please refer to the previous text. It will not be repeated here.

[0090] Step 402: The electronic device obtains the input data for the classification model based on the foreground feature set and background features of the video clip; Electronic devices construct input data suitable for classification models based on the foreground features of each video frame in the foreground feature set and the background features of the video clip. For example, the electronic device uses the foreground features of each video frame in the foreground feature set and the background features of the video clip as input data for the classification model. As another example, taking a ping-pong video clip, the electronic device concatenates the ping-pong ball features (i.e., ball features), athlete features, and temporal features from multiple video frames in the foreground feature set along the frame dimension to form the input sequence corresponding to the foreground feature set (also known as the second concatenated feature). Then, the input sequence corresponding to the foreground feature set and the background features of the video clip are used as input data for the classification model.

[0091] Step 403: The electronic device inputs the acquired input data into the classification model, and performs self-attention processing and cross-fusion processing through the classification model to obtain the classification result output by the classification model; The electronic device classifies video clips from a competition video based on a foreground feature set and background features, using a classification model. During the classification process, the electronic device cross-fuses the foreground and background features to obtain a fused feature (i.e., a second fused feature), which is then used by the classification model to classify the second video clip. For example, as shown... Figure 4 As shown, the electronic device inputs the input data into a pre-trained classification model, and uses the pre-trained classification model to perform self-attention processing and cross-fusion processing on the input data to obtain the classification result output by the classification model.

[0092] In some alternative embodiments, the electronic device inputs the input sequence into the encoder of the trained classification model, and the encoder encodes the input sequence through a self-attention mechanism to obtain the self-attention features (also known as second self-attention features) output by the encoder.

[0093] The trained classification model can extract the spatiotemporal dependencies between the ball position, athlete posture and time information in each video frame through the self-attention mechanism, thereby enhancing the classification model's ability to understand dynamic features such as continuous actions and motion trajectories.

[0094] It is understandable that in the video classification process, the electronic device uses a trained classification model for self-attention processing, which is consistent with the process described above where the electronic device uses an untrained classification model for self-attention processing. Please refer to the relevant description above, which will not be repeated here.

[0095] The electronic device inputs the self-attention features obtained by the encoder and the background features of the video segment into the decoder of the classification model. The decoder then cross-fuses the self-attention features and the background features through a cross-attention mechanism to obtain the fused features (i.e., the second fused features) output by the decoder.

[0096] The decoder can perform linear transformation on the background features to generate a query vector, and generate key vectors and value vectors based on the self-attention features. Then, through the cross-attention mechanism, the decoder outputs the fused features based on the query vector, key vector, and value vector, thus realizing the fusion of background information (i.e., background features) and spatiotemporal dynamic features (i.e., self-attention features).

[0097] It is understandable that in the video classification process, the electronic device uses a trained classification model for cross-fusion, which is consistent with the process of the electronic device using an untrained classification model for cross-fusion mentioned above. Please refer to the relevant description above, which will not be repeated here.

[0098] After obtaining the fused features, the electronic device inputs the fused features into the fully connected layer of the trained classification model. The fully connected layer activates the fused features, such as by using the softmax activation function, to obtain the classification result of the trained classification model for the video segment.

[0099] In this embodiment, by adding temporal features to the foreground feature set, not only is the static information of the video frame extracted, but also the dynamic information in the competition video clip is captured based on the temporal features. By comprehensively utilizing the static information and dynamic features in the video frame, the correlation between the ball trajectory, the athlete's actions and the field environment in the competition video clip can be effectively established, and the accurate identification of dynamic events such as continuous hitting and powerful smashes can be achieved.

[0100] In the feature extraction stage, instance segmentation technology and human pose detection technology are used to identify and separate various objects in the video frame and extract the position information of the athlete's key points, thereby significantly improving the classification model's ability to capture subtle movements and overcoming the limitations in complex movement recognition.

[0101] Meanwhile, by introducing a mask autoencoder to encode the background features of the match video (such as the features of the table and net), the semantic expression of static scene elements such as the table and net is enhanced, and the background features and foreground features are effectively integrated, which greatly improves the accuracy of judging exciting events such as continuous rallies and ultra-long rallies, and effectively reduces the false recognition rate.

[0102] Furthermore, throughout the classification process, not only are the static image features of video frames utilized, but also temporal information and athlete posture information are combined for classification reasoning, which significantly improves the classification model's ability to capture exciting events of various categories, especially in the classification of exciting events in complex sports scenarios, thereby enhancing the robustness of exciting event classification.

[0103] In some optional embodiments, after obtaining the classification result of a video clip, if the video clip is determined to be a highlight clip with a highlight event based on the classification result, the electronic device can set descriptive information (such as tags or descriptive text) for the highlight event based on the classification result. For example, after classifying a video clip from a table tennis match video, if the classification result of the video clip is "continuous rally," the electronic device sets a tag or descriptive text corresponding to "continuous rally" for the video clip; if the classification result of the video clip is "diving save," the electronic device sets a tag or descriptive text corresponding to "diving save" for the video clip.

[0104] Descriptive information from video clips can be used for video clip retrieval, video content display, or generating highlight reels. For example, electronic devices can generate corresponding highlight event markers on the playback progress bar of a match video based on the descriptive information of the video clips. When a user terminal detects a user action pointing to a highlight event marker on the playback progress bar, descriptive information for highlights such as "continuous rally" or "diving save" can appear, allowing users to quickly locate the highlight segments of the match video. As another example, electronic devices can extract similar highlight segments from multiple match videos based on the descriptive information, generate highlight reels, and assign corresponding titles or descriptions to the highlight reels based on the descriptive information, such as "Powerful Smash Highlights" or "Extreme Long-Range Rally Highlights." Highlight reel compilations can appear as short video collections on the homepage or special pages of video platforms. In this way, users can efficiently watch the highlights of match videos, improving the user experience.

[0105] To implement the model training method provided in this application embodiment, this application embodiment also provides a model training device, which is installed on an electronic device, such as... Figure 5 As shown, the model training device includes: The first acquisition module 501 is used to acquire a foreground feature set and background features of a first video segment, wherein the first video segment is a video segment in a training video, and the training video includes a match video of a ball sport; the first video segment includes multiple first video frames, and the foreground feature set includes a first feature, a second feature, and a third feature, wherein the first feature is obtained based on the ball in the first video segment, the second feature is obtained based on the athlete in the first video segment, and the third feature is obtained based on the frame number of the multiple first video frames; Training module 502 is used to train a classification model based on the foreground feature set and the background features; wherein, during the training process of the classification model, the foreground feature set and the background features are cross-fused to obtain a first fused feature, and the first fused feature is used by the classification model to classify the first video segment.

[0106] In some optional embodiments, the first acquisition module 501 is specifically used for: For any one of the plurality of first video frames, determine the first position information of the sphere and the second position information of the athlete; The first location information, the second location information, and the frame number of the first video frame are embedded respectively to obtain the first feature corresponding to the first location information, the second feature corresponding to the second location information, and the third feature of the first video frame. The foreground feature set is obtained by combining the first feature, second feature, and third feature of each of the multiple first video frames.

[0107] In some optional embodiments, the first acquisition module 501 is specifically used for: Obtain the first mask of the sphere in the first video frame; Based on the first mask, determine the minimum bounding rectangle of the sphere; The first position information of the sphere is determined based on the image position of the minimum bounding rectangle in the first video frame.

[0108] In some optional embodiments, the first acquisition module 501 is specifically used for: Obtain the second mask of the athlete in the first video frame; The image of the athlete is obtained from the first video frame based on the second mask; Key point detection is performed on the athlete's image to obtain the second location information, wherein the second location information includes the location information of multiple key points of the athlete.

[0109] In some optional embodiments, the training module 502 is specifically used for: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the first self-attention feature; The classification model is used to cross-fuse the background features and the first self-attention features to obtain the first fused feature.

[0110] In some optional embodiments, the classification model includes an encoder; the training module 502 is specifically used for: The first feature, the second feature, and the third feature in the foreground feature set are spliced ​​together to obtain the spliced ​​feature after splicing. The splicing feature is input into the encoder, and self-attention processing is performed using the encoder to obtain the first self-attention feature output by the encoder.

[0111] In some optional embodiments, the classification model includes a decoder; the training module 502 is specifically used for: The background features and the first self-attention features are input into the decoder, and cross-fusion is performed using the decoder to obtain the first fused feature output by the decoder.

[0112] In some optional embodiments, the training module 502 is specifically used for: Generate a query vector based on the background features; Generate a key vector and a value vector based on the first self-attention feature vector; Based on the query vector, the key vector, and the value vector, the first fusion feature output by the decoder is obtained.

[0113] In practical applications, the first acquisition module 501 can be implemented by the processor in the model training device in conjunction with the communication interface, and the training module 502 can be implemented by the processor in the model training device.

[0114] To implement the video classification method provided in this application embodiment, this application embodiment also provides a video classification device, which is installed on an electronic device, such as... Figure 6 As shown, the device includes: The second acquisition module 601 is used to acquire the foreground feature set and background features of the second video segment, wherein the second video segment includes video segments from a ball game competition video, the second video segment includes multiple second video frames, the foreground feature set of the second video segment includes a fourth feature, a fifth feature and a sixth feature, the fourth feature is obtained based on the ball in the second video segment, the fifth feature is obtained based on the athlete in the second video segment, and the sixth feature is obtained based on the frame number of the multiple second video frames; The classification module 602 is used to classify the second video segment based on the foreground feature set and the background features, and using a classification model; wherein, in the process of classifying the second video segment using the classification model, the foreground feature set and the background features are cross-fused to obtain a second fused feature, and the second fused feature is used by the classification model to classify the second video segment.

[0115] In some optional embodiments, the second acquisition module 601 is specifically used for: For any one of the plurality of second video frames, determine the third position information of the sphere and the fourth position information of the athlete; The third location information, the fourth location information, and the frame number of the second video frame are embedded respectively to obtain the fourth feature corresponding to the third location information, the fifth feature corresponding to the fourth location information, and the sixth feature of the second video frame. The foreground feature set is obtained by combining the fourth, fifth, and sixth features of each of the multiple second video frames.

[0116] In some optional embodiments, the second acquisition module 601 is specifically used for: Obtain the fourth mask of the sphere in the second video frame; determine the minimum bounding rectangle of the sphere based on the fourth mask; determine the second position information of the sphere according to the image position of the minimum bounding rectangle in the second video frame; In some optional embodiments, the second acquisition module 601 is specifically used for: Obtain the fifth mask of the athlete in the second video frame; based on the fifth mask, obtain the image of the athlete from the second video frame; Key point detection is performed on the athlete's image to obtain the fourth location information, wherein the fourth location information includes the location information of multiple key points of the athlete.

[0117] In some optional embodiments, the classification module 602 is specifically used for: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the second self-attention feature; The classification model is used to cross-fuse the background features and the second self-attention features to obtain the second fused feature.

[0118] In some optional embodiments, the classification model includes an encoder; the classification module 602 is specifically used for: The fourth, fifth, and sixth features in the foreground feature set are spliced ​​together to obtain the second spliced ​​feature. The second splicing feature is input into the encoder, and self-attention processing is performed using the encoder to obtain the second self-attention feature output by the encoder.

[0119] In some optional embodiments, the classification model includes a decoder; the classification module 602 is specifically used for: The background feature and the second self-attention feature are input into the decoder, and cross-fusion is performed using the decoder to obtain the second fused feature output by the decoder.

[0120] In practical applications, the second acquisition module 601 can be implemented by the processor in the video classification device in combination with the communication interface, and the classification module 602 can be implemented by the processor in the video classification device.

[0121] It should be noted that the model training device provided in the above embodiments is only illustrated by the division of the above-described program units during model training. In practical applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the device can be divided into different program units to complete all or part of the processing described above. Furthermore, the model training device and the model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here. Correspondingly, the video classification device provided in the above embodiments is only illustrated by the division of the above-described program units during video classification. In practical applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the device can be divided into different program units to complete all or part of the processing described above. Furthermore, the video classification device and the video classification method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0122] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 7 As shown, the electronic device includes: The communication interface 701 enables information exchange with other electronic devices (such as user terminals); The processor 702 is connected to the communication interface 701 to enable information interaction with other electronic devices (such as user terminals) and to execute the methods provided by one or more of the above-mentioned technical solutions when running computer programs; The computer program is stored in memory 703.

[0123] When the electronic device is used for model training, the processor 702, in conjunction with the communication interface 701, is used to acquire the foreground feature set and background features of a first video segment. The first video segment is a video segment from a training sample video, which includes video recordings of ball sports matches. The first video segment includes multiple first video frames. The foreground feature set includes a first feature, a second feature, and a third feature. The first feature is obtained based on the ball in the first video segment, the second feature is obtained based on the athlete in the first video segment, and the third feature is obtained based on the frame number of the multiple first video frames. The processor 702 is used to train a classification model based on the foreground feature set and the background features; wherein, during the training process of the classification model, the foreground feature set and the background features are cross-fused to obtain a first fused feature, and the first fused feature is used by the classification model to classify the first video segment.

[0124] In some alternative embodiments, the processor 702, in conjunction with the communication interface 701, is specifically used for: For any one of the plurality of first video frames, determine the first position information of the sphere and the second position information of the athlete; The first location information, the second location information, and the frame number of the first video frame are embedded respectively to obtain the first feature corresponding to the first location information, the second feature corresponding to the second location information, and the third feature of the first video frame. The foreground feature set is obtained by combining the first feature, second feature, and third feature of each of the multiple first video frames.

[0125] In some alternative embodiments, the processor 702, in conjunction with the communication interface 701, is specifically used for: Obtain the first mask of the sphere in the first video frame; Based on the first mask, determine the minimum bounding rectangle of the sphere; The first position information of the sphere is determined based on the image position of the minimum bounding rectangle in the first video frame.

[0126] In some alternative embodiments, the processor 702, in conjunction with the communication interface 701, is specifically used for: Obtain the second mask of the athlete in the first video frame; The image of the athlete is obtained from the first video frame based on the second mask; Key point detection is performed on the athlete's image to obtain the second location information, wherein the second location information includes the location information of multiple key points of the athlete.

[0127] In some alternative embodiments, the processor 702 is specifically used for: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the first self-attention feature; The classification model is used to cross-fuse the background features and the first self-attention features to obtain the first fused feature.

[0128] In some alternative embodiments, the processor 702 is specifically used for: The first feature, the second feature, and the third feature in the foreground feature set are spliced ​​together to obtain the spliced ​​feature after splicing. The concatenated features are input into the encoder of the classification model, and self-attention processing is performed using the encoder to obtain the first self-attention feature output by the encoder.

[0129] In some alternative embodiments, the processor 702 is specifically used for: The background features and the first self-attention features are input into the decoder of the classification model, and cross-fusion is performed using the decoder to obtain the first fused feature output by the decoder.

[0130] In some alternative embodiments, the processor 702 is specifically used for: Generate a query vector based on the background features; Generate a key vector and a value vector based on the first self-attention feature vector; Based on the query vector, the key vector, and the value vector, the first fusion feature output by the decoder is obtained.

[0131] When the electronic device is used for video classification, the processor 702, in conjunction with the communication interface 701, is used to acquire the foreground feature set and background features of a second video segment. The second video segment is a video segment from a ball game video. The second video segment includes multiple second video frames. The foreground feature set of the second video segment includes a fourth feature, a fifth feature, and a sixth feature. The first feature is obtained based on the ball in the second video segment, the second feature is obtained based on the athlete in the second video segment, and the third feature is obtained based on the frame number of the multiple second video frames. The processor 702 is configured to obtain the classification result of the classification model for the second video segment based on the foreground feature set and the background features; wherein, during the classification process of the classification model, the foreground feature set and the background features are cross-fused to obtain a second fused feature, and the second fused feature is used by the classification model to classify the second video segment.

[0132] In some alternative embodiments, the processor 702, in conjunction with the communication interface 701, is specifically used for: For any one of the plurality of second video frames, determine the third position information of the sphere and the fourth position information of the athlete; The third location information, the fourth location information, and the frame number of the second video frame are embedded respectively to obtain the fourth feature corresponding to the third location information, the fifth feature corresponding to the fourth location information, and the sixth feature of the second video frame. The foreground feature set is obtained by combining the fourth, fifth, and sixth features of each of the multiple second video frames.

[0133] In some optional embodiments, the processor 702, in conjunction with the communication interface 701, is specifically used to: obtain a fourth mask of the sphere in the second video frame; determine the minimum bounding rectangle of the sphere based on the fourth mask; and determine the second position information of the sphere according to the image position of the minimum bounding rectangle in the second video frame.

[0134] In some optional embodiments, the processor 702, in conjunction with the communication interface 701, is specifically used to: acquire a fifth mask of the athlete in the second video frame; acquire an image of the athlete from the second video frame based on the fifth mask; perform key point detection on the image of the athlete to obtain the fourth position information, wherein the fourth position information includes position information of multiple key points of the athlete.

[0135] In some alternative embodiments, the processor 702 is specifically used for: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the second self-attention feature; The classification model is used to cross-fuse the background features and the second self-attention features to obtain the second fused feature.

[0136] In some optional embodiments, the classification model includes an encoder; the processor 702 is specifically used for: The fourth, fifth, and sixth features in the foreground feature set are spliced ​​together to obtain the second spliced ​​feature. The second splicing feature is input into the encoder, and self-attention processing is performed using the encoder to obtain the second self-attention feature output by the encoder.

[0137] In some optional embodiments, the classification model includes a decoder; the processor 702 is specifically used for: The background feature and the second self-attention feature are input into the decoder, and cross-fusion is performed using the decoder to obtain the second fused feature output by the decoder.

[0138] It should be noted that the specific processing procedures of the processor 702 and the communication interface 701 can be understood by referring to the above model training method or the above video classification method.

[0139] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0140] The memory 703 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0141] The methods disclosed in the embodiments of this application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 702 or by instructions in the form of software. The processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 703. The processor 702 reads the information in the memory 703 and combines its hardware to complete the steps of the aforementioned method.

[0142] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0143] It is understood that the memory 703 in this embodiment can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0144] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 703 storing a computer program. This computer program can be executed by a processor 702 of an electronic device to complete the steps described in the aforementioned model training method, or to complete the steps described in the aforementioned video classification method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0145] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 702 of an electronic device to complete the steps of the aforementioned model training method or the steps of the aforementioned video classification method.

[0146] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0147] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0148] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Obtain the foreground feature set and background features of a first video segment, wherein the first video segment is a video segment in a training video, the training video including a match video of a ball sport; the first video segment includes multiple first video frames, the foreground feature set includes a first feature, a second feature and a third feature, the first feature is obtained based on the ball in the first video segment, the second feature is obtained based on the athlete in the first video segment, and the third feature is obtained based on the frame number of the multiple first video frames; A classification model is trained based on the foreground feature set and the background features; wherein, during the training process of the classification model, the foreground feature set and the background features are cross-fused to obtain a first fused feature, and the first fused feature is used by the classification model to classify the first video segment.

2. The method according to claim 1, characterized in that, Obtain the foreground feature set of the first video segment, including: For any one of the plurality of first video frames, determine the first position information of the sphere and the second position information of the athlete; The first location information, the second location information, and the frame number of the first video frame are embedded to obtain the first feature corresponding to the first location information, the second feature corresponding to the second location information, and the third feature of the first video frame. The foreground feature set is obtained by combining the first feature, second feature, and third feature of each of the multiple first video frames.

3. The method according to claim 2, characterized in that, Determining the first position information of the sphere includes: obtaining a first mask of the sphere in the first video frame; determining the minimum bounding rectangle of the sphere based on the first mask; and determining the first position information of the sphere according to the image position of the minimum bounding rectangle in the first video frame. And / or, Determining the athlete's second position information includes: obtaining a second mask of the athlete in the first video frame; and obtaining an image of the athlete from the first video frame based on the second mask. Key point detection is performed on the athlete's image to obtain the second location information, wherein the second location information includes the location information of multiple key points of the athlete.

4. The method according to any one of claims 1 to 3, characterized in that, The step of cross-fusing the foreground feature set and the background features to obtain the first fused feature includes: Based on the foreground feature set, the classification model is used to perform self-attention processing to obtain the first self-attention feature; The classification model is used to cross-fuse the background features and the first self-attention features to obtain the first fused feature.

5. The method according to claim 4, characterized in that, The classification model includes an encoder; the step of performing self-attention processing based on the foreground feature set using the classification model to obtain the first self-attention feature includes: The first feature, the second feature, and the third feature in the foreground feature set are spliced ​​together to obtain the first spliced ​​feature after splicing. The first splicing feature is input into the encoder, and self-attention processing is performed using the encoder to obtain the first self-attention feature output by the encoder.

6. The method according to claim 4, characterized in that, The classification model includes a decoder; the background features and the first self-attention features are cross-fused using the classification model to obtain the first fused features, including: The background features and the first self-attention features are input into the decoder, and cross-fusion is performed using the decoder to obtain the first fused feature output by the decoder.

7. A video classification method, characterized in that, include: Obtain the foreground feature set and background features of the second video segment, wherein the second video segment includes video segments from a ball game competition video, the second video segment includes multiple second video frames, and the foreground feature set of the second video segment includes a fourth feature, a fifth feature, and a sixth feature, wherein the fourth feature is obtained based on the ball in the second video segment, the fifth feature is obtained based on the athlete in the second video segment, and the sixth feature is obtained based on the frame number of the multiple second video frames; Based on the foreground feature set and the background features, the second video segment is classified using a classification model; wherein, during the classification process of the second video segment using the classification model, the foreground feature set and the background features are cross-fused to obtain a second fused feature, and the second fused feature is used by the classification model to classify the second video segment.

8. An electronic device, characterized in that, include: Processor and memory used to store computer programs that can run on the processor; When the processor is used to run a computer program, it performs the steps of the method according to any one of claims 1 to 6, or performs the steps of the method according to claim 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6, or the steps of the method according to claim 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6, or the steps of the method according to claim 7.

Images