Video data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a video data processing method and device, equipment and a storage medium, and the method comprises the steps: determining K candidate events described by a target video image according to the image features of the target video image and the attribute features corresponding to M candidate events; and when K is an integer greater than 1, performing mutual exclusion relation detection on K candidate events described by the target video image to obtain a mutual exclusion detection result. And when the mutual exclusion detection result indicates that candidate events with mutual exclusion relationship exist in the K candidate events, adjusting the K candidate events described by the target video image according to the mutual exclusion detection result to obtain a target event described by the target video image. And dividing the to-be-detected video according to the target event described by each video image in the to-be-detected video and the playing time corresponding to each video image to obtain at least one video clip. According to the invention, the accuracy of video time sequence positioning can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of artificial intelligence (AI) technology, it is being applied to an increasing number of businesses. For example, video temporal localization is a technique used to identify events occurring in a video and pinpoint the time of those events within the video. Currently, video temporal localization can be implemented using dedicated localization network models.

[0003] During implementation, it was found that different types of events typically require training dedicated localization network models based on a large amount of video data to identify these different categories. However, the video data in publicly available datasets is of low quality, and online video data is usually protected by copyright, making it difficult and costly to obtain. Therefore, the training performance of dedicated localization network models is poor, resulting in low accuracy in video temporal localization. Summary of the Invention

[0004] This application provides a video data processing method, apparatus, device, and storage medium that can improve the accuracy of video timing positioning.

[0005] One embodiment of this application provides a video data processing method, including:

[0006] Obtain the image features of the target video image and the attribute features corresponding to M candidate events; M is an integer greater than 1; the target video image is a video image in the video to be detected.

[0007] Based on the image features of the target video image and the attribute features corresponding to the M candidate events, determine the K candidate events described by the target video image.

[0008] When K is an integer greater than 1, mutual exclusion detection is performed on the K candidate events described by the target video image to obtain the mutual exclusion detection result;

[0009] When the mutual exclusion detection result indicates that there are mutually exclusive candidate events among the K candidate events, the K candidate events described by the target video image are adjusted according to the mutual exclusion detection result to obtain the target event described by the target video image.

[0010] Based on the target events described by each video image in the video to be detected, and the playback time corresponding to each video image, the video to be detected is divided to obtain at least one video segment; the target events described by the video images within the same video segment are the same.

[0011] One embodiment of this application provides a video data processing apparatus, including:

[0012] The acquisition module is used to acquire the image features of the target video image and the attribute features corresponding to M candidate events; M is an integer greater than 1; the target video image is a video image in the video to be detected.

[0013] The first processing module is used to determine K candidate events described by the target video image based on the image features of the target video image and the attribute features corresponding to the M candidate events respectively.

[0014] The detection module is used to perform mutual exclusion detection on the K candidate events described by the target video image when K is an integer greater than 1, and obtain the mutual exclusion detection result.

[0015] The adjustment module is used to adjust the K candidate events described by the target video image according to the mutual exclusion detection results when the mutual exclusion detection results indicate that there are candidate events with mutual exclusion relationships among the K candidate events, so as to obtain the target event described by the target video image.

[0016] The second processing module is used to divide the video to be detected into at least one video segment based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image; the target events described by the video images in the same video segment are the same.

[0017] One embodiment of this application provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0018] One embodiment of this application provides a computer storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0019] One aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0020] In this application, the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events is determined by using the image features of the target video image and the attribute features corresponding to the candidate events. Based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events, K candidate events described by the target video image are determined. Since the attribute features of candidate events with mutual exclusion have a relatively high matching degree, it is easy to identify two candidate events with mutual exclusion as candidate events described by the target video image. Two candidate events with mutual exclusion can refer to candidate events that cannot appear in the target video image at the same time, which leads to a relatively low accuracy of the candidate events described by the target video image. Based on this, when K is greater than 1, mutual exclusion detection is performed on the K candidate events to obtain mutual exclusion detection results. When the mutual exclusion detection results indicate that there are candidate events with mutual exclusion among the K candidate events, the K candidate events are adjusted based on the mutual exclusion detection results to obtain the target event described by the target video image. This helps to improve the accuracy of obtaining the target event described by the target video image. Based on the target events described by each video image, the video to be detected is divided to obtain at least one video segment, improving the accuracy of video temporal localization. This method improves the accuracy of video temporal localization by detecting mutual exclusion relationships, eliminating the need to train a dedicated localization network model based on a large video dataset, thus reducing the cost and difficulty of video temporal localization. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of a video data processing system provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating a video data processing method provided in an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating another video data processing method provided in an embodiment of this application;

[0024] Figure 4 This is a flowchart illustrating a method for determining a target event based on mutually exclusive text, as provided in an embodiment of this application.

[0025] Figure 5 This is a flowchart illustrating another video data processing method provided in an embodiment of this application;

[0026] Figure 6 This is a flowchart illustrating a video data processing method in a live streaming scenario provided in an embodiment of this application;

[0027] Figure 7 This is a flowchart illustrating a feature extraction method for a live streaming scenario provided in an embodiment of this application;

[0028] Figure 8 This is a flowchart illustrating a method for determining a target event in a live streaming scenario, as provided in an embodiment of this application.

[0029] Figure 9 This is a flowchart illustrating a method for determining video segments in a live streaming scenario, as provided in an embodiment of this application.

[0030] Figure 10 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of the structure of a computer device provided in this embodiment. Detailed Implementation

[0032] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0033] Currently, dedicated localization network models are commonly used for video temporal localization. Specifically, these models extract video segments of varying lengths and locations from the video to be detected, predict the events described by each segment, and finally adjust the boundaries of the video segments based on the prediction results to obtain the video temporal localization result.

[0034] However, this video temporal positioning method has the following drawbacks:

[0035] 1. Low accuracy of video temporal localization. Training dedicated localization network models typically requires a large amount of video data. If training data is obtained from existing public datasets, the amount of labeled video data in these datasets is limited, and the quality of the video data is generally low. Therefore, the training effect of dedicated localization network models is poor, resulting in low accuracy of video temporal localization.

[0036] 2. Video temporal localization is costly and challenging. During the training of dedicated localization network models, using online video data as training data is difficult and costly due to the fact that online video data is usually protected by copyright. Therefore, video temporal localization is costly and challenging.

[0037] It should be noted that the dedicated localization network here can refer to a model that performs video temporal localization based on video segments in the video to be detected. In other words, this dedicated localization network model is a model specifically designed for video temporal localization.

[0038] Based on this, this application proposes a video data processing method. This method determines K candidate events described by the target video image based on the image features of the target video image and the attribute features corresponding to candidate events. Specifically, it determines the K candidate events based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events. The attribute features are the textual features of the attribute description information of the candidate events. Typically, the matching degree between the attribute description information of two mutually exclusive candidate events is relatively high, resulting in a relatively high matching degree between the attribute features of the two mutually exclusive candidate events. Consequently, the attribute features of two mutually related candidate events will both have a relatively high matching degree with the image features of the target video image, making it likely that mutually exclusive candidate events will appear among the K candidate events described by the target video image. Here, two mutually exclusive candidate events refer to two candidate events that cannot appear simultaneously in the target video image; that is, among two mutually exclusive candidate events, one candidate event is not the event described by the target video image.

[0039] For example, candidate events A1 and A2 are mutually exclusive. Candidate event A1 is "a coin heads up," and its attribute description is "a coin is on the table with the number face up." Candidate event A2 is "a coin tails up," and its attribute description is "a coin is on the table with the suit face up." When the target video image includes a coin, candidate events A1 and A2 cannot both appear in the target video image simultaneously; that is, they are mutually exclusive. However, the matching degree of their attribute descriptions is relatively high. Therefore, when the image features of the target video image match the attribute features of candidate event A1 well, the matching degree between the image features of the target video image and the attribute features of candidate event A2 is also relatively high. In this case, candidate events A1 and A2 will be included among the K candidate events described by the target video image; that is, mutually exclusive candidate events will appear among the K candidate events.

[0040] Therefore, in this application, when K is greater than 1, mutual exclusion detection needs to be performed on K candidate events to obtain mutual exclusion detection results. When the mutual exclusion detection results indicate that there are candidate events with mutual exclusion relationships among the K candidate events, the K candidate events are adjusted based on the mutual exclusion detection results to obtain the target event described by the target video image. According to the target event described by each video image in the video to be detected and the playback time corresponding to each video image, the video to be detected is divided to obtain at least one video segment, so as to realize video temporal localization of the video to be detected.

[0041] In summary, this application has the following advantages:

[0042] 1. Adjusting K candidate events based on mutual exclusion detection results to obtain the target event described by the target video image is beneficial to improving the accuracy of obtaining the target event described by the target video image, thereby improving the accuracy of video temporal localization.

[0043] 2. By using mutual exclusion detection, the accuracy of video temporal localization can be improved. This eliminates the need to train a dedicated localization network model based on a large video dataset, thus reducing the cost and difficulty of video temporal localization.

[0044] First, let's introduce Figure 1 , Figure 1 This is a schematic diagram of the structure of a video data processing system provided in an embodiment of this application, as shown below. Figure 1 As shown, the video data processing system may include a server 10 and a terminal cluster, which may include multiple terminals. Figure 1 As shown, taking a terminal cluster consisting of three terminals as an example, the terminal cluster can specifically include a first terminal, a second terminal, and a third terminal. It can be understood that the first terminal, the second terminal, and the third terminal can all connect to the server via the network so that each terminal can interact with the server through the network connection.

[0045] Server 10 can detect the target events described by each video image in the video to be detected, and divide the video to be detected to obtain video segments describing the target events.

[0046] The video to be detected can be downloaded from the network by the terminal (taking the first terminal as an example), or it can be obtained locally by the first terminal. The server 10 can receive the video to be detected sent by the first terminal, detect the target events described by each video image in the video to be detected, and divide the video to be detected into video segments describing the target events. Specifically, the detection process of the video to be detected can also be implemented by each terminal.

[0047] Understandably, a server can be a single physical server, or a server cluster or distributed system consisting of at least two physical servers. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud knowledge bases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. A terminal can specifically refer to in-vehicle terminals, smartphones, tablets, laptops, desktop computers, smart speakers, speakers with screens, smart TVs, smartwatches, etc., but is not limited to these. Various terminals and servers can be directly or indirectly connected via wired or wireless communication. Furthermore, the number of terminals and servers can be one or at least two; this application does not impose any restrictions.

[0048] Please see Figure 2 , Figure 2 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 2 As shown, this method can be derived from... Figure 1 It can be executed by any terminal in the system, or by... Figure 1 The server in the middle can be used to execute it, or it can be executed by... Figure 1 The terminal and server in the process work together to execute the video data processing method. The device used to execute this method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0049] S101. Obtain the image features of the target video image and the attribute features corresponding to the M candidate events.

[0050] In this application, the computer device can obtain the video to be detected by acquiring it locally or downloading it from the network, extract a video image from the video to be detected, and determine it as the target video image in the video to be detected. The computer device can also extract image features of the target video image, obtain attribute description information corresponding to M candidate events, extract attribute features corresponding to the M attribute description information, and obtain attribute features corresponding to the M candidate events.

[0051] The aforementioned local acquisition can be achieved in any of the following ways: 1. from the local storage of a computer device; 2. by taking a picture with a computer device.

[0052] It should be noted that the target video image can be a video image in the video to be detected, or the target video image can be obtained by extracting frames from the video to be detected using a frame-sampling method.

[0053] Frame extraction methods can refer to either interval-based or continuous frame extraction. Interval-based extraction involves extracting one video image from the video to be tested at intervals of playback duration. For example, if the video to be tested has a playback duration of 10 seconds, and the interval is 0.5 seconds, the computer device can start playback from second 0 and extract one video image from the video to be tested every 0.5 seconds, resulting in 20 video images. This means the playback time difference between any two adjacent video images is 0.5 seconds. Continuous frame extraction involves acquiring each frame of the video to be tested based on its frame rate. The frame rate of the video to be tested indicates the number of video images that can be displayed per second. For example, if the frame rate of the video to be tested is 24 fps (frames per second), then the video to be tested can display 24 video images per second.

[0054] It should be noted that candidate events can refer to objects in an image, stories depicted in an image, scenes depicted in an image, and so on. Objects can refer to items, people, scenery, buildings, etc. Attribute description information can be textual information used to describe the attribute characteristics of candidate events, reflecting the state of objects (such as color, shape, clothing, etc.), the content of the story (such as the time and location of the story), specific information about the scene (such as the people and items in the scene, etc.), and so on.

[0055] For example, candidate event A3 could be "snow mountain", and its attribute description could be "There are mountains in the image, with thick snow covering the peaks, and the peaks shimmering silver in the sunlight". Candidate event A4 could be "desert", and its attribute description could be "Endless sand dunes, with the sand surface showing a wavy texture as the wind blows".

[0056] It should be noted that this application can be applied to film and television scenarios, live streaming scenarios (such as live sports events and live game events), traffic safety scenarios, and other scenarios.

[0057] In film and television scenarios, the video to be detected can refer to film and television videos, such as movies, documentaries, and advertisements. Candidate events can refer to film and television objects (such as movie characters, advertised products, etc.), storylines within the film and television video (such as dialogues between characters, chase scenes, etc.), and scenery within the film and television video (such as snow-capped mountains, deserts, forests, etc.). In sports event live streaming scenarios, the video to be detected can refer to sports competition videos (such as basketball games, swimming competitions, marathon races, etc.). Candidate events can refer to athletes in the competition, key moments in the race (such as basketball goals, swimming sprints, marathon finish lines, etc.). In game event live streaming scenarios, the video to be detected can refer to game competition videos. Candidate events can refer to game characters, game scenes, and the output of game skills, etc. In traffic safety scenarios, the video to be detected can be traffic safety videos. Candidate events can refer to vehicles, pedestrians, pedestrians crossing the road, vehicles running red lights, vehicles crossing lane lines, etc.

[0058] In one embodiment, the candidate events differ for different application scenarios. The candidate events for each application scenario, along with their attribute descriptions, can be stored in a candidate event library. The computer device can retrieve M candidate events from the candidate event library, along with their corresponding attribute descriptions, based on the application scenario of the video to be detected.

[0059] The M candidate events and their descriptions can be generated by the generative model based on the scene description information of the application scenario of the video to be detected, or the M candidate events and their descriptions can be manually generated. The generative model can be any of the following: generative adversarial network model, transformer, long short-term memory network model, etc.

[0060] In one embodiment, the acquisition of image features of the target video image and attribute features corresponding to the M candidate events includes: a computer device extracting initial image features of the target video image and initial attribute features of the attribute description information corresponding to the M candidate events using a feature extraction network. The computer device can apply a regularization formula to regularize the initial image features of the target video image to obtain the image features of the target video image. The computer device can also apply a regularization formula to regularize the initial attribute features corresponding to the M candidate events to obtain the attribute features corresponding to the M candidate events. Regularizing the extracted features using a regularization formula helps prevent overfitting of the feature data and improves the generalization ability of the video data processing method in this application on unseen data, where unseen data refers to training data that did not appear during the training process of the general localization network model.

[0061] It should be noted that initial image features can reflect all content in the target video image, such as the state of objects, the story being told, and the scene being described. Initial attribute features can reflect all content in the attribute description information of candidate events, such as the state of objects, the content of the story, and specific information about the scene. Image features can reflect important content in the target video image, and attribute features can reflect important content in the attribute description information of candidate events. For example, if the target video image is a football match video image, initial image features can reflect the state of the football (such as its shape, color, and position), the state of the football players (such as their clothing and position), and whether the football players are passing the ball. Image features can reflect important content in the target video image, such as the position of the football, the position of the football players, and whether the football players are passing the ball.

[0062] The feature extraction network can be classified as a general localization network model, which can refer to any model with image and text recognition capabilities. Specifically, it can be a vision-language pre-trained model. General localization network models may include contrastive language-image pre-training (CLIP) models, foundational language and vision alignment (FLAVA) models, and so on.

[0063] It should be noted that the visual-language pre-trained model is a technique that combines computer vision and natural language processing to achieve multimodal learning and zero-shot learning. Multimodal learning refers to combining data from two different modalities—images and text—through specific algorithms and techniques for joint learning and representation to achieve the understanding and processing of complex information. Zero-shot learning is a machine learning paradigm that enables visual-language models to recognize and classify events that did not appear during training. Therefore, this application has the following advantages:

[0064] 1. Wide applicability. Combining multimodal learning and zero-shot learning capabilities, feature extraction networks can identify different categories of events, meaning they have a wide range of applications. Therefore, they demonstrate high accuracy in identifying abstract or scene-specific events, which is beneficial for improving the accuracy of video temporal localization.

[0065] 2. Avoids the difficulties of data collection and annotation. Since the feature extraction network can identify different categories of events, it eliminates the need to train dedicated localization network models for identifying different event categories. Therefore, data collection and annotation are unnecessary, avoiding the difficulties associated with these processes.

[0066] It should be noted that the above regularization formula can be formula (1):

[0067]

[0068] When v in formula (1) is the initial image feature, regularize(v) in formula (1) is the image feature. In this case, ||v||2 is the Euclidean norm of the initial image feature, that is, ||v||2 is the square root of the sum of squares of the components in the initial image feature. When v in formula (1) is the initial attribute feature, regularize(v) in formula (1) is the attribute feature. In this case, ||v||2 is the Euclidean norm of the initial attribute feature, that is, ||v||2 is the square root of the sum of squares of the components in the initial attribute feature.

[0069] S102. Based on the image features of the target video image and the attribute features corresponding to the M candidate events, determine the K candidate events described by the target video image.

[0070] In this application, a computer device can compare the image features of a target video image with the attribute features of a first candidate event. When the image features of the target video image match the attribute features of the first candidate event, it indicates that the event described in the target video image may include the first candidate event. Therefore, the computer device can determine that the first candidate event is the candidate event described by the target video image. When the image features of the target video image do not match the attribute features of the first candidate event, it indicates that the event described in the target video image does not include the first candidate event. By sequentially performing the above operations on M candidate events, K candidate events described by the target video image can be determined.

[0071] It should be noted that the matching of the image features of the target video image with the attribute features of the first candidate event can mean that the feature matching degree between the image features of the target video image and the attribute features of the first candidate event is greater than a matching degree threshold. The matching degree threshold can be a default value in the computer device, or it can be manually set.

[0072] In one embodiment, determining K candidate events described by the target video image based on the image features of the target video image and the attribute features corresponding to the M candidate events includes: a computer device using a matching degree calculation formula to perform feature matching degree calculation on the image features of the target video image and the attribute features corresponding to the M candidate events, obtaining the feature matching degree between the target video image and the M candidate events respectively. The K candidate events described by the target video image are determined based on the feature matching degrees between the target video image and the M candidate events respectively. The matching degree calculation formula may include the cosine matching degree formula, the Euclidean distance formula, etc. Quantifying the similarity between the target video image and the candidate events into feature matching degree through the matching degree calculation formula is beneficial for determining the K candidate events described by the target video image.

[0073] It should be noted that feature matching degree can be used to reflect the correlation between the target video image and the candidate events. Generally, when the feature matching degree is greater than a matching degree threshold, a correlation between the target video image and the candidate events can be determined; when the feature matching degree is less than or equal to the matching degree threshold, no correlation can be determined. In one embodiment, determining K candidate events described by the target video image based on the feature matching degrees corresponding to the target video image and the M candidate events includes: the computer device summing the feature matching degrees corresponding to the target video image and the M candidate events to obtain a total matching degree. The computer device can determine the occurrence probability of each of the M candidate events in the target video image based on the total matching degree and the feature matching degrees corresponding to the target video image and the M candidate events. The computer device can determine the K candidate events described by the target video image based on the occurrence probabilities of each of the M candidate events in the target video image.

[0074] It should be noted that the above method of determining the occurrence probability of each of the M candidate events in the target video image based on the sum of matching degrees and the feature matching degrees between the target video image and the M candidate events can include any of the following: 1. Dividing the feature matching degree between the target video image and each candidate event by the sum of matching degrees to obtain the occurrence probability of each candidate event in the target video image; 2. Performing exponential division on the feature matching degree between the target video image and each candidate event by the sum of matching degrees to obtain the occurrence probability of each candidate event in the target video image.

[0075] It should be noted that the occurrence probabilities of the above M candidate events in the above target video image, respectively, and the determination of the K candidate events described by the above target video image, may include the following operations: when the occurrence probability of the first candidate event in the target video image is greater than the occurrence probability threshold, the first candidate event can be determined as the candidate event described by the target video image, and the above operations are performed on the M candidate events in turn until the K candidate events described by the target video image are determined.

[0076] S103. When K is an integer greater than 1, perform mutual exclusion detection on the K candidate events described by the target video image to obtain the mutual exclusion detection result.

[0077] It should be noted that mutual exclusion detection can be used to detect whether there are mutually exclusive candidate events among K candidate events. When mutually exclusive candidate events are detected among the K candidate events, the mutual exclusion detection result can be used to indicate that mutually exclusive candidate events exist among the K candidate events; when no mutually exclusive candidate events are detected among the K candidate events, the mutual exclusion detection result can be used to indicate that no mutually exclusive candidate events exist among the K candidate events.

[0078] It should be noted that mutually exclusive candidate events refer to two logically mutually exclusive candidate events among K candidate events. This logical mutual exclusion can mean that the events are semantically opposite. Specifically, two semantically opposite candidate events cannot appear simultaneously in the target video image.

[0079] Therefore, the existence of mutually exclusive candidate events among the K candidate events indicates that two of the K candidate events cannot appear simultaneously in the target video image. Conversely, the absence of mutually exclusive candidate events among the K candidate events indicates that all K candidate events can appear simultaneously in the target video image.

[0080] For example, candidate event A1 is "the coin is heads up" and candidate event A2 is "the coin is tails up". When there is only one coin in the target video image, a coin cannot be both heads up and tails up at the same time. That is, candidate event A1 and candidate event A2 cannot appear in the target video image at the same time. Candidate event A1 and candidate event A2 are mutually exclusive.

[0081] Therefore, in this application, when there are multiple candidate events described by the target video image, i.e., K is an integer greater than 1, the computer device can perform mutual exclusion detection on every two candidate events among the K candidate events to determine the mutual exclusion relationship between every two candidate events among the K candidate events; generate mutual exclusion detection results to indicate the mutual exclusion relationship between every two candidate events among the K candidate events. By using mutual exclusion detection, the accuracy of video temporal localization can be improved, without the need to train a dedicated localization network model based on a large video dataset, thus reducing the cost and difficulty of video temporal localization.

[0082] For example, given K candidate events, including candidate events A5, A6, and A7, the computer device can first perform mutual exclusion detection on candidate events A5 and A6, that is, detect whether candidate events A5 and A6 can appear simultaneously in the target video image, thus determining the mutual exclusion relationship between candidate events A5 and A6. The computer device can then perform mutual exclusion detection on candidate events A5 and A7, that is, detect whether candidate events A5 and A7 can appear simultaneously in the target video image, thus confirming the mutual exclusion relationship between candidate events A5 and A7. Finally, the computer device can perform mutual exclusion detection on candidate events A6 and A7, that is, detect whether candidate events A6 and A7 can appear simultaneously in the target video image, thus confirming the mutual exclusion relationship between candidate events A6 and A7. If a mutual exclusion relationship is detected between candidate events A5 and A6, a mutual exclusion detection result is generated to indicate that a mutual exclusion relationship exists between candidate events A5 and A6.

[0083] In one embodiment, when K equals 1, the computer device can directly determine one candidate event described by the target video image as the target event described by the target video image.

[0084] S104. When the mutual exclusion detection result indicates that there are mutually exclusive candidate events among the K candidate events, adjust the K candidate events described by the target video image according to the mutual exclusion detection result to obtain the target event described by the target video image.

[0085] It should be noted that the mutual exclusion detection result can also indicate which two candidate events among the K candidate events are mutually exclusive. For example, if the K candidate events include candidate events A5, A6, and A7, when the computer device detects that candidate events A5 and A6 are mutually exclusive, the obtained mutual exclusion detection result can indicate that candidate events A5 and A6 are mutually exclusive.

[0086] In this application, when the mutual exclusion detection result indicates that there are mutually exclusive candidate events among K candidate events, the computer device can determine two mutually exclusive candidate events among the K candidate events described by the target video image based on the mutual exclusion detection result. From the two mutually exclusive candidate events, candidate events that are not related to the target video image are filtered out. The computer device can delete candidate events that are not related to the target video image from the K candidate events to obtain the target event described by the target video image. It can adjust the logically mutually exclusive events among the K candidate events to improve the accuracy of obtaining the target event described by the target video image.

[0087] In one embodiment, taking two mutually exclusive candidate events, candidate events A5 and A6, as an example, the process of filtering out candidate events that are not associated with the target video image from these two mutually exclusive candidate events includes: the computer device can determine the matching degree between candidate events A5 and A6 and the target video image. When the matching degree between candidate event A5 and the target video image is greater than the matching degree between candidate event A6 and the target video image, the computer device can determine candidate event A6 as a candidate event that is not associated with the target video image. When the matching degree between candidate event A6 and the target video image is greater than the matching degree between candidate event A5 and the target video image, the computer device can determine candidate event A5 as a candidate event that is not associated with the target video image.

[0088] It should be noted that the matching degree between the above candidate events A5 and A6 and the target video image can refer to the feature matching degree between the candidate events A5 and A6 and the target video image, respectively.

[0089] In one embodiment, when the mutual exclusion detection result indicates that there are no mutually exclusive candidate events among the K candidate events, the computer device can directly determine the K candidate events described by the target video image as the target event described by the target video image.

[0090] S105. Based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image, the video to be detected is divided to obtain at least one video segment.

[0091] In this application, a computer device can perform steps 101 to 104 as described above on each video image in the video to be detected, thereby obtaining the target event described by each video image in the video to be detected. The computer device can cluster video images that describe the same target event to obtain at least one set of video images. Based on the playback time corresponding to each video image in each set, a video segment corresponding to the target event of that set of video images is extracted from the video to be detected.

[0092] For example, the video to be detected includes video images B1, B2, B3, B4, and B5. The target event described by video image B1 includes target event C1; the target event described by video image B2 includes target event C1; the target event described by video image B3 includes both target events C1 and C2; the target event described by video image B4 includes target event C2; and the target event described by video image B5 includes target event C2. The computer device can cluster video images B1, B2, and B3 to obtain a first set of video images; and cluster video images B3, B4, and B5 to obtain a second set of video images. Furthermore, if the playback time of video image B1 is the 1st second, the playback time of video image B2 is the 2nd second, and the playback time of video image B3 is the 3rd second, the computer device can extract a video segment from the 1st second to the 3rd second from the video to be detected. This extracted video segment is the video segment corresponding to target event C1.

[0093] In this application, the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events is determined by using the image features of the target video image and the attribute features corresponding to the candidate events. Based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events, K candidate events described by the target video image are determined. Since the attribute features of candidate events with mutual exclusion have a relatively high matching degree, it is easy to identify two candidate events with mutual exclusion as candidate events described by the target video image. Two candidate events with mutual exclusion can refer to candidate events that cannot appear in the target video image at the same time, which leads to a relatively low accuracy of the candidate events described by the target video image. Based on this, when K is greater than 1, mutual exclusion detection is performed on the K candidate events to obtain mutual exclusion detection results. When the mutual exclusion detection results indicate that there are candidate events with mutual exclusion among the K candidate events, the K candidate events are adjusted based on the mutual exclusion detection results to obtain the target event described by the target video image. This helps to improve the accuracy of obtaining the target event described by the target video image. Based on the target events described by each video image, the video to be detected is divided to obtain at least one video segment, improving the accuracy of video temporal localization. This method improves the accuracy of video temporal localization by detecting mutual exclusion relationships, eliminating the need to train a dedicated localization network model based on a large video dataset, thus reducing the cost and difficulty of video temporal localization.

[0094] Please see Figure 3 , Figure 3 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 3 As shown, this method can be derived from... Figure 1 It can be executed by any terminal in the system, or by... Figure 1 The server in the middle can be used to execute it, or it can be executed by... Figure 1 The terminal and server in the process work together to execute the video data processing method. The device used to execute this method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0095] S201. Obtain the image features of the target video image and the attribute features corresponding to the M candidate events.

[0096] S202. Based on the image features of the target video image and the attribute features corresponding to the M candidate events, determine the K candidate events described by the target video image.

[0097] S203. When K is an integer greater than 1, obtain the attribute description information corresponding to the K candidate events described by the target video image.

[0098] In this application, when K is an integer greater than 1, and the computer device determines that the number of candidate events described by the target video image is greater than 1, the computer device can obtain the attribute description information corresponding to each of the K candidate events.

[0099] S204. Based on the attribute description information corresponding to each of the K candidate events, determine the joint probability of each pair of candidate events occurring simultaneously.

[0100] It should be noted that joint probability can refer to the probability that two candidate events occur simultaneously in the target video image.

[0101] In this application, a computer device can acquire attribute description information corresponding to each pair of K candidate events. Based on the attribute description information, it analyzes the correlation and matching degree between each pair of candidate events. Based on the correlation and matching degree, it predicts the joint probability of each pair of candidate events occurring simultaneously. By using the joint probability, the simultaneous occurrence of two candidate events in a target video image is quantified, which helps improve the accuracy of mutual exclusion detection.

[0102] In one embodiment, determining the joint probability of the simultaneous occurrence of any two candidate events among the K candidate events based on their respective attribute descriptions includes: The computer device can use semantic recognition methods such as sentiment analysis, entity recognition, relation extraction, and natural language reasoning to perform semantic recognition on the attribute descriptions corresponding to each pair of candidate events among the K candidate events, obtaining the semantic matching degree for each pair of candidate events. The computer device can use a text matching degree formula to determine the text matching degree for each pair of candidate events among the K candidate events based on their respective attribute descriptions. The computer device can determine the joint probability of the simultaneous occurrence of each pair of candidate events among the K candidate events based on the text matching degree and semantic matching degree. Further identifying the semantic state (such as semantic similarity, semantic oppositeness, etc.) between the attribute descriptions corresponding to the two candidate events using semantic recognition methods such as sentiment analysis, entity recognition, relation extraction, and natural language reasoning helps improve the accuracy of mutual exclusion relationship detection. It should be noted that the semantic matching degree can reflect the semantic state (such as semantic similarity, semantic oppositeness, etc.) between candidate events. The aforementioned semantic recognition can include sentiment analysis, entity recognition, relation extraction, natural language reasoning, and so on.

[0103] Sentiment analysis refers to determining the emotional tendency expressed by attribute descriptions. This emotional tendency can include positive, negative, and neutral sentiment. For example, if the attribute description is "The sofa in the living room is fluffy and comfortable to sit on," the emotional tendency expressed by this attribute description can be positive. As another example, if the attribute description is "The river has become murky due to industrial pollution," the emotional tendency expressed by this attribute description can be negative. And as yet another example, if the attribute description is "During the day, the weather is cloudy turning sunny," the emotional tendency expressed by this attribute description can be neutral.

[0104] Entity recognition can refer to identifying entities in attribute description information. Entities can include names of people, places, organizations, objects, etc. For example, when the attribute description information is "The sofa in the living room is fluffy and comfortable to sit on," the entities in the attribute description information are "sofa" and "living room."

[0105] Relation extraction can refer to identifying the relationship between entities in the attribute descriptions of two candidate events. For example, if one attribute description is "the sofa in the living room is fluffy and comfortable to sit on," and the other attribute description is "the sofa in the living room is wooden and has a natural wood color," then both attributes describe entities as "sofa" and "living room," and the relationship between the entities in the two attributes is a parallel relationship.

[0106] Natural language reasoning can refer to judging the logical relationship between two attribute descriptions based on the emotional tendency expressed by attribute description information, the entities in the attribute description information, and the relationship between the entities in the attribute description information. Logical relationships can include implication, contradiction, neutrality, etc.

[0107] S205. Based on the joint probability of each pair of candidate events occurring simultaneously in the K candidate events, perform mutual exclusion detection on the K candidate events to obtain the mutual exclusion detection result.

[0108] In this application, a computer device can filter out candidate events from K candidate events whose joint probability of two candidate events occurring simultaneously is less than a probability threshold, based on the joint probability of each pair of candidate events occurring simultaneously among K candidate events. The computer device can determine that the two selected candidate events are mutually exclusive and generate a mutual exclusion detection result indicating that the two selected candidate events are mutually exclusive.

[0109] In one embodiment, taking a first candidate event and a second candidate event as examples, the above-mentioned mutual exclusion detection is performed on the K candidate events based on the joint probability of each pair of candidate events occurring simultaneously, to obtain a mutual exclusion detection result. This includes: when the joint probability of the first candidate event and the second candidate event occurring simultaneously is less than a probability threshold, it indicates that the first candidate event and the second candidate event have a mutual exclusion relationship, and the computer device can generate a mutual exclusion detection result indicating that the first candidate event and the second candidate event have a mutual exclusion relationship. Conversely, when the joint probability of the first candidate event and the second candidate event occurring simultaneously is not less than the probability threshold, it indicates that there are no mutually exclusive candidate events among the K candidate events, and the computer device can generate a mutual exclusion detection result indicating that there are no mutually exclusive candidate events among the K candidate events. Using a probability threshold to further analyze the joint probability of two candidate events occurring simultaneously, and thus determine the mutual exclusion relationship between the two candidate events, helps to improve the accuracy of mutual exclusion detection.

[0110] S206. When the mutual exclusion detection result indicates that there are mutually exclusive candidate events among the K candidate events, adjust the K candidate events described by the target video image according to the mutual exclusion detection result to obtain the target event described by the target video image.

[0111] In one embodiment, adjusting the K candidate events described by the target video image based on the mutual exclusion detection result to obtain the target event described by the target video image includes: when the mutual exclusion detection result also indicates that the first candidate event and the second candidate event have a mutual exclusion relationship, the computer device can query the first mutual exclusion text of the first candidate event and the second mutual exclusion text of the second candidate event from the mutual exclusion text library. Further, the computer device can filter candidate events that are not related to the target video image from the first and second candidate events based on the first and second mutual exclusion texts. The computer device can then delete the filtered candidate events from the K candidate events to obtain the target event described by the target video image. Both the first and second candidate events belong to the K candidate events. Deleting candidate events that are not related to the target video image from the K candidate events helps improve the accuracy of the target event described by the target video image and improves the accuracy of video temporal localization.

[0112] It should be noted that the mutual exclusion text library can be used to store the mutual exclusion texts corresponding to two candidate events that have a mutual exclusion relationship. Specifically, assuming a first candidate event and a second candidate event are mutually exclusive, the mutual exclusion text of the first candidate event can be used to indicate that the target video image includes the first candidate event but does not include the second candidate event; similarly, the mutual exclusion text of the second candidate event can be used to indicate that the target video image includes the second candidate event but does not include the first candidate event. Because the first and second mutual exclusion texts have opposite semantics, the matching degree between them is low.

[0113] For example, the first candidate event is "river" and the second candidate event is "stream". The first mutually exclusive text is "An image that contains a river instead of a stream", and the second mutually exclusive text is "An image that contains a stream instead of a river". Since the first and second mutually exclusive texts have opposite semantics, their match is low.

[0114] In one embodiment, the process of filtering candidate events unrelated to the target video image from first and second candidate events based on first and second mutually exclusive texts includes: a computer device extracting first mutually exclusive features of the first mutually exclusive text, extracting second mutually exclusive features of the second mutually exclusive text, and extracting image features of the target video image using a feature extraction network. The computer device substitutes the first mutually exclusive features and the image features of the target video image into a feature matching degree formula to obtain a first matching degree between the first mutually exclusive text and the target video image; it also substitutes the second mutually exclusive features and the image features of the target video image into the feature matching degree formula to obtain a first matching degree between the second mutually exclusive text and the target video image. The computer device can compare the first matching degree and the second matching degree. When the first matching degree is greater than the second matching degree, the second candidate event is determined as a candidate event unrelated to the target video image. When the second matching degree is greater than the first matching degree, the first candidate event is determined as a candidate event unrelated to the target video image. The feature matching degree formula may include a cosine matching degree formula, an Euclidean distance formula, etc. By using mutually exclusive text, the matching degree between the target video image and the mutually exclusive event is determined. Based on this, candidate events that are not related to the target video image are filtered out, which helps to improve the accuracy of obtaining the target event described by the target video image and improve the accuracy of video temporal localization.

[0115] Specifically, taking the K candidate events described by the target video image, including the first candidate event and the second candidate event, where the first candidate event is "pedestrian crossing the road" and the second candidate event is "pedestrian not crossing the road", as an example, the process of determining the target event described by the target video image based on mutually exclusive text is explained.

[0116] The target video image can be extracted from traffic safety videos. The K candidate events described by the target video image include the first candidate event being "pedestrian crossing the road" and the second candidate event being "pedestrian not crossing the road". The first candidate event and the second candidate event are mutually exclusive.

[0117] For further information, please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart illustrating a process for determining a target event based on mutually exclusive text, as provided in an embodiment of this application. Figure 4 The video image 41a in the video is the target video image, which includes the road, zebra crossing and pedestrians. Figure 4 The "pedestrian crossing the road" event is the first candidate event. For example... Figure 4 As shown, the computer device can obtain the first mutually exclusive text of the first candidate event from the mutually exclusive text library, that is... Figure 4 The text states that "the pedestrian's position is on the zebra crossing, not on the sidewalk." A feature extraction network is used to extract features from the first mutually exclusive text, yielding the first mutually exclusive features. The computer device can then use a feature extraction network to extract features from the video image 41a, obtaining image features. The computer device can then use a matching degree calculation formula to perform a matching degree calculation on the image features and the first mutually exclusive features, obtaining the first matching degree between the video image 41a and the first candidate event. Figure 4 The first match score is 0.85.

[0118] further, Figure 4 The second candidate event is "the pedestrian did not cross the road". Figure 4 As shown, the computer device can obtain the second mutually exclusive text of the second candidate event from the mutually exclusive text library, that is... Figure 4 The text states that "the pedestrian's position is on the sidewalk, not on the zebra crossing." A feature extraction network is used to extract features from the second mutually exclusive text, yielding the second mutually exclusive features. The computer device can then use a matching degree calculation formula to perform matching degree calculations on the image features and the second mutually exclusive features, obtaining the second matching degree between video image 41a and the second candidate event. Figure 4 The second match score is 0.65.

[0119] Based on this, the computer device can obtain a first matching degree (i.e., "0.85") between the first mutually exclusive text and the target video image, and a second matching degree (i.e., "0.65") between the second mutually exclusive text and the target video image. Since the first matching degree is greater than the second matching degree, the computer device can determine the second candidate event as a candidate event that is not related to the target video image. The second candidate event is deleted from the first and second candidate events described by the target video image, resulting in the target event described by the target video image being the first candidate event (i.e., 0.85). Figure 4 (The phrase "pedestrians crossing the road" in the text).

[0120] S207. Based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image, the video to be detected is divided to obtain at least one video segment.

[0121] In one embodiment, the process of dividing the video to be detected into at least one video segment based on the target events described by each video image and the corresponding playback time of each video image includes: A computer device can cluster the video images in the video to be detected based on the target events described by each video image, grouping video images with the same described target events into a single video image set, thus obtaining N video image sets. Next, the computer device can determine the playback time of each video image based on its occurrence time in the video to be detected. The computer device can then arrange the video images in the N video image sets according to their respective playback times, obtaining N video sequences. Based on the N video sequences, at least one video segment can be determined. Here, video images within the same video image set describe the same target event, and N is a positive integer. This ensures that the target events described by each video image in each video sequence are the same, improving the accuracy of video temporal localization.

[0122] It should be noted that the arrangement rule can be that video images played earlier are placed earlier in the video image set, and video images played later are placed later in the video image set.

[0123] In one embodiment, the method of determining at least one video segment based on N video sequences includes: a computer device performing a difference operation on the playback times of each pair of adjacent video images in the i-th video sequence to obtain the playback time interval between each pair of adjacent video images; where i is a positive integer less than or equal to N. Next, the computer device can group two adjacent video images in the i-th video sequence whose playback time interval is less than a time interval threshold into the same video segment, obtaining the initial video segment corresponding to the i-th video sequence. Ensuring that the playback time interval between each video image in each initial video segment is small helps to guarantee the continuity of the events described by the initial video segment and improves the accuracy of video temporal positioning. Furthermore, when the playback duration of the initial video segment corresponding to the i-th video sequence is greater than a duration threshold, the computer device can determine the initial video segment corresponding to the i-th video sequence as the video segment corresponding to the i-th video sequence. Based on this, invalid initial video segments with excessively short playback durations are excluded, improving the accuracy of video temporal positioning.

[0124] The time interval threshold can be a default value in the computer device, or it can be manually set. The duration threshold can be a default value in the computer device, or it can be manually set.

[0125] For example, the i-th video sequence includes 5 video images: B6, B7, B8, B9, and B10. The playback time of video image B6 is 1 second, B7 is 2 seconds, B8 is 4 seconds, B9 is 7 seconds, and B10 is 8 seconds. The time interval threshold is 3 seconds. Since the playback time interval between video images B6 and B7, and between video images B7 and B8, is less than 3 seconds, video images B6, B7, and B8 are grouped into the same video segment. Since the playback time interval between video images B9 and B10 is equal to 3 seconds, video images B9 and B10 are grouped into the same video segment. Therefore, two initial video segments can be obtained, namely initial video segment D1 and initial video segment D2; initial video segment D1 includes video images B6, B7 and B8, and initial video segment D2 includes video images B9 and B10.

[0126] Next, a duration threshold can be set to 3 seconds. The playback time of video image B6 is 1 second. Since the playback time of video image B7 is 2 seconds and the playback time of video image B8 is 4 seconds, the playback duration of the initial video segment D1 is 4 seconds, which is greater than the duration threshold. The playback time of video image B9 is 7 seconds and the playback time of video image B10 is 8 seconds. The playback duration of the initial video segment D2 is 2 seconds, which is less than the duration threshold. Therefore, the computer device can determine the initial video segment E1 as the video segment corresponding to the i-th video sequence.

[0127] In one embodiment, after determining that the playback duration of the initial video segment corresponding to the i-th video sequence is greater than a duration threshold, the computer device can further obtain the number of video images in the i-th video sequence. When the number of video images in the i-th video sequence is greater than a quantity threshold, the computer device can determine the initial video segment corresponding to the i-th video sequence as the video segment corresponding to the i-th video sequence. A larger number of video images indicates a higher continuity of events described by the initial video segment, thereby improving the accuracy of video temporal positioning.

[0128] The quantity threshold can be a default value in the computer equipment, or it can be set manually.

[0129] For example, after determining that the playback duration of the initial video segment corresponding to the i-th video sequence is greater than the duration threshold, the initial video segments corresponding to the i-th video sequence include initial video segment E1 and initial video segment E2. Initial video segment E1 includes video images B6, B7, and B8, and initial video segment E2 includes video images B9 and B10. When the quantity threshold is 2, since the number of video images in initial video segment E1 is 3, which is greater than the quantity threshold, and the number of video images in initial video segment E2 is 2, which is equal to the quantity threshold, the computer device can determine initial video segment E1 as the video segment corresponding to the i-th video sequence.

[0130] In one embodiment, assuming the number of video segments is S, where S is a positive integer, the computer device can obtain the feature matching degree between each video image in the j-th video segment and its corresponding target event, as well as the probability of the corresponding target event appearing in each video image in the j-th video segment; j is a positive integer less than or equal to S. The computer device can perform an average calculation on the feature matching degrees between each video image in the j-th video segment and its corresponding target event to obtain the average feature matching degree corresponding to the j-th video segment. The computer device can also perform an average calculation on the probability of the target event appearing in each video image in the j-th video segment to obtain the average occurrence probability corresponding to the j-th video segment. The computer device can then sort the S video segments according to their respective average occurrence probabilities and average feature matching degrees to obtain a video segment sequence.

[0131] The mean operation can include weighted average operation, arithmetic average operation, etc.

[0132] For example, the j-th video segment includes video images B6, B7, and B8. The feature matching degree between video image B6 and its corresponding target event is 45, the feature matching degree between video image B7 and its corresponding target event is 50, and the feature matching degree between video image B8 and its corresponding target event is 55. The probability of occurrence of the corresponding target event in video image B6 is 0.2, the probability of occurrence of the corresponding target event in video image B7 is 0.1, and the probability of occurrence of the corresponding target event in video image B8 is 0.15. Therefore, using the arithmetic mean, the computer device can obtain that the average occurrence probability of the j-th video segment is 0.15 and the average feature matching degree is 50.

[0133] In one embodiment, the above-mentioned sorting of the S video segments to obtain a video segment sequence based on the feature matching degree between each video segment and its corresponding target event, and the occurrence probability of the corresponding target event in each video segment, includes: the computer device performing a weighted calculation on the feature matching degree between each video segment and its corresponding target event, and the occurrence probability of the corresponding target event in each video segment, to obtain the importance level of each video segment. Then, the computer device can sort the S video segments according to their respective importance levels to obtain a video segment sequence. This video segment sequence allows users to quickly find video segments with different levels of importance.

[0134] The sorting can be based on the degree of importance. The earlier a video clip appears in the sequence, the more important it is; the later a video clip appears in the sequence, the less important it is.

[0135] In one embodiment, when the importance of the j-th video segment is greater than an importance threshold, the computer device can identify the j-th video segment as key content in the video to be detected.

[0136] In one embodiment, after obtaining the video segment sequence, the computer device can acquire attribute description information of the target events corresponding to the S video segments respectively, and generate key content information corresponding to the S video segments respectively based on the attribute description information. The key content information allows users to easily find the desired video segment from the obtained video segment sequence.

[0137] In this application, the matching degree between the attribute features corresponding to the target video image and the candidate events is determined. Based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events, K candidate events described by the target video image are determined. Since the attribute features of candidate events with mutual exclusion have a relatively high matching degree, it is easy to identify two candidate events with mutual exclusion as candidate events described by the target video image. Two candidate events with mutual exclusion can refer to candidate events that cannot appear in the target video image at the same time, which leads to a relatively low accuracy of the candidate events described by the target video image.

[0138] Furthermore, since the joint probability of two mutually exclusive candidate events in the target video image is low, when K is greater than 1, the joint probability of each pair of candidate events occurring simultaneously is determined by using the attribute description information corresponding to each of the K candidate events. Based on the joint probability of each pair of candidate events occurring simultaneously, mutual exclusion detection is performed on the K candidate events, which improves the detection accuracy. Adjusting the K candidate events based on the mutual exclusion detection results yields the target event described by the target video image, thus improving the accuracy of obtaining the target event described by the target video image. Based on the target event described by each video image, the video to be detected is divided to obtain at least one video segment, improving the accuracy of video temporal localization. In other words, improving the accuracy of video temporal localization through mutual exclusion detection eliminates the need to train a dedicated localization network model based on a large video dataset, reducing the cost and difficulty of video temporal localization.

[0139] Please see Figure 5 , Figure 5This is a flowchart illustrating a video data processing method provided in an embodiment of this application. Figure 5 As shown, this method can be derived from... Figure 1 It can be executed by any terminal in the system, or by... Figure 1 The server in the middle can be used to execute it, or it can be executed by... Figure 1 The terminal and server in the process work together to execute the video data processing method. The device used to execute this method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0140] S301. Obtain the image features of the target video image and the attribute features corresponding to the M candidate events.

[0141] S302. Based on the image features of the target video image and the attribute features corresponding to the M candidate events, determine the K candidate events described by the target video image.

[0142] S303. When K is an integer greater than 1, retrieve the mutex event library.

[0143] It should be noted that the mutual exclusion event library can include one or more mutual exclusion event pairs, and a mutual exclusion event pair can include two events that are mutually exclusive. These mutual exclusion event pairs can be manually configured. By using two known mutually exclusive events and K candidate events to perform mutual exclusion detection, the accuracy of mutual exclusion detection can be improved.

[0144] It should be noted that the mutex event library can be stored in the local storage of the computer device, or it can be an external database, that is, stored on other devices besides the computer device. These other devices can include terminals, servers, server clusters, etc.

[0145] In this application, when the mutex event library is stored in the local storage of a computer device, and K is detected as an integer greater than 1, the computer device can retrieve the mutex event pairs from the mutex event library from the local storage. When the mutex event library can be stored in a device other than a computer device, and K is detected as an integer greater than 1, the computer device can retrieve the mutex event pairs from the mutex event library by establishing a network connection with the other device through the network connection.

[0146] S304. Determine the event matching degree between each pair of candidate events in the K candidate events and the mutual exclusion event pair in the mutual exclusion event library.

[0147] It should be noted that event matching degree can be used to quantify the similarity between each pair of candidate events and mutually exclusive event pairs.

[0148] In this application, the computer device can calculate the matching degree between each pair of candidate events in the K candidate events and the mutual exclusion event pair in the mutual exclusion event library, thereby obtaining the event matching degree between each pair of candidate events in the K candidate events and the mutual exclusion event pair in the mutual exclusion event library.

[0149] For example, there are K candidate events, including candidate event A11 and candidate event A12, where candidate event A11 is "river" and candidate event A12 is "stream". The mutual exclusion event library includes two mutual exclusion event pairs, namely mutual exclusion event pair F1 and mutual exclusion event pair F. Mutual exclusion event pair F1 consists of events A13 (i.e., "river") and A14 (i.e., "stream"), and mutual exclusion event pair F2 consists of events A15 (i.e., "running") and A16 (i.e., "lying down"). The computer device can calculate the event matching degree between candidate events A11 and A12 and events A13 and A14, based on candidate events A11 and A12 (i.e., "river" and "stream"), and events A13 and A14 (i.e., "river" and "stream"). The computer device can calculate the event matching degree between candidate events A11 and A12 and events A15 and A16, based on candidate events A11 and A12 (i.e., “river” and “stream”) and events A15 and A16 (i.e., “running” and “lying down”).

[0150] S305. Based on the event matching degree between every two candidate events in the K candidate events, perform mutual exclusion detection on the K candidate events to obtain the mutual exclusion detection result.

[0151] In this application, a computer device can identify two candidate events with a matching degree greater than a matching degree threshold from among K candidate events as two candidate events with a mutual exclusion relationship, and generate a mutual exclusion detection result indicating that the two candidate events have a mutual exclusion relationship. The matching degree threshold can be a default value in the computer device or a manually set value.

[0152] In one embodiment, the above-mentioned mutual exclusion detection of the K candidate events based on the event matching degree corresponding to every two candidate events among the K candidate events, to obtain a mutual exclusion detection result, includes: when there is an event matching degree greater than a matching degree threshold corresponding to both the first candidate event and the second candidate event among the K candidate events, it indicates that the first candidate event and the second candidate event are a mutually exclusive event pair in the mutually exclusive event library, that is, there is a mutual exclusion relationship between the first candidate event and the second candidate event. The computer device can generate a mutual exclusion detection result indicating that there is a mutual exclusion relationship between the first candidate event and the second candidate event.

[0153] S306. When the mutual exclusion detection result indicates that there are mutually exclusive candidate events among the K candidate events, adjust the K candidate events described by the target video image according to the mutual exclusion detection result to obtain the target event described by the target video image.

[0154] S307. Based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image, the video to be detected is divided to obtain at least one video segment.

[0155] In this application, the matching degree between the attribute features corresponding to the target video image and the candidate events is determined. Based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events, K candidate events described by the target video image are determined. Since the attribute features of candidate events with mutual exclusion have a relatively high matching degree, it is easy to identify two candidate events with mutual exclusion as candidate events described by the target video image. Two candidate events with mutual exclusion can refer to candidate events that cannot appear in the target video image at the same time, which leads to a relatively low accuracy of the candidate events described by the target video image.

[0156] Furthermore, known mutually exclusive event pairs can be pre-stored in a mutual exclusion event database. Based on the event matching degree between each pair of candidate events and the mutually exclusive event pairs in the database, mutual exclusion relationship detection is performed on the K candidate events, improving the accuracy of mutual exclusion relationship detection. Adjusting the K candidate events based on the mutual exclusion detection results yields the target event described by the target video image, further improving the accuracy of obtaining the target event described by the target video image. Based on the target event described by each video image, the video to be detected is divided into segments, obtaining at least one video fragment, improving the accuracy of video temporal localization. In other words, improving the accuracy of video temporal localization through mutual exclusion relationship detection eliminates the need to train a dedicated localization network model based on a large video dataset, reducing the cost and difficulty of video temporal localization.

[0157] Please see Figure 6 , Figure 6 This is a flowchart illustrating a video data processing method for a live streaming scenario provided in an embodiment of this application. Figure 6 As shown, this method can be derived from... Figure 1 It can be executed by any terminal in the system, or by... Figure 1 The server in the middle can be used to execute it, or it can be executed by... Figure 1 The terminal and server in the process work together to execute the video data processing method. The device used to execute this method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0158] S601, Feature Extraction.

[0159] It should be noted that, in the context of live streaming, computer equipment can refer to the backend equipment of the live streaming platform, which can be used to manage various live videos on the platform.

[0160] Therefore, in this application, the computer device can acquire live video from a live streaming platform (i.e., the video to be detected in this application), as well as attribute description information corresponding to M candidate events. The computer device can perform frame extraction processing on the video to be detected to obtain Q video images.

[0161] For example, if the live video is of a basketball game, the computer can perform frame extraction on the live video to obtain video images 61, 62, 63, and 64. The M candidate events can include basketball goal events, basketball miss events, basketball free throw events, basketball steal events, etc.

[0162] Furthermore, the computer device can extract features from Q video images to obtain image features corresponding to each of the Q video images; and extract features from the attribute description information corresponding to each of the M candidate events to obtain attribute features corresponding to each of the M candidate events.

[0163] For the detailed feature extraction process, please refer to [link / reference]. Figure 7 , Figure 7 This is a flowchart illustrating a feature extraction method for a live streaming scenario provided in an embodiment of this application. Figure 7 As shown, the computer device can input Q video images into the image feature extractor 71 in the feature extraction network 70 to obtain the image features corresponding to the Q video images respectively. The computer device can also input the attribute description information corresponding to M candidate events into the attribute feature extractor 72 in the feature extraction network 70 to obtain the attribute features corresponding to the M candidate events respectively.

[0164] For example, a computer device can input video images 61, 62, 63, and 64 into the image feature extractor 71 of the feature extraction network 70 to obtain the image features of video image 61, video image 62, video image 63, and video image 64, respectively. Similarly, the computer device can input the attribute description information corresponding to a basketball goal event into the attribute feature extractor 72 of the feature extraction network 70 to obtain the attribute features corresponding to the basketball goal event.

[0165] S602. Determine the target event described by the video image.

[0166] In this application, a computer device can determine the candidate event described by each video image based on the image features corresponding to Q video images and the attribute features corresponding to M candidate events. Through mutual exclusion detection, the candidate events described by each video image are adjusted to obtain the target event described by each video image.

[0167] It should be noted that the specific operation procedure for step S602 above can be found in [reference needed]. Figure 8 , Figure 8 This is a flowchart illustrating a method for determining a target event in a live streaming scenario, as provided in an embodiment of this application. Figure 8 As shown, the specific operation process includes the following steps:

[0168] S81. Determine the feature matching degree between video images and candidate events; In this application, the computer device can perform feature matching degree calculation on the image features corresponding to Q video images and the attribute features corresponding to M candidate events respectively, to determine the feature matching degree between each video image and each candidate event. The feature matching degree calculation formula (2) can be as follows:

[0169] score i,j =100f i V f i L (2)

[0170] It should be noted that f in formula (2) i V It can refer to the image features of the i-th video image; f i L It can refer to the attribute features of the j-th candidate event; score i,j 100 represents the feature matching degree between the i-th video image and the j-th candidate event; 100 is an adjustable value used to amplify the feature matching degree, making the difference between the feature matching degrees larger.

[0171] For example, computer equipment can... Figure 6 The image features of video image 62 and the attribute features corresponding to the basketball goal event are used to perform feature matching degree calculation to obtain... Figure 6 The feature matching degree between video image 62 and the basketball goal event. Computer equipment can... Figure 6 The image features of video image 62 and the attribute features corresponding to the basketball missed shot event are used to perform feature matching degree calculation to obtain... Figure 6 The feature matching degree between video image 63 and the basketball missed goal event.

[0172] S82. Determine the probability of occurrence of candidate events in video images; In this application, the computer device can also determine the probability of occurrence of each candidate event in each video image based on the feature matching degree corresponding to each video image and each candidate event. The formula (3) for calculating the probability of occurrence can be as follows:

[0173]

[0174] It should be noted that the score in formula (3) i,j It can refer to the feature matching degree between the i-th video image and the j-th candidate event; score i It can refer to the sum of matching degrees corresponding to the i-th video image; prob i,j It can refer to the probability of the j-th candidate event occurring in the i-th video image.

[0175] S83. Mutual Exclusion Detection: In this application, a computer device can determine K candidate events described by the first video image based on the occurrence probabilities corresponding to the first video image and M candidate events. The computer device can perform mutual exclusion detection on the K candidate events described by the first video image to obtain the mutual exclusion detection result corresponding to the first video image. Based on the mutual exclusion detection result corresponding to the first video image, the K candidate events described by the first video image are adjusted to obtain the target event described by the first video image. The above operation is performed on the K candidate events described by Q video images respectively until the target event described by Q video images is obtained.

[0176] For example, if M candidate events include basketball goal events and basketball not scored events. Figure 6 The K candidate events described in video image 62 include basketball goal and basketball miss events. Since basketball goal and basketball miss events cannot occur simultaneously, mutual exclusion detection is performed on the K candidate events described in the first video image to obtain a mutual exclusion detection result indicating the existence of mutually exclusive candidate events among the K candidate events described in video image 62. Based on this mutual exclusion detection result, the K candidate events described in video image 62 are adjusted to obtain the target event described in video image 62. The specific adjustment method can be referred to the previous description; repetitions will not be repeated.

[0177] S603, Clustering Images and Segmenting Videos.

[0178] For the specific operation process of step S603 in this application, please refer to [reference needed]. Figure 9 , Figure 9This is a flowchart illustrating a method for determining video segments in a live streaming scenario, as provided in an embodiment of this application. Figure 9 As shown, the specific operation process includes the following steps:

[0179] S91, Clustering Images; In this application, a computer device can cluster video images that describe the same target event based on Q video images to obtain N video image sets.

[0180] For example, Figure 6 The target events described in video image 61 include basketball goal events. Figure 6 The target events depicted in video image 62 include basketball goal events. Figure 6 The target events depicted in video image 62 include basketball goal events. Figure 6 The target event described by video image 64 includes a basketball goal event. The computer device can cluster video images 61, 62, 63, and 64 to obtain a set of video images.

[0181] S92. Video segmentation: In this application, the video images in the N video image sets are arranged according to the playback time corresponding to each video image to obtain N video sequences, based on which at least one video segment can be determined.

[0182] For example, a collection of video images includes Figure 6 The video images 61, 62, 63, and 64 are shown in the image. Video image 61 plays at the 1st second, video image 62 at the 3rd second, video image 63 at the 5th second, and video image 64 at the 7th second. The computer device can arrange video images 61, 62, 63, and 64 according to their respective playback times to obtain a video sequence. Based on this, the video segment of the basketball goal event can be determined. The method for determining the video segment can be referred to the previous description; repeated details will not be repeated here.

[0183] In one embodiment, taking the number of video segments as S, where S is a positive integer, the computer device can also obtain the feature matching degree between each video image in each video segment and its corresponding target event, as well as the occurrence probability of the corresponding target event in each video image in each video segment. The computer device can also use formula (4) to obtain the feature matching degree between each video segment and its corresponding target event based on the feature matching degree between each video image in each video segment and its corresponding target event.

[0184]

[0185] It should be noted that S in formula (4) refers to the number of video images in each video segment, and the score is... i The feature matching degree between the i-th video image in each video segment and its corresponding target event. Average feature matching degree for each video segment.

[0186] The computer device can also use formula (5) to obtain the average occurrence probability in each video segment based on the occurrence probability of the target event in each video image in each video segment.

[0187]

[0188] It should be noted that S in formula (5) refers to the number of video images in each video segment, and prob i Let be the probability of occurrence of the target event in the i-th video image of each video segment. The average probability of occurrence for each video segment.

[0189] The computer device can, based on the above operations, obtain the average feature matching degree and average occurrence probability of each of the S video segments. Using the average feature matching degree and average occurrence probability of each of the S video segments, as well as the occurrence probability of the target event in each video segment, the computer device can sort the S video segments to obtain a video segment sequence.

[0190] In this application, in a live basketball game scenario, image features of the target video image (i.e., any one of Q video images) and attribute features corresponding to candidate events (such as a successful goal event, a failed goal event, etc.) can be obtained from the video to be detected, thus determining K candidate events described by the target video image. When K is greater than 1, mutual exclusion detection is performed on the K candidate events to obtain mutual exclusion detection results. Based on the mutual exclusion detection results, the K candidate events are adjusted to obtain the target event described by the target video image. Based on this, video segments including successful goal events and video segments including failed goal events in the live video can be determined by the target events described by the Q video images, making it convenient for users to directly obtain the video segments corresponding to each event in the live video.

[0191] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application. Figure 10 As shown, the video data processing device may include:

[0192] The acquisition module 1011 is used to acquire the image features of the target video image and the attribute features corresponding to M candidate events; M is an integer greater than 1; the target video image is a video image in the video to be detected.

[0193] The first processing module 1012 is used to determine the K candidate events described by the target video image based on the image features of the target video image and the attribute features corresponding to the M candidate events respectively.

[0194] The detection module 1013 is used to perform mutual exclusion detection on the K candidate events described by the above target video image when K is an integer greater than 1, and obtain the mutual exclusion detection result;

[0195] The adjustment module 1014 is used to adjust the K candidate events described by the target video image according to the mutual exclusion detection result when the mutual exclusion detection result indicates that there are candidate events with mutual exclusion relationship among the K candidate events, so as to obtain the target event described by the target video image.

[0196] The second processing module 1015 is used to divide the video to be detected into at least one video segment based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image; the target events described by the video images in the same video segment are the same.

[0197] Optionally, the detection module 1013 described above can also be used to perform the following operations:

[0198] Obtain the attribute description information corresponding to the K candidate events described by the target video image above;

[0199] Based on the attribute description information corresponding to the above K candidate events, determine the joint probability of each pair of candidate events occurring simultaneously among the above K candidate events;

[0200] Based on the joint probability of each pair of candidate events occurring simultaneously among the above K candidate events, mutual exclusion detection is performed on the above K candidate events to obtain the mutual exclusion detection results.

[0201] Optionally, the detection module 1013 described above can also be used to perform the following operations:

[0202] Semantic recognition is performed on the attribute description information corresponding to each pair of candidate events in the above K candidate events to obtain the semantic matching degree corresponding to each pair of candidate events in the above K candidate events;

[0203] The text matching degree is calculated by performing text matching degree calculation on the attribute description information corresponding to each pair of candidate events in the above K candidate events;

[0204] Based on the text matching degree and semantic matching degree of each pair of candidate events among the above K candidate events, determine the joint probability of each pair of candidate events occurring simultaneously among the above K candidate events.

[0205] Optionally, the detection module 1013 described above can also be used to perform the following operations:

[0206] When the joint probability of the first candidate event and the second candidate event occurring simultaneously among the above K candidate events is less than the probability threshold, it is determined that the first candidate event and the second candidate event have a mutually exclusive relationship.

[0207] Generate mutually exclusive detection results to indicate that there is a mutually exclusive relationship between the first candidate event and the second candidate event.

[0208] Optionally, the detection module 1013 described above can also be used to perform the following operations:

[0209] Obtain the mutual exclusion event library; the mutual exclusion event library includes mutual exclusion event pairs with mutual exclusion relationships;

[0210] Determine the event matching degree between each pair of candidate events from the above K candidate events and the mutual exclusion event pair in the above mutual exclusion event library;

[0211] Based on the event matching degree between each pair of candidate events among the above K candidate events, mutual exclusion relationship detection is performed on the above K candidate events to obtain mutual exclusion detection results.

[0212] Optionally, the detection module 1013 described above can also be used to perform the following operations:

[0213] When there is an event matching degree that corresponds to both the first candidate event and the second candidate event among the above K candidate events that is greater than the matching degree threshold, it is determined that there is a mutual exclusion relationship between the first candidate event and the second candidate event.

[0214] Generate mutually exclusive detection results to indicate that there is a mutually exclusive relationship between the first candidate event and the second candidate event.

[0215] Optionally, the adjustment module 1014 described above can also be used to perform the following operations:

[0216] When the above mutual exclusion detection result also indicates that the first candidate event and the second candidate event have a mutual exclusion relationship, query the first mutual exclusion text of the first candidate event and the second mutual exclusion text of the second candidate event from the mutual exclusion text library; the first candidate event and the second candidate event both belong to the above K candidate events;

[0217] Based on the first mutually exclusive text and the second mutually exclusive text, candidate events that are not related to the target video image are filtered from the first candidate event and the second candidate event.

[0218] From the above K candidate events, delete the candidate events that are not related to the above target video image to obtain the target event described by the above target video image.

[0219] Optionally, the adjustment module 1014 described above can also be used to perform the following operations:

[0220] Extract the first mutually exclusive features of the first mutually exclusive text, extract the second mutually exclusive features of the second mutually exclusive text, and extract the image features of the target video image;

[0221] Based on the first mutually exclusive feature and the image feature of the target video image, a first matching degree between the first mutually exclusive text and the target video image is determined;

[0222] Based on the aforementioned second mutually exclusive features and the image features of the aforementioned target video image, a second matching degree between the aforementioned second mutually exclusive text and the aforementioned target video image is determined;

[0223] Based on the first matching degree and the second matching degree, candidate events that are not related to the target video image are selected from the first candidate event and the second candidate event.

[0224] Optionally, the adjustment module 1014 described above can also be used to perform the following operations:

[0225] When the first matching degree is greater than the second matching degree, the second candidate event is determined as a candidate event that is not related to the target video image.

[0226] When the first matching degree is less than the second matching degree, the first candidate event is determined as a candidate event that is not related to the target video image.

[0227] Optionally, the second processing module 1015 described above can also be used to perform the following operations:

[0228] Based on the target events described by each video image in the video to be detected, the video images in the video to be detected are clustered to obtain N video image sets; the video images in the same video image set describe the same target events, and N is a positive integer.

[0229] According to the playback time corresponding to each video image, the video images in the above N video image sets are arranged to obtain N video sequences;

[0230] Based on the above N video sequences, determine at least one video segment.

[0231] Optionally, the second processing module 1015 described above can also be used to perform the following operations:

[0232] The playback time interval between any two adjacent video images is determined based on the playback time corresponding to each pair of adjacent video images in the i-th video sequence; i is a positive integer less than or equal to N.

[0233] In the i-th video sequence, two adjacent video images whose playback time interval is less than the time interval threshold are divided into the same video segment to obtain the initial video segment corresponding to the i-th video sequence.

[0234] When the playback duration of the initial video segment corresponding to the i-th video sequence exceeds the duration threshold, the initial video segment corresponding to the i-th video sequence is determined as the video segment corresponding to the i-th video sequence.

[0235] Optionally, the first processing module 1012 described above can also be used to perform the following operations:

[0236] The feature matching degree is calculated by performing feature matching degree calculation on the image features of the target video image and the attribute features corresponding to the M candidate events, respectively, to obtain the feature matching degree between the target video image and the M candidate events respectively;

[0237] Based on the feature matching degrees between the target video image and the M candidate events, the K candidate events described by the target video image are determined.

[0238] Optionally, the first processing module 1012 described above can also be used to perform the following operations:

[0239] The feature matching degrees corresponding to the target video image and the M candidate events are summed to obtain the total matching degree.

[0240] Based on the sum of the matching degrees and the feature matching degrees between the target video image and the M candidate events, the probability of the M candidate events appearing in the target video image is determined.

[0241] Based on the probability of occurrence of the M candidate events in the target video image, the K candidate events described by the target video image are determined.

[0242] Optionally, the second processing module 1015 described above can also be used to perform the following operations:

[0243] Obtain the feature matching degree between each video image in the j-th video segment and its corresponding target event, as well as the probability of the corresponding target event appearing in each video image in the j-th video segment; j is a positive integer less than or equal to S;

[0244] The average feature matching degree between each video image in the j-th video segment and its corresponding target event is calculated by averaging the feature matching degree to obtain the average feature matching degree corresponding to the j-th video segment.

[0245] The average occurrence probability of the target event in each video image of the j-th video segment is calculated by averaging the probability of occurrence in the j-th video segment.

[0246] Based on the average occurrence probability and average feature matching degree of the S video segments respectively, the S video segments are sorted to obtain a video segment sequence.

[0247] Optionally, the second processing module 1015 described above can also be used to perform the following operations:

[0248] The importance of each of the S video segments is determined by weighting the average occurrence probability and average feature matching degree of the corresponding S video segments.

[0249] Based on the importance of the S video segments, the S video segments are sorted to obtain a video segment sequence.

[0250] Optionally, the acquisition module 1011 described above can also be used to perform the following operations:

[0251] The initial image features of the target video image are extracted through a feature extraction network, as well as the initial attribute features of the attribute description information corresponding to M candidate events.

[0252] The initial image features of the target video image are regularized to obtain the image features of the target video image.

[0253] The initial attribute features corresponding to the above M candidate events are regularized to obtain the attribute features corresponding to the above M candidate events.

[0254] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0255] In this application, the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events is determined by using the image features of the target video image and the attribute features corresponding to the candidate events. Based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events, K candidate events described by the target video image are determined. Since the attribute features of candidate events with mutual exclusion have a relatively high matching degree, it is easy to identify two candidate events with mutual exclusion as candidate events described by the target video image. Two candidate events with mutual exclusion can refer to candidate events that cannot appear in the target video image at the same time, which leads to a relatively low accuracy of the candidate events described by the target video image. Based on this, when K is greater than 1, mutual exclusion detection is performed on the K candidate events to obtain mutual exclusion detection results. When the mutual exclusion detection results indicate that there are candidate events with mutual exclusion among the K candidate events, the K candidate events are adjusted based on the mutual exclusion detection results to obtain the target event described by the target video image. This helps to improve the accuracy of obtaining the target event described by the target video image. Based on the target events described by each video image, the video to be detected is divided to obtain at least one video segment, improving the accuracy of video temporal localization. This method improves the accuracy of video temporal localization by detecting mutual exclusion relationships, eliminating the need to train a dedicated localization network model based on a large video dataset, thus reducing the cost and difficulty of video temporal localization.

[0256] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 11As shown, the aforementioned computer device 1100 can refer to a server or terminal, including: a processor 1101, a network interface 1104, and a memory 1105. Furthermore, the aforementioned computer device 1100 may also include: a user interface 1103, and at least one communication bus 1102. The communication bus 1102 is used to implement communication between these components. In some embodiments, the user interface 1103 may include a display screen and a keyboard; optionally, the user interface 1103 may also include a standard wired interface or a wireless interface. The network interface 1104 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1105 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. The memory 1105 may also optionally be at least one storage device located remotely from the aforementioned processor 1101. Figure 11 As shown, the memory 1105, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and computer programs.

[0257] exist Figure 11 In the computer device 1100 shown, the network interface 1104 provides network communication functionality; the user interface 1103 is mainly used to provide an input interface; and the processor 1101 can be used to call computer programs stored in the memory 1105 to execute:

[0258] Obtain the image features of the target video image and the attribute features corresponding to M candidate events; M is an integer greater than 1; the target video image is a video image in the video to be detected.

[0259] Based on the image features of the target video image and the attribute features corresponding to the M candidate events, determine the K candidate events described by the target video image.

[0260] When K is an integer greater than 1, mutual exclusion detection is performed on the K candidate events described by the target video image to obtain the mutual exclusion detection result;

[0261] When the mutual exclusion detection result indicates that there are mutually exclusive candidate events among the K candidate events, the K candidate events described by the target video image are adjusted according to the mutual exclusion detection result to obtain the target event described by the target video image.

[0262] Based on the target events described by each video image in the video to be detected, and the playback time corresponding to each video image, the video to be detected is divided to obtain at least one video segment; the target events described by the video images within the same video segment are the same.

[0263] In this application, the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events is determined by using the image features of the target video image and the attribute features corresponding to the candidate events. Based on the matching degree between the image features of the target video image and the attribute features corresponding to the candidate events, K candidate events described by the target video image are determined. Since the attribute features of candidate events with mutual exclusion have a relatively high matching degree, it is easy to identify two candidate events with mutual exclusion as candidate events described by the target video image. Two candidate events with mutual exclusion can refer to candidate events that cannot appear in the target video image at the same time, which leads to a relatively low accuracy of the candidate events described by the target video image. Based on this, when K is greater than 1, mutual exclusion detection is performed on the K candidate events to obtain mutual exclusion detection results. When the mutual exclusion detection results indicate that there are candidate events with mutual exclusion among the K candidate events, the K candidate events are adjusted based on the mutual exclusion detection results to obtain the target event described by the target video image. This helps to improve the accuracy of obtaining the target event described by the target video image. Based on the target events described by each video image, the video to be detected is divided to obtain at least one video segment, improving the accuracy of video temporal localization. This method improves the accuracy of video temporal localization by detecting mutual exclusion relationships, eliminating the need to train a dedicated localization network model based on a large video dataset, thus reducing the cost and difficulty of video temporal localization.

[0264] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program executed by the aforementioned data processing apparatus. This computer program includes program instructions, which, when executed by the processor, enable the execution of the data processing method described in the corresponding embodiments above. Therefore, further details will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0265] As an example, the above program instructions can be deployed and executed on a computer device, or deployed and executed on at least two computer devices in one location, or executed on at least two computer devices distributed in at least two locations and interconnected by a communication network. At least two computer devices distributed in at least two locations and interconnected by a communication network can form a blockchain network.

[0266] The aforementioned computer-readable storage medium may be a data processing apparatus provided in any of the foregoing embodiments or a central storage unit of the aforementioned computer device, such as a hard disk or central storage of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both the central storage unit and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0267] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish content in different media, rather than to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0268] In practice, the collection and processing of data in this application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the data subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the data subject.

[0269] This application also provides a computer program product, including a computer program. When executed by a processor, the computer program implements the data processing method and decoding method described in the preceding embodiments, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0270] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0271] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable network-connected device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable network-connected device, generate instructions for implementing the process... Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable network-connected device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable network-connected device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0272] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A video data processing method, characterized in that, include: Obtain the image features of the target video image, and the attribute features corresponding to the M candidate events; M is an integer greater than 1; the target video image is a video image in the video to be detected; Based on the image features of the target video image and the attribute features corresponding to the M candidate events, determine the K candidate events described by the target video image; When K is an integer greater than 1, mutual exclusion detection is performed on the K candidate events described by the target video image to obtain the mutual exclusion detection result; When the mutual exclusion detection result indicates that there are mutually exclusive candidate events among the K candidate events, the K candidate events described by the target video image are adjusted according to the mutual exclusion detection result to obtain the target event described by the target video image; Based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image, the video to be detected is divided into at least one video segment; the target events described by the video images within the same video segment are the same.

2. The method according to claim 1, characterized in that, The step of performing mutual exclusion relationship detection on the K candidate events described by the target video image to obtain mutual exclusion detection results includes: Obtain the attribute description information corresponding to the K candidate events described by the target video image; Based on the attribute description information corresponding to the K candidate events, determine the joint probability of each pair of candidate events occurring simultaneously. Based on the joint probability of each pair of candidate events occurring simultaneously among the K candidate events, mutual exclusion detection is performed on the K candidate events to obtain the mutual exclusion detection result.

3. The method according to claim 2, characterized in that, The step of determining the joint probability of every two candidate events occurring simultaneously among the K candidate events based on the attribute description information corresponding to each of the K candidate events includes: Semantic recognition is performed on the attribute description information corresponding to each pair of candidate events in the K candidate events to obtain the semantic matching degree corresponding to each pair of candidate events in the K candidate events; The text matching degree is calculated by performing a text matching degree calculation on the attribute description information corresponding to each pair of candidate events in the K candidate events; Based on the text matching degree and semantic matching degree of each pair of candidate events in the K candidate events, determine the joint probability of each pair of candidate events occurring simultaneously.

4. The method according to claim 2, characterized in that, The step of performing mutual exclusion detection on the K candidate events based on the joint probability of each pair of candidate events occurring simultaneously, and obtaining the mutual exclusion detection result, includes: When the joint probability of the first candidate event and the second candidate event occurring simultaneously among the K candidate events is less than the probability threshold, it is determined that the first candidate event and the second candidate event have a mutually exclusive relationship. Generate a mutual exclusion detection result to indicate that there is a mutual exclusion relationship between the first candidate event and the second candidate event.

5. The method according to claim 1, characterized in that, The step of performing mutual exclusion relationship detection on the K candidate events described by the target video image to obtain mutual exclusion detection results includes: Obtain a mutual exclusion event library; the mutual exclusion event library includes mutual exclusion event pairs with mutual exclusion relationships; Determine the event matching degree between every two candidate events among the K candidate events and the mutual exclusion event pairs in the mutual exclusion event library; Based on the event matching degree between every two candidate events among the K candidate events, mutual exclusion detection is performed on the K candidate events to obtain mutual exclusion detection results.

6. The method according to claim 5, characterized in that, The step of performing mutual exclusion detection on the K candidate events based on the event matching degree between every two candidate events to obtain mutual exclusion detection results includes: When there is an event matching degree that is greater than the matching degree threshold for both the first candidate event and the second candidate event among the K candidate events, it is determined that there is a mutual exclusion relationship between the first candidate event and the second candidate event. Generate a mutual exclusion detection result to indicate that there is a mutual exclusion relationship between the first candidate event and the second candidate event.

7. The method according to claim 1, characterized in that, The step of adjusting the K candidate events described by the target video image based on the mutual exclusion detection results to obtain the target event described by the target video image includes: When the mutual exclusion detection result also indicates that the first candidate event and the second candidate event have a mutual exclusion relationship, the first mutual exclusion text of the first candidate event and the second mutual exclusion text of the second candidate event are queried from the mutual exclusion text library; both the first candidate event and the second candidate event belong to the K candidate events; Based on the first mutually exclusive text and the second mutually exclusive text, candidate events that are not related to the target video image are filtered from the first candidate events and the second candidate events; From the K candidate events, delete the candidate events that are not related to the target video image to obtain the target event described by the target video image.

8. The method according to claim 7, characterized in that, The step of filtering candidate events that are not related to the target video image from the first candidate events and the second candidate events based on the first mutually exclusive text and the second mutually exclusive text includes: Extract the first mutual exclusion feature of the first mutually exclusive text, extract the second mutual exclusion feature of the second mutually exclusive text, and extract the image features of the target video image; Based on the first mutually exclusive feature and the image features of the target video image, a first matching degree between the first mutually exclusive text and the target video image is determined; Based on the second mutually exclusive feature and the image features of the target video image, a second matching degree between the second mutually exclusive text and the target video image is determined; Based on the first matching degree and the second matching degree, candidate events that are not related to the target video image are filtered from the first candidate events and the second candidate events.

9. The method according to claim 8, characterized in that, The step of filtering candidate events that are not related to the target video image from the first candidate events and the second candidate events based on the first matching degree and the second matching degree includes... When the first matching degree is greater than the second matching degree, the second candidate event is determined as a candidate event that is not related to the target video image; When the first matching degree is less than the second matching degree, the first candidate event is determined as a candidate event that is not related to the target video image.

10. The method according to claim 1, characterized in that, The step of dividing the video to be detected into at least one video segment based on the target event described by each video image in the video to be detected and the playback time corresponding to each video image includes: Based on the target events described by each video image in the video to be detected, the video images in the video to be detected are clustered to obtain N video image sets; the video images in the same video image set describe the same target events, and N is a positive integer; According to the playback time corresponding to each video image, the video images in the N video image sets are arranged to obtain N video sequences; Based on the N video sequences, at least one video segment is determined.

11. The method according to claim 10, characterized in that, The step of determining at least one video segment based on the N video sequences includes: The playback time interval between any two adjacent video images is determined based on the playback time corresponding to each pair of adjacent video images in the i-th video sequence; i is a positive integer less than or equal to N. In the i-th video sequence, two adjacent video images whose playback time interval is less than the time interval threshold are divided into the same video segment to obtain the initial video segment corresponding to the i-th video sequence; When the playback duration of the initial video segment corresponding to the i-th video sequence is greater than the duration threshold, the initial video segment corresponding to the i-th video sequence is determined as the video segment corresponding to the i-th video sequence.

12. The method according to claim 1, characterized in that, The step of determining K candidate events described by the target video image based on the image features of the target video image and the attribute features corresponding to the M candidate events includes: The feature matching degree is calculated by performing feature matching degree calculation on the image features of the target video image and the attribute features corresponding to the M candidate events respectively, to obtain the feature matching degree between the target video image and the M candidate events respectively; Based on the feature matching degrees between the target video image and the M candidate events, K candidate events described by the target video image are determined.

13. The method according to claim 12, characterized in that, The step of determining the K candidate events described by the target video image based on the feature matching degrees corresponding to the target video image and the M candidate events includes: The feature matching degrees corresponding to the target video image and the M candidate events are summed to obtain the total matching degree. Based on the sum of the matching degrees and the feature matching degrees between the target video image and the M candidate events respectively, the probability of the M candidate events appearing in the target video image is determined. Based on the occurrence probabilities of the M candidate events in the target video image, K candidate events describing the target video image are determined.

14. The method according to claim 1, characterized in that, The number of video segments is S, where S is a positive integer; the method further includes: Obtain the feature matching degree between each video image in the j-th video segment and its corresponding target event, as well as the probability of the corresponding target event appearing in each video image in the j-th video segment; j is a positive integer less than or equal to S; The average feature matching degree between each video image in the j-th video segment and its corresponding target event is calculated by averaging the feature matching degree to obtain the average feature matching degree corresponding to the j-th video segment. The average occurrence probability of the target event in each video image of the j-th video segment is calculated by averaging the probability of occurrence in the j-th video segment. Based on the average occurrence probability and average feature matching degree of the S video segments respectively, the S video segments are sorted to obtain a video segment sequence.

15. The method according to claim 1, characterized in that, The acquisition of image features of the target video image and attribute features corresponding to the M candidate events includes: The initial image features of the target video image are extracted through a feature extraction network, as well as the initial attribute features of the attribute description information corresponding to M candidate events. The initial image features of the target video image are regularized to obtain the image features of the target video image; The initial attribute features corresponding to the M candidate events are regularized to obtain the attribute features corresponding to the M candidate events.

16. A video data processing apparatus, characterized in that, include: The acquisition module is used to acquire the image features of the target video image in the video to be detected, as well as the attribute features corresponding to the M candidate events; M is an integer greater than 1; The first processing module is used to determine K candidate events described by the target video image based on the image features of the target video image and the attribute features corresponding to the M candidate events respectively; The detection module is used to perform mutual exclusion detection on K candidate events described by the target video image when K is an integer greater than 1, and obtain mutual exclusion detection results. An adjustment module is used to adjust the K candidate events described by the target video image according to the mutual exclusion detection result when the mutual exclusion detection result indicates that there are candidate events with mutual exclusion relationship among the K candidate events, so as to obtain the target event described by the target video image; The second processing module is used to divide the video to be detected into at least one video segment based on the target events described by each video image in the video to be detected and the playback time corresponding to each video image; the target events described by the video images in the same video segment are the same.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 15.

18. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, performs the method as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 15.

Images