Video recognition method and device, computer device and storage medium

Abstract

This application relates to a video recognition method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring a target video and a reference video from a video series; performing image matching of video frames between the target video and the reference video to obtain video frame pairs; each video frame pair includes a video frame to be identified belonging to the target video, and a reference video frame in the reference video whose image matches the video frame to be identified; determining a time offset of the video frame pair based on the time attributes of the video frames to be identified and the reference video frames in the video frame pair; filtering video frame pairs with matching time offsets, and determining similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs. This method can improve the accuracy of identifying similar video segments in a video.

Description

[0001] This application is a divisional application of the invention patent application with application number 202210461009.7, application date April 28, 2022, entitled "Video Recognition Method, Apparatus, Computer Equipment and Storage Medium". Technical Field

[0002] This application relates to the field of computer technology, and in particular to a video recognition method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0003] With the development of computer technology, various online video platforms have emerged. Besides online film and television resources, people can also create a wide variety of videos on these platforms, including lecture series, knowledge sharing, arts and culture classes, current affairs commentary, and entertainment videos, satisfying viewers' new viewing needs. In series videos, similar video segments such as opening and closing credits are often created, but currently, the accuracy of identifying these similar segments is relatively low. Summary of the Invention

[0004] Therefore, it is necessary to provide a video recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of identifying similar video segments in a video, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a video recognition method. The method includes:

[0006] Retrieve target and reference videos from video series;

[0007] The target video and the reference video are image matched frame by frame to obtain video frame pairs; the video frame pairs include the video frame to be identified belonging to the target video, and also the reference video frame in the reference video that matches the image of the video frame to be identified.

[0008] The time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair.

[0009] Filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs.

[0010] Obtain a list of similar video segments from the target video relative to each reference video; in the list of similar video segments, each similar video segment is sorted from largest to smallest statistical value, and similar video segments with the same statistical value are sorted from earliest to latest start time;

[0011] In the list of similar segments, segment intersection analysis is performed on each similar video segment to obtain the analysis results;

[0012] By updating the earlier similar video segments in the target video through the later similar video segments in each similar video segment that have an intersection relationship, a common video segment in the target video relative to each reference video is obtained.

[0013] In the list of similar segments, the preceding similar video segments are positioned before the following similar video segments.

[0014] In one embodiment, determining the start time and end time based on the timestamps of the video frames to be identified in a video frame pair belonging to the same video segment includes: determining a start video frame pair and an end video frame pair from the video frame pairs belonging to the same video segment based on the timestamps of the video frames to be identified in the video frame pair belonging to the same video segment; obtaining the start time based on the timestamps of the video frames to be identified in the start video frame pair; and obtaining the end time based on the timestamps of the video frames to be identified in the end video frame pair.

[0015] In one embodiment, the video recognition method further includes: determining the start time and end time of the public video; in response to a video comparison trigger event, extracting a non-public video from the target video based on the start time and end time; and performing a video comparison between the non-public video and the video to be compared.

[0016] In one embodiment, the video recognition method further includes: determining the jump time point of the public video; playing the target video in response to a video playback event for the target video; and skipping the public video and playing it when the playback progress of the target video reaches the jump time point.

[0017] Secondly, this application also provides a video recognition device. The device includes:

[0018] The video acquisition module is used to acquire target videos and reference videos from video series.

[0019] The video frame pair acquisition module is used to perform image matching of video frames between the target video and the reference video to obtain video frame pairs; the video frame pairs include the video frames to be identified belonging to the target video, and also include the reference video frames in the reference video that match the images of the video frames to be identified.

[0020] The offset determination module is used to determine the time offset of the video frame pair based on the time attributes of the video frame to be identified and the time attributes of the reference video frame.

[0021] The similar segment determination module is used to filter video frame pairs with matching time offsets, and based on the time attributes of the video frames to be identified in the filtered video frame pairs, determine the similar video segments in the target video relative to the reference video.

[0022] The common video segment determination module is used to obtain a list of similar video segments in the target video relative to each reference video. In the list of similar video segments, each similar video segment is sorted from largest to smallest according to its statistical value, and similar video segments with the same statistical value are sorted from earliest to latest according to their start time. In the list of similar video segments, segment intersection analysis is performed on each similar video segment to obtain the analysis results. By using the later similar video segments among the similar video segments with intersection relationships, the earlier similar video segments are updated to obtain the common video segments in the target video relative to each reference video. Among them, the earlier similar video segments are in the similar segment list in a sorted position before the later similar video segments.

[0023] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0024] Retrieve target and reference videos from video series;

[0025] The target video and the reference video are image matched frame by frame to obtain video frame pairs; the video frame pairs include the video frame to be identified belonging to the target video, and also the reference video frame in the reference video that matches the image of the video frame to be identified.

[0026] The time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair.

[0027] Filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs.

[0028] Obtain a list of similar video segments from the target video relative to each reference video; in the list of similar video segments, each similar video segment is sorted from largest to smallest statistical value, and similar video segments with the same statistical value are sorted from earliest to latest start time;

[0029] In the list of similar segments, segment intersection analysis is performed on each similar video segment to obtain the analysis results;

[0030] By updating the earlier similar video segments in the target video through the later similar video segments in each similar video segment that have an intersection relationship, a common video segment in the target video relative to each reference video is obtained.

[0031] In the list of similar segments, the preceding similar video segments are positioned before the following similar video segments.

[0032] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0033] Retrieve target and reference videos from video series;

[0034] The target video and the reference video are image matched frame by frame to obtain video frame pairs; the video frame pairs include the video frame to be identified belonging to the target video, and also the reference video frame in the reference video that matches the image of the video frame to be identified.

[0035] The time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair.

[0036] Filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs.

[0037] Obtain a list of similar video segments from the target video relative to each reference video; in the list of similar video segments, each similar video segment is sorted from largest to smallest statistical value, and similar video segments with the same statistical value are sorted from earliest to latest start time;

[0038] In the list of similar segments, segment intersection analysis is performed on each similar video segment to obtain the analysis results;

[0039] By updating the earlier similar video segments in the target video through the later similar video segments in each similar video segment that have an intersection relationship, a common video segment in the target video relative to each reference video is obtained.

[0040] In the list of similar segments, the preceding similar video segments are positioned before the following similar video segments.

[0041] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0042] Retrieve target and reference videos from video series;

[0043] The target video and the reference video are image matched frame by frame to obtain video frame pairs; the video frame pairs include the video frame to be identified belonging to the target video, and also the reference video frame in the reference video that matches the image of the video frame to be identified.

[0044] The time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair.

[0045] Filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs.

[0046] Obtain a list of similar video segments from the target video relative to each reference video; in the list of similar video segments, each similar video segment is sorted from largest to smallest statistical value, and similar video segments with the same statistical value are sorted from earliest to latest start time;

[0047] In the list of similar segments, segment intersection analysis is performed on each similar video segment to obtain the analysis results;

[0048] By updating the earlier similar video segments in the target video through the later similar video segments in each similar video segment that have an intersection relationship, a common video segment in the target video relative to each reference video is obtained.

[0049] In the list of similar segments, the preceding similar video segments are positioned before the following similar video segments.

[0050] The aforementioned video recognition method, apparatus, computer equipment, storage medium, and computer program product perform image matching of video frames between a target video and a reference video in a video series to obtain video frame pairs, including a video frame to be identified belonging to the target video and a reference video frame that matches the image of the video frame to be identified. The time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair, and video frame pairs with matching time offsets are selected. Based on the time attributes of the video frame to be identified in the selected video frame pairs, similar video segments relative to the reference video are determined from the target video. For target and reference videos in a video series, the time offset of video frame pairs is determined based on the time attributes of the video frames to be identified and the time attributes of the reference video frames in the image matching video frames. By selecting video frame pairs with matching time offsets, the time attributes of the video frames to be identified in the target video are used to determine similar video segments in the target video relative to the reference video. Similar video segments of different durations can be flexibly determined based on the video frame pairs of image matching. Then, based on the list of similar segments composed of similar video segments in the target video relative to each reference video, the earlier similar video segments are updated by using the later similar video segments among the similar video segments with intersection relationships. This ensures that similar video segments with high statistical value can be accurately retained, which can improve the effect of segment update and help improve the accuracy of identifying common video segments from the target video. Attached Figure Description

[0051] Figure 1 This is a diagram illustrating the application environment of the video recognition method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a video recognition method in one embodiment;

[0053] Figure 3 This is a flowchart illustrating the process of determining similar video segments in one embodiment;

[0054] Figure 4 This is a schematic diagram of the intro and outro extraction process in one embodiment;

[0055] Figure 5 This is a flowchart illustrating the application of video matching for intro and outro data mining in one embodiment;

[0056] Figure 6 This is a flowchart illustrating the application of intro / outro tag skipping in one embodiment;

[0057] Figure 7 This is a flowchart illustrating the process of extracting intro and outro segments for different business types in one embodiment.

[0058] Figure 8This is a schematic diagram of the processing flow of an image multi-task model in one embodiment;

[0059] Figure 9 This is a schematic diagram of the intro and outro extraction process in another embodiment;

[0060] Figure 10 This is a flowchart illustrating the process of correcting the beginning and end credits in one embodiment;

[0061] Figure 11 This is a flowchart illustrating the intro and outro extraction process in yet another embodiment.

[0062] Figure 12 This is a schematic diagram of the structure of an image multitasking model in one embodiment;

[0063] Figure 13 This is a schematic diagram of model branch shared inference in one embodiment;

[0064] Figure 14 This is a schematic diagram illustrating non-shared inference in model branches in one embodiment;

[0065] Figure 15 This is a schematic diagram of matching segment information in one embodiment;

[0066] Figure 16 This is a schematic diagram illustrating the time period in one embodiment;

[0067] Figure 17 This is a schematic diagram illustrating the overlapping time periods in one embodiment for updating the endpoint time;

[0068] Figure 18 This is a schematic diagram illustrating the overlapping time periods in one embodiment, where the starting time for updating is updated.

[0069] Figure 19 This is a schematic diagram illustrating the update process when time periods overlap in one embodiment.

[0070] Figure 20 This is a schematic diagram illustrating the storage of video and intro / outro information in one embodiment;

[0071] Figure 21 This is a structural block diagram of a video recognition device in one embodiment;

[0072] Figure 22 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0074] The video recognition method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another server. Server 104 can perform image matching of video frames between target video and reference video in a video series, obtaining video frame pairs including the video frame to be identified belonging to the target video and the reference video frame that matches the image of the video frame to be identified. Based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair, the time offset of the video frame pair is determined, and video frame pairs with matching time offsets are filtered. Based on the time attributes of the video frame to be identified in the filtered video frame pairs, similar video segments relative to the reference video are determined from the target video. When the target video is played on terminal 102, server 104 can send segment information of similar video segments in the target video relative to the reference video to terminal 102. Terminal 102 can skip the playback of similar video segments in the target video based on the received segment information. If the similar video segment is the beginning or end of a video, the beginning or end can be skipped, improving the video playback efficiency of terminal 102. Furthermore, the video recognition method provided in this application can be executed by the terminal or server alone, or by both the terminal and server, to achieve video recognition processing.

[0075] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server, a server cluster consisting of multiple servers, or a cloud server.

[0076] In one embodiment, such as Figure 2 As shown, a video recognition method is provided. This method can be executed by a terminal or a server, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0077] Step 202: Obtain the target video and reference video from the video series.

[0078] In this context, a video series refers to a collection of multiple videos belonging to the same series. The classification within a series can be tailored to specific needs. For example, a single TV series can be considered part of the same series, and the collection of individual videos from that series constitutes its video series. Similarly, videos created by the same creator can be considered part of the same series, and the collection of those videos can be considered a video series. Furthermore, a series can also consist of videos on the same theme, videos produced in the same location, and so on. A video series includes multiple videos, and these videos may have similar segments. For instance, videos created by the same creator might begin with an intro introducing the creator or end with a summary of the video. The intro typically indicates the start of the video, while the outro indicates its end. Intros and outros can take many forms, including but not limited to audio / video materials, text, and logos.

[0079] The target video is the video in the video series that needs to be identified. This means identifying video segments from the target video that are similar to other videos in the series, such as the opening and closing credits. Reference videos are used as a reference for target video identification. These reference videos can be sampled from the video series. By comparing the target video and the reference video, similar video segments relative to the reference video are identified from the target video. Both the target and reference videos belong to the same video series. Similar video segments may exist between videos within the same series, allowing for skipping during playback or effective editing of the video content. The number of reference videos can be set according to actual needs. For example, the longer the target video, the larger the number of reference videos can be; the more videos in the video series, the larger the number of reference videos can be. You can also set the number of reference videos to a fixed percentage of the number of videos included in the video series. For example, you can set it to 50%. If the number of videos included in the video series is 20, then the number of reference videos can be 10, that is, 10 videos are extracted from the video series excluding the target video as reference videos.

[0080] Specifically, when a video recognition event is triggered, the server obtains the target video and reference video from the video series. Specifically, the server can determine the video series to which the video recognition event is targeted, query the video series, determine the target video from the video series, and extract the reference video from the video series, thereby obtaining the target video and reference video belonging to the same video series.

[0081] Step 204: Perform image matching of video frames between the target video and the reference video to obtain video frame pairs; the video frame pairs include the video frames to be identified belonging to the target video, and also include the reference video frames in the reference video that match the images of the video frames to be identified.

[0082] In this context, a video frame refers to each image frame in a video. A video is composed of multiple video frames, meaning it contains multiple video frames. Image matching refers to matching video frames in a target video with video frames in a reference video to determine the matching video frames. For example, similarity matching can be performed between video frames in the target video and video frames in the reference video. Video frames that meet a similarity threshold are identified as matching video frames between the target and reference videos. A video frame pair is an image pair consisting of successfully matched video frames determined through image matching between the target and reference videos. A video frame pair includes the video frame to be identified from the target video and a reference video frame from the reference video that matches the image frame to be identified. In other words, the video frame to be identified and the reference video frame in the video frame pair are obtained through successful image matching; the video frame to be identified comes from the target video, and the reference video frame comes from the reference video.

[0083] Specifically, the server performs image matching of video frames between the target video and the reference video. This matching can be done by comparing video frames from the target video with frames from the reference video, for example, based on image similarity. The matching results then determine video frame pairs. A video frame pair is an image pair consisting of successfully matched video frames. In video frame pairs determined by similarity-based image matching, the image similarity between the target video frame and the reference video frame is high; that is, the target video frame and the reference video frame are quite similar and may contain the same video content, such as frames from the opening or closing credits.

[0084] Step 206: Determine the time offset of the video frame pair based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair.

[0085] The time attribute describes the timing information of the corresponding video frame, representing its position within the video. Specifically, the time attribute can be a timestamp or a frame number. For example, a time attribute of 2.0S indicates that the video frame is at the 2.0-second mark in its video; similarly, a time attribute of 500 indicates that the video frame is the 500th frame in its video. The time attribute marks the position of a video frame within its video and determines its appearance time. A video is composed of multiple video frames combined according to their timing information, and each video frame has a time attribute carrying this information. The time offset represents the time interval between the appearance time of the video frame to be identified in the target video and the appearance time of the reference video frame in the reference video. The time offset is derived from the time attributes of both the video frame to be identified and the reference video frame. For example, in a pair of video frames, the time attribute of the video frame to be identified can be the 2nd second, meaning the video frame to be identified is the 2nd second video frame in the target video frame, while the time attribute of the reference video frame can be the 3rd second, meaning the reference video frame is the 3rd second video frame in the reference video frame. In other words, the 2nd second video frame in the target video and the 3rd second video frame in the reference video frame match, so the time offset of the video frame pair can be obtained as 1 second based on the difference between the time attributes of the video frame to be identified and the time attributes of the reference video frame.

[0086] Specifically, for each obtained video frame pair, the server determines the time attribute of the video frame to be identified and the time attribute of the reference video frame. This can be done by querying the frame information of both the video frame to be identified and the reference video frame. Based on the time attributes of the obtained video frame to be identified and the reference video frame, the server determines the time offset of the video frame pair. For example, when the time attribute is a quantized value, the server can obtain the time offset of the video frame pair based on the numerical difference between the time attributes of the video frame to be identified and the reference video frame.

[0087] Step 208: Filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs.

[0088] In this context, similar video clips refer to video segments in the target video that are similar to segments in the reference video. If a similar video clip in the target video is similar to a segment in the reference video, then the similar video clip may be duplicated content between the target and reference videos, such as reused content from the target and reference videos, specifically reused content from intros, outros, or advertisements. The similar video clips are determined based on the temporal attributes of the video frames to be identified in the selected video frame pairs, thus allowing for flexible identification of similar video clips of varying lengths and adapting to various types of repetitive or reused video content in different video formats.

[0089] Specifically, the server filters video frame pairs based on time offsets, selecting those with matching time offsets. Specifically, the server can select video frame pairs with the same or similar time offset values ​​within a certain range. Based on the selected video frame pairs, the server determines the time attributes of the video frames to be identified within those pairs, and uses these time attributes to obtain similar video segments in the target video relative to the reference video. For example, after determining the time attributes of the video frames to be identified within the selected pairs, the server can determine the start and end times based on the numerical values ​​of the time attributes of each frame, thereby identifying similar video segments in the target video based on the start and end times.

[0090] In practical applications, the server can group video frames according to their time offset values, obtaining sets of video frame pairs corresponding to different time offsets. Each set includes video frame pairs with matching time offsets. For example, if the obtained video frame pairs have time offsets of 1s, 4s, and 5s, the server can use video frame pairs with a time offset of 1s as the first set and determine similar video segments in the target video based on the time attributes of the video frames to be identified within this first set. The server can also use video frame pairs with time offsets of 4s and 5s as the second set and determine similar video segments in the target video based on the time attributes of the video frames to be identified within this second set. The server can determine similar video segments for each set by analyzing the time attributes of the video frames to be identified in each set, and then merge these similar video segments. For example, the server can delete duplicate similar video segments and update partially overlapping similar video segments to obtain similar video segments in the target video relative to the reference video.

[0091] Furthermore, after identifying similar video segments in the target video, these similar video segments may be reused within the target video, such as intros, outros, or advertisements. When playing the target video, these similar video segments can be skipped, thereby improving playback efficiency. Additionally, in video comparison applications, if multiple videos in a video series contain reused intros, outros, or advertisements, and comparison is not required, similar video segments can be extracted from the target video. This allows for video comparison of other video segments within the target video, reducing the amount of data processed and improving processing efficiency.

[0092] In the aforementioned video recognition method, image matching of video frames is performed between the target video and the reference video in a video series. This yields video frame pairs, including the video frame to be identified belonging to the target video and the reference video frame that matches the image of the video frame to be identified. The time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the reference video frame in the video frame pair. Video frame pairs with matching time offsets are then selected. Based on the time attributes of the video frame to be identified in the selected video frame pairs, similar video segments relative to the reference video are determined from the target video. For the target video and reference video in a video series, the time offset of the video frame pair is determined based on the time attributes of the video frame to be identified and the reference video frame obtained through image matching. By using the time attributes of the video frame to be identified in the selected video frame pairs with matching time offsets, similar video segments relative to the reference video in the target video are determined. This method allows for flexible determination of similar video segments of varying durations based on image-matched video frame pairs, improving the accuracy of similar video segment recognition in various types of videos.

[0093] In one embodiment, such as Figure 3 As shown, the process of identifying similar video segments involves filtering video frame pairs with matching time offsets. Based on the time attributes of the video frames to be identified in the filtered video frame pairs, similar video segments in the target video relative to the reference video are determined, including:

[0094] Step 302: Perform numerical matching of the time offset of each video frame pair, and filter out the video frame pairs whose time offset values ​​match based on the numerical matching results.

[0095] The time offset represents the time interval between the appearance time of the video frame to be identified in the target video and the appearance time of the reference video frame in the reference video. The time offset is a quantized numerical value, such as a value in seconds, representing the time difference in seconds between the appearance times of the video frame to be identified and the reference video frame in their respective videos. Numerical matching refers to matching the numerical values ​​of the time offsets of each video frame pair to obtain a numerical matching result. The numerical matching result can include the numerical difference between the time offsets of each video frame pair, that is, the difference in the time offset values.

[0096] Specifically, when identifying similar video segments, the server performs numerical matching on the time offsets of each pair of video frames. Specifically, it matches the time offsets of every pair of video frames to obtain numerical matching results. Based on these results, the server determines video frame pairs whose time offsets match. For example, the numerical matching results may include the numerical difference between the time offsets of each pair of video frames. The server can determine time offsets where the difference between the time offsets of each pair of video frames is less than a preset threshold as numerically matched time offsets, thus selecting video frame pairs based on these numerically matched time offsets.

[0097] Step 304: Determine the start and end times based on the time attributes of the video frames to be identified in the selected video frame pairs.

[0098] The start time refers to the time when the video segment begins, and the end time refers to the time when the video segment ends. Based on the start and end times, the start time is taken as the beginning time of the video, and the end time is taken as the end time of the video. Thus, the duration of the video from the start time to the end time can be used to determine the corresponding video segment.

[0099] Specifically, after obtaining the video frame pairs with matching time offset values, the server determines the time attribute of the video frame to be identified within the selected video frame pair. This can be done by querying the frame information of each video frame to be identified, thus obtaining the time attribute. Based on the time attribute of the video frame to be identified, the server determines the start and end times. In practical applications, after obtaining the time attributes of the video frames to be identified from the selected video frame pair, the server can determine the time attribute with the smallest value and use it as the start time. Similarly, the server can determine the time attribute with the largest value and use it as the end time. For example, in one application, if the array of time attributes of the video frames to be identified in the selected video frame pair is {1,3,4,5,6,7,8,9,10,12,15}, the server can use 1 second as the start time and 15 seconds as the end time.

[0100] Step 306: Based on the start and end times, determine similar video segments from the target video relative to the reference video.

[0101] Specifically, the server determines similar video segments in the target video relative to the reference video based on the start and end times. For example, the server can identify video segments in the target video between the start and end times as similar video segments. For instance, if the server determines that 1 second is the start time and 15 seconds is the end time, the server can identify the video segment from second 1 to second 15 of the target video as a similar video segment relative to the reference video.

[0102] In this embodiment, the time offsets of video frame pairs are numerically matched, and video frame pairs with matching time offset values ​​are selected based on the numerical matching results. Based on the time attributes of the video frames to be identified in the selected video frame pairs, the start and end times are determined, and similar video segments in the target video are determined based on the start and end times. Thus, based on the video frames to be identified in the selected video frame pairs, similar video segments are determined from the target video. This method can flexibly determine similar video segments based on the video frames to be identified at the frame level, and is applicable to videos that include similar video segments of varying durations, thereby improving the accuracy of similar video segment identification in various videos.

[0103] In one embodiment, the time offsets of each video frame pair are numerically matched, and video frame pairs with matching time offset values ​​are selected based on the numerical matching results. This includes: comparing the time offsets of each video frame pair numerically to obtain a numerical comparison result; selecting video frame pairs from each video frame pair whose time offset difference is less than a numerical difference threshold based on the numerical comparison result; and updating the offsets of the video frame pairs whose time offset difference is less than the numerical difference threshold to obtain video frame pairs with matching time offset values.

[0104] Numerical comparison refers to comparing the numerical values ​​of the time offsets of each video frame pair to obtain a comparison result, which can include the numerical difference between the time offsets of each video frame pair. For example, if the time offset of video frame pair 1 is 1 second and the time offset of video frame pair 2 is 2 seconds, then the numerical difference in time offset between video frame pair 1 and video frame pair 2 is 1 second. The numerical difference threshold can be flexibly set according to actual needs. The numerical difference threshold is used to match the time offsets of each video frame pair. Specifically, video frame pairs whose numerical difference in time offset is less than the numerical difference threshold can be selected as the filtered video frame pairs. Offset update updates the time offsets of video frame pairs whose numerical difference in time offset is less than the numerical difference threshold to ensure that the time offsets of the video frame pairs match. For example, the time offsets of video frame pairs can be updated to be uniformly the same.

[0105] Specifically, the server compares the time offsets of each video frame pair numerically, obtaining comparison results. These results include the differences between the time offsets of each pair, which can be obtained by the server subtracting each pair of time offsets. The server determines a pre-set threshold for these differences and, based on the comparison results, selects video frame pairs whose time offset differences are less than the threshold. Specifically, the server compares the differences in the comparison results with the threshold to identify video frame pairs associated with time offsets whose differences are less than the threshold, and selects these video frame pairs from among all video frame pairs. The server updates the time offsets of these video frame pairs whose time offset differences are less than the threshold. Specifically, the time offsets of these video frame pairs can be updated to the same value, such as updating them to the minimum time offset among video frame pairs whose time offset differences are less than the threshold, thus obtaining video frame pairs with matching time offset values. For example, if the numerical difference threshold is 2s, among the video frame pairs whose time offset numerical difference is less than the threshold, the time offset includes two types: 1s and 2s. Then the server can update the time offset of the video frame pair with a time offset of 2s to 1s, thereby obtaining each video frame pair with a time offset of 1s, that is, obtaining video frame pairs with matching time offset values.

[0106] In this embodiment, based on the numerical comparison results obtained by comparing the time offsets of each video frame pair, video frame pairs with a time offset difference less than a threshold are selected from the video frame pairs. The selected video frame pairs are then updated with offsets to obtain video frame pairs with matching time offset values. This process is used to select video frame pairs for determining similar video segments. The selected video frame pairs can accurately identify similar video segments relative to the reference video from the target video.

[0107] In one embodiment, determining the start and end times based on the time attributes of the video frames to be identified in the selected video frame pairs includes: obtaining a list of video frame pairs composed of the selected video frame pairs; in the video frame pair list, each video frame pair is sorted in ascending order of its time offset value, and video frame pairs with the same time offset are sorted in ascending order of the timestamp values ​​of the included video frames to be identified, the timestamps being determined based on the time attributes of the included video frames to be identified; in the video frame pair list, determining the time attribute distance between the time attributes of the video frames to be identified in adjacent video frame pairs; identifying adjacent video frame pairs whose time attribute distance does not exceed a distance threshold as video frame pairs belonging to the same video segment; and determining the start and end times based on the timestamps of the video frames to be identified in the video frame pairs belonging to the same video segment.

[0108] The video frame pair list consists of sorted selected video frame pairs. Within this list, the selected video frame pairs are sorted in ascending order of their time offset values. For video frame pairs with the same time offset, they are further sorted in ascending order of the timestamp values ​​of the included video frames to be identified. The timestamp is determined based on the temporal attributes of the included video frames to be identified; it represents the time point when the video frame to be identified appears in the target video. In the video frame pair list, the smaller the time offset, the higher the ranking. For video frame pairs with the same time offset, the smaller the timestamp of the included video frames to be identified, the higher the ranking. The temporal attribute distance is determined by the temporal attributes of adjacent video frame pairs in the video frame pair list, representing the time interval between adjacent video frame pairs. The distance threshold is preset according to actual needs and is used to determine whether they belong to the same video segment. Specifically, adjacent video frame pairs whose time attribute distance does not exceed the distance threshold can be identified as video frame pairs belonging to the same video segment. This allows for the aggregation of video segments from each video pair, thereby determining the start and end times.

[0109] Specifically, the server obtains a list of video frame pairs obtained by sorting the selected video frame pairs. In this list, each video frame pair is sorted in ascending order of its time offset value, and video frame pairs with the same time offset are sorted in ascending order of the timestamp values ​​of the included video frames to be identified. In practical applications, after selecting the video frame pairs, the server can sort them in ascending order of their time offset values. For video frame pairs with the same time offset, the server can determine the timestamps of the video frames to be identified included in that pair and sort them in ascending order of their timestamp values, thus obtaining the list of video frame pairs. Within this list, the server compares the time attributes of the video frames to be identified in adjacent video frame pairs, specifically by calculating the difference between their respective time attributes to obtain the time attribute distance. The server determines a preset distance threshold, compares the temporal attribute distance with this threshold, and, based on the comparison result, identifies adjacent video frame pairs from the list whose temporal attribute distance does not exceed the threshold. These adjacent video frame pairs are then identified as belonging to the same video segment; that is, if the temporal attribute distance between the video frames to be identified in adjacent video frame pairs is small, they can be considered to belong to the same video segment. Thus, video segments are aggregated based on the video frames to be identified within the video frame pairs. The server determines the timestamps of the video frames to be identified within the video frame pairs belonging to the same video segment, and determines the start and end times based on these timestamps. For example, the server can determine the start time based on the timestamp with the smallest value and the end time based on the timestamp with the largest value. The determined start and end times are the start and end times of the video segment to which the video frame pairs belonging to the same video segment belong.

[0110] In this embodiment, based on the video frame pair list composed of selected video frame pairs, the video frame pairs belonging to the same video segment are determined according to the time attribute distance between the time attributes of the video frames to be identified in adjacent video frame pairs. The start time and end time are determined according to the timestamp of the video frames to be identified in the video frame pairs belonging to the same video segment. This can realize the reasoning and mining from the video frames to be identified to the video segments, which is beneficial to accurately identify similar video segments from the target video.

[0111] In one embodiment, determining the start time and end time based on the timestamps of the video frames to be identified in a video frame pair belonging to the same video segment includes: determining a start video frame pair and an end video frame pair from the video frame pairs belonging to the same video segment based on the timestamps of the video frames to be identified in the video frame pair belonging to the same video segment; obtaining the start time based on the timestamps of the video frames to be identified in the start video frame pair; and obtaining the end time based on the timestamps of the video frames to be identified in the end video frame pair.

[0112] The timestamp of the video frame to be identified is determined based on its time attribute, representing the point in time when it appears in the target video. The starting and ending video frame pairs are determined based on the timestamp values ​​of the video frames to be identified within each video frame pair belonging to the same video segment. The timestamp of the video frame to be identified in the starting video frame pair can be the smallest among the timestamps of the video frames to be identified in each video frame pair belonging to the same video segment; while the timestamp of the video frame to be identified in the ending video frame pair can be the largest. Therefore, the video frames to be identified in the starting video frame pair are determined as the starting video frames of the same video segment, and the video frames to be identified in the ending video frame pair are determined as the ending video frames of the same video segment.

[0113] Specifically, the server determines the timestamps of the video frames to be identified within a pair of video frames belonging to the same video segment. Based on the values ​​of each timestamp, the server determines the start and end video frame pairs from the pair of video frames belonging to the same video segment. Specifically, the server determines the video frame pair containing the video frame with the smallest timestamp as the start video frame pair, and the video frame pair containing the video frame with the largest timestamp as the end video frame pair. The server obtains the start time based on the timestamps of the video frames to be identified in the start video frame pair; for example, the time point corresponding to that timestamp can be determined as the start time. Similarly, the server obtains the end time based on the timestamps of the video frames to be identified in the end video frame pair; for example, the time point corresponding to that timestamp can be determined as the end time.

[0114] In this embodiment, the server determines the starting video frame pair and the ending video frame pair based on the timestamp of the video frame to be identified in the video frame pair belonging to the same video segment. The server then determines the start time and the end time according to the video frame to be identified included in each of the starting video frame pair and the ending video frame pair. This enables reasoning and mining from the video frame to be identified belonging to the same video segment to the video segment, thereby improving the accuracy of identifying similar video segments from the target video.

[0115] In one embodiment, the video recognition method further includes: determining the segment intersection relationship between the similar video segments based on their respective start and end times; and updating the segments of each similar video segment based on the segment intersection relationship to obtain the updated similar video segments in the target video relative to the reference video.

[0116] If multiple similar video segments are identified from the target video relative to the reference video, these segments can be updated based on their intersection relationships to obtain updated similar video segments. The intersection relationship refers to the shared relationships between similar video segments. For example, if the time range of similar video clip A is (2,5), that is, from the 2nd second to the 5th second of the target video, and the time range of similar video clip B is (3,4), then similar video clip A completely covers similar video clip B. In this case, similar video clip B can be deleted, while similar video clip A is retained. If the time range of similar video clip C is (2,6) and the time range of similar video clip D is (5,8), then similar video clip C and similar video clip D have some overlap. In this case, similar video clip C and similar video clip D can be expanded and updated to obtain the updated similar video clip CD (2,8). If the time range of similar video clip F is (4,8) and the time range of similar video clip F is (1,5), then similar video clip E and similar video clip F have some overlap. In this case, similar video clip E and similar video clip F can be expanded and updated to obtain the updated similar video clip EF (1,8). Furthermore, if multiple similar video segments do not intersect, such as (2,5) and (7,10), then merging these segments is unnecessary; the segments without intersection can be directly identified as video recognition results. Different update methods can be set for different segment intersection relationships to ensure the accuracy of updating similar video segments.

[0117] Specifically, when multiple similar video segments are obtained, the server can determine the segment intersection relationship between the similar video segments based on their respective start and end times. This intersection relationship can be inclusive, partial, or nonexistent. Based on this intersection relationship, the server updates each similar video segment. Specifically, it can perform processes such as merging, deleting, or retaining segments to obtain the updated similar video segments in the target video relative to the reference video.

[0118] In this embodiment, when multiple similar video segments are identified, the segments are updated based on the intersection relationship between the segments, thereby obtaining more accurate similar video segments and improving the accuracy of identifying similar video segments from the target video.

[0119] In one embodiment, the video recognition method further includes: updating the segments of similar video segments in the target video that have an intersection relationship with each reference video, so as to obtain a common video segment in the target video relative to each reference video.

[0120] The system requires at least two reference videos. This means that similar video segments relative to the target video can be obtained by performing video recognition processing on the target video and one or more reference videos. Intersection relationship refers to the common connection between similar video segments identified based on different reference videos. Specifically, it can be determined based on the time endpoints of each identified similar video segment, including start and end times. Common video segments are pre-made video segments shared by all videos in a video series. These pre-made video segments can be reused across all videos in the video series. In practical applications, common video segments can be various reusable video segments such as video intros, outros, and commercials.

[0121] Specifically, the server can acquire more than one reference video and perform video recognition processing on the target video and each of the reference videos separately to obtain similar video segments in the target video relative to each reference video. The server then updates the segments of the target video that have intersection relationships with the reference videos, thereby obtaining common video segments in the target video relative to all the reference videos. In practical applications, the server determines the start and end times of each similar video segment in the target video relative to each reference video, and based on the start and end times of each similar video segment, determines the similar video segments that have intersection relationships. The server then updates the segments of these overlapping similar video segments, such as by deleting, merging, or retaining them, to obtain common video segments in the target video relative to all the reference videos.

[0122] In this embodiment, video recognition is performed on the target video using multiple reference videos, and the segments of each similar video segment are updated based on the intersection relationship of the identified similar video segments to obtain common video segments in the target video relative to each of the reference videos. This makes the accuracy of identifying common video segments by referring to multiple reference videos higher, thereby improving the accuracy of identifying common video segments from the target video.

[0123] In one embodiment, updating the segments of similar video segments in the target video that have an intersection relationship with each reference video to obtain a common video segment in the target video relative to each reference video includes: performing segment intersection analysis on the similar video segments in the target video relative to each reference video to obtain analysis results; and updating the segments of the similar video segments that have an intersection relationship as determined by the analysis results to obtain a common video segment in the target video relative to each reference video.

[0124] Common video clips are reusable video segments within a video series, specifically including intros, outros, and commercials. Segment intersection analysis involves analyzing the intersections between similar video clips identified from various reference videos. The analysis results can include whether there are intersection relationships between the similar video clips identified from each reference video. If intersection relationships exist, the segments with intersection relationships can be updated to obtain the common video clips in the target video relative to each reference video.

[0125] Specifically, the server identifies similar video segments in the target video relative to each reference video. The server performs segment intersection analysis on these similar video segments, determining the start and end times of each segment and then performing the analysis based on these times. If the analysis indicates no intersection relationship, these similar video segments do not require processing and can be retained as common video segments in the target video relative to each reference video. If the analysis indicates an intersection relationship, meaning there is segment overlap between the similar video segments, the server identifies these overlapping segments and updates them, such as deleting, merging, or retaining segments, to obtain the common video segments in the target video relative to each reference video.

[0126] In this embodiment, by performing segment intersection analysis on similar video segments in the target video relative to each reference video, and updating the segments of each similar video segment that have an intersection relationship as determined by the analysis results, common video segments can be determined using similar video segments identified from multiple reference videos, thereby improving the accuracy of identifying common video segments from the target video.

[0127] In one embodiment, updating the segments of similar video segments that have an intersection relationship as determined by the analysis results to obtain common video segments in the target video relative to each reference video includes: determining the similar video segments that have an intersection relationship as determined by the analysis results; updating the segments of the similar video segments that have an intersection relationship based on the intersection duration and statistics of the similar video segments that have an intersection relationship to obtain common video segments in the target video relative to each reference video.

[0128] The intersection duration refers to the duration of the intersecting segments between similar video segments that have an intersection relationship. For example, if the time range of similar video segment A determined from the first reference video is (2, 8), and the time range of similar video segment B determined from the second reference video is (5, 10), then there is an intersection relationship between similar video segment A and similar video segment B. The intersecting segment is (5, 8), and the intersection duration is 4 seconds between the 5th and 8th seconds. The statistical measure can include the number of times the same similar video segment is identified relative to each reference video in the target video. The larger the statistical value, the more times the corresponding similar video segment is identified, and the greater the probability that the similar video segment belongs to a shared video segment.

[0129] Specifically, when updating segments of similar video clips that have an intersection relationship as determined by the analysis results, the server identifies these segments and determines their respective statistics and the duration of the intersection. Based on the intersection duration and statistics, the server updates the segments to obtain common video clips in the target video relative to the reference videos. The server can then determine whether merging is necessary based on the length of the intersection duration and whether to retain or merge segments based on the statistical values.

[0130] In this embodiment, the segments are updated based on the intersection duration and statistics of the segments that have an intersection relationship. This update combines the characteristics of the segments and improves the effectiveness of the segment update, which is beneficial for improving the accuracy of identifying common video segments from the target video.

[0131] In one embodiment, segment intersection analysis is performed on similar video segments in the target video relative to each reference video to obtain analysis results. This includes: obtaining a list of similar segments composed of similar video segments in the target video relative to each reference video; in the list of similar segments, each similar video segment is sorted from largest to smallest according to its statistical value, and similar video segments with the same statistical value are sorted from earliest to latest according to their start time; and segment intersection analysis is performed on each similar video segment in the list of similar segments to obtain analysis results.

[0132] The similarity segment list is composed of sorted video segments from the target video relative to each reference video. Within the list, similar video segments are sorted from largest to smallest statistical value, and segments with the same statistical value are sorted from earliest to latest start time. In other words, the similarity segment list is first sorted by statistical value from largest to smallest, and then segments with the same statistical value are sorted by start time from earliest to latest.

[0133] Specifically, the server obtains a list of similar video segments from the target video relative to each reference video. This list can be pre-sorted by the server based on the similar video segments; specifically, it can first sort the segments according to their statistical magnitude from largest to smallest. For segments with the same statistical magnitude, the server sorts them according to their start time from earliest to latest, thus obtaining the list of similar video segments. The server then performs segment intersection analysis on these segments to obtain the analysis results. In practical applications, the server can perform segment intersection analysis sequentially from earliest to latest, according to the sorting of the similar video segments in the list, to obtain the analysis results.

[0134] Furthermore, the analysis results show that each similar video segment has an intersection relationship. The segments are then updated to obtain common video segments in the target video relative to each reference video. This includes updating the segments of the preceding similar video segments by using the later similar video segments among the similar video segments with an intersection relationship. The preceding similar video segments are in the sorted position before the later similar video segments in the similar segment list.

[0135] In this context, "preceding similar video clips" refers to the segment that appears before "following similar video clips" in the list of similar video clips. Specifically, compared to "preceding similar video clips," "following similar video clips" refers to the segment that appears later in the list of similar video clips with overlapping relationships. Conversely, compared to "following similar video clips," "preceding similar video clips" refers to the segment that appears earlier in the list of similar video clips. For example, if the list of similar video clips includes clips A and B, and clip A has a higher statistical value than clip B, then clip A will appear before clip B in the list of similar video clips. Therefore, clip B can be considered "following similar video clips," while clip A can be considered "preceding similar video clips."

[0136] Specifically, the server can determine the later similar video segments and the earlier similar video segments among the similar video segments that have an intersection relationship. The server updates the earlier similar video segments by determining the later similar video segments, such as by deleting, merging, retaining and other update processes, to obtain the common video segments in the target video relative to each reference video.

[0137] In this embodiment, based on the list of similar segments composed of similar video segments in the target video relative to each reference video, the preceding similar video segments are updated by using the later similar video segments among the similar video segments that have an intersection relationship. This can ensure that similar video segments with high statistical value can be accurately retained, which can improve the effect of segment update and help improve the accuracy of identifying common video segments from the target video.

[0138] In one embodiment, updating the segments of similar video segments in the target video that have an intersection relationship with each reference video to obtain a common video segment in the target video relative to each reference video includes: updating the segments of similar video segments in the target video that have an intersection relationship with each reference video to obtain an updated similar video segment; determining the statistics of the updated similar video segment; and when the statistics of the updated similar video segment exceed a statistics threshold, obtaining a common video segment in the target video relative to each reference video based on the updated similar video segment.

[0139] The statistics can include the cumulative number of times the same similar video segment is identified relative to each reference video within the target video. The statistical threshold is used to determine whether the updated similar video segments are valid common video segments; the statistical threshold can be set according to actual needs.

[0140] Specifically, the server updates the similar video segments in the target video that have intersection relationships with each reference video, obtaining updated similar video segments. The server determines the statistical value of the updated similar video segments by performing statistical processing on them. The server determines a preset statistical threshold; when the statistical value of the updated similar video segments exceeds the threshold, they are considered valid common video segments. Based on the updated similar video segments, the server obtains the common video segments in the target video relative to each reference video. For example, the server can use the updated similar video segments as common video segments in the target video relative to each reference video.

[0141] In this embodiment, the validity of the updated similar video segments is determined by a statistical threshold. After the validity determination, the common video segments in the target video relative to each reference video are obtained based on the updated similar video segments, which can ensure the validity of the identified common video segments.

[0142] In one embodiment, the video recognition method further includes: when a public video segment meets the determination criteria for a public video type, obtaining a public video in the target video that matches the public video type based on the public video segment.

[0143] The term "public video type" refers to the type of public video, which may include, but is not limited to, intros, outros, and advertisements. The public video type can be set according to actual needs. The criteria for determining the public video type are used to determine whether the type of a public video segment matches the public video type. Specifically, the distribution area of ​​public videos associated with the public video type is compared with the public video segment to determine whether the public video segment matches the public video type, thus identifying the type of the public video segment. A public video matching a public video type means that the types of public videos match each other. A public video is a reusable video segment of a defined type. For example, a public video can be intros, outros, or advertisements—video content that can be reused in various videos.

[0144] Specifically, the server determines the criteria for identifying the common video type. If a common video meets these criteria, the server retrieves the common video in the target video that matches the common video type. For example, the criteria for identifying the common video type could be that it falls within a common video distribution range associated with that common video type. The server determines the time period of the common video segment and whether that time period falls within the common video distribution range. If the time period falls within the common video distribution range, the server retrieves the common video that matches the common video type. If the common video type is a title sequence, then the title sequence in the target video can be retrieved from that common video segment; specifically, that common video segment can be used as the title sequence of the target video.

[0145] In this embodiment, when the identified public video segment meets the determination condition of the public video type, the public video in the target video that matches the public video type is obtained based on the public video segment, thereby identifying the public video that matches the public video type from the target video, which improves the recognition accuracy of identifying the public video from the target video.

[0146] In one embodiment, when a public video segment meets the determination criteria for a public video type, the public video in the target video that matches the public video type is obtained based on the public video segment, including: determining the distribution range of public videos associated with the public video type of the target video; when the time period of the public video segment is within the distribution range of public videos, the public video in the target video that matches the public video type is obtained based on the public video segment.

[0147] The public video distribution range refers to the time distribution range of public videos belonging to the public video type within the target video. For example, when the public video type is the intro type, its associated time distribution range can be the first N seconds of the target video, such as the first 20 seconds of the target video, i.e., the time distribution range is 0s-20s. The time period of the public video segment refers to the time span of the identified public video segment in the target video, which can be determined based on the start and end times of the public video segment, specifically the time span from the start time to the end time.

[0148] Specifically, the server determines the common video distribution range associated with the common video type of the target video. Different common video types correspond to different common video distribution ranges. For example, for a common video type like "intro," the associated common video distribution range could be the first N seconds of the video, while for a common video type like "outro," the associated common video distribution range could be the last M seconds of the video. The server determines the time period of the common video segment, specifically based on the start and end times of the common video segment. If the time period of the common video segment falls within the common video distribution range associated with the common video type, it indicates that the common video segment is within the time span corresponding to the common video type. Based on the common video segment, the server obtains the common video in the target video that matches the common video type. For example, the server can use this common video segment as the common video in the target video that matches the common video type. If the common video type is "outro," the server uses this common video segment as the outro of the target video.

[0149] In this embodiment, based on the public video distribution range associated with the public video type and the comparison result of the time period of the public video segment, the public video in the target video that matches the public video type is determined according to the public video segment. Thus, based on the preset public video distribution range, the accuracy of identifying the public video that matches the public video type from the target video is ensured, and the recognition accuracy of identifying public videos from the target video is improved.

[0150] In one embodiment, the video recognition method further includes: identifying a corrected video segment adjacent to a common video segment in the target video; determining the segment type of the corrected video segment based on the classification results of the video frames in the corrected video segment; when the segment type of the corrected video segment is a corrected segment type, updating the common video segment through the corrected video segment to obtain an updated common video segment; the updated common video segment covers the corrected video segment and the common video segment.

[0151] The corrected video segment is a video segment in the target video adjacent to the identified common video segment. It is used to correct the segment range of the common video segment, further improving the accuracy of its identification. The classification result is obtained by classifying the video frames in the corrected video segment. Specifically, a pre-trained image classification model can be used to classify the video frames in the corrected video segment. Based on the classification results of the video frames, the segment type of the corrected video segment can be determined, such as whether the corrected video segment is a segment type that requires correction of the common video segment. The corrected segment type is the segment type that requires correction of the common video segment.

[0152] Specifically, the server identifies corrected video segments adjacent to the common video segment from the target video. This can be done by identifying corrected video segments on either side of the common video segment. The time span of the corrected video segments can be preset as needed or dynamically adjusted. The server determines the classification results of the video frames in the corrected video segments. Specifically, the server extracts video frames from the corrected video segments and performs image classification on the extracted video frames to obtain classification results. Based on these classification results, the server determines the segment type of the corrected video segment. Specifically, it determines whether the corrected video segment belongs to the segment type that requires correction of the common video segment. When the server determines that the segment type of the corrected video segment is a correction segment type, it indicates that the common video segment needs to be corrected using the corrected video segment. The server then updates the common video segment using the corrected video segment, such as by merging the common video segment and the corrected video segment to obtain an updated common video segment, thus ensuring that the updated common video segment covers both the corrected video segment and the common video segment.

[0153] In a specific application, the time range of the common video segment identified from the target video is (2,7), which is the title sequence segment. The server determines the corrected video segment adjacent to this common video segment, with a time range of (8,9). The server determines that the classification result of the video frames in the corrected video segment is a text frame. Therefore, the server determines that the segment type of the corrected video segment is the corrected segment type. The server updates the common video segment using this corrected video segment, and the time range of the updated common video segment is (2,9), which covers both the corrected video segment and the common video segment. This merges the video segments containing text content adjacent to the common video segment into the title sequence segment, improving the recognition accuracy of the title sequence.

[0154] In this embodiment, the common video segment is updated by using a corrected video segment that is of the corrected segment type and is adjacent to the common video segment in the target video. This updates the common video segment to cover both the corrected video segment and the common video segment, thereby using adjacent video segments to correct the identified common video segment and further improving the accuracy of the identified common video segment.

[0155] In one embodiment, the video recognition method further includes: determining a video frame to be processed in a public video segment whose classification result belongs to the correction type; when the video frame to be processed matches a video frame in a non-public video segment of the target video, determining the video frame to be processed as a correction video frame; and updating the time segment of the public video segment according to the time attributes of each correction video frame to obtain the updated public video segment.

[0156] The classification result of the video frame to be processed belongs to the correction type, indicating that the common video segment needs to be corrected using the video frame to be processed. The classification result of the video frame to be processed can be obtained by performing image classification processing on the video frame to be processed. Specifically, the video frame to be processed can be classified using a pre-trained image classification model to obtain the classification result of the video frame to be processed. Non-common video segments are other video segments in the target video besides the common video segments. When the video frame to be processed is matched with video frames in non-common video segments, it indicates that the video frame to be processed has duplicated video frames in non-common video segments. In this case, the video frame to be processed is identified as the correction video frame for correcting the common video segment.

[0157] Specifically, the server can obtain the classification results of each video frame in the public video segment. This can be achieved using a pre-trained image classification model to classify the video frames and obtain the classification results. Based on the classification results, it determines whether the video frame belongs to the correction type, i.e., whether the video frame requires correction processing for the public video segment. If the classification result of the video frame belongs to the correction type, then the video frame is identified as the video frame to be processed in the public video segment whose classification result belongs to the correction type. The server identifies non-public video segments from the target video and matches the video frame to be processed with each video frame in the non-public video segments. Specifically, it can perform similarity matching or video frame feature matching between the video frame to be processed and each video frame in the non-public video segments to obtain the matching results. When the matching result shows that the video frame to be processed matches a video frame in the non-public video segments of the target video, the server identifies the video frame to be processed as a correction video frame that requires correction processing for the public video segment. The server determines the temporal attributes of each correction video frame, for example, by querying the frame information of each correction video frame. The server updates the time segment of the common video clip based on the time attributes of each corrected video frame, resulting in an updated common video clip. Specifically, the server can update the start or end time of the common video clip based on the time attributes of each corrected video frame to obtain the updated common video clip.

[0158] In a specific application, the time range of the common video segment identified from the target video is (2,7), which is the opening segment. The server determines that the classification result of the video frame to be processed from the common video segment is a person frame, and the video frame to be processed matches the video frame in the non-common video segment of the target video. The server determines the video frame to be processed as the corrected video frame. The server updates the time period of the common video segment according to the time attribute of each corrected video frame. For example, if the time attribute of the corrected video frame is 5s, the time range of the common video segment obtained after updating the common video segment is (2,5), thereby removing the video segment containing the person content in the opening segment and improving the recognition accuracy of the opening segment.

[0159] In this embodiment, the public video segment is updated by using video frames in the public video segment whose classification result belongs to the correction type and matches video frames in the non-public video segment. This allows the video content in the non-public video segment to be used to correct the public video segment, further improving the accuracy of the identified public video segment.

[0160] In one embodiment, the video recognition method further includes: determining the start time and end time of the public video; in response to a video comparison trigger event, extracting a non-public video from the target video based on the start time and end time; and performing a video comparison between the non-public video and the video to be compared.

[0161] In this context, "public video" refers to reusable video segments of a defined type. For example, public video could be intros, outros, or advertisements—video content that can be reused across various videos. The start time of a public video refers to the point in time when it begins, and the end time refers to the point in time when it ends. The video comparison trigger event is the event that initiates the comparison of videos; by comparing the videos, the similarity between them can be determined. "Non-public video" refers to segments of the target video other than public videos. Non-public videos are not reusable video segments and can be considered part of the target video's main content. The video to be compared is the video that needs to be compared; by comparing the non-public video with the video to be compared, the degree of similarity between the two can be determined.

[0162] Specifically, the server determines the start and end times of the public video. In response to a video comparison trigger event, such as a video comparison event triggered by a user on a terminal, the server extracts the non-public video from the target video based on the start and end times of the public video. Specifically, the server removes the public video from the target video based on its start and end times, thus extracting the non-public video from the target video. The server then obtains the video to be compared and performs a video comparison with the extracted non-public video to obtain the video comparison result. The video comparison result reflects the degree of content similarity between the video to be compared and the extracted non-public video.

[0163] In this embodiment, based on the start and end times of the public video, non-public videos are extracted from the target video for video comparison with the video to be compared. This allows for accurate and rapid location of non-public videos in the target video, which is beneficial for improving the accuracy and processing efficiency of video comparison.

[0164] In one embodiment, the video recognition method further includes: determining the jump time point of the public video; playing the target video in response to a video playback event for the target video; and skipping the public video and playing it when the playback progress of the target video reaches the jump time point.

[0165] The jump point refers to the point in time during the playback of the target video where, when the playback reaches the common video, a jump is required; that is, the point in time where the common video needs to be skipped without further playback. The video playback event is the trigger event for playing the target video. Specifically, the server determines the jump point in the common video, which can be at least one of the start or end times of the common video. The server responds to the video playback event for the target video. Specifically, the user can trigger the video playback event on the terminal to play the target video. When the playback progress of the target video reaches the jump point, the common video is skipped, and playback proceeds directly to the non-common video within the target video. In specific applications, if the common video is the intro, the jump point can be the start time of the common video; that is, when playing the target video, the intro is skipped, and the non-common video following the intro is played directly. Similarly, if the common video is the outro, the jump point can be the end time of the common video; that is, when playing the target video, the outro is skipped, and playback either ends directly or switches to another video.

[0166] In this embodiment, when the playback progress reaches the jump time point of the common video during the playback of the target video, the playback of the common video is skipped, thereby skipping the playback of repeated common videos during video playback and improving video playback efficiency.

[0167] In one embodiment, performing image matching of video frames between a target video and a reference video to obtain a video frame pair includes: extracting a video frame to be identified from the target video and extracting a reference video frame from the reference video; extracting video frame features of the video frame to be identified and the video frame features of the reference video frame respectively; performing feature matching between the video frame features of the video frame to be identified and the video frame features of the reference video frame; and obtaining a video frame pair based on the successfully matched video frame to be identified and the reference video frame.

[0168] Specifically, after acquiring the target video and the reference video, the server extracts video frames from both videos. Specifically, it extracts the video frame to be identified from the target video and the reference video frame from the reference video. The server extracts video frame features from both the video frame to be identified and the reference video frame. This extraction can be performed using an image processing model. The server then performs feature matching between the video frame features of the video frame to be identified and the video frame features of the reference video frame. If feature distance matching is possible, the video frames to be identified and the reference video frames whose feature distance is less than a threshold are considered a successful feature match. The server then obtains video frame pairs based on the successfully matched video frames to be identified and the reference video frames.

[0169] In this embodiment, video frames are extracted from the target video and the reference video for feature matching. Based on the successfully matched video frames to be identified and the reference video frames, video frame pairs are obtained. Then, similar video segments are identified based on the video frame pairs obtained by image matching, ensuring the accuracy of similar video segment identification.

[0170] In one embodiment, extracting video frame features of the video frame to be identified and video frame features of the reference video frame respectively includes: extracting video frame features of the video frame to be identified and video frame features of the reference video frame respectively using an image processing model.

[0171] The image processing model can be a pre-trained artificial neural network model, such as a convolutional neural network, residual network, or other types of network models. Specifically, the server uses the pre-trained image processing model to extract video frame features from the video frame to be identified and the video frame features from the reference video frame. In practical applications, the image processing model can be a pre-trained triplet neural network model or a multi-task model.

[0172] Furthermore, the training steps of the image processing model include: acquiring training sample images with classification labels; performing feature extraction and image classification on the training sample images using the image processing model to be trained, to obtain the sample image features and sample image categories of the training sample images; determining the model loss based on the sample image features, sample image categories, and classification labels; updating the image processing model to be trained based on the model loss and continuing training; and obtaining the trained image processing model when training is complete.

[0173] The training sample images carry classification labels, and the training dataset can be set according to actual needs. The sample image features are image features extracted from the training sample images by the image processing model to be trained, and the sample image categories are the classification results obtained by classifying the training sample images based on the image processing model to be trained. The model loss is used to update the model parameters in the image processing model to be trained, ensuring that the image processing model to be trained can converge, thereby completing model training. Specifically, the server acquires training sample images with classification labels, and performs feature extraction and image classification on the training sample images using the image processing model to be trained, obtaining the sample image features and sample image categories output by the image processing model to be trained. The server determines the model loss based on the sample image features, sample image categories, and classification labels. Specifically, it can determine the triplet loss based on the sample image features, and determine the classification loss (specifically, cross-entropy loss) based on the sample image categories and classification labels, and obtain the model loss based on the triplet loss and classification loss. The server updates the image processing model to be trained based on the model loss and continues training. When training is complete, a trained image processing model is obtained. The trained image processing model can extract image features from input image frames and perform image classification processing on input image frames.

[0174] In this embodiment, the image processing model to be trained is updated and trained based on the sample image features, the sample image category, and the model loss determined by the classification label. The trained image processing model extracts the video frame features of the video frame to be identified and the video frame features of the reference video frame. By fully mining the video frame features of the input video frame through the image processing model, the accuracy of video frame matching can be improved.

[0175] This application also provides an application scenario in which the above-described video recognition method is applied. Specifically, the video recognition method is applied in this scenario as follows:

[0176] Video similarity identification is used to protect the main content of films and television dramas from being stolen and misused. The system needs to deduplicate query videos within a database of similar videos. If duplicates are found, the videos are considered similar. However, since only the main content needs protection, the video's opening and closing credits are not included in the deduplication process. Meanwhile, mid-length videos created and uploaded by self-media users on video platforms are important content for independent users, including lecture series, knowledge sharing, arts and culture classes, and current affairs commentary. Mid-length videos refer to self-produced drama-style videos by self-media or platform media. These videos are generally 1-10 minutes long, with opening and closing credits lasting about 5-20 seconds. Because these videos are shorter than television dramas and movies, but longer than short videos casually shot by individual users, they are called mid-length videos. The opening and closing credits of these videos differ from those of regular films and television dramas; they lack rich video footage and unique, long-lasting music. They often begin with text, a simple logo, or a simple background introducing the characters, accompanied by a short ending with similar material.

[0177] At this point, accurately locating the opening and closing credits of self-produced dramas is crucial for improving the accuracy of video similarity identification. Traditional opening and closing credits processing often suffers from limitations. Since these credits may contain music, different visuals, different textual introductions, logos, and the main theme of the video, it's difficult to use machines to uniformly identify specific styles. Traditional methods typically rely on manual annotation, which requires significant resources. Furthermore, traditional sliding window-based opening and closing credits identification only supports fixed styles and cannot flexibly identify credits with varying lengths and formats. Considering that opening and closing credits not only share similarities across multiple videos within a drama series but also possess different characteristics for different dramas, this embodiment proposes a video recognition method to solve the problem of identifying general self-produced drama opening and closing credits. This method can accurately identify similar video segments within a video, specifically identifying the opening and closing credits.

[0178] The video recognition method provided in this embodiment can be applied to scenarios such as video intro and outro recognition, filtering intros and outros of identical or similar videos, and jumping to intro and outros playback. Figure 4 As shown, for the application scenario of video title and end credit recognition, if the video is a self-media drama series, then each video is extracted from the series, including video 1, video 2... video N. Features are extracted from each video, and title and end credit mining is performed based on the extracted features. The mined title and end credits are then corrected at the business layer to obtain the title and end credit times of each video. Figure 5As shown, in applications involving video similarity or identical content recognition, the intro and outro times of each video in a series can be pre-mined and stored in a database. When video similarity or identical content matching is required, video segments in the series are matched by querying the videos. Based on the matching results, the video is determined. The intro and outro times of the video are removed, and the main content of the video is then used for video similarity or identical content matching to obtain the matching result for the main video. For example... Figure 6 As shown, in applications that require skipping between intros and outros, intro and outros can be pre-processed for each video in a video series to obtain the intro and outros times for each video and store them in a database. When a user browses videos on their device, recommended videos are retrieved, their intros and outros are removed, and a pop-up recommendation window appears on the device interface, allowing the user to skip the intro of the recommended videos and start playback from the beginning of the recommended videos. For example... Figure 7 As shown, the video recognition method provided in this embodiment can also support the mining of intro and outro sequences defined by multiple different business types. The first business type can be the processing of character and text correction in self-made dramas by self-media, and the second business type can be the processing of time point correction for the intro and outro sequences of TV dramas. Specifically, the video that needs to be identified for intro and outro sequence recognition and the corresponding business type are determined. Videos in the same series are identified. Based on the features obtained by feature extraction, intro and outro sequence mining is performed on the video through videos in the same series, and correction is performed through the first business type correction to obtain the intro and outro sequence recognition result corresponding to business 1; correction can also be performed through the second business type correction to obtain the intro and outro sequence recognition result corresponding to business 2.

[0179] The video recognition method provided in this embodiment offers a scalable overall framework. Specifically, it provides the ability to mine intro and outro titles for self-produced dramas by separating the core module from the business module. To address changes in business logic requirements, a new format for intro and outro titles for self-produced dramas is introduced. A corresponding module for addressing these changes can be added to the upper-level correction module based on this solution. Furthermore, an efficient and low-cost feature preparation step is proposed through a shared network, reducing resource consumption. Moreover, through efficient intro and outro title mining core modules and business correction modules, intro and outro titles can be quickly obtained, achieving a rapid mining process. Finally, it supports user-friendly bad case repair, allowing for the addition of different intro and outro title effects based on bad case feedback, including image classification processing.

[0180] Specifically, the video recognition method provided in this embodiment, such as Figure 8As shown, video frame feature extraction is achieved through a convolutional neural network model. For a target video, six video frames are extracted. Each video frame is then processed through the convolutional neural network model for feature extraction, yielding the video frame features output by the feature extraction task and the video frame classification results output by the classification task. Further, as... Figure 9 As shown, for each video in a video series, intro and outro are extracted. For each target video in the series, reference videos are identified. Video time segments are matched between the target video and the reference videos, and common matching segments are determined based on the matched video time segments of each reference video. Intro and outro alignment is performed based on the determined common matching segments to finally determine the intro and outro positions. Figure 10 As shown, for the application scenario of self-produced drama correction, the classification results of each time point in the main video can be obtained. The identified opening sequence time points are merged with the text time points, and then video frames related to the main video characters are removed from the opening sequence time period. For the identified closing sequence time points, the file time points are merged, and video frames related to the main video characters are removed from the closing sequence time period to obtain the location results of the opening and closing sequences. By correcting the opening and closing sequences, the uncertainty in location caused by mismatches in pure text image types and large-scale mismatches in mid-video lecturing types can be reduced, ensuring the accuracy of opening and closing sequence recognition.

[0181] Furthermore, the video recognition method provided in this embodiment can identify the opening and closing credits of a video. When applied to an opening and closing credits recognition system, it can include a feature extraction module, an opening and closing credits mining module based on time period matching, and a business correction module for self-produced dramas. In addition, to facilitate the expansion of opening and closing credits mining to other business types of videos such as TV dramas, the system framework can separate the opening and closing credits mining module from the business correction module. This allows the system to quickly support opening and closing credits mining for self-produced dramas by only adjusting the business layer correction module, while also supporting opening and closing credits mining for TV dramas or other shows. Specifically, in the opening and closing credits mining process, multiple videos of a certain drama series are input. For each target video, it is paired with other reference videos in the drama series to generate video pairs. These video pairs are then subjected to video time period matching processing based on image features to obtain matching time periods that are the same or have variable speed or internal truncation relationships between the two videos. For the matching time periods generated from multiple video pairs, the same matching time periods are obtained. Incorrect time periods are eliminated according to the preset time range for opening and closing credits, and finally, the mined opening and closing credits times are determined. Figure 11As shown, the target video and reference video undergo frame-level similarity matching. Frame matches with the same time deviation are merged into segment matches, and matching segments with a time deviation threshold are merged. The temporal sequence of multiple matching segments is then merged, and the intro and outro alignment is performed to determine the intro and outro positions. The frame-level features used for time segment matching can be obtained based on the feature extraction branch of a deep learning image multi-task model. The image multi-task learning model can also include an image classification branch to correct the determined intro and outro positions based on the image classification results.

[0182] First, for frame-level feature extraction, the video is frame-by-frame extracted. For example, in a 6-second video at 25 FPS (Frames Per Second), one frame is extracted every second, resulting in 6 images. These extracted images are then processed by a feature extractor to obtain the video frame features for each frame, resulting in 6 video frame features. The video frames can be extracted using an image multi-task model. For image multi-task models, such as... Figure 12 As shown, the image multi-task model is based on a convolutional neural network (CNN), including a feature extraction branch and an image classification branch. The model is trained using triplet inputs and anchor samples within the triplets. The triplet loss is determined through the feature extraction branch, and the cross-entropy loss is determined through the image classification branch. The model loss is obtained by summing the triplet loss and the cross-entropy loss, and this is used to train the image multi-task model. Specifically, in the model structure, CNNs are used to extract deep image representations. The feature extraction branch performs feature extraction based on these deep representations, and the image classification branch performs image classification. The CNN deep representation module and model weights can reuse the ResNet-101 neural network pre-trained on the large-scale open-source dataset ImageNet. Furthermore, the feature extraction model can also use different network structures and different pre-trained model weights as the base model. Simultaneously, the image embedding model can be optimized through training to improve the performance of the video segment matching algorithm, demonstrating optimizability. ImageNet is a large-scale, general-purpose open-source dataset for object recognition.

[0183] Image embedding is used to describe the features of image information, including low-level image representation and semantic representation. Embedding is not limited to floating-point features; it can also be an image representation composed of binary feature vectors, i.e., deep hash features. In this embodiment, the embedding feature can be a binary deep hash feature. Low-level image representation is the image embedding derived from deep learning low-level features, mainly describing the overall image texture, feature layout, and other representational information. Semantic image representation is the image embedding derived from semantic learning, mainly describing the representation of parts of the image containing certain specified semantic content. For example, to describe the embedding of a dog, the features of the dog's location in the image are extracted as the image representation.

[0184] The structure of the deep representation module of ResNet-101 CNN is shown in Table 1 below.

[0185] Table 1

[0186]

[0187] The feature extraction branch can adopt the fully connected layer structure shown in Table 2 below.

[0188] Table 2

[0189]

[0190] The image classification branch can adopt the fully connected layer structure shown in Table 3 below.

[0191] Table 3

[0192]

[0193] When preparing training data, triplet data is prepared. Training requires triplet data input, so triplet data is labeled. A triplet consists of an anchor point, a positive sample, and a negative sample (a, p, n). In a triplet, a and p form a positive sample pair, and a and n form a negative sample pair. These triplet data can be randomly selected from all images. Figure 1For each set of three images, label whether they form a triplet and which image each triplet's aPN corresponds to (for two similar images in a triplet, one can be randomly selected as 'a' and the other as 'p'). Since the model is used to match opening and closing sequences, the opening and closing sequences of the same series are usually similar; therefore, two samples need to be extremely similar to be considered similar samples 'a' and 'p'. Next, perform classification labeling. For the labeled triplet data, assign each image a three-category classification: pure text, upper body or face, and other. Since 'a' and 'p' in a triplet are similar, only one needs to be labeled. Finally, label all images 'a' and 'n' in the triplet sets, assigning the category of 'a' to 'p' to obtain the classification of the three samples in the triplet.

[0194] During training, parameter initialization is performed first. In the pre-training stage, Conv1-Conv5 use the parameters of ResNet-101 pre-trained on the ImageNet dataset. The newly added layers are shown in Tables 2 and 3, and are initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0. Learning parameters are set for all parameters of the image multi-task model. A learning rate of lr=0.0005 is used for all parameters, and lr is reduced to 0.1 times after every 10 iterations. During the learning process, N triples of the full dataset are processed in epochs. The full dataset is processed once in each iteration until the average epoch loss no longer decreases. For each epoch iteration, training is performed in batches. Specifically, the full dataset is processed in batches of bs triples, for a total of N / bs batches. One batch is taken each time and input into the model to update the parameters (updating N / bs times in total, completing one epoch iteration). In one batch of forward computation, during training, the neural network performs forward computation on the input triplet image to obtain the feature extraction layer and classification prediction results, denoted by E and C, where E is a 1x64 vector representing floating-point features, and C is a 1x3 vector representing the predicted values ​​for the three categories. The outputs are the floating-point feature representations of the triples (Ea, Ep, En) and the classification predictions (Ca, Cp, Cn). The sum of the average metric learning loss and the classification loss is calculated as the model loss. Using SGD stochastic gradient descent, the model loss is used for gradient backward computation to obtain the updated parameter values, and the parameters of the network to be learned at the corresponding stage are updated to complete one batch of model parameter updates. This batch forward computation, model loss calculation, and model parameter update process is repeated to complete all N / bs batch model updates.

[0195] The model loss can be expressed as Equation (1).

[0196] (1)

[0197] in, For model loss, and For coefficients, For triple loss, The classification loss is calculated using the video frame features of these triplet samples in the batch, as shown in equation (2).

[0198] (2)

[0199] in, For margin (interval), it can be set to 0.6. This represents the L2 distance between the features of two video frames. The goal of triplet loss is to ensure that the distance between the anchor sample and the negative sample is greater than 0.6 compared to the distance to the positive sample.

[0200] For the probability vector output by the classification layer, calculate its cross-entropy loss with the labeled classification label, as shown in equation (3).

[0201] (3)

[0202] in, This represents the truth label of sample i. This represents the predicted probability under the ground truth label in the classification output ci. n represents the number of images in this batch participating in the classification calculation.

[0203] Furthermore, feature extraction and image classification can be achieved using forward computation of deep learning models. For example... Figure 13 As shown, the feature extraction branch and the image classification branch share the same post-inference module. (As...) Figure 14 As shown, the feature extraction branch and the image classification branch do not share inference. When applying inference, for a certain image frame, the video frame features and image classification information can be directly obtained through the three modules. In this case, the CNN model shares modules, which can avoid excessive resource consumption caused by two models.

[0204] Furthermore, when forming query video pairs, for a given TV series, the videos for that series are obtained. For example, if a TV series has 46 episodes, then there are 46 videos. The task of intro and outro mining is to find the intro and outro of each video. The video recognition method in this embodiment mines each video separately. For each video i, 10 videos are randomly selected from the remaining videos to form video pairs with video i, so that each video has 10 video pairs for mining. Time segment matching is performed on these 10 video pairs, so that each video pair generates 0 or more matching time segments. When a certain time segment is matched more than twice and appears at the beginning or end of the video, then that matching time segment is the target intro and outro.

[0205] When performing video time segment matching, for each video pair (i, r), where i represents the target video whose opening and closing sequences are to be determined, and r represents the reference video, and r takes values ​​from 1 to 10 according to the above video pair composition method, a total of 10 time segment matching algorithms are required for the query video i, processing one video pair each time. The specific steps of the time segment matching algorithm are as follows: A distance threshold t0 = 0.5 is preset for the video frame feature embedding, meaning that when the Euclidean distance between two embeddings is less than 0.5, it indicates that the two embeddings come from similar frames; frames are extracted from the two videos in the video pair, and the embedding of each frame is obtained. For each frame j in video i, the Euclidean distance between it and the embedding of each frame in video r is calculated, and frames with a distance less than t0 are taken as similar frames of j, obtaining the list of similar frames or matching frames of j (sim-id-list), and simultaneously recording the corresponding similar frame time deviation (diff-time-list). For frame j=1, the similar frame list sim-id-list is [1,2,3], indicating similarity to the 1st, 2nd, and 3rd seconds of video r; the time deviation diff-time-list is [0,1,2], representing the time distance between the similar frames in sim-id-list and the time represented by frame j=1. The default frame extraction is 1 frame per second, so the frame number is the number of seconds. Therefore, we obtain the similar frame list SL and the time deviation list TL for all frames i.

[0206] Iterate through all frames and count the number of frames that match between video i and video r, i.e., the number of frames that match video j in video r. If the number of matching frames is less than 1, then video i and video r do not have the same video segment, and the beginning and end of the title cannot be found. Otherwise, reorder the time difference dt to obtain the SL list. Specifically, sort all matching frames in SL by diff-time (i.e., dt) from smallest to largest. When dt is the same, sort by the sequence number of video i from smallest to largest. At the same time, reorganize the corresponding diff-time-list in this order, i.e., the time difference of 0 is placed at the beginning, the time difference of 1 is placed at the end, and so on. For example, the new SL list is [10,11],[11,12],[2,4], [3,5],[4,6],[6,9],[7,10].

[0207] The data is reconstructed using dt to obtain match-dt-list. Specifically, for all frames of i, the list of similar frames in SL is reconstructed using time deviation as the primary key to obtain a list of dt from smallest to largest, resulting in match-dt-list of similar frames with time deviations of 0, 1, 2, ...: {0:{count, start-id, match-id-list},...}, for example {2:{3, 2, [[2,4],[3,5],[4,6]]}, 3:{2,6,[ [6,9],[7,10]]}}, where 2 refers to a time difference of 2. For example, if the 2nd frame of i is similar to the 4th frame of video vid2, then the time difference between these two frames is 1; count is the number of similar frames under this time deviation. For example, if the 2nd frame of i is similar to the 4th frame of vid2, then count is incremented by 1; start-id refers to the smallest frame ID of i under this time difference. For example, if the 1st frame of i is not similar to vid2, but the 2nd frame of i is similar to the 4th frame of video vid2, then start-id is 2.

[0208] Merge two dt lists in the match-dt-list where the preceding and following dt values ​​are less than 3 (i.e., merge matching pairs with a matching deviation within 3 seconds). The larger dt list is merged into the smaller dt list. Simultaneously, the similar frames with the larger dt list are updated, and the matching frame list SL is also updated. For example, dt=2 and dt=3 can be merged, resulting in {2:{5, 2, [[2,4],[3,5],[4,6],[6,8],[7,9]]}}, where coun t is the sum of counts for dt=2 and dt=3. start-id finds the smallest frame of video i from the similar frame lists of dt=2 and dt=3. For the list of dt=3, the sequence number of the matched frame is rewritten and merged, such as rewriting [6,9] as [6,8] and merging it into the similar frame list of dt=2. At the same time, the similar frame pairs with rewritten frame sequence numbers are synchronously updated to the SL matching frame list in step 5), such as updating it to: [10,11],[11,12],[2,4], [3,5],[4,6],[6,8],[7,9]. Since the above merging of the frame list may scramble the order of dt or frame ID, it needs to be reordered. Specifically, dt is reordered, that is, the dt reordering process is performed again on the new SL list to obtain the SL list, resulting in a matching frame list sorted by dt from smallest to largest (the frame ID of video i from smallest to largest). Reorganize the data with dt to obtain match-dt-list: Execute the process of reorganizing the data with dt to obtain match-dt-list again.

[0209] The matching time period list `match-duration-list` is calculated. Specifically, the time interval between two matching segments is preset to be greater than T2 (e.g., 8s, which means the frame numbers differ by 8 for 1 frame per second). For each `dt` in `match-dt-list` (e.g., `dt=2`), for each frame `srcT` of video `i` under `dt` (e.g., 2 in examples 2, 3, 4, 6, 7 above), if the difference between `srcT` and the previous `srcT` is greater than T2 (e.g., 2 differs by 9 from the previous `srcT` 11, which is greater than the interval threshold), then the previous similar frame pairs are merged into a matching segment. New similar frame pair statistics begin from the current `srcT`, and the similar frames are stored in a temporary list `tmplist`. If dt=2 and srcT=2, similar frames from the previous temporary frame list are stored as matching segments. For example, similar frames from the previous tmplist=[[10,11],[11,12]] are added to the match-duration-list as matching segments. For instance, adding matching segment information like [10,11,11,12,1,2,2], where each value represents [src-startTime, src-endTime, ref-startTime, ref-endTime, dt, duration, count], meaning the matching segment stores the start and end frames of video i, the start and end frames of the matching video, the dt of the matching segment, the duration of the matching segment, and the number of similar frames matched. Figure 15 As shown, the matching segment information includes the start frame time and last frame time of the target video, the start frame time and last frame time of the matching video, etc. The similar frames in this instance are stored in a temporary list tmplist=[[2,4]]. When the difference between srcT and the previous srcT is less than T2, the similar frames in this instance are stored in the temporary list tmplist. For example, for dt2, srcT=3, 4, 6, and 7 are all stored in the temporary list, resulting in tmplist=[[2,4], [3,5],[4,6],[6,8],[7,9]]. When the current frame is the last similar frame in this dt (e.g., srcT=7), the accumulated similar frames in tmplist form a matching segment and are added to match-duration-list, such as adding [2,7,4,9,2,6,5], where the duration is 7-2+1 and count=5 is the similar frame count, thus match-duration-list=[[10,11,11,12,1,2,2],[2,7,4,9,2,6,5]]. The above match-duration-list is then sorted in descending order by the count of similar frames, such as match-duration-list=[[2,7,4,9,2,6,5],[10,11,11,12,1,2,2]].

[0210] Overlapping time segments in the match-duration-list need to be handled. Since similarity frame calculation involves iterating through all frames of two videos, calculating distances, and selecting similarity within a certain threshold range, it's easy for a single frame to be similar to multiple frames. This can lead to overlapping time segments in the match-duration-list, requiring handling. Specifically, a minimum matching segment duration T3 (e.g., 5, meaning a minimum matching duration of 5 seconds) is set. For time segment i in the match-duration-list (the segment consisting of src-startTime and src-endTime), time segment j in the match-duration-list is incremented by 1. If time segment i contains time segment j, then j is deleted. Figure 16 As shown, the start time of time period i is before the start time of time period j, and the end time of time period i is after the end time of time period j. That is, time period i contains time period j, and j needs to be deleted. When i and j intersect, and the start time of i is the earliest start time, the start time of j is moved to the end time of i, and j is updated. If the duration of time period j is less than T3, then j is deleted; otherwise, the new j replaces the old j. For example... Figure 17 As shown, the start time of time period i is before the start time of time period j, but the end time of time period i is before the end time of time period j. i and j intersect, so the end time of time period i needs to be updated to match the end time of time period j. When i and j intersect, and the start time of j is the earliest start time, the end time of j is moved forward to the start time of i, and j is updated. If the duration of time period j is less than T3, then j is deleted; otherwise, the new j replaces the old j. For example... Figure 18 As shown, the start time of time period i is after the start time of time period j, and the end time of time period i is after the end time of time period j. i and j have an intersection, so the start time of time period i needs to be updated to the start time of time period j. Finally, the matching time period information is returned, such as match-duration-list=[[2,7,4,9,2,6,5], [10,11,11,12,1,2,2]], or only the matching segment [[2,7,4,9],[10,11,11,12]].

[0211] For processing the same matching segments, for video i, assuming it needs to be mined from video vid2, video vid3, and video vid4, then for N=3 pairs of video pairs [i,vid2][i,vid3], [i,vid4], the above video segment matching processing is performed respectively, resulting in 3 matching information. For example, the first pair of video matching segments returns: [[2,7,4,9],[10,11,11,12]], the second pair of matching segments returns [[2,7,4,9]], and the third pair returns [[2,7,4,10]]. The matching segments are counted; for example, [2,7,4,9] appears twice, [2,7,4,10] appears once, and [10,11,11,12] appears once. Sort the matching segments in reverse order of their counts. When the counts are the same, sort them in ascending order of src-startTime: resulting in match-list = [[2,7,4,9],[2,7,4,10],[10,11,11,12]],count-list=[2,1,1].

[0212] For overlapping matching segments in the match-list, merge them. Specifically, set the effective intersection ratio T4, such as 0.5, meaning that if the intersection duration of two time periods is greater than T4 of the target segment duration, the counts of the two segments need to be merged. Set the effective matching count T5, such as 3, meaning that if the count of a matching segment is greater than T5, then that segment cannot be ignored. For time period i in the match-list (referring to the time period consisting of src-startTime and src-endTime), for time period j in the match-list = i+1, if time period i contains time period j, and the duration of j is greater than 0.5 * the duration of i, then delete j, and the count of i equals the original count of i plus the count of j. If i and j intersect, and the intersection duration is greater than 0.5 * the duration of i, and the count of j is greater than T5, then merge i and j into the longest start and end time, and the count of i equals the original count of i plus the count of j. If the count of j is less than T5, then delete j, and the count of i equals the original count of i plus the count of j. This means that at this point, segments i and j are not merged; only segment i, which appears most frequently, is retained, but the frequency of segment j is reflected in the new segment i count. If segments i and j intersect, and the intersection duration is less than 0.5 * segment i duration, segment j is discarded. For example... Figure 19 As shown, if the start time of time period i is before the start time of time period j, but the end time of time period i is before the end time of time period j, i and j have an intersection, and the end time of time period i needs to be updated to the end time of time period j. On the other hand, if the start time of time period i is after the start time of time period j, and the end time of time period i is after the end time of time period j, i and j have an intersection, and the start time of time period i needs to be updated to the start time of time period j.

[0213] This yields a new match-list of video segments (e.g., [[2,7,4,9], [10,11,11,12]]) and a count-list (e.g., [3,1]). A threshold T6 is set for the effective recurrence rate, indicating that in the mining of N video pairs, if the recurrence rate x of a matching video segment is greater than N*T6, it is considered a valid recurrence segment (e.g., T6=0.5). For the match-list, the valid time periods are retained, resulting in match-list = [[2,7,4,9]] and count = [3].

[0214] The intro and outro times are aligned and output. Specifically, for query video i, the first 20 seconds of the video are taken as the intro time range, and the last 10 seconds are taken as the outro time range. For each matching segment in the above match-list, it is determined whether the src-startTime and src-endTime time intervals are at the intro or outro position. If they are at the intro position, an intro matching segment is returned; if they are at the outro position, an outro matching segment is returned. The intro and outro information of the returned result is shown, such as [2,7] being the intro, which means that the 2nd to 7th seconds of video i are the intro. When a video has multiple intros or outros that meet the above requirements, such as the two time intervals [2,7] and [19,15] that meet the intro time range, both are considered to be intros. For example, the intro with the opening animation + the theme image of this video + the white background of the host's introduction will match the opening animation and the white background of the host's introduction. However, the theme image that changes in each video cannot be matched. In this case, multiple segments need to be merged, and [2,19] is output as the intro.

[0215] Further correction is performed through a business-level correction system. For self-produced dramas, the classification output of a multi-task model can be used for business correction. Specifically, the classification information at each time point of the main video is acquired and processed. While extracting features, the classification branch output of the multi-task model is also retained to obtain the classification information at each time point of the video, resulting in the classification of each frame. Here, the categories include text, characters, and others. The text time point processing for the opening sequence is as follows: For the opening sequence deadline, starting from the next time point, it is determined whether it is text. If it is, the opening sequence is added at that time point, and the new opening sequence deadline is incremented by 1; otherwise, the next step is performed. For example, in the opening sequence [2,7] above, if the 8th and 9th seconds are text, they are merged into the opening sequence, and finally [2,9] is the opening sequence.

[0216] The opening sequence time period is processed by removing character frames from the main content. The opening sequence character time point list L is initialized to be empty. The embedding features of all time points in the main content form a database feature db. For the i-th feature qi in the frame feature queries (there may be multiple frames with characters) at the opening sequence time point: query the number of features in the database whose Euclidean distance to the qi feature is less than thr (e.g., 0.3), and record it as Nqi. When Nqi > Nthr (Nthr is preset to 3), it indicates that there is a character in the opening sequence that appears in the main content. The time point ti corresponding to qi is saved in the L table, resulting in the list L of all opening sequence times containing characters. The minimum time point in L is taken, and the opening sequence end time is corrected to that time point. For example, for the above example of text time point merging opening sequence [2,9], assuming that the minimum L is 5, the final opening sequence time is [2,5], that is, from the 2nd to the 5th second.

[0217] Furthermore, the same method is used to merge the text time points at the end of the video, and the same method is used to remove the main video's character frames from the end-of-video time period. The main purpose of this text correction is to merge the text-type time points at the end of the main video into the end credits, thereby shifting the start time of the end credits forward—avoiding inaccurate matching and recall due to inaccurate embedding retrieval of text-type images, leading to inaccurate localization. The purpose of correcting the character frames is to prevent video segments featuring lectures from being incorrectly identified as the beginning or end credits.

[0218] Furthermore, an extended supplementary correction system can be applied. Specifically, by adding customized correction systems for other dramas to the overall data mining system, it is possible to support the mining of opening and closing credits for multiple business scenarios within this framework. For example, if additional support for TV series mining is required, an audio correction system can be added to the correction system. Considering that TV series opening credits all have an opening theme song, and the main content will begin when the opening theme song ends, the system can determine whether each second of audio is music. If it is not music, it means that the current time is not the opening credits, and this time is used to correct the opening credits time.

[0219] When a video is added to the database, its intro and outro information are recorded. Specifically, the intro and outro information of the mined videos is first stored. For the intro and outro, the steps for adding them to the database are: extracting the intro and outro positions from video i (e.g., extracting the intro at [2,7] in the example above), recording the video ID (video id, i.e., vid), and the intro time 2 and outro time 7. Figure 20As shown, the database stores the intro and outro information for each video. Specifically, the database stores K video segments, from video 1 to video K. For each video segment, it also stores the corresponding intro and outro information, such as the intro time information t1 and outro time information t2 for video 1, and the intro time information t3 and outro time information t4 for video K. In the comparison of identical or similar videos, it is necessary to extract the matching time of the intro and outro. For video i, when it is necessary to determine whether video j is identical or similar to it, the time segment matching algorithm in this embodiment can be used to calculate. After obtaining the matching segment, the matching segments located at the intro and outro of video i are removed to obtain the final matching segments that constitute identical or similar videos. In the application of recommending videos for playback, after the recommendation system obtains the recommended video, it can directly jump to the intro position in the stored intro and outro information and automatically play it for the user, thereby avoiding the use of useless information at the beginning to occupy the video playback time, thus consuming the user's attention or interest in continuing to watch the video.

[0220] The video recognition method provided in this embodiment is applied to the mining of intro and outro titles. Based on cross-video similarity feature video segment matching and text and image correction, it extracts commonalities from various video styles to design a unified mining system, and designs time positioning correction schemes for different video characteristics, ultimately achieving the location of intro and outro titles in self-made dramas. In the intro and outro title mining process of self-made dramas proposed in this embodiment, it is first necessary to train an image similarity embedding model and a related image classification model, then establish a video segment intro and outro title matching mining system based on image embedding, and finally correct for the image characteristics of the drama. To improve the efficiency of data preparation, including feature extraction and classification, this embodiment can merge similarity embedding and image classification into different task branches of the same network structure, so that the application can reduce the resources required for embedding and classification inference by sharing a part of the network. To improve the effect of intro and outro title mining, this embodiment designs a video segment matching scheme based on its characteristics during the initial mining of intro and outro titles; finally, it performs time point correction based on the characteristics of intro and outro titles in the video to improve the overall recognition accuracy and recall.

[0221] In this embodiment, a high-efficiency and low-cost feature preparation step is proposed through a shared network, enabling efficient computation of the core module for opening and closing title sequence mining. By leveraging video frame retrieval and the multi-video frame sequence matching technique used in this solution for opening and closing title sequence location, the identification and location of opening and closing titles under conditions of time misalignment or unequal length are achieved. A frame-level embedding-based opening and closing title sequence video segment matching algorithm quickly finds preliminary matching targets from the entire video. A relevant upper-layer correction module is applied, utilizing classification information prepared by the shared network and image embedding characteristics to further correct mid-videos, improving the mining effect. Frame-level image classification information is used to optimize erroneous location points in the matching results, thereby calibrating the image retrieval results and improving the accuracy of opening and closing title sequence location time points. The scalable overall framework allows for the addition of modules to address changes in business logic requirements or the introduction of new self-produced drama opening and closing title sequence formats within the upper-layer correction module. By using a multi-task model to share underlying parameters, the required features and classification inference are accelerated, improving the overall system processing speed.

[0222] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0223] Based on the same inventive concept, this application also provides a video recognition device for implementing the video recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more video recognition device embodiments provided below can be found in the limitations of the video recognition method described above, and will not be repeated here.

[0224] In one embodiment, such as Figure 21 As shown, a video recognition device 2100 is provided, including: a video acquisition module 2102, a video frame pair acquisition module 2104, an offset determination module 2106, and a similar segment determination module 2108, wherein:

[0225] The video acquisition module 2102 is used to acquire target videos and reference videos in a video series.

[0226] The video frame pair acquisition module 2104 is used to perform image matching of video frames between the target video and the reference video to obtain video frame pairs; the video frame pairs include the video frames to be identified belonging to the target video, and also include the reference video frames in the reference video that match the images of the video frames to be identified.

[0227] The offset determination module 2106 is used to determine the time offset of the video frame pair based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair.

[0228] The similar segment determination module 2108 is used to filter video frame pairs with matching time offsets and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs.

[0229] In one embodiment, the similar segment determination module 2108 includes a video frame pair filtering module, a time attribute processing module, and a start and end time processing module; wherein: the video frame pair filtering module is used to perform numerical matching of the time offsets of each video frame pair, and filter out video frame pairs with matching time offset values ​​according to the numerical matching results; the time attribute processing module is used to determine the start and end times based on the time attributes of the video frames to be identified in the filtered video frame pairs; the start and end time processing module is used to determine similar video segments relative to the reference video from the target video according to the start and end times.

[0230] In one embodiment, the video frame pair filtering module includes a numerical comparison module, a numerical comparison result processing module, and an offset update module; wherein: the numerical comparison module is used to compare the time offsets of each video frame pair to obtain a numerical comparison result; the numerical comparison result processing module is used to filter video frame pairs from each video frame pair whose numerical difference in time offset is less than a numerical difference threshold based on the numerical comparison result; and the offset update module is used to update the offsets of video frame pairs whose numerical difference in time offset is less than the numerical difference threshold to obtain video frame pairs with matching time offset values.

[0231] In one embodiment, the time attribute processing module includes a video frame pair list acquisition module, a time attribute distance determination module, a time attribute distance processing module, and a start and end time determination module; wherein: the video frame pair list acquisition module is used to acquire a video frame pair list composed of filtered video frame pairs; in the video frame pair list, each video frame pair is sorted in ascending order of time offset value, and video frame pairs with the same time offset are sorted in ascending order of the timestamp values ​​of the included video frames to be identified, the timestamps being determined based on the time attributes of the included video frames to be identified; the time attribute distance determination module is used to determine the time attribute distance between the time attributes of the video frames to be identified in adjacent video frame pairs in the video frame pair list; the time attribute distance processing module is used to determine adjacent video frame pairs whose time attribute distance does not exceed a distance threshold as video frame pairs belonging to the same video segment; the start and end time determination module is used to determine the start time and end time based on the timestamps of the video frames to be identified in the video frame pairs belonging to the same video segment.

[0232] In one embodiment, the start and end time determination module includes a start and end video frame pair determination module, a start time determination module, and an end time determination module; wherein: the start and end video frame pair determination module is used to determine the start video frame pair and the end video frame pair from the video frame pairs belonging to the same video segment based on the timestamp of the video frame to be identified in the video frame pair belonging to the same video segment; the start time determination module is used to obtain the start time based on the timestamp of the video frame to be identified in the start video frame pair; and the end time determination module is used to obtain the end time based on the timestamp of the video frame to be identified in the end video frame pair.

[0233] In one embodiment, the system further includes a segment intersection relationship determination module and a segment intersection relationship processing module; wherein: the segment intersection relationship determination module is used to determine the segment intersection relationship between similar video segments based on the start time and end time of each similar video segment; the segment intersection relationship processing module is used to update each similar video segment based on the segment intersection relationship to obtain the updated similar video segments in the target video relative to the reference video.

[0234] In one embodiment, the system further includes a common video segment determination module, which updates the segments of similar video segments in the target video that have an intersection relationship with each reference video to obtain a common video segment in the target video relative to each reference video.

[0235] In one embodiment, the common video segment determination module includes an intersection analysis module and an analysis result processing module; wherein: the intersection analysis module is used to perform segment intersection analysis on similar video segments in the target video relative to each reference video to obtain analysis results; the analysis result processing module is used to update the segments of each similar video segment whose analysis results show an intersection relationship to obtain the common video segments in the target video relative to each reference video.

[0236] In one embodiment, the analysis result processing module is further configured to determine that the analysis results are similar video segments with an intersection relationship; based on the intersection duration and statistics of the similar video segments with an intersection relationship, the module updates the segments of the similar video segments with an intersection relationship to obtain the common video segments in the target video relative to each reference video.

[0237] In one embodiment, the intersection analysis module includes a similar segment list acquisition module and a similar segment intersection analysis module; wherein: the similar segment list acquisition module is used to acquire a similar segment list consisting of similar video segments in the target video relative to each reference video; in the similar segment list, each similar video segment is sorted from largest to smallest according to its statistical value, and similar video segments with the same statistical value are sorted from first to last according to their start time; the similar segment intersection analysis module is used to perform segment intersection analysis on each similar video segment in the similar segment list to obtain the analysis result; the analysis result processing module is further used to update the earlier similar video segments by using the later similar video segments among the similar video segments with intersection relationships, to obtain the common video segments in the target video relative to each reference video; wherein, the later similar video segments are the similar video segments that are sorted later in the similar segment list among the similar video segments with intersection relationships; the earlier similar video segments are the similar video segments that are sorted earlier in the similar segment list.

[0238] In one embodiment, the common video segment determination module includes an updated segment acquisition module, a statistical quantity determination module, and a statistical quantity judgment module; wherein: the updated segment acquisition module is used to update the segments of similar video segments in the target video that have an intersection relationship with each reference video, and obtain the updated similar video segments; the statistical quantity determination module is used to determine the statistical quantity of the updated similar video segments; the statistical quantity judgment module is used to obtain the common video segments in the target video relative to each reference video based on the updated similar video segments when the statistical quantity of the updated similar video segments exceeds the statistical quantity threshold.

[0239] In one embodiment, the system further includes a public video determination module, which, when a public video segment meets the determination criteria for a public video type, obtains a public video in the target video that matches the public video type based on the public video segment.

[0240] In one embodiment, the public video determination module includes a distribution interval determination module and a distribution interval determination module; wherein: the distribution interval determination module is used to determine the public video distribution interval associated with the public video type of the target video; the distribution interval determination module is used to obtain the public video in the target video that matches the public video type based on the public video segment when the time period of the public video segment is within the public video distribution interval.

[0241] In one embodiment, the system further includes a corrected video segment determination module, a segment type determination module, and a corrected video segment update module; wherein: the corrected video segment determination module is used to determine corrected video segments adjacent to common video segments in the target video; the segment type determination module is used to determine the segment type of the corrected video segment based on the classification results of video frames in the corrected video segment; the corrected video segment update module is used to update the common video segment by the corrected video segment when the segment type of the corrected video segment is a corrected segment type, thereby obtaining an updated common video segment; the updated common video segment covers the corrected video segment and the common video segment.

[0242] In one embodiment, the system further includes a video frame to be processed determination module, a corrected video frame determination module, and a corrected video frame processing module; wherein: the video frame to be processed determination module is used to determine the video frames in the public video segment whose classification results belong to the corrected type; the corrected video frame determination module is used to determine the video frame to be processed as a corrected video frame when the video frame to be processed matches a video frame in a non-public video segment of the target video; the corrected video frame processing module is used to update the time period of the public video segment according to the time attributes of each corrected video frame to obtain the updated public video segment.

[0243] In one embodiment, the system further includes a video comparison module for determining the start and end times of the public video; in response to a video comparison trigger event, extracting a non-public video from the target video based on the start and end times; and performing a video comparison between the non-public video and the video to be compared.

[0244] In one embodiment, the system further includes a video skipping module, which is used to determine the jump time point of the public video; play the target video in response to a video playback event for the target video; and skip the public video and play it when the playback progress of the target video reaches the jump time point.

[0245] In one embodiment, the video frame pair acquisition module 2104 includes a video frame extraction module, a frame feature extraction module, and a frame feature matching module; wherein: the video frame extraction module is used to extract the video frame to be identified from the target video and extract the reference video frame from the reference video; the frame feature extraction module is used to extract the video frame features of the video frame to be identified and the video frame features of the reference video frame respectively; the frame feature matching module is used to perform feature matching between the video frame features of the video frame to be identified and the video frame features of the reference video frame, and obtain a video frame pair based on the successfully matched video frame to be identified and the reference video frame.

[0246] In one embodiment, the video frame acquisition module 2104 is further configured to extract video frame features of the video frame to be identified and video frame features of the reference video frame using an image processing model; it also includes a model training module for acquiring training sample images carrying classification labels; performing feature extraction and image classification on the training sample images using the image processing model to be trained to obtain sample image features and sample image categories of the training sample images; determining the model loss based on the sample image features, sample image categories, and classification labels; updating the image processing model to be trained based on the model loss and continuing training; and obtaining the trained image processing model when training is complete.

[0247] Each module in the aforementioned video recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0248] In one embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows: Figure 22 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores video data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a video recognition method.

[0249] Those skilled in the art will understand that Figure 22 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0250] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0251] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0252] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0253] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0254] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0255] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0256] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of video recognition, characterized by, The method includes: Retrieve target and reference videos from video series; The target video and the reference video are image matched frame by frame to obtain a video frame pair; the video frame pair includes a video frame to be identified belonging to the target video, and also includes a reference video frame in the reference video that matches the image of the video frame to be identified. Based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair, the time offset of the video frame pair is determined. Filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs. Obtain a list of similar video segments composed of similar video segments in the target video relative to each of the reference videos; in the list of similar video segments, each similar video segment is sorted from largest to smallest according to its statistical value, and similar video segments with the same statistical value are sorted from first to last according to their start time; In the list of similar segments, segment intersection analysis is performed on each similar video segment to obtain the analysis results; By updating the earlier similar video segments in each similar video segment that has an intersection relationship, a common video segment in the target video relative to each of the reference videos is obtained. The preceding similar video segment is positioned in the similar segment list before the following similar video segment.

2. The method according to claim 1, characterized in that, The process of filtering video frame pairs with matching time offsets, and determining similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs, includes: The time offsets of each video frame pair are numerically matched, and video frame pairs with matching time offset values ​​are selected based on the numerical matching results. Based on the time attributes of the video frames to be identified in the selected video frame pairs, the start and end times are determined. Based on the start time and the end time, similar video segments relative to the reference video are determined from the target video.

3. The method according to claim 2, characterized in that, The step of numerically matching the time offsets of each video frame pair and filtering out video frame pairs with matching time offset values ​​based on the numerical matching results includes: The time offset of each video frame pair is compared numerically to obtain the comparison results. Based on the numerical comparison results, video frame pairs whose time offset difference is less than the numerical difference threshold are selected from each video frame pair. For video frame pairs whose time offset values ​​are less than a threshold, offset updates are performed to obtain video frame pairs with matching time offset values.

4. The method according to claim 2, characterized in that, The process of determining the start and end times based on the time attributes of the video frames to be identified in the selected video frame pairs includes: Obtain a list of video frame pairs consisting of the filtered video frame pairs; in the list of video frame pairs, each video frame pair is sorted in ascending order of time offset value, and video frame pairs with the same time offset are sorted in ascending order of the timestamp value of the included video frames to be identified, wherein the timestamp is determined based on the time attribute of the included video frames to be identified. In the video frame pair list, the temporal attribute distance between the temporal attributes of the video frames to be identified in adjacent video frame pairs is determined. Adjacent video frame pairs whose temporal distance does not exceed a distance threshold are identified as video frame pairs belonging to the same video segment; The start and end times are determined based on the timestamps of the video frames to be identified in the video frame pairs belonging to the same video segment.

5. The method according to claim 4, characterized in that, The step of determining the start and end times based on the timestamps of the video frames to be identified in a pair of video frames belonging to the same video segment includes: Based on the timestamps of the video frames to be identified in the video frame pairs belonging to the same video segment, the starting video frame pair and the ending video frame pair are determined from the video frame pairs belonging to the same video segment. The start time is obtained based on the timestamp of the video frame to be identified in the starting video frame pair; The termination time is obtained based on the timestamp of the video frame to be identified in the termination video frame pair.

6. The method according to claim 1, characterized in that, The method further includes: Based on the start and end times of each of the similar video segments, the segment intersection relationship between the similar video segments is determined; Based on the intersection relationship of the segments, each of the similar video segments is updated to obtain the updated similar video segments in the target video relative to the reference video.

7. The method according to claim 1, characterized in that, The method further includes: For the similar video segments in the target video that have an intersection relationship with each of the reference videos, segment updates are performed to obtain the common video segments in the target video that are relative to each of the reference videos.

8. The method according to claim 7, characterized in that, The step of updating the segments of similar video segments in the target video that have an intersection relationship with each of the reference videos to obtain the common video segments in the target video relative to each of the reference videos includes: Perform segment intersection analysis on similar video segments in the target video relative to each of the reference videos to obtain the analysis results; For each similar video segment that has an intersection relationship as determined by the analysis, the segment is updated to obtain a common video segment in the target video relative to each of the reference videos.

9. The method according to claim 8, characterized in that, The step of updating the segments of similar video segments that have an intersection relationship according to the analysis results to obtain the common video segments in the target video relative to each of the reference videos includes: The analysis results were determined to be similar video segments that had an intersection relationship; Based on the intersection duration and statistics of similar video segments that have an intersection relationship, the segments of each similar video segment with an intersection relationship are updated to obtain the common video segments in the target video relative to each of the reference videos.

10. The method according to claim 7, characterized in that, The step of updating the segments of similar video segments in the target video that have an intersection relationship with each of the reference videos to obtain the common video segments in the target video relative to each of the reference videos includes: For the similar video segments in the target video that have an intersection relationship with each of the reference videos, the segments are updated to obtain the updated similar video segments; Determine the statistics of the updated similar video segments; When the statistics of the updated similar video segments exceed the statistics threshold, a common video segment in the target video relative to each of the reference videos is obtained based on the updated similar video segments.

11. The method according to claim 7, characterized in that, The method further includes: When the public video segment meets the determination criteria for the public video type, the public video in the target video that matches the public video type is obtained based on the public video segment.

12. The method according to claim 11, characterized in that, When the public video segment meets the determination criteria for the public video type, the public video in the target video that matches the public video type is obtained based on the public video segment, including: Determine the public video distribution range associated with the public video type of the target video; When the time period of the public video segment falls within the public video distribution range, the public video in the target video that matches the public video type is obtained based on the public video segment.

13. The method according to claim 7, characterized in that, The method further includes: Identify the corrected video segment that is adjacent to the common video segment in the target video; Based on the classification results of the video frames in the corrected video segment, the segment type of the corrected video segment is determined; When the corrected video segment is of the corrected segment type, the common video segment is updated using the corrected video segment to obtain the updated common video segment; The updated public video clip covers both the corrected video clip and the public video clip.

14. The method according to claim 7, characterized in that, The method further includes: The video frames in the public video segment whose classification results belong to the correction type are identified; When the video frame to be processed matches a video frame in a non-public video segment of the target video, the video frame to be processed is determined to be a corrected video frame. Based on the time attributes of each corrected video frame, the time period of the common video segment is updated to obtain the updated common video segment.

15. The method according to any one of claims 1 to 14, characterized in that, The step of performing image matching of video frames between the target video and the reference video to obtain video frame pairs includes: Extract the video frame to be identified from the target video, and extract the reference video frame from the reference video; Extract the video frame features of the video frame to be identified and the video frame features of the reference video frame, respectively. The video frame features of the video frame to be identified are matched with the video frame features of the reference video frame, and a video frame pair is obtained based on the video frame to be identified and the reference video frame that successfully match the features.

16. The method according to claim 15, characterized in that, The step of extracting the video frame features of the video frame to be identified and the video frame features of the reference video frame includes: The video frame features of the video frame to be identified and the video frame features of the reference video frame are extracted using an image processing model. The training steps of the image processing model include: Obtain training sample images with classification labels; The training sample images are subjected to feature extraction and image classification by the image processing model to be trained, so as to obtain the sample image features and sample image categories of the training sample images; The model loss is determined based on the sample image features, the sample image category, and the classification label; Based on the model loss, the image processing model to be trained is updated and then trained again. When training is complete, the trained image processing model is obtained.

17. A video recognition device, characterized in that, The device includes: The video acquisition module is used to acquire target videos and reference videos from video series. The video frame pair acquisition module is used to perform image matching of video frames between the target video and the reference video to obtain video frame pairs; the video frame pairs include video frames to be identified belonging to the target video, and also include reference video frames in the reference video that match the image of the video frames to be identified; The offset determination module is used to determine the time offset of the video frame pair based on the time attributes of the video frame to be identified and the time attributes of the reference video frame in the video frame pair. The similar segment determination module is used to filter video frame pairs with matching time offsets, and determine similar video segments in the target video relative to the reference video based on the time attributes of the video frames to be identified in the filtered video frame pairs. A common video segment determination module is used to obtain a list of similar segments composed of similar video segments in the target video relative to each of the reference videos; in the list of similar video segments, each similar video segment is sorted from largest to smallest according to its statistical value, and similar video segments with the same statistical value are sorted from earliest to latest according to their start time; in the list of similar video segments, segment intersection analysis is performed on each similar video segment to obtain the analysis result; through the later similar video segments among the similar video segments with intersection relationships, the earlier similar video segments are updated to obtain the common video segments in the target video relative to each of the reference videos; wherein, the earlier similar video segments are in the similar segment list in a sorted position before the later similar video segments.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 16.

Images