A video segmentation method and system based on cross attention and sequential attention

By combining cross-attention and sequence attention in video segmentation, the problem of inaccurate product information recognition in live videos is solved, achieving efficient video segmentation and product classification.

CN118658104BActive Publication Date: 2024-11-19XIAMEN LIMAYAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411124069.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-19
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately identify product information in live-stream videos, resulting in inefficient video editing and segmentation.

Method used

We employ a video segmentation method based on cross-attention and sequence attention, detect time points using an autoregressive coordinate point method, combine speech and text information with image matching technology, and utilize multimodal fusion to capture the correlation between products and time points, thereby re-dividing the video time segments.

Benefits of technology

It improves the accuracy and efficiency of video segmentation, enabling more accurate identification of product information in live videos, reducing timing errors, and achieving precise product classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658104B_ABST
    Figure CN118658104B_ABST
Patent Text Reader

Abstract

The present invention provides a video segmentation method and system based on cross attention and sequence attention, which relates to the field of video processing technology. The present invention obtains live video and product information in a certain time period; performs time point detection on product information and live video to obtain all time points; performs multimodal fusion on product information and all time points to capture the relationship and key information between the two through cross attention and sequence attention to obtain a time point product feature vector; according to product information and all time points, traverses and compares the time point product feature vectors of adjacent time points, re-divides the video time period of the live video, and obtains a second segmented video with more accurate product classification. The present invention can recall the time points of product-related videos to the greatest extent, reduce subsequent misjudgments, fully capture the correlation between video time points and products, more accurately identify the time points of live videos, and solve the video segmentation problem in the product dimension.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video segmentation method and system based on cross attention and sequential attention. Background Art

[0002] Since live product videos are long, contain many categories of product information, and each frame has similar image information, manual video editing and segmentation is labor-intensive and inefficient. Therefore, it is necessary to accurately segment live product videos with the help of artificial intelligence and image processing technology.

[0003] The existing technology mainly uses information such as text and pictures to obtain the key time points of the video based on rules, and then divides the video according to the time points; or, it uses a video-based shot division method (i.e., montage technique) to divide the video, and determines whether the scene is switched by analyzing the similarity of the pictures captured from the previous and next frames of the video or performing face recognition. However, the above technical solutions can only determine the switching of faces or scenes, which are relatively obvious features, but cannot accurately identify small features in the video, such as product information.

[0004] In view of this, the applicant filed this application after studying the existing technology. Summary of the invention

[0005] The present invention aims to provide a video segmentation method and system based on cross attention and sequential attention, so as to solve the problem that the existing methods cannot accurately extract the video time segments corresponding to the products in the video.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] A video segmentation method based on cross attention and sequence attention, comprising:

[0008] S1, obtain live video and product information in a certain period of time;

[0009] S2, performing time point detection on the product information and the live video to obtain all time points; wherein all time points include: time points of the first segmented video, audio time points and video screenshot time points; the first segmented video is obtained by using an autoregressive coordinate point method to perform time point detection, product classification marking and time period division on the live video according to a preset time window length; the audio time point is obtained by detecting and matching the product information with the voice and text information extracted from the live video; the video screenshot time point is obtained by detecting and matching the product screenshot in the first segmented video with the live video;

[0010] S3, multimodally fusing the product information with all time points to capture the contextual relationship and key information between the two through a cross attention mechanism and a sequence attention mechanism, and obtaining a time point product feature vector of the association relationship between the time point and the product;

[0011] S4, according to the product information and all time points, traverse and compare the time point product feature vectors of adjacent time points, re-divide the video time period of the live video, and obtain a second segmented video with more accurate product classification.

[0012] The present invention also provides a video segmentation system based on cross attention and sequence attention, comprising:

[0013] Data acquisition module, used to obtain live video and product information in a certain period of time;

[0014] A time point detection module is used to detect the time points of the product information and the live video to obtain all time points; wherein all time points include: the time point of the first segmented video, the audio time point and the video screenshot time point; the first segmented video is obtained by using the autoregressive coordinate point method to detect the time points of the live video, mark the product classification and divide the time period according to the preset time window length; the audio time point is obtained by detecting and matching the product information with the voice text information extracted from the live video; the video screenshot time point is obtained by detecting and matching the product screenshot in the first segmented video with the live video;

[0015] A multimodal fusion module is used to perform multimodal fusion on the product information and all time points, so as to capture the contextual relationship and key information between the two through a cross attention mechanism and a sequence attention mechanism, and obtain a time point product feature vector of the association relationship between the time point and the product;

[0016] The re-division confirmation module is used to traverse and compare the time point product feature vectors of adjacent time points based on the product information and all time points, and re-divide the video time period of the live video to obtain a second segmented video with more accurate product classification.

[0017] The present invention also provides a video segmentation device based on cross attention and sequential attention, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a video segmentation method based on cross attention and sequential attention as described above.

[0018] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a video segmentation method based on cross attention and sequential attention is implemented as described above.

[0019] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) The present invention adopts a variety of time point matching methods to obtain video time points, such as using an autoregressive coordinate point method to detect the time points of product information and live video to obtain preliminary start and end time points, that is, the first segmented video; then using voice technology to obtain the voice time points in the video, and using an image matching model to obtain the video screenshot time points, and obtain the remaining scattered time points in the video, so as to recall the start and end time points of the product-related video to the greatest extent and reduce the misjudgment of subsequent time points.

[0021] (2) Based on cross attention and sequential attention, the present invention performs multimodal feature fusion on product information and all time points to fully capture the correlation characteristics between video time points and products, distinguish the time points of the same product and those of different products, and more accurately identify the time points of live video.

[0022] (3) The present invention fully considers the attributes of the product and solves the problem of video segmentation in the product dimension from the perspectives of product information, voice and text information, and product brief information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a video segmentation method based on cross attention and sequential attention provided in Example 1 of the present invention.

[0025] Figure 2 A flowchart of a video segmentation method based on cross attention and sequential attention is provided in Example 1 of the present invention.

[0026] Figure 3 A schematic diagram of a time point detection process provided in Embodiment 1 of the present invention.

[0027] Figure 4A schematic diagram of the fusion and splicing of text and image information in the multimodal fusion provided in the first embodiment of the present invention.

[0028] Figure 5 A schematic diagram of cross-attention fusion of product information and time points provided in Example 1 of the present invention.

[0029] Figure 6 This is a schematic diagram of the result of multimodal fusion of commodity information and time points provided in the first embodiment of the present invention.

[0030] Figure 7 A schematic diagram of a video segmentation system based on cross attention and sequential attention provided in Example 2 of the present invention.

[0031] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] Embodiment 1

[0034] Embodiment 1 of the present invention provides a video segmentation method based on cross attention and sequential attention, which can be implemented by a video segmentation device based on cross attention and sequential attention (hereinafter referred to as the video segmentation device), and in particular, executed by one or more processors in the video segmentation device.

[0035] In this embodiment, the video segmentation device may be an electronic device equipped with a processor, the processor having a computer program of the video segmentation method based on cross attention and sequential attention and the computer program can be executed, such as a computer, a smart phone, a smart tablet, etc.

[0036] During the e-commerce live broadcast process, it is often necessary to review the product live broadcast, summarize the product live broadcast sales in a certain period of time or every day or week, analyze and evaluate the impact of the anchor's speech on product sales, etc., in order to make the next decision analysis. Therefore, it is necessary to find the video of the relevant products introduced by the anchor in the live broadcast video and extract it for product classification analysis.

[0037] like Figure 1-Figure 2 As shown, a video segmentation method based on cross attention and sequential attention includes steps S1 to S5.

[0038] S1, obtain live video and product information in a certain time period.

[0039] In this embodiment, the live video of a certain time period is obtained, and the historical live video of the certain time period to be analyzed can be obtained.

[0040] The product information includes product title, product nickname, product image, product model and product style, etc.

[0041] S2, performing time point detection on the product information and the live video to obtain all time points; wherein all time points include: time points of the first segmented video, audio time points and video screenshot time points; the first segmented video is obtained by using an autoregressive coordinate point method to perform time point detection, product classification marking and time period division on the live video according to a preset time window length; the audio time point is obtained by detecting and matching the product information with the voice and text information extracted from the live video; the video screenshot time point is obtained by detecting and matching the product screenshot in the first segmented video with the live video.

[0042] Preferably, when the autoregressive coordinate point method is used to obtain the first segmented video, the specific operations are as follows:

[0043] Initialize the first segmented video, set the time window length to c seconds, use the SeqTrack model to track the target product in each frame of the live video within the current time window, and obtain its regression coordinates; wherein the first segmented video is a video time period list;

[0044] For each frame of video within the c-second time window, calculate the coordinate change between it and the previous frame of video;

[0045] If the coordinate change of the current frame is less than the preset threshold, it means that the change is not large within the c-second window, and the current frame is added to the current time period; otherwise, the previous frame of the current frame is set as the end point of the current time period, and the current time period is added to the first segmented video, and the current frame is used as the starting point of the next time period;

[0046] Continue to calculate and compare the next frame of video until the end point of the current window;

[0047] Then, the subsequent video frames are traversed in a c-second window manner until the end point of the live video, and the first segmented video divided into time periods according to the products is obtained.

[0048] In this embodiment, the autoregression model is often used in statistics and data analysis to predict and analyze time series data. When processing coordinate point data, if there is temporal dependence or spatial autocorrelation between coordinate points, the idea of ​​the autoregression model can be used for analysis or prediction.

[0049] In this embodiment, according to actual needs, c seconds is set to 5 seconds. If the 5-second window does not change much, it means that the same product is shown in the 5-second video, and the 5 seconds are set as a time period. If the 5-second window changes too much, if there is no starting point, the current frame is set as the starting point of the time period, and then the end point is found in the 5-second window mode; if there is a starting point, it is set as the end point of the time period.

[0050] Furthermore, the audio time point is obtained by detecting and matching the product information with the voice text information extracted from the live video:

[0051] According to the first segmented video, obtaining a screenshot of the product in the video;

[0052] Using voice ASR (Automatic Speech Recognition) technology, converting the audio information in the live video into text information to obtain audio text information;

[0053] A speech-to-text model can also be used to convert audio information into text information.

[0054] Using NLP (natural language processing) technology, identifying and extracting brief product information from the product information, wherein the brief product information includes product name, product brand and product alias;

[0055] The audio text information is matched with the brief information of the commodity, and the time point of the brief information of the commodity in the audio is extracted to obtain the audio time point.

[0056] When detecting and matching the product screenshot in the first segmented video with the live video, an image matching model is used to perform matching to obtain the time point of the product screenshot in the live video, that is, the video screenshot time point.

[0057] In this embodiment, the image matching model is used to identify and compare key features in the image to achieve matching. A pre-trained CNN model (such as VGG, ResNet), OCR text recognition, image recognition technology, etc. can be used, without limitation here.

[0058] like Figure 3 As shown in the figure, the A and C points of the first segmented video are obtained after the autoregressive model is used for time point detection; the audio time points detected after the audio-to-text conversion and the product matching are points B, E, and G; the video screenshot time points matching the product image are points D and F. Through the above time point detection method, possible time points can be recalled as much as possible to reduce detection errors.

[0059] S3, multimodally fuse the product information with all time points to capture the contextual relationship and key information between the two through a cross-attention mechanism and a sequence attention mechanism, and obtain a time point product feature vector of the association relationship between the time point and the product.

[0060] Further details:

[0061] like Figure 4 As shown, the image information of all time points and the adjacent text information of the corresponding time points are obtained;

[0062] The adjacent text information is subjected to a text coding model to obtain text features, the picture information is subjected to an image coding model to obtain image features, and the image features are fused and spliced ​​with the text features to obtain product features.

[0063] Then, the product features are respectively concatenated with the product information and all time points in an ordered single-dimensional vector to obtain a product information concatenation vector Q and a time point information concatenation vector; and the time point information concatenation vector is mapped into a key vector K and a value vector V;

[0064] like Figure 5 As shown in the figure, the relationship between Q and K is captured through the cross-attention mechanism, so as to obtain the relational attention weight vector Z between Q and K, that is, the feature of the relationship between the time point and the product.

[0065] like Figure 6 As shown, through the sequence attention mechanism, the relational attention weight vector Z is used to perform weighted summation on the value vector V, and the key information between the product information and the time point information is captured by assigning weights to obtain the time point product feature vector of the relationship between the time point and the product.

[0066] In this embodiment, the adjacent text information is a piece of text adjacent to the time point (such as the text corresponding to 2 seconds before and after the time point, etc., which can be set according to actual needs) or a 5-second audio text, which can be set according to actual needs.

[0067] In this embodiment, a text encoding model, such as a word embedding model, a BERT model, etc., is a process of converting text data (such as words, sentences, or paragraphs) into a numerical form that can be understood and processed by a computer.

[0068] The image coding model is a collection of algorithms and technologies that convert image data into digital form and compress it, such as JPEG\PNG and other image encodings.

[0069] The product information, all time points, and the texts and images corresponding to the time points are taken as input and input into the multimodal fusion model to re-determine the time period for video segmentation.

[0070] In this embodiment, the cross-attention mechanism is an attention mechanism that handles the dependency between two different sequences. In the cross-attention, one sequence (usually the query sequence) focuses on another sequence (usually the key and value sequence) through the attention mechanism, thereby capturing the relationship between the two sequences.

[0071] Sequence attention is an attention mechanism used in sequence modeling tasks, which allows the model to focus on the part of the sequence that is most relevant to the current prediction. Generally, there are self-attention and multi-head attention. In sequence modeling tasks, the attention mechanism focuses on key information by assigning weights to each sequence element.

[0072] S4, according to the product information and all time points, traverse and compare the time point product feature vectors of adjacent time points, re-divide the video time period of the live video, and obtain a second segmented video with more accurate product classification.

[0073] Furthermore, the time point product feature vectors of adjacent time points are traversed and compared. When the live video is re-divided into video time periods, the time point product feature vector of the current time point is compared with the time point product feature vector of the previous time point to re-divide the video into time periods. If the comparison result does not exceed the threshold, it is considered that the current time point and the previous time point belong to the same product, and the current time point is merged into the current time period; otherwise, the current time point is set as the starting point of the next time period and compared with the next time point until all time points are calculated and compared to obtain the second segmented video of the re-divided time period.

[0074] S5, confirming the association between the video time period and the product based on the second segmented video and the product information, that is: matching the product information with the audio text information in the second segmented video, identifying and recording the products and the corresponding number of times that appear in each video time period, and taking the product that appears the most times in each video time period as the associated product of the corresponding video time period.

[0075] When matching the product information with the audio and text information in the second segmented video, in order to reduce the amount of calculation, the product information is analyzed and judged before the matching calculation. If the product information is for similar products, simple keywords of the product information are used for matching; if the product information is for non-similar products, the brief information of the product extracted by NLP technology is used for matching; wherein the simple keywords include product models and product styles.

[0076] When matching the video time period with the product, you can use keywords to match the translated text in the video time period to get the number of votes for the product in each video time period. A successful match counts as one vote, and multiple products are allowed in one video time period. You can also use the main image of the product and the random frames of the video to match to get the number of votes for the product in the video time period. Products with high voting values ​​are used as the products introduced in this video.

[0077] The reason for using this mechanism is that the deeper the connection between the video and the product, the more product information the video reveals and the more times it appears.

[0078] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0079] In the time point detection part, in addition to using OCR text recognition, image recognition and translated text matching technology, the present invention also uses an autoregressive model to detect the time point when the product appears, increase the recall possibility of the time point, and make the start and end time of the detection and identification more accurate.

[0080] In the multimodal fusion part, cross attention is used to match the product information in the video with the related product information to solve the problem of associating video clips and products. Sequential attention is used to let the model learn the relationship between previous and next time points, so as to distinguish whether the previous and next time points belong to the same product.

[0081] Finally, use voting to associate the complete video clip with the actual product. The deeper the association between the video clip and the product, the more product information the video reveals, and the higher the voting value.

[0082] The present invention can fully consider the attributes of commodities and solve the problem of dividing videos by commodity dimensions in live broadcast videos.

[0083] Embodiment 2

[0084] like Figure 7 As shown, the second embodiment of the present invention further provides a video segmentation system based on cross attention and sequence attention, including:

[0085] Data acquisition module, used to obtain live video and product information in a certain period of time;

[0086] A time point detection module is used to detect the time points of the product information and the live video to obtain all time points; wherein all time points include: the time point of the first segmented video, the audio time point and the video screenshot time point; the first segmented video is obtained by using the autoregressive coordinate point method to detect the time points of the live video, mark the product classification and divide the time period according to the preset time window length; the audio time point is obtained by detecting and matching the product information with the voice text information extracted from the live video; the video screenshot time point is obtained by detecting and matching the product screenshot in the first segmented video with the live video;

[0087] A multimodal fusion module is used to perform multimodal fusion on the product information and all time points, so as to capture the contextual relationship and key information between the two through a cross attention mechanism and a sequence attention mechanism, and obtain a time point product feature vector of the association relationship between the time point and the product;

[0088] The re-division confirmation module is used to traverse and compare the time point product feature vectors of adjacent time points based on the product information and all time points, and re-divide the video time period of the live video to obtain a second segmented video with more accurate product classification.

[0089] Furthermore, it also includes:

[0090] The video product association module is used to confirm the association between the video time period and the product based on the second segmented video and the product information, that is, matching the product information with the audio text information in the second segmented video, identifying and recording the products and the corresponding number of times that appear in each video time period, and taking the product that appears the most times in each video time period as the associated product of the corresponding video time period.

[0091] Embodiment 3

[0092] The third embodiment of the present invention also provides a video segmentation device based on cross-attention and sequential attention, which includes a memory and a processor, wherein a computer program is stored in the memory, and the computer program can be executed by the processor to implement the video segmentation method based on cross-attention and sequential attention as described above.

[0093] Embodiment 4

[0094] The fourth embodiment of the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the video segmentation method based on cross attention and sequential attention as described above is implemented.

[0095] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0097] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0098] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0099] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0100] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0101] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video segmentation method based on cross attention and sequential attention, characterized in that: include: S1, obtain live video and product information in a certain period of time; S2, performing time point detection on the product information and the live video to obtain all time points; wherein all time points include: time points of the first segmented video, audio time points and video screenshot time points; the first segmented video is obtained by using an autoregressive coordinate point method to perform time point detection, product classification marking and time period division on the live video according to a preset time window length; the audio time point is obtained by detecting and matching the product information with the voice and text information extracted from the live video; the video screenshot time point is obtained by detecting and matching the product screenshot in the first segmented video with the live video; S3, multimodally fusing the product information with all time points to capture the contextual relationship and key information between the two through a cross attention mechanism and a sequence attention mechanism, and obtaining a time point product feature vector of the association relationship between the time point and the product; S4, according to the product information and all time points, traverse and compare the time point product feature vectors of adjacent time points, re-divide the video time period of the live video, and obtain a second segmented video with more accurate product classification; Wherein, the S3 is specifically: Get the image information of all time points and the adjacent text information of the corresponding time points; The image information is subjected to an image coding model to obtain image features, the adjacent text information is subjected to a text coding model to obtain text features, and the image features and the text features are fused and spliced ​​to obtain product features; The product features are respectively concatenated with the product information and all time points in an ordered single-dimensional vector to obtain a product information concatenation vector Q and a time point information concatenation vector; and the time point information concatenation vector is mapped into a key vector K and a value vector V; Through the cross attention mechanism, the relationship between Q and K is captured, thereby obtaining the relationship attention weight vector Z between Q and K; Through the sequential attention mechanism, the relational attention weight vector Z is used to perform weighted summation on the value vector V. The key information between product information and time point information is captured by assigning weights to obtain the time point product feature vector of the relationship between the time point and the product.

2. According to claim 1, a video segmentation method based on cross attention and sequential attention is characterized in that , also includes: S5, confirming the association between the video time period and the product based on the second segmented video and the product information, that is, matching the product information with the audio text information in the second segmented video, identifying and recording the products and the corresponding times that appear in each video time period, and taking the product that appears the most times in each video time period as the associated product of the corresponding video time period.

3. The video segmentation method based on cross attention and sequential attention according to claim 2 is characterized in that The product information includes product title, product nickname, product image, product model and product style.

4. The video segmentation method based on cross attention and sequential attention according to claim 3 is characterized in that ,When the autoregressive coordinate point method is used to obtain the first segmented video, the details are as follows: Initialize the first segmented video, set the time window length to c seconds, use the SeqTrack model to track the target product in each frame of the live video within the current time window, and obtain its regression coordinates; wherein the first segmented video is a video time period list; For each frame of video within the c-second time window, calculate the coordinate change between it and the previous frame of video; If the coordinate change of the current frame is less than the preset threshold, it means that the change is not large within the c-second window, and the current frame is added to the current time period; otherwise, the previous frame of the current frame is set as the end point of the current time period, and the current time period is added to the first segmented video, and the current frame is used as the starting point of the next time period; Continue to calculate and compare the next frame of video until the end point of the current window; Then, the subsequent video frames are traversed in a c-second window manner until the end point of the live video, and the first segmented video divided into time periods according to the products is obtained.

5. The video segmentation method based on cross attention and sequential attention according to claim 4 is characterized in that The audio time point is obtained by detecting and matching the product information with the voice text information extracted from the live video, specifically: According to the first segmented video, obtaining a screenshot of the product in the video; Using voice ASR technology, converting the audio information in the live video into text information to obtain audio text information; Using NLP technology, identifying and extracting brief information of the product from the product information, wherein the brief information of the product includes the product name, product brand and product alias; The audio text information is matched with the brief information of the commodity, and the time point of the brief information of the commodity in the audio is extracted to obtain the audio time point.

6. The video segmentation method based on cross attention and sequential attention according to claim 5 is characterized in that When detecting and matching the product screenshot in the first segmented video with the live video, an image matching model is used to match the time point of the product screenshot in the live video, that is, the video screenshot time point.

7. The video segmentation method based on cross attention and sequential attention according to claim 6 is characterized in that In S4, the time point product feature vectors of adjacent time points are traversed and compared. When the live video is re-divided into video time periods, the time point product feature vector of the current time point is compared with the time point product feature vector of the previous time point to re-divide the video into time periods. If the comparison result does not exceed the threshold, it is considered that the current time point and the previous time point belong to the same product, and the current time point is merged into the current time period; otherwise, the current time point is set as the starting point of the next time period and compared with the next time point until all time points are calculated and compared to obtain the second segmented video of the re-divided time period.

8. The video segmentation method based on cross attention and sequential attention according to claim 7, characterized in that: When matching the product information with the audio and text information in the second segmented video, in order to reduce the amount of calculation, the product information is analyzed and judged before the matching calculation. If the product information is for similar products, simple keywords of the product information are used for matching; if the product information is for non-similar products, the brief information of the product extracted by NLP technology is used for matching; wherein the simple keywords include product models and product styles.

9. A video segmentation system based on cross attention and sequential attention, characterized in that: include: Data acquisition module, used to obtain live video and product information in a certain period of time; A time point detection module is used to detect the time points of the product information and the live video to obtain all time points; wherein all time points include: the time point of the first segmented video, the audio time point and the video screenshot time point; the first segmented video is obtained by using the autoregressive coordinate point method to detect the time points of the live video, mark the product classification and divide the time period according to the preset time window length; the audio time point is obtained by detecting and matching the product information with the voice text information extracted from the live video; the video screenshot time point is obtained by detecting and matching the product screenshot in the first segmented video with the live video; The multimodal fusion module is used to perform multimodal fusion on the product information and all time points, so as to capture the contextual relationship and key information between the two through the cross attention mechanism and the sequence attention mechanism, and obtain the time point product feature vector of the association relationship between the time point and the product, specifically: Get the image information of all time points and the adjacent text information of the corresponding time points; The image information is subjected to an image coding model to obtain image features, the adjacent text information is subjected to a text coding model to obtain text features, and the image features and the text features are fused and spliced ​​to obtain product features; The product features are respectively concatenated with the product information and all time points in an ordered single-dimensional vector to obtain a product information concatenation vector Q and a time point information concatenation vector; and the time point information concatenation vector is mapped into a key vector K and a value vector V; Through the cross attention mechanism, the relationship between Q and K is captured, thereby obtaining the relationship attention weight vector Z between Q and K; Through the sequence attention mechanism, the relational attention weight vector Z is used to perform weighted summation on the value vector V. The key information between product information and time point information is captured by assigning weights, and the time point product feature vector of the relationship between the time point and the product is obtained. The re-division confirmation module is used to traverse and compare the time point product feature vectors of adjacent time points based on the product information and all time points, and re-divide the video time period of the live video to obtain a second segmented video with more accurate product classification.