Video detection method, system, storage medium and server
By acquiring and fusing the single-modal attention features of multiple modal object detection data of target video, calculating similarity feature information, and determining video clips, the problems of low detection accuracy and insufficient robustness in the prior art are solved, and the problem of low detection accuracy and insufficient robustness are realized. Higher video detection accuracy and adaptability.
Patent Information
- Application Number
- CN202210166453.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-23
AI Technical Summary
The existing video detection method only considers a single related video clip when processing target videos, and there is a bias in the network model training data set, resulting in low detection accuracy, insufficient robustness and generalization, and cannot be effectively applied to complex Real scene.
By obtaining multiple modal object detection data of the target video, single-modal attention features are extracted and fused to obtain global features, and then similarity feature information is calculated with the specific detection information to determine the relevant video clips in the video.
It improves the accuracy and robustness of video detection, can obtain information in the target video more accurately, and is suitable for complex real scenes and multiple modal inputs, enhancing the generalization ability of the model.
Smart Images

Figure CN114581821B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information processing technology based on artificial intelligence, and in particular to a video detection method, system, storage medium and server. Background Art
[0002] Current video detection technology can generally detect the parts of the target video that meet specific conditions. It can be widely used in scenarios such as video clip retrieval, highlight moment detection, video mixing and video cover generation. The accuracy of target video detection must be guaranteed to meet user needs in various scenarios.
[0003] Existing video detection methods may include: using an artificial intelligence network model to detect relevant video clips from a target video according to specific conditions. However, the existing video detection methods only consider that there is only one relevant video clip in the video, and the data set used in training the network model also has a serious bias, that is, most target clips are located at the beginning of the video, which makes the accuracy of target video detection not high, and the robustness and generalization of the network model are also poor, and it cannot be well applied to more complex real-life scenarios. Summary of the invention
[0004] The embodiments of the present invention provide a video detection method, system, storage medium and server, which improve the accuracy of video detection.
[0005] An embodiment of the present invention provides a video detection method, including:
[0006] Obtain target detection data of multiple modes contained in the target video, and obtain specific detection information;
[0007] Obtaining unimodal attention features corresponding to the target detection data of the multiple modalities respectively;
[0008] fusing the single-modal attention features corresponding to the target detection data of the multiple modalities respectively to obtain the global features of the target video;
[0009] Acquire similarity feature information between the global feature and specific detection information;
[0010] A video segment in the target video that is related to the specific detection information is determined according to the similarity feature information.
[0011] Another aspect of the present invention provides a video detection system, including:
[0012] An information acquisition unit, used to acquire target detection data of multiple modes contained in a target video, and to acquire specific detection information;
[0013] A unimodal feature unit, used to obtain unimodal attention features corresponding to the target detection data of the multiple modalities respectively;
[0014] A fusion unit, used for fusing the single-modal attention features corresponding to the target detection data of the multiple modalities to obtain the global features of the target video;
[0015] A similarity unit, used to obtain similarity feature information between the global feature and specific detection information;
[0016] A detection unit is used to determine a video segment in the target video that is related to the specific detection information based on the similarity feature information.
[0017] Another aspect of the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the video detection method as described in one aspect of the embodiment of the present invention.
[0018] Another aspect of the embodiment of the present invention further provides a server, including a processor and a memory;
[0019] The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the video detection method as described in one aspect of an embodiment of the present invention; the processor is used to implement each computer program in the multiple computer programs.
[0020] It can be seen that in the method of this embodiment, the video detection system will divide the target video into multiple modal target detection data, obtain the corresponding unimodal attention features respectively, and then fuse these unimodal attention features to obtain the global features of the target video, and then obtain the similarity feature information between the global features and the specific detection information, and determine the video segments related to the specific detection information based on the similarity feature information. In this way, by processing the target detection data of each modality separately, all types of information contained in the target video can be obtained more accurately, and then the global features obtained by fusing the unimodal attention features can more realistically reflect the target video, and the video segments finally determined are also more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0022] Figure 1is a schematic diagram of a video detection method provided by an embodiment of the present invention;
[0023] Figure 2 is a flow chart of a video detection method provided by an embodiment of the present invention;
[0024] Figure 3 is a flow chart of a method for training a video detection model in one embodiment of the present invention;
[0025] Figure 4 is a schematic diagram of an initial model for video detection determined in one embodiment of the present invention;
[0026] Figure 5 is a schematic diagram of a video detection model obtained by training in an application embodiment of the present invention;
[0027] Figure 6 is a flow chart of a video detection method provided by an application embodiment of the present invention;
[0028] Figure 7 It is a schematic diagram of the structure of a cross-modal encoder and a single-modal encoder in an application embodiment of the present invention;
[0029] Figure 8 is a curve graph of highlight significance scores detected in an application embodiment of the present invention;
[0030] Fig. 9 is a schematic diagram of a distributed system to which a video detection method in another application embodiment of the present invention is applied;
[0031] Fig.10 is a schematic diagram of a block structure in another application embodiment of the present invention;
[0032] Fig.11 is a logical structure diagram of a video detection system provided by an embodiment of the present invention;
[0033] Fig.12 It is a schematic diagram of the logical structure of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0036] The embodiment of the present invention provides a video detection method, which can be applied to scenes such as video clip retrieval, highlight moment detection, video remixing and video cover generation. Specifically, Figure 1 As shown, the video detection system can detect the target video through the following steps:
[0037] Obtain target detection data of multiple modes (n modes are taken as an example in the figure) contained in the target video, and obtain specific detection information; obtain the single-modal attention features corresponding to the target detection data of the multiple modes; fuse the single-modal attention features corresponding to the target detection data of the multiple modes to obtain the global features of the target video; obtain similarity feature information between the global features and the specific detection information; determine the video segment in the target video related to the specific detection information according to the similarity feature information.
[0038] In actual applications, the video detection system can be applied to but not limited to the following forms of user terminals: mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc. It can also be applied to servers without limiting the specific devices used.
[0039] In this way, by processing the target detection data of each modality separately, all types of information contained in the target video can be obtained more accurately. Then, the global features obtained by fusing the attention features of each single modality can more realistically reflect the target video, and the final determined video clips are also more accurate.
[0040] The embodiment of the present invention provides a video detection method, which is mainly a method executed by a video detection system. The flow chart is as follows: Figure 2 As shown, including:
[0041] Step 101, obtaining target detection data of multiple modes contained in a target video, and obtaining specific detection information.
[0042] It can be understood that the video detection system will provide a user interface so that the user can operate the user interface, input specific detection information from the user interface, and initiate a video detection request based on the specific detection information, and the video detection request is used to request the video detection system to detect video clips related to the specific detection information. After obtaining the video detection request, the video detection system can initiate the video detection process of this embodiment. For example, the video detection system provides a user search interface. When the user enters the keyword to be searched through the user search interface and triggers the start search control, the video detection system can obtain the video detection request based on the keyword to be searched.
[0043] Furthermore, in the process of initiating a video detection request through the above-mentioned user interface, the user can also specify a video as a target video through the user interface provided by the video detection system. The initiated video detection request is then used to request the video detection system to detect video clips related to specific detection information in the target video.
[0044] The specific detection information refers to certain conditions satisfied by the detected video clip, and may be a voice, picture or short text input by the user.
[0045] When the video detection system initiates the process of this embodiment, any video in the system can be used as the target video, or the video specified in the video detection request initiated by the user can be used as the target video, and target detection data of multiple modalities contained in the target video can be obtained. The target detection data of each modality is data in a format contained in the target video, such as multi-frame image data contained in the target video, or audio data in the target video, or text data corresponding to subtitle information in the target video, etc.
[0046] Step 102, obtaining unimodal attention features corresponding to target detection data of multiple modalities.
[0047] The video detection system needs to obtain the corresponding unimodal attention features for each modality of target detection data. The unimodal attention features refer to the features obtained after paying special attention to (i.e., enhancing) some aspects of the feature information of the target detection data. Specifically, it is necessary to set weight values for each sub-feature in the feature information of the target detection data, such as setting a larger weight value for the feature that is focused on. In the specific implementation, multiple groups of weight matrices can be learned according to the feature information of the target detection data of each modality, and each group of weight matrices can enhance the effective information of the feature information of the unimodal target detection data.
[0048] In this embodiment, the unimodal attention feature is mainly obtained by weighted fusion of feature information of unimodal target detection data after global context aggregation, so as to obtain the unimodal attention feature.
[0049] Specifically, the video detection system can first extract the feature information of the target detection data of each modality, and then use the attention mechanism to fuse the weighted values of each sub-feature in the feature information to obtain the unimodal attention feature of the target detection data.
[0050] Step 103, merging the single-modal attention features corresponding to the target detection data of multiple modalities to obtain the global features of the target video.
[0051] Specifically, the video detection system can first compress the single-modal attention features corresponding to the target detection data of multiple modalities into a low-dimensional feature space to obtain multiple low-dimensional features; then fuse the multiple low-dimensional features to obtain fused features; finally, expand the fused features to a high-dimensional feature space, and the obtained high-dimensional fused features are the global features of the target video, where it is assumed that the dimension of the high-dimensional feature space is N v , needs to be consistent with the dimension of the above unimodal attention feature, and the dimension of the low-dimensional feature space is N b , in the specific implementation, the dimension N b Can be much smaller than dimension N v , that is, N b N v .
[0052] Since the video detection system mainly performs feature fusion in low-dimensional space, the computational complexity of feature fusion is relatively low.
[0053] Step 104, obtaining similarity feature information between the global feature and the specific detection information.
[0054] Specifically, the similarity feature information is used to represent the similarity between each sub-feature in the global feature and the feature represented by the specific detection information. For example, the distance information between each sub-feature and the feature represented by the specific detection information can be calculated.
[0055] Step 105: determine the video segment in the target video that is related to the specific detection information according to the similarity feature information.
[0056] Furthermore, in addition to determining the video segments, the video detection system may also have other detection tasks for the target video, such as detecting the highlight saliency score, etc. Specifically, the video detection system may obtain the highlight saliency score of each sub-segment included in the target video according to the similarity feature information. In this way, the same similarity feature information may be used for multiple detection tasks of the target video, thereby simplifying the detection of the target video.
[0057] The highlight prominence score is used to indicate whether each sub-segment in the target video is a significant highlight segment. The length of each sub-segment may be the same, for example, all sub-segments are 2 seconds long.
[0058] It should be noted that the above steps 102 to 104 can be implemented by a preset video detection model, and the video detection model is a machine learning model based on artificial intelligence, which can be trained in advance by a certain training method, and its operation logic is stored in the video detection system. When the video detection system initiates the process of this embodiment, after executing step 101, the preset video detection model can be directly called to execute steps 102 to 105. Among them, the training samples used in training the video detection system can include: multiple sample videos, specific detection samples corresponding to each sample video, and video segment annotations based on specific detection samples in each sample video.
[0059] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0060] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0061] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0062] It can be seen that in the method of this embodiment, the video detection system will divide the target video into multiple modal target detection data, obtain the corresponding unimodal attention features respectively, and then fuse these unimodal attention features to obtain the global features of the target video, and then obtain the similarity feature information between the global features and the specific detection information, and determine the video segments related to the specific detection information based on the similarity feature information. In this way, by processing the target detection data of each modality separately, all types of information contained in the target video can be obtained more accurately, and then the global features obtained by fusing the unimodal attention features can more realistically reflect the target video, and the video segments finally determined are also more accurate.
[0063] In a specific embodiment, in order to implement the above steps 102 to 105, the video detection model can be trained in advance according to the following steps. The flowchart is as follows: Figure 3 As shown, including:
[0064] Step 201, determine the initial model of video detection, the initial model of video detection includes: a first feature module, a second feature module, a cross-modal feature module, a query feature module and a fragment output module corresponding to multiple modalities respectively.
[0065] It can be understood that when determining the initial video detection model, the video detection system will determine the multi-layer structure included in the initial video detection model and the initial values of the parameters in each layer mechanism.
[0066] Specifically, Figure 4 As shown, the initial model for video detection may include: a plurality of first feature modules 10, a second feature module 11, a cross-modal feature module 12, a query feature module 13 and a segment output module 14, wherein:
[0067] A first feature module 10 of any modality is used to obtain unimodal attention features corresponding to target detection data of any modality contained in a video sample; a cross-modal feature module 12 is used to fuse the unimodal attention features obtained by each first feature module 10 to obtain a global feature of the video sample; a second feature module 11 is used to obtain feature information of a specific detection sample; a query feature module 13 is used to obtain similarity feature information between the global feature obtained by the cross-modal feature module 12 and the feature information of the corresponding specific detection sample obtained by the second feature module 11; a segment output module 14 is used to determine a video segment based on the corresponding specific detection sample in each video sample according to the similarity feature information obtained by the query feature module 13.
[0068] Furthermore, the determined initial model for video detection may also include: a highlight information output module 15 for determining highlight significance scores of respective segments in the target video in the video sample according to the similarity feature information obtained by the query feature module 13 .
[0069] The parameters of the initial model of video detection refer to the fixed parameters used by each layer structure in the initial model of video detection during the calculation process, which do not need to be assigned at any time, such as parameter scale, number of network layers, user vector length and other parameters.
[0070] Step 202 : determining training samples, where the training samples include a plurality of video samples, specific detection samples corresponding to each video sample, and video segment annotations based on the specific detection samples in each video sample.
[0071] Furthermore, if the initial video detection model determined in step 201 includes a highlight information output module 15, in order to train the highlight information output module 15, the training samples also include highlight saliency annotations of each sub-segment in each video sample.
[0072] Step 203, obtain the unimodal attention features corresponding to the target detection data of any modality contained in the video sample through the first feature module 10 of any modality, fuse the unimodal attention features obtained by the first feature module through the cross-modal feature module 12 to obtain the global features of the video sample, obtain the feature information of the specific detection sample through the second feature module 11, obtain the similarity feature information between the global feature and the feature information of the corresponding specific detection sample through the query feature module 13, and determine the video segment based on the corresponding specific detection sample in the video sample through the segment output module 14.
[0073] Furthermore, the highlight information output module 15 may also determine the highlight significance score of each sub-segment included in the video sample.
[0074] Step 204 , adjusting the initial video detection model according to the video segment annotations of the corresponding video samples in the video segment training samples of each video sample obtained by the initial video detection model to obtain a preset video detection model.
[0075] Specifically, the video detection system will first calculate the first loss function related to the initial video detection model based on the video clips obtained by the initial video detection model in the above step 203 and the video clip annotations in the training samples. The first loss function is used to indicate the error between the video clips of each video detected by the initial video detection model and the actual video clips (obtained according to the video clip annotations) based on the specific detection samples in each video sample, such as the cross entropy loss function.
[0076] Specifically, the video detection system will first calculate the error of the video detection initial model based on the video segments of each video sample output by the segment output module 14 in the video detection initial model and the video segment annotations of the corresponding video samples in the training samples; and then calculate the first loss function related to the video detection initial model based on the error of the video detection initial model.
[0077] The training process of the video detection model is to minimize the value of the above error. The training process continuously optimizes the parameter values of the parameters in the initial video detection model determined in the above step 201 through a series of mathematical optimization methods such as back propagation derivation and gradient descent, and reduces the calculated value of the above first loss function to the minimum.
[0078] Specifically, when the function value of the calculated first loss function is large, such as greater than a preset value, it is necessary to change the parameter value, such as reducing the weight value of a certain neuron connection, so that the function value of the loss function calculated according to the adjusted parameter value is reduced.
[0079] Furthermore, in a specific embodiment, if the initial model of video detection determined in the above step 201 includes a highlight information output module 15, then when adjusting the initial model of video detection, it is also necessary to make adjustments in combination with the highlight significance score output by the highlight information output module 15. Specifically, in addition to calculating the above-mentioned first loss function, the video detection system also needs to calculate the error of the highlight information output module 15 in the initial model of video detection based on the highlight significance scores of the sub-segments in each video sample output by the highlight information output module 15 in the initial model of video detection and the highlight significance labels of each sub-segment of the corresponding video sample in the training sample; then calculate the second loss function related to the highlight information output module 15 in the initial model of video detection based on the error, and calculate the overall loss function of the initial model of video detection based on the first loss function and the second loss function, and then adjust the parameter values of the parameters in the initial model of video detection based on the overall loss function.
[0080] In this case, the training process of the video detection model is to continuously optimize the parameter values of the parameters in the initial video detection model determined in the above step 201 through a series of mathematical optimization methods such as back propagation derivation and gradient descent, and minimize the calculated value of the above overall loss function.
[0081] It should also be noted that the above steps 203 to 204 are an adjustment of the parameter values in the video detection initial model based on the video segments of each video sample determined by the video detection initial model. In actual applications, it is necessary to continuously loop through the above steps 203 to 204 until the adjustment of the parameter values meets certain stop conditions.
[0082] Therefore, after executing steps 201 to 204 of the above embodiment, the video detection system needs to determine whether the current adjustment of the parameter value meets the preset stop condition. If it does, the process ends and the video detection initial model adjusted in step 204 is used as the preset video detection model; if it does not, the video detection initial model after the parameter value is adjusted returns to execute the above steps 203 to 204. The preset stop condition includes but is not limited to any one of the following conditions: the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold value, that is, the adjusted parameter value reaches convergence; and the number of parameter value adjustments is equal to the preset number of times, etc.
[0083] The following is a specific application example to illustrate the video detection method of the present invention. The video detection model trained in this embodiment can be as follows: Figure 5 As shown, it includes: a modal feature module 20 and a single modal encoder 21 corresponding to two modalities (image format and audio format), a specific feature module 22, a cross-modal encoder 23, a query generation module 24, a query encoder 25, a fragment output 26 and a highlight information output 27, wherein:
[0084] The combination of the modal feature modules 20 of each modality and the unimodal encoder 21 is similar to the above-mentioned first feature module, wherein the modal feature module 20 is used to extract feature information of target detection data of the corresponding modality in any video, and the unimodal encoder 21 is used to obtain the unimodal attention features of the target detection data based on the feature information extracted by the modal feature module 20.
[0085] Specifically, the unimodal encoder 21 can be implemented by one or more transformer encoder layers, each of which can include a multi-head self-attention module and a feedforward neural network. In order to avoid gradient diffusion and ensure stability during the training phase, residual connections and layer normalization modules are added to each layer.
[0086] The specific feature module 22 is similar to the above-mentioned second feature module, and is used to obtain feature information of specific detection information.
[0087] The cross-modal encoder 23 is similar to the above-mentioned cross-modal feature module, and is used to fuse various unimodal attention features to obtain the global features of any video.
[0088] The query generation module 24 and the query encoder 25 are combined to be similar to the above-mentioned query feature module, wherein the query generation module 24 is used to obtain the similarity information between the global feature and the feature information of the specific detection information, and the query encoder 25 is used to use the attention mechanism to process the similarity information of the query generation module 24 to obtain similarity feature information, which is feature information obtained after focusing on certain information in the similarity information.
[0089] Specifically, the query generation module 24 can be implemented by one or more attention layers, specifically, the similarity between each sub-feature in the global feature and the feature information of the specific detection information can be estimated; the query encoder 25 can be implemented by a certain number of Transformer decoder layers, each layer including a multi-head self-attention module, a multi-head attention module and a feedforward neural network.
[0090] The segment output 26 and the highlight information output 27 are similar to the segment output module and the highlight information output module described above, respectively. Specifically, the segment output 26 can obtain the temporal center point, length, and center point offset of a video segment based on specific detection information in any video.
[0091] After the above video detection model is trained, the operation logic of the video detection model can be preset in the video detection system in advance. Figure 6 As shown, the video detection system can detect the target video through the following steps:
[0092] In step 301, a user inputs specific detection information through a user interface provided by a video detection system, designates a video as a target video, and initiates a video detection request for detecting a video segment related to the specific detection information in the target video.
[0093] Step 302: After the video detection system obtains the video detection request, it calls a video detection model preset in the system.
[0094] Step 303: The video detection system obtains target detection data of two modes in the target video, specifically target detection data in image format and target detection data in audio format, and inputs them into the modality feature module 20 of the corresponding modality of the video detection model to obtain feature information of the target detection data of each modality. and Among them, N v is the length of the target detection data. At the same time, the video detection system will execute the following step 304.
[0095] Step 304: the video detection system inputs the specific detection information into the specific feature module 22 to extract the feature information of the specific detection information. Among them, N t is the length of the specific detection information.
[0096] Step 305: The single-mode encoder 21 of each mode will respectively obtain the characteristic information of the target detection data of each mode. and Unimodal attention features and Then, the cross-modal encoder 23 fuses the single-modal attention features to obtain the global features of the target video.
[0097] Specifically, in order to reduce the noise transmission between modes and reduce the computational cost, the cross-modal encoder 23 can adopt the information compression-expansion mechanism to propagate features when implemented. In general, the cross-modal encoder 23 can include multiple layers of cross-modal encoding and a fusion submodule, and the multiple layers of cross-modal encoding are connected in series, that is, the output of a single layer of cross-modal encoding is connected to the input of another single layer of cross-modal encoding. The first single layer of cross-modal encoding in the multi-layer cross-modal encoding is used to perform feature attention processing on the single-modal attention features of each modality to obtain cross-modal attention features, and the second single layer of cross-modal encoding in each subsequent layer is used to perform feature attention processing on the cross-modal attention features of each modality obtained by the previous single layer of cross-modal encoding to obtain another cross-modal attention feature; and the fusion submodule is used to fuse the cross-modal attention features of each modality obtained by the last single layer of cross-modal encoding in the multi-layer cross-modal encoding to obtain the global features of the target video.
[0098] In this way, the unimodal attention features obtained by each unimodal encoder 21 are processed by multiple layers of cross-modal encoding to obtain cross-modal attention features. Finally, the fusion sub-module fuses the cross-modal attention features to obtain the global features of the target video.
[0099] like Figure 7 The figure shows a single-layer cross-modal encoding, which may include: multiple multi-head attention sub-modules, multiple layer normalization, superposition sub-modules and a feedforward neural network, specifically:
[0100] The first multi-head attention submodule of the multiple multi-head attention submodules in the cross-modal encoder 23 uses a shorter sequence of media features (N b Much smaller than N v ), respectively, the unimodal attention features of target detection data of multiple modalities ( and ) is compressed into a low-dimensional feature space, and after attention processing is performed on each modal attention feature in the low-dimensional feature space, the first layer of standardization in multiple layers of standardization standardizes the features after low-dimensional attention processing to obtain the first standardized features, wherein the first multi-head attention submodule mainly uses the media feature sequence as q and the unimodal feature information as k and v for compression; the superposition submodule uses the first standardized features obtained by the first layer of standardization corresponding to each modality and the above-mentioned media feature sequence After superposition, the compressed cross-modal feature sequence is obtained The compressed cross-modal feature sequence They are respectively input into the second multi-head attention sub-module in the multiple multi-head attention sub-modules.
[0101] Each second multi-head attention submodule first converts the compressed cross-modal feature sequence The features after attention processing are expanded to high-dimensional space, and the second layer of multi-layer normalization is based on the unimodal attention features of the target detection data of the corresponding modality ( or ) and the features after attention processing are summed and standardized to obtain the second standardized features, thereby realizing the expansion of the features of the low-dimensional space to the high-dimensional space, specifically to the corresponding modality, wherein the second multi-head multi-attention submodule takes the unimodal attention feature as q, and the compressed cross-modal feature sequence as k and v, and expands the compressed cross-modal feature sequence to the original high-dimensional feature space of each modality after attention processing; then a feedforward neural network is used to filter the second standardized features, that is, to refine and improve them, and finally the third standardization in the multi-layer standardization is used to sum and standardize the filtered features to output the cross-modal attention features of the corresponding modality and
[0102] In this process, the single-layer cross-modal encoding in the cross-modal encoder 23 uses a compression-expansion mechanism to limit the feature calculation for each single modality to a low-dimensional space, and then expands and propagates it to each single modality. The computational complexity is reduced from flat to linear, and the cross-modal performance is effectively improved.
[0103] Step 306: the query generation module 24 obtains each sub-feature in the global feature. Characteristic information related to specific detection information Similarity information between The query encoder 25 will generate similarity information based on the query generation module 24. And the global feature information obtained by the cross-modal encoder 23 Get the similarity feature information between the target video and the specific detection information
[0104] The similarity feature information It is suitable for processing the fragment output 26 and the highlight information output 27 , and the format, dimension and other information of the input information required by the fragment output 26 and the highlight information output 27 are consistent.
[0105] Step 307, the segment output 26 is based on the similarity feature information Determine the information of the video clip related to the specific detection information in the target video, and output the highlight information 27 according to the similarity feature information Determine the highlight saliency score of each sub-segment in the target video, especially the video segment related to the specific detection information.
[0106] It can be seen that in the method of this embodiment, the two tasks of detecting the video clip and highlight information of the target video are unified through the video detection model, and the model processes the target detection information of multiple modalities separately, so that the characteristics of the target video can be accurately described, and the detection accuracy of the target video is improved. At the same time, the video detection model can automatically adapt to the input-output combination of multiple different modalities, which not only improves the robustness and generalization of the model in the face of multiple queries, but also can be applied to scenarios where there is insufficient available information and single or multiple modal inputs are missing.
[0107] Regarding the detection of highlight information of a video clip, when the single-mode target detection data in the target video is processed to detect the highlight saliency score, and the multi-modal target detection data in the embodiment of the present invention is used to detect the highlight saliency score, a curve graph of the change of the highlight saliency score with time in the target video is shown as follows: Figure 8 As shown, curves 1 to 4 respectively represent the highlight saliency score detected by the target detection data in picture format, the highlight saliency score detected by the target detection data in audio format, the true value of the highlight saliency score, and the highlight saliency score detected by the target detection data in multi-modality. It can be seen that the highlight saliency score detected by the target detection data in multi-modality is closer to the true value, and the video detection model of the embodiment of the present invention has higher robustness and generalization.
[0108] Furthermore, a variety of datasets, such as QVHighlights, YouTube Highlights and TVSum, are used as training samples to train the video detection model. The performance of the video detection model in detecting related video clips in the target video is measured by the recall rate (R1) and the mean average precision (mAP). The detection performance of the highlight information of sub-clips in the target video is evaluated by the mean average precision (mAP).
[0109] When the trained video detection model adopts a variety of different network structures (such as MCN, CAL, XML, etc.), the performance comparison results of the obtained video detection model are shown in Table 1 below, and the performance comparison structure of the video detection model with different network structures trained on different data sets is shown in Table 2 below. It can be seen that the various performance indicators of the video detection model obtained on each data set show that no matter which data set is used, when the structure of the video detection model in the embodiment of the present invention is used, the detection of video clips and highlight information is better (the indicators in bold font in Tables 1 and 2 below are better).
[0110]
[0111]
[0112] Table 1
[0113]
[0114] Table 2
[0115] The following is another specific application example to illustrate the video detection method in the present invention. The video detection system in the embodiment of the present invention is mainly a distributed system 100, which may include a client 300 and multiple nodes 200 (any form of computing device in the access network, such as a server, a user terminal), and the client 300 and the node 200 are connected through network communication.
[0116] Take the distributed system as the blockchain system as an example, see Fig. 9 This is an optional structural diagram of the distributed system 100 provided by the embodiment of the present invention applied to the blockchain system, which is formed by multiple nodes 200 (any form of computing devices in the access network, such as servers and user terminals) and clients 300, and a peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In a distributed system, any machine such as a server or a terminal can join and become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.
[0117] See also Fig. 9 The functions of each node in the blockchain system shown include:
[0118] 1) Routing: a basic function of a node, used to support communication between nodes.
[0119] In addition to the routing function, the node can also have the following functions:
[0120] 2) Applications, which are deployed in the blockchain to implement specific businesses according to actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to the temporary block when they successfully verify the source and integrity of the record data.
[0121] For example, the service implemented by the application includes the code for implementing the video detection function, which mainly includes:
[0122] Acquire target detection data of multiple modalities contained in a target video, and acquire specific detection information; acquire unimodal attention features respectively corresponding to the target detection data of the multiple modalities; fuse the unimodal attention features respectively corresponding to the target detection data of the multiple modalities to obtain a global feature of the target video; acquire similarity feature information between the global feature and the specific detection information; and determine a video segment in the target video that is related to the specific detection information based on the similarity feature information.
[0123] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.
[0124] See also Fig.10 An optional schematic diagram of the block structure provided by the embodiment of the present invention, each block includes the hash value of the transaction record stored in this block (the hash value of this block) and the hash value of the previous block, and each block is connected by the hash value to form a blockchain. In addition, the block can also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains relevant information for verifying the validity of its information (anti-counterfeiting) and generating the next block.
[0125] The embodiment of the present invention also provides a video detection system, the structural diagram of which is shown in FIG. Fig.11 As shown, it may specifically include:
[0126] The information acquisition unit 30 is used to acquire target detection data of multiple modes contained in the target video, and to acquire specific detection information.
[0127] The unimodal feature unit 31 is used to obtain unimodal attention features corresponding to the target detection data of multiple modes obtained by the information acquisition unit 30.
[0128] The unimodal feature unit 31 is specifically used to extract feature information of the target detection data; and to obtain the unimodal attention feature of the target detection data by fusing the weighted values of each sub-feature in the feature information.
[0129] The fusion unit 32 is used to fuse the unimodal attention features corresponding to the target detection data of multiple modes obtained by the unimodal feature unit 31 to obtain the global features of the target video.
[0130] The fusion unit 32 is specifically used to compress the unimodal attention features corresponding to the target detection data of the multiple modalities into low-dimensional feature spaces respectively to obtain multiple low-dimensional features; fuse the multiple low-dimensional features to obtain fused features; expand the fused features to high-dimensional feature space, and the obtained high-dimensional fused features are the global features of the target video, and the dimension of the high-dimensional feature space is consistent with the dimension of the unimodal attention features.
[0131] The similarity unit 33 is used to obtain similarity feature information between the global feature obtained by the fusion unit 32 and the specific detection information.
[0132] The detection unit 34 is used to determine the video segment related to the specific detection information in the target video according to the similarity feature information obtained by the similarity unit 33.
[0133] The detection unit 34 is further configured to determine a highlight saliency score of each sub-segment included in the target video according to the similarity feature information.
[0134] Furthermore, the video detection system in this embodiment also includes:
[0135] The training unit 35 is used to determine an initial model for video detection, wherein the initial model for video detection includes: a first feature module, a second feature module, a cross-modal feature module, a query feature module and a segment output module corresponding to the multiple modalities respectively; determine training samples, wherein the training samples include multiple video samples, specific detection samples corresponding to each video sample and video segment annotations based on the specific detection samples in each video sample; obtain a single-modal attention feature corresponding to the target detection data of any modality contained in the video sample through the first feature module of any modality, fuse the single-modal attention feature obtained by the first feature module through the cross-modal feature module to obtain a global feature of the video sample, obtain feature information of the specific detection sample through the second feature module, obtain similarity feature information between the global feature and the feature information of the corresponding specific detection sample through the query feature module, and determine a video segment based on the corresponding specific detection sample in the video sample through the segment output module; adjust the initial model for video detection according to the video segments of each video sample obtained by the initial model for video detection and the video segment annotations of the corresponding video sample in the training sample to obtain a preset video detection model. In this way, after acquiring the target detection data and specific detection information of each modality, the information acquisition unit 30 can call the video detection model trained by the training unit 35.
[0136] Furthermore, the training samples also include: highlight significance annotations of the video segments in the respective video samples, and the video detection initial model also includes a highlight information output module; the training unit 35 is also used to determine the highlight significance scores of each sub-segment of each video sample through the highlight information output module; and according to the video segments of each video sample determined by the segment output model in the video detection initial model and the video segment annotations of the corresponding video samples in the training samples, and the highlight significance scores of each video sample determined by the highlight information output module in the video detection initial model and the video segment annotations of the corresponding video samples in the training samples, the video detection initial model is adjusted.
[0137] Among them, the cross-modal feature module determined by the training unit 35 includes multiple layers of cross-modal encoding and a fusion sub-module, wherein: the multiple layers of cross-modal encoding are connected in series, the first single layer of cross-modal encoding in the multiple layers of cross-modal encoding is used to perform feature attention processing on the single-modal attention features of each modality respectively to obtain a cross-modal attention feature, and the second single layer of cross-modal encoding in the multiple layers of cross-modal encoding is used to perform feature attention processing on the cross-modal attention features of each modality obtained by the previous single layer of cross-modal encoding respectively to obtain another cross-modal attention feature; the fusion sub-module is used to fuse the cross-modal attention features of each modality obtained by the last single layer of cross-modal encoding in the multiple layers of cross-modal encoding to obtain the global features of the target video.
[0138] Among them, the cross-modal coding of any single layer in the multi-layer cross-modal coding includes: multiple multi-head attention sub-modules, multiple layer normalization, superposition sub-modules and feedforward neural network; specifically:
[0139] The first multi-head attention submodule among the multiple multi-head attention submodules is used to perform attention processing on the unimodal attention features of the target detection data of each modality in a low-dimensional feature space through a media feature sequence, the first layer of standardization among the multiple layers of standardization is used to standardize the features after attention processing to obtain first standardized features, the superposition submodule is used to superimpose the first standardized features obtained by each first layer of standardization and the media feature sequence to obtain a compressed cross-modal feature sequence, and the compressed cross-modal feature sequence is input into the second multi-head attention submodule among the multiple multi-attention submodules; the second multi-head attention submodule expands the compressed cross-modal feature sequence to a high-dimensional feature space, the second layer of standardization among the multiple layers of standardization is used to sum and standardize the features after attention processing obtained by the unimodal attention features and the second multi-head attention submodule to obtain second standardized features; the feedforward neural network is used to filter the second standardized features, and the third layer of standardization among the multiple layers of standardization is used to output the cross-modal attention features of the corresponding modality according to the filtered features.
[0140] Furthermore, the training unit 35 is also used to stop adjusting the parameter value when the number of adjustments to the parameter value is equal to a preset number, or when the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold.
[0141] It can be seen that in the system of this embodiment, the unimodal feature unit 31 will divide the target video into target detection data of multiple modes, and obtain the corresponding unimodal attention features respectively. The fusion unit 32 then fuses these unimodal attention features to obtain the global features of the target video, and then the similarity unit 33 obtains the similarity feature information between the global features and the specific detection information, and the detection unit 34 determines the video segment related to the specific detection information based on the similarity feature information. In this way, by processing the target detection data of each mode separately, all types of information contained in the target video can be obtained more accurately, and then the global features obtained by fusing the various unimodal attention features can more realistically reflect the target video, and the video segment finally determined is also more accurate.
[0142] The embodiment of the present invention further provides a server, the structural diagram of which is shown in Fig.12 As shown, the server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 40 (for example, one or more processors) and memory 41, and one or more storage media 42 (for example, one or more mass storage devices) storing application programs 421 or data 422. Among them, the memory 41 and the storage medium 42 can be short-term storage or permanent storage. The program stored in the storage medium 42 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 40 can be configured to communicate with the storage medium 42 to execute a series of instruction operations in the storage medium 42 on the server.
[0143] Specifically, the application 421 stored in the storage medium 42 includes an application for video detection, and the program may include the information acquisition unit 30, the unimodal feature unit 31, the fusion unit 32, the similarity unit 33, the detection unit 34 and the training unit 35 in the above-mentioned video detection system, which are not described in detail here. Furthermore, the central processing unit 40 can be configured to communicate with the storage medium 42, and execute a series of operations corresponding to the application for video detection stored in the storage medium 42 on the server.
[0144] The server may also include one or more power supplies 43, one or more wired or wireless network interfaces 44, one or more input and output interfaces 45, and / or one or more operating systems 423, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.
[0145] The steps performed by the video detection system in the above method embodiment can be based on the Fig.12 The structure of the server is shown.
[0146] Furthermore, another aspect of the embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the video detection method executed by the above-mentioned video detection system.
[0147] Another aspect of the embodiment of the present invention further provides a server, including a processor and a memory;
[0148] The memory is used to store a plurality of computer programs, and the computer programs are used to be loaded by the processor and execute the video detection method executed by the above-mentioned video detection system; the processor is used to implement each computer program in the plurality of computer programs.
[0149] In addition, according to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the video detection method provided in the above various optional implementations.
[0150] A person skilled in the art may understand that all or part of the steps in the various methods of the above-mentioned embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0151] The above is a detailed introduction to a video detection method, system, storage medium and server provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A video detection method, characterized in that: include: Obtain target detection data of multiple modes contained in the target video, and obtain specific detection information; The specific detection information is a certain condition satisfied by the detected video segment; Obtaining unimodal attention features corresponding to the target detection data of the multiple modalities respectively; fusing the single-modal attention features corresponding to the target detection data of the multiple modalities respectively to obtain the global features of the target video; Acquire similarity feature information between the global feature and the specific detection information; the similarity feature information is used to identify the similarity between each sub-feature in the global feature and the feature identified by the specific detection information; Determining a video segment in the target video that is related to the specific detection information according to the similarity feature information; The fusing of the unimodal attention features corresponding to the target detection data of the multiple modalities specifically includes: the unimodal attention features of each modality are processed by multi-layer cross-modal encoding to obtain cross-modal attention features, and the fusion submodule fuses the cross-modal attention features; Among them, the cross-modal coding of any single layer in the multi-layer cross-modal coding includes: multiple multi-head attention sub-modules, multiple layer standardization, superposition sub-modules and feedforward neural network; The first multi-head attention submodule among the multiple multi-head attention submodules is used to perform attention processing on the single-modal attention features of the target detection data of each modality in a low-dimensional feature space through a media feature sequence, the first layer of standardization among the multiple layers of standardization is used to standardize the features after attention processing to obtain first standardized features, the superposition submodule is used to superimpose the first standardized features obtained by each first layer of standardization and the media feature sequence to obtain a compressed cross-modal feature sequence, and the compressed cross-modal feature sequence is input into the second multi-head attention submodule among the multiple multi-attention submodules; The second multi-head attention submodule expands the compressed cross-modal feature sequence to a high-dimensional feature space, and the second layer of normalization in the multiple layers of normalization is used to sum and normalize the single-modal attention features and the attention-processed features obtained by the second multi-head attention submodule to obtain a second normalized feature; The feedforward neural network is used to filter the second standardized features, and the third layer of standardization in the multiple layers of standardization is used to output cross-modal attention features of the corresponding modality based on the filtered features.
2. The method according to claim 1, characterized in that The obtaining of the single-modal attention features corresponding to the target detection data of the multiple modalities respectively comprises: Extracting feature information of the target detection data; The weighted values of each sub-feature in the feature information are fused to obtain the unimodal attention feature of the target detection data.
3. The method according to claim 1, characterized in that The step of fusing the single-modal attention features corresponding to the target detection data of the multiple modalities to obtain the global features of the target video specifically includes: Compressing the single-modal attention features corresponding to the target detection data of the multiple modalities into a low-dimensional feature space respectively to obtain multiple low-dimensional features; Fusing the multiple low-dimensional features to obtain fused features; The fused features are expanded to a high-dimensional feature space, and the obtained high-dimensional fused features are the global features of the target video. The dimension of the high-dimensional feature space is consistent with the dimension of the unimodal attention feature.
4. The method according to claim 1, characterized in that The method further comprises: Determine a highlight saliency score of each sub-segment included in the target video according to the similarity feature information.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine an initial video detection model, wherein the initial video detection model includes: a first feature module, a second feature module, a cross-modal feature module, a query feature module, and a segment output module corresponding to the multiple modalities respectively; Determine a training sample, wherein the training sample includes a plurality of video samples, a specific detection sample corresponding to each video sample, and a video segment annotation based on the specific detection sample in each video sample; Obtaining a unimodal attention feature corresponding to the target detection data of any modality contained in the video sample through a first feature module of any modality, fusing the unimodal attention feature obtained by the first feature module through the cross-modal feature module to obtain a global feature of the video sample, obtaining feature information of the specific detection sample through the second feature module, obtaining similarity feature information between the global feature and the feature information of the corresponding specific detection sample through the query feature module, and determining a video segment based on the corresponding specific detection sample in the video sample through the segment output module; According to the video segments of each video sample obtained by the video detection initial model and the video segment annotations of the corresponding video samples in the training samples, the video detection initial model is adjusted to obtain a preset video detection model.
6. The method according to claim 5, characterized in that The training samples further include: highlight saliency annotations of each sub-segment in each video sample, and the video detection initial model further includes a highlight information output module; The method further comprises: determining, by the highlight information output module, a highlight significance score of each sub-segment included in each video sample; The video detection initial model is adjusted according to the video segments of each video sample obtained according to the video detection initial model and the video segment annotations of the corresponding video samples in the training samples, specifically including: adjusting the video detection initial model according to the video segments of each video sample determined by the segment output model in the video detection initial model and the video segment annotations of the corresponding video samples in the training samples, and the highlight significance scores of each video sample determined by the highlight information output module in the video detection initial model and the video segment annotations of the corresponding video samples in the training samples.
7. The method according to claim 5, characterized in that The cross-modal feature module includes multiple layers of cross-modal encoding and fusion sub-modules, wherein: The multi-layer cross-modal encoding is connected in series, the first single-layer cross-modal encoding in the multi-layer cross-modal encoding is used to perform feature attention processing on the single-modal attention features of each modality respectively to obtain a cross-modal attention feature, and the second single-layer cross-modal encoding in the multi-layer cross-modal encoding is used to perform feature attention processing on the cross-modal attention features of each modality obtained by the previous single-layer cross-modal encoding respectively to obtain another cross-modal attention feature; The fusion submodule is used to fuse the cross-modal attention features of each modality obtained by the last single-layer cross-modal encoding in the multi-layer cross-modal encoding to obtain the global features of the target video.
8. The method according to claim 5, characterized in that When the number of times the parameter value is adjusted is equal to the preset number, or if the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold, the adjustment of the parameter value is stopped.
9. A video detection system, characterized in that: An information acquisition unit, used to acquire target detection data of multiple modes contained in a target video, and to acquire specific detection information; the specific detection information is a certain condition satisfied by the detected video segment; A unimodal feature unit, used to obtain unimodal attention features corresponding to the target detection data of the multiple modalities respectively; A fusion unit, used for fusing the single-modal attention features corresponding to the target detection data of the multiple modalities to obtain the global features of the target video; A similarity unit, used to obtain similarity feature information between the global feature and the specific detection information; the similarity feature information is used to identify the similarity between each sub-feature in the global feature and the feature identified by the specific detection information; A detection unit, configured to determine a video segment in the target video that is related to the specific detection information according to the similarity feature information; The fusion unit is specifically used for processing the single-modal attention features of each modality through multi-layer cross-modal encoding to obtain cross-modal attention features, and the fusion submodule fuses the cross-modal attention features; Among them, the cross-modal coding of any single layer in the multi-layer cross-modal coding includes: multiple multi-head attention sub-modules, multiple layer standardization, superposition sub-modules and feedforward neural network; The first multi-head attention submodule among the multiple multi-head attention submodules is used to perform attention processing on the single-modal attention features of the target detection data of each modality in a low-dimensional feature space through a media feature sequence, the first layer of standardization among the multiple layers of standardization is used to standardize the features after attention processing to obtain first standardized features, the superposition submodule is used to superimpose the first standardized features obtained by each first layer of standardization and the media feature sequence to obtain a compressed cross-modal feature sequence, and the compressed cross-modal feature sequence is input into the second multi-head attention submodule among the multiple multi-attention submodules; The second multi-head attention submodule expands the compressed cross-modal feature sequence to a high-dimensional feature space, and the second layer of normalization in the multiple layers of normalization is used to sum and normalize the single-modal attention features and the attention-processed features obtained by the second multi-head attention submodule to obtain a second normalized feature; The feedforward neural network is used to filter the second standardized features, and the third layer of standardization in the multiple layers of standardization is used to output cross-modal attention features of the corresponding modality based on the filtered features.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor and executing the video detection method according to any one of claims 1 to 8.
11. A server, characterized in that: including a processor and a memory; The memory is used to store a plurality of computer programs, and the computer programs are used to be loaded by a processor and execute the video detection method according to any one of claims 1 to 8; the processor is used to implement each computer program in the plurality of computer programs.
12. A computer program product, characterized in that The computer program product comprises computer instructions, which are stored in a computer-readable storage medium, and are suitable for being loaded by a processor to execute the video detection method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Target retrieval method and device, computer readable storage medium and computer device
CN110866491A
Video processing method and device
CN112738557A
Video editing method, device and equipment and storage medium
CN113709561A
Video classification method, device and equipment based on multi-modal representation, and storage medium
CN113762322A
Video object positioning method and system based on mixed attention mechanism
CN113971208A
