Video content analysis method, equipment, medium and product based on deep learning

Abstract

The present application relates to the field of video processing, and discloses a video content analysis method based on deep learning. The original video file is obtained, and the original video file is decoded to obtain the target video file; the target video file is preliminarily detected to obtain the basic information of the target video file; adaptive sampling is performed based on the basic information of the target video file, and the target video file is converted into a frame image array; the spatiotemporal pyramid attention network is used to detect the shot connection points, and the frame image array is divided into at least two segments based on the detected shot connection points; a dynamic contrast learning framework is used to extract key frame images from each segment; a context-aware multi-task learning network is used to remove subtitles from the key frame images, and a cyclic progressive multimodal alignment network is used to generate image descriptions for the key frame images. It can at least be used to solve the limitations of the existing technology and provide a more efficient and accurate video content analysis solution.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video content analysis method, device, medium and product based on deep learning. Background Art

[0002] With the rapid development of multimedia technology, the demand for automated intelligent analysis of video content is growing. However, existing technologies still face many challenges in the field of video content analysis. In terms of shot detection, traditional methods based on visual features have poor adaptability to complex scenes, while existing deep learning methods mainly rely on single-scale features and are difficult to effectively handle different types of shot transitions. At the same time, they lack effective use of temporal information, resulting in unstable detection accuracy.

[0003] In terms of key frame extraction, existing technologies are mostly based on low-level visual features or predefined rules for selection, which makes it difficult to capture important information at the semantic level. They also lack modeling of inter-frame relationships and cannot guarantee the representativeness and diversity of the selected key frames. They have limited adaptability to changes in video content and are prone to redundancy or omission of key information.

[0004] There are also obvious deficiencies in subtitle processing technology. Traditional subtitle removal methods often destroy the original image content, and existing methods generally treat subtitle detection and removal as independent tasks, which is inefficient. The lack of full use of contextual information around the subtitle area seriously affects the processing effect.

[0005] In the field of video content description, existing methods mostly adopt a single generation strategy, which makes it difficult to ensure the accuracy and completeness of the description. The alignment of image and text features is not precise enough, and the video temporal context is not considered, resulting in the quality and coherence of the generated description being difficult to meet actual needs.

[0006] From the perspective of system performance and resource utilization, existing systems generally use fixed processing parameters, which cannot be optimized according to video characteristics. There is a lack of effective coordination mechanism between processing modules, resulting in overall low efficiency. At the same time, the use of computing resources is not flexible enough, which is prone to resource waste or processing bottlenecks.

[0007] These technical defects seriously restrict the application and development of video content intelligent analysis technology. In order to solve the above problems, it is urgent to develop a new video content intelligent analysis method. Summary of the invention

[0008] One purpose of the present application is to provide a video content analysis method, device, medium and product based on deep learning, at least to address the limitations of the existing technology and provide a more efficient and accurate video content analysis solution.

[0009] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0010] In a first aspect, some embodiments of the present application provide a video content analysis method based on deep learning, characterized in that the method includes:

[0011] Obtaining an original video file, and decoding the original video file to obtain a target video file;

[0012] Performing a preliminary detection on the target video file to obtain basic information of the target video file; the basic information includes basic attribute information and quality feature information;

[0013] Adaptively sampling the target video file according to the basic information of the target video file, and converting the target video file into a frame image array;

[0014] Using a spatiotemporal pyramid attention network to detect shot connection points, and dividing the frame image array into at least two segments according to the detected shot connection points;

[0015] A dynamic contrastive learning framework is used to extract key frame images from each segment;

[0016] A context-aware multi-task learning network is used to remove captions from key-frame images, and a cyclic progressive multimodal alignment network is used to generate image descriptions for key-frame images.

[0017] In a second aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0018] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0019] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the method described above when executed by a processor.

[0020] Compared with the related art, in the solution provided by the embodiment of the present application, a spatiotemporal pyramid attention network is used to accurately model shot switching, which greatly improves the robustness and accuracy of shot connection point detection; a dynamic contrast learning framework is used to innovatively construct an inter-frame dynamic contrast mechanism, achieving more representative and informative key frame selection; a context-aware multi-task learning network is used to introduce a multi-task learning paradigm in subtitle processing for the first time, and context-aware capabilities are integrated, significantly improving the processing efficiency and accuracy of subtitle removal; a cyclic progressive multimodal alignment network is used to break through the limitations of traditional single alignment, achieve fine alignment of image-text features, and generate more accurate, detailed and coherent image descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] One or more embodiments are exemplarily described by the images in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the images in the drawings do not constitute proportional limitations.

[0022] Figure 1 An exemplary flow chart of a video content analysis method based on deep learning according to some embodiments of the present application;

[0023] Figure 2 A schematic diagram of the structure of another deep learning-based video content analysis method provided according to some embodiments of the present application;

[0024] Figure 3 An exemplary flow chart of a spatiotemporal pyramid attention network algorithm provided according to some embodiments of the present application;

[0025] Figure 4 An exemplary flow chart of a dynamic contrastive learning framework provided according to some embodiments of the present application;

[0026] Figure 5 An exemplary flow chart of a context-aware multi-task learning network provided according to some embodiments of the present application;

[0027] Figure 6 This is a schematic diagram of an exemplary structure of a cyclic progressive multimodal alignment network provided according to some embodiments of the present application;

[0028] Figure 7 This is a schematic diagram of an exemplary structure of a video content analysis system based on deep learning according to some embodiments of the present application;

[0029] Figure 8 This is a schematic diagram of an exemplary structure of an electronic device provided according to some embodiments of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The following terms are used in this article:

[0032] The core idea of ​​Spatial Temporal Pyramid Attention Networks (STPAN) is to use the pyramid structure to process spatiotemporal features of different scales and to focus on important spatiotemporal information through the attention mechanism. This network can process any number of input frames and realize the interactive perception of local features of different scales input in each layer of the deep convolutional neural network and the fusion of spatiotemporal features.

[0033] The Dynamic Contrastive Learning Framework (DCLF) is an emerging machine learning paradigm that dynamically adjusts the positive and negative sample selection strategies in contrastive learning to adapt to changing data distributions and learning objectives. This approach aims to improve the generalization ability of the model in complex and dynamic environments, especially in processing time series data, video analysis, online learning and other scenarios.

[0034] Context-Aware Multi-Task Learning Network (CAMLN) is an advanced neural network architecture that combines context-awareness and multi-task learning. This architecture aims to improve the generalization and efficiency of the model by learning shared representations between different tasks. In multi-task learning, different tasks share some parameters, which can capture common domain knowledge, so that shared parameters can be effectively trained on more data and improve the generalization of the model. Context-awareness means that the system can respond adaptively based on the system itself and its environmental state information.

[0035] The Incremental Progressive Multi-modal Alignment Network (IPMAN) is an advanced multimodal learning technology that aims to achieve the ability to process and understand multi-source modal information through machine learning methods. The core idea of ​​IPMAN is to gradually align data of different modalities in a cyclic and progressive manner to achieve more accurate pattern recognition and information fusion. This method is particularly effective when processing multi-modal data such as images, videos, audio, and semantics.

[0036] First embodiment

[0037] The first embodiment of the present application relates to a video content analysis method based on deep learning. Figure 1 As shown, the method may include the following steps:

[0038] S110: Acquire an original video file, and decode the original video file to obtain a target video file.

[0039] The original video file is obtained through compression and transmission, and decoding the original video file refers to converting the compressed original video file into a processable original data stream to provide basic data for subsequent analysis and processing.

[0040] Optionally, the present application uses adaptive buffering technology to decode the original video file. By calculating the appropriate buffer size, the system dynamically adjusts the memory allocation according to the size of the input original video file, while ensuring the minimum processing space and avoiding occupying too many system resources, thereby optimizing the memory usage efficiency of the decoding process. The calculation formula of the buffer size B is:

[0041] B=min(max(N*α,Bmin),Bmax)

[0042] Where B is the buffer size, N is the original video file size, α is the scaling factor, and Bmin and Bmax are the minimum and maximum values ​​of the buffer size, respectively.

[0043] In this application, the buffer size is dynamically adjusted according to the size of the original video file, while ensuring the minimum buffer size, avoiding an excessively large buffer from occupying too much memory, thereby improving decoding efficiency and optimizing system resource usage.

[0044] S120: Perform a preliminary detection on the target video file to obtain basic information of the target video file.

[0045] The preliminary detection may refer to detecting and analyzing the header metadata and pixel data of the target video file to obtain basic information of the target video file. The basic information includes basic attribute information and quality feature information; the basic attribute information is used to characterize the attributes of the target video file, including but not limited to video duration, resolution, frame rate, encoding format, bit rate and file size; the quality feature information includes but not limited to clarity, brightness contrast, picture stability and noise level.

[0046] S130 , performing adaptive sampling according to basic information of the target video file, and converting the target video file into a frame image array.

[0047] Among them, adaptive sampling can automatically adjust the mathematical model of its behavior and structure according to input data or environmental conditions. In this application, the frame sampling interval is dynamically adjusted according to the basic information of the target video file; the target video file is sampled according to the dynamically adjusted sampling interval and saved as a frame image array.

[0048] S140, using a spatiotemporal pyramid attention network to detect shot connection points, and dividing the frame image array into at least two segments according to the detected shot connection points.

[0049] The detecting of the shot connection point may refer to detecting the shots in the frame image array, determining whether there is a shot switch and the probability of the shot switch, and taking the point when the shot switches as the shot connection point. After determining the shot connection point, the frame image array is segmented to divide the frame image array into at least two segments. One segment represents one shot.

[0050] S150, using a dynamic contrast learning framework to extract key frame images from each segment.

[0051] The key frame image may refer to a frame image that can represent a shot. The key frame image that can represent a segment is determined from a segment through a dynamic contrast learning framework, and the corresponding key frame image is extracted from each segment to construct a key frame image set.

[0052] S160, using a context-aware multi-task learning network to remove subtitles from the key frame image, and using a cyclic progressive multimodal alignment network to generate image descriptions for the key frame image.

[0053] The key frame images of each segment are subjected to subtitle removal and image description generation. A context-aware multi-task learning network is used to remove subtitles from the key frame images and identify corresponding text information; a cyclic progressive multimodal alignment network is used to generate image descriptions for the key frame images after subtitle removal. The generation of the image description may refer to generating an image description of the key frame image based on the key frame image information, other frame image information in the same shot, frame image information in associated shots, and identified text information.

[0054] It is not difficult to find that compared with the related art, in the solution provided by the embodiment of the present application, a spatiotemporal pyramid attention network is used to accurately model shot switching, which greatly improves the robustness and accuracy of shot connection point detection; a dynamic contrast learning framework is used to innovatively construct an inter-frame dynamic contrast mechanism, achieving more representative and informative key frame selection; a context-aware multi-task learning network is used to introduce a multi-task learning paradigm in subtitle processing for the first time, and context-aware capabilities are integrated, significantly improving the processing efficiency and accuracy of subtitle removal; a cyclic progressive multimodal alignment network is used to break through the limitations of traditional single alignment, achieve fine alignment of image-text features, and generate more accurate, detailed and coherent image descriptions.

[0055] Second embodiment

[0056] The second embodiment of the present application relates to a video content analysis method based on deep learning. The second implementation is a further optimization based on the first embodiment. The embodiment of the present invention can be combined with each optional solution in one or more of the above embodiments. Figure 2 As shown, the video content analysis method based on deep learning provided in the embodiment of the present invention may include the following steps:

[0057] S210: Acquire an original video file, and decode the original video file to obtain a target video file.

[0058] S220: Perform a preliminary detection on the target video file to obtain basic information of the target video file.

[0059] After decoding the original video file, a preliminary detection is performed on the obtained target video file to obtain basic information of the target video file.

[0060] As an optional but non-limiting implementation, the preliminary detection of the target video file to obtain basic information of the target video file includes:

[0061] Reading the target video file header metadata and analyzing the target video file pixel data to obtain basic attribute information and quality feature information of the target video file;

[0062] Among them, by reading the target video file header metadata and analyzing the target video file pixel data, the basic attribute information and quality feature information of the target video file are obtained, the basic attribute information includes video duration, resolution, frame rate, encoding format, bit rate and file size, and the quality feature information includes clarity, brightness contrast, picture stability and noise level. The basic attribute information and quality feature information of the target video file provide important reference for subsequent sampling calculation, buffer adjustment and various analysis tasks.

[0063] S230 , performing adaptive sampling according to basic information of the target video file, and converting the target video file into a frame image array.

[0064] Among them, when converting the target video file into a frame image array, the present application adopts an intelligent adaptive sampling algorithm to dynamically calculate the sampling interval for sampling and save it as an image array.

[0065] As an optional but non-limiting implementation, the adaptive sampling is performed according to the basic information of the target video file to convert the target video file into a frame image array, including but not limited to steps A1-A2:

[0066] Step A1: Adopt an adaptive sampling algorithm to dynamically calculate the sampling interval of the target video file.

[0067] Step A2: adaptively sampling the target video file according to the sampling interval, and saving the sampled image as a frame image array.

[0068] The sampling interval of the target video file is dynamically calculated by using an adaptive sampling algorithm, which is expressed as:

[0069] S=max(1,round(F*H / (T*K)))

[0070] Among them, S is the sampling interval, F is the total number of video frames, H is the content complexity factor, T is the processing time of the target video file, and K is the benchmark value of the number of frames processed per second.

[0071] The sampling interval is dynamically adjusted according to the basic information of the target video file, taking into account factors such as the total number of video frames, content complexity, and target processing time. The present application samples the target video file according to the calculated sampling interval and saves it as an image array. This method can automatically adjust the sampling density according to the characteristics of the target video file, balancing processing efficiency and information retention.

[0072] S240, using a spatiotemporal pyramid attention network to detect shot connection points, and dividing the frame image array into at least two segments according to the detected shot connection points.

[0073] Among them, a spatiotemporal pyramid attention network is used to detect the shot switching probability value at each moment to detect the shot connection point; based on the detected shot connection point, the frame image array is divided into at least two segments, each segment corresponding to a shot.

[0074] As an optional but non-limiting implementation, the spatiotemporal pyramid attention network is used to detect the shot connection points, and the frame image array is divided into at least two segments according to the detected shot connection points, including but not limited to steps B1-B2:

[0075] Step B1: The preset frame sequence at each moment is analyzed using a spatiotemporal pyramid attention network to determine the shot switching probability value at each moment so as to detect the shot connection point; the preset frame sequence includes a sequence of preset frames before and after each moment.

[0076] Step B2: using a multi-modal segmentation calibration technique on the detected shot junctions to divide the frame image array into at least two segments.

[0077] Among them, see Figure 3 , the frame image array is input into the spatiotemporal pyramid attention network. For each moment t to be analyzed, a preset frame sequence Ft-k:t+k consisting of k frames before and after moment t is intercepted from the frame image array. This preset frame sequence is input into the network, and the probability of shot switching at that moment is calculated by analyzing its spatiotemporal characteristics. This method enables detection to achieve a high accuracy in complex situations such as fast scene switching and gradual lens changes.

[0078] Optionally, the shot switching probability value at each moment is determined, which can be expressed as:

[0079] P(S t )=Softmax(STPAN(Ft-k:t+k))

[0080] Among them, P(S t ) is the probability of a shot switch at time t, Ft-k:t+k is the frame sequence of k frames before and after time t, and STPAN is the spatiotemporal pyramid attention network function. The spatiotemporal characteristics of the preset frame sequence at each moment are analyzed by the formula, and the probability of a shot switch at the current moment is calculated to determine the shot connection point.

[0081] According to the detected shot connection points, the video is divided into multiple segments, and the video segmentation adopts a multimodal segmentation calibration technology to segment the frame image array into at least two segments. Optionally, the multimodal segmentation calibration technology is adopted to segment the frame image array into at least two segments, which can be expressed as:

[0082] CP=argmax(λ1*V(t)+λ2*A(t)+λ3*M(t))

[0083] Among them, CP is the final segmentation point, V(t) is the visual change score at time t, A(t) is the audio change score at time t, M(t) is the motion change score at time t, and λ1, λ2, and λ3 are the corresponding weight coefficients. The function of this formula is to comprehensively consider visual, audio, and motion information to determine the best shot segmentation point; the visual change, audio change, and motion change scores in the formula are calculated by analyzing the differences in visual features between frames (such as color, texture, edges, etc.), audio feature changes (such as waveform, spectrum, energy, etc.), and scene motion features (such as optical flow, object motion, camera motion, etc.). These scores comprehensively reflect the changing characteristics of the video in different modes and help determine the best segmentation position.

[0084] This application uses a spatiotemporal pyramid attention network to detect shot connection points. The feature pyramid captures spatiotemporal information of different scales. Residual connections and multi-scale fusion improve feature representation capabilities. The self-attention mechanism effectively captures the relationship between long-distance frame images. The entire spatiotemporal pyramid attention network can be optimized end-to-end through back propagation, which greatly improves the robustness and accuracy of shot boundary detection. Finally, the best shot segmentation position is determined by integrating multimodal information, thereby improving the accuracy of video segmentation.

[0085] S250, adopts a dynamic contrast learning framework to extract key frame images from each segment.

[0086] Among them, a dynamic contrast learning framework is used to extract key frame images for each shot.

[0087] As an optional but non-limiting implementation, the dynamic contrast learning framework is used to extract key frame images from each segment, including but not limited to steps C1-C3:

[0088] Step C1: Select candidate frame images from the first segment, and select representative frame images from the second segment and construct a representative frame image set; wherein the second segment refers to other segments of at least two segments except the first segment.

[0089] Step C2: using a dynamic contrast learning framework to determine the importance score of the first segment candidate frame image, and determine the differences between the first segment candidate frame image and other frame images in the first segment and the representative frame image set.

[0090] Step C3: Determine the key frame image in the first segment according to the difference.

[0091] Taking the extraction of key frame images from the first segment as an example, a candidate frame image is selected from the first segment; a representative frame image of the second segment other than the first segment in at least two segments is obtained, and a representative frame image set is constructed. The candidate frame image in the first segment is compared with other frame images in the first segment and frame images in the representative frame image set to determine the difference; thereby determining the key frame image in the first segment.

[0092] Optionally, if the difference between the first candidate frame image and other frame images in the first segment or frame images in a representative frame image set is small, the next frame image is selected from the first segment as the second candidate frame image, and the difference between the second candidate frame image and other frame images in the first segment or frame images in a representative frame image set is determined again to determine the key frame image in the first segment.

[0093] As an optional but non-limiting implementation, the method of extracting key frame images from each segment using a dynamic contrast learning framework also includes but is not limited to steps D1-D2:

[0094] Step D1: According to the extraction method of the key frame images in the first segment, the importance score of the candidate frame images of each segment is determined using a dynamic contrast learning framework.

[0095] Step D2: Determine the key frame image in each segment according to the importance score of the candidate frame image of each segment.

[0096] The key frame images of other segments are determined by analogy based on the extraction method of the key frame images of the first segment.

[0097] Optionally, a dynamic contrast learning framework is used to determine the importance score of each candidate frame image in each segment, so as to determine the difference between the candidate frame image and other frame images in the current segment or frame images in the representative frame image set, so as to determine the key frame image. The dynamic contrast learning framework is used to determine the importance score of each candidate frame image in each segment, which is expressed as:

[0098] Score(f)=DCLF(f,{f i},{f j})

[0099] Among them, Score(f) represents the importance score of the candidate frame image f, f represents the candidate frame image, {f i} represents the set of other frame images in the same segment, {f j} represents a set of representative frame images of other fragments.

[0100] Optional, see Figure 4 , the learning rate can be dynamically adjusted according to the importance of the current frame image to update the model parameters to adapt to changes in video content.

[0101] The dynamic contrastive learning framework adopted in this application can dynamically adjust the learning rate to adapt to changes in video content; through contrastive learning, it provides the discriminative ability of feature representation; through context perception, the selection of key frame images takes into account the relationship between images between frames; by determining the importance scores of candidate frame images, efficient and accurate positioning of more representative and informative key frame images is achieved.

[0102] S260, using a context-aware multi-task learning network to remove subtitles from the key frame image.

[0103] Among them, the present application adopts a context-aware multi-task learning network to perform subtitle removal processing on key frame images, and outputs the key frame images after subtitle removal and the recognized text content; the subtitle removal processing includes subtitle detection, subtitle recognition and subtitle removal processing.

[0104] Optionally, the context-aware multi-task learning network is used to perform subtitle removal processing on the key frame image, which can be expressed as:

[0105] I restored ,T=CAMLN(I)

[0106] Among them, I restored is the key frame image after removing subtitles, T is the recognized text content, I is the input key frame image, and CAMLN ​​is the context-aware multi-task learning network function.

[0107] See also Figure 5 This application adopts the context-aware multi-task learning network to simultaneously perform subtitle detection, subtitle recognition, and subtitle removal tasks to improve the speed, processing efficiency, and accuracy of subtitle removal processing while retaining key semantic information; utilizes global and local context to improve subtitle removal processing performance; and generates an adversarial network during subtitle removal to determine natural image restoration.

[0108] S270, using a cyclic progressive multimodal alignment network to generate an image description for the key frame image.

[0109] Among them, the present application adopts a cyclic progressive multimodal alignment network to align the context information of the key frame image after the subtitles are removed with the image features, and generates an image description of the key frame image after the subtitles are removed.

[0110] Optionally, the method of using a cyclic progressive multimodal alignment network to generate an image description for a key frame image can be expressed as:

[0111] D=IPMAN(I restored ,{C i},n)

[0112] Among them, D is the final image description generated, I restored is the key frame image after removing the subtitles from the input, {C i} is the context information set, n is the number of iterations, and IPMAN is the cyclic progressive multimodal alignment network function. i It contains other key frame information of the same shot, scene information of adjacent shots, text content obtained by subtitle recognition, and temporal context of the video. Through multiple rounds of iterations, the alignment between image and text is gradually refined, resulting in accurate and detailed image description.

[0113] See also Figure 6 This application uses a cyclic progressive multimodal alignment network to generate image descriptions for keyframe images, and can adjust the number of iterations and description details as needed; effectively fuse visual and contextual information through multimodal alignment; enhance image descriptions using contextual information of scenes and objects; improve image description quality through multiple iterations to generate more accurate, detailed and coherent image descriptions.

[0114] It is not difficult to find that in the embodiment of the present application, the spatiotemporal pyramid attention network is used to detect the shot connection points, which greatly improves the robustness and accuracy of the shot boundary detection; the optimal shot segmentation position is determined by integrating multimodal information, thereby improving the accuracy of video segmentation. A dynamic contrast learning framework is adopted to evaluate the differences between candidate frames and frames within the same shot and representative frames across shots, thereby achieving the selection of more representative and informative key frame images. The context-aware multi-task learning network is used to simultaneously perform subtitle detection, subtitle recognition, and subtitle removal tasks, thereby improving the speed, processing efficiency, and accuracy of the subtitle removal process, while retaining key semantic information. A cyclic progressive multimodal alignment network is used to improve the quality of image description through multiple iterations to generate more accurate, detailed, and coherent image descriptions.

[0115] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0116] Third embodiment

[0117] This application also provides a video content analysis system based on deep learning. Figure 7 The system includes a video input module, a preliminary detection module, a frame extraction module, a shot detection module, a video segmentation module, a key frame extraction module, a subtitle removal module and a text-image pairing module. The modules exchange data through a high-speed cache mechanism to improve processing efficiency.

[0118] In the shot detection module, the problem of poor adaptability of traditional methods to complex scenes is effectively solved through the spatiotemporal pyramid attention network. The network can adaptively capture multi-scale spatiotemporal features and model the temporal information in combination with the attention mechanism, which effectively improves the accuracy of shot boundary detection. The detection accuracy is significantly improved in complex situations such as fast scene switching and gradual lens changes, and can achieve real-time processing performance.

[0119] In terms of key frame extraction, the frame extraction module innovatively adopts a dynamic contrast learning framework, breaking through the limitations of traditional low-level feature selection. This framework reduces the redundancy of key frames and improves information coverage by establishing a dynamic contrast mechanism between frames. Combined with adaptive sampling technology, the processing speed is improved while maintaining the accuracy of key frame extraction, which is significantly better than existing technologies.

[0120] In terms of subtitle processing, the subtitle removal module uses a context-aware multi-task learning network, innovatively integrating subtitle detection, recognition and removal into a unified framework. By making full use of the contextual information around the subtitle area, the subtitle recognition accuracy is significantly improved, the regional restoration quality is improved compared to traditional methods, and the image structure integrity is improved. The multi-task parallel processing design improves overall efficiency and significantly reduces processing latency.

[0121] In terms of content description generation, the text-image pairing module uses a cyclic progressive multimodal alignment network to overcome the limitations of traditional single-shot generation methods. Through multiple rounds of iterative optimization, the description accuracy is improved, and the BLEU-4 score is significantly higher than the level of existing technologies. The description completeness is improved, and an average of 3-4 accurate description sentences can be generated for each scene, and the generation speed is significantly faster than traditional methods.

[0122] In terms of system performance, the adaptive resource management strategy proposed in this application realizes the dynamic optimization of processing parameters. Through adaptive buffering and sampling technology, the system memory usage efficiency is improved, the processing capacity is improved, and the operation failure rate is reduced. The application of multi-modal segmentation calibration technology has significantly improved the system's processing accuracy on different types of videos and reduced processing latency.

[0123] From the application effect point of view, this application significantly improves the automation level and processing efficiency of video content analysis, reduces manual participation, and reduces processing costs. In large-scale video tests, the system performs stably, and the fluctuation range of various indicators is controlled within the preset range. The system can be applied to multiple industry scenarios, including video surveillance, content review, and media resource management. The deployment cost is lower than that of traditional solutions, and the maintenance cost is lower, which has significant economic benefits and social value.

[0124] Through deployment and testing in actual application environments, the various technical solutions of this application have demonstrated excellent performance indicators and stability, which has effectively verified its innovation and practical value in the field of intelligent video content analysis. The promotion and application of the system will bring important technological breakthroughs and practical value to the field of video content analysis.

[0125] Fourth embodiment

[0126] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0127] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 8 An exemplary structural diagram of the electronic device is disclosed. Figure 8As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0128] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0129] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0130] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0131] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0132] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0133] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0136] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0138] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0139] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A video content analysis method based on deep learning, characterized in that: The method comprises: Obtaining an original video file, and decoding the original video file to obtain a target video file; Performing a preliminary detection on the target video file to obtain basic information of the target video file; the basic information includes basic attribute information and quality feature information; Adaptively sampling the target video file according to the basic information of the target video file, and converting the target video file into a frame image array; Using a spatiotemporal pyramid attention network to detect shot connection points, and dividing the frame image array into at least two segments according to the detected shot connection points; A dynamic contrastive learning framework is used to extract key frame images from each segment; A context-aware multi-task learning network is used to remove captions from key-frame images, and a cyclic progressive multimodal alignment network is used to generate image descriptions for key-frame images.

2. The method according to claim 1, characterized in that The preliminary detection of the target video file to obtain basic information of the target video file includes: Reading the target video file header metadata and analyzing the target video file pixel data to obtain basic attribute information and quality feature information of the target video file; Among them, the basic attribute information includes video length, resolution, frame rate, encoding format, bit rate and file size, and the quality feature information includes clarity, brightness contrast, picture stability and noise level.

3. The method according to claim 1, characterized in that The step of performing adaptive sampling according to basic information of the target video file to convert the target video file into a frame image array includes: Adopt adaptive sampling algorithm to dynamically calculate the sampling interval of target video file; Adaptively sampling the target video file according to the sampling interval, and saving the sampled image as a frame image array; The sampling interval of the target video file is dynamically calculated by using an adaptive sampling algorithm, which is expressed as: S=max(1,round(F*H / (T*K))) Among them, S is the sampling interval, F is the total number of video frames, H is the content complexity factor, T is the processing time of the target video file, and K is the benchmark value of the number of frames processed per second.

4. The method according to claim 1, characterized in that The method of using a spatiotemporal pyramid attention network to detect shot connection points and dividing the frame image array into at least two segments according to the detected shot connection points includes: The preset frame sequence at each moment is analyzed using a spatiotemporal pyramid attention network to determine the shot switching probability value at each moment so as to detect the shot connection point; the preset frame sequence includes a sequence of preset frames before and after each moment; The detected shot junction points are subjected to a multi-modal segmentation calibration technique to divide the frame image array into at least two segments.

5. The method according to claim 1, characterized in that: The dynamic contrast learning framework is used to extract key frame images from each segment, including: Selecting candidate frame images from the first segment, and selecting representative frame images from the second segment and constructing a representative frame image set; wherein the second segment refers to other segments of the at least two segments except the first segment; A dynamic contrastive learning framework is used to determine the importance score of the candidate frame image of the first segment, and to determine the differences between the candidate frame image of the first segment and other frame images in the first segment and a representative frame image set; According to the difference, a key frame image in the first segment is determined.

6. The method according to claim 5, characterized in that The method of extracting key frame images from each segment using a dynamic contrast learning framework also includes: According to the extraction method of the key frame images in the first segment, a dynamic contrastive learning framework is used to determine the importance score of the candidate frame images of each segment; Determine the key frame image in each segment according to the importance score of the candidate frame image of each segment; The dynamic contrast learning framework is used to determine the importance score of each candidate frame image, which is expressed as: Score(f)=DCLF(f,{f i },{f j }) Among them, Score(f) represents the importance score of the candidate frame image f, f represents the candidate frame image, {f i } represents the set of other frame images in the same segment, {f j } represents a set of representative frame images of other fragments.

7. The method according to claim 1, characterized in that The method of using a context-aware multi-task learning network to remove captions from key frame images and using a cyclic progressive multimodal alignment network to generate image descriptions for key frame images includes: Using a context-aware multi-task learning network to perform subtitle removal processing on the key frame image, and outputting the key frame image after subtitle removal and the recognized text content; the subtitle removal processing includes subtitle detection, subtitle recognition and subtitle removal processing; A cyclic progressive multimodal alignment network is used to align context information and image features of the key frame image after subtitles are removed, so as to generate an image description of the key frame image after subtitles are removed.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the deep learning-based video content analysis method as described in any one of claims 1 to 7.

9. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the deep learning-based video content analysis method described in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the deep learning-based video content analysis method described in any one of claims 1 to 7 are implemented.

Images