Video processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a video processing method and device, computer equipment and a computer storage medium, and relates to the technical field of videos. Comprising the following steps: inputting a to-be-processed video into a hybrid deep neural network, and extracting image semantic features of each image sequence frame in the to-be-processed video based on the hybrid deep neural network; the image semantic feature extraction module comprises a feature map extraction module and a target adaptive attention module, the feature map extraction module is used for extracting a corresponding multi-modal feature map, and the target adaptive attention module is used for determining image semantic features of each image sequence frame according to a weight value and an association degree value of each modal feature map in the multi-modal feature map; inputting the image semantic features into a region proposal network to determine a target region of each image sequence frame based on the region proposal network; and cutting the target area of each image sequence frame so as to convert the target video between a horizontal screen frame and a vertical screen frame. The accuracy of target recognition can be improved, and the quality and integrity of the vertical screen video are ensured.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, a video processing apparatus, a computer device, and a computer storage medium. Background Art

[0002] With the rapid development of electronic and internet technologies, a wide variety of terminals have been widely used, and the applications running on these terminals are also becoming increasingly diverse. In some video scenarios, such as live streaming, remote video surveillance and analysis, and intelligent video content creation and distribution, there is a need to convert video resources displayed horizontally (i.e., landscape videos) to vertically (i.e., portrait videos). When converting horizontal videos to vertical, preserving the target object or area in the video image is a key factor influencing the video conversion effect.

[0003] Related technical solutions typically use static rules or shallow feature extraction models to locate target areas within a video to convert horizontal to vertical video. For example, horizontal videos are cropped to a fixed aspect ratio, or target areas are extracted using threshold segmentation combined with edge detection.

[0004] However, the above method is difficult to adapt to complex scene images, and it is very easy to lead to high target recognition error rate, frequent misjudgment and missed judgment, or loss of key details, affecting the quality of the vertical screen image and the integrity of the content. Summary of the Invention

[0005] In the embodiments of the present application, a video processing method, a video processing method apparatus, a computer device, and a computer storage medium are provided, thereby overcoming, at least to a certain extent, the technical problems that are difficult to adapt to complex scene images due to the limitations and defects of related technologies, which easily lead to high target recognition error rates, frequent misjudgments and missed judgments, or loss of key details, affecting the quality of vertical screen images and the integrity of content.

[0006] A first aspect of an embodiment of the present application provides a video processing method, which includes: obtaining a video to be processed; inputting the video to be processed into a pre-trained hybrid deep neural network, and extracting image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model includes at least a feature map extraction module and a target adaptive attention module, the feature map extraction module is used to extract corresponding multimodal feature maps from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame based on the weight value and correlation value of each modal feature map in the multimodal feature map; inputting the image semantic features of each image sequence frame into a region proposal network to determine the target region of each image sequence frame based on the region proposal network; cropping the target region of each image sequence frame to obtain a target video, so as to convert the target video into a horizontal screen frame and a vertical screen frame.

[0007] According to a second aspect of an embodiment of the present application, a video processing device is provided, which includes: a video acquisition module for acquiring a video to be processed; a semantic feature extraction module for inputting the video to be processed into a pre-trained hybrid deep neural network, and extracting image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model includes at least a feature map extraction module and a target adaptive attention module, the feature map extraction module is used to extract a corresponding multimodal feature map from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame based on the weight value and correlation value of each modal feature map in the multimodal feature map; a target area determination module is used to input the image semantic features of each image sequence frame into a region proposal network, so as to determine the target area of each image sequence frame based on the region proposal network; and a video conversion module is used to crop the target area of each image sequence frame to obtain a target video, so as to convert the target video into a horizontal frame and a vertical frame.

[0008] According to a third aspect of an embodiment of the present application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above video processing methods are implemented.

[0009] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above video processing methods are implemented.

[0010] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps of any one of the above video processing methods.

[0011] The technical solution of this application has the following beneficial effects:

[0012] The video processing method described above feeds the video to be processed into a hybrid deep neural network, which extracts image semantic features from each image sequence frame in the video. The method comprises a feature map extraction module for extracting the corresponding multimodal feature map and a target adaptive attention module for determining the image semantic features of each image sequence frame based on the weights and correlation values of each modal feature map in the multimodal feature map. The image semantic features are then fed into a region proposal network to determine the target region of each image sequence frame based on the region proposal network. The target region of each image sequence frame is then cropped to convert the target video between horizontal and vertical frames. This method, through a deep neural network architecture, deeply mines image semantic features, analyzes the inherent logic of scenes and targets, and accurately identifies and distinguishes primary and secondary targets in complex video scenes. Furthermore, it effectively improves target recognition rate and positioning accuracy in video processing in low-light, occluded, or noisy environments, thereby improving target recognition accuracy and ensuring the quality and integrity of vertical video. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 An architectural diagram of a video processing system provided in one embodiment of the present application;

[0015] Figure 2 A flowchart of a video processing method provided in one embodiment of the present application;

[0016] Figure 3 A flowchart of a method for extracting image semantic features provided in one embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of a video processing device provided in one embodiment of the present application;

[0018] Figure 5 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0020] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0021] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0022] With the rapid development of electronic and internet technologies, a wide variety of terminals have been widely used, and correspondingly, the types and functionality of applications on these terminals have increased. In some video scenarios, such as live streaming, remote video surveillance and analysis, and intelligent video content creation and distribution, there is a need to convert video resources displayed in landscape mode (i.e., landscape videos / horizontal videos) into video resources displayed in portrait mode (i.e., portrait videos / vertical videos). This provides users with a superior visual experience in portrait mode, meeting the diverse demands for vertical video quality, content integrity, and viewing comfort in various scenarios in the mobile internet era. For example, live quiz applications are a very common type of video application. To enhance the interactive experience, an encoding server can convert video frames captured by a camera that are suitable for landscape display (referred to as landscape video frames) into live video streams suitable for portrait display (referred to as portrait video streams). For example, video frames captured by a camera with a 16:9 aspect ratio can be converted into video frames with a 9:16 aspect ratio. Another example: short videos are usually vertical videos. For the massive amount of horizontal video resources, there is a need to convert horizontal video resources into vertical short videos for traffic diversion and promotion.

[0023] When converting a horizontal video to a vertical video, there is usually a loss of image information. Whether the key information in the image (i.e., the target object / target area) can be retained is an important factor affecting the video conversion effect. In related technical solutions, static rule-based or shallow feature extraction models are often used to handle video conversion tasks. For example, horizontal videos are cut according to a fixed screen ratio, or the target area in the video screen is extracted using threshold segmentation combined with edge detection to achieve vertical conversion.

[0024] However, the above-mentioned related technical solutions have at least the following problems:

[0025] 1) Regarding the static rule-based approaches in the aforementioned related technical solutions: These techniques rely on preset parameters and fixed rules, such as universal cropping ratios or fixed template matching, making them difficult to adapt to diverse scenarios and content changes. This is particularly true in complex dynamic scenes (such as sports events and action movies), where they can easily lead to object clipping, image imbalance, or loss of key details. These techniques fail to meet the requirements for content integrity and visual aesthetics required for vertical videos in dynamic scenes.

[0026] 2) Regarding the shallow feature extraction model method in the above-mentioned related technical solutions: it has weak robustness to complex lighting, perspective changes, background interference, etc. When low-light surveillance videos or special effects-rich film and television content are converted to vertical screens, the target recognition error rate is high and misjudgments and missed judgments are frequent, resulting in poor quality of the vertical screen image and serious information loss, which limits the improvement of user experience.

[0027] 3) Both of the aforementioned methods suffer from a lack of semantic understanding. That is, none of the aforementioned technical solutions involve analyzing the relationship between semantic understanding of video scenes and target objects, making it impossible to optimize vertical screen composition layout based on video themes, character relationships, and event logic. Furthermore, when processing videos with multi-subject interactions or strong narratives, vertical screens struggle to effectively convey core information and direct audience attention, hindering the efficient dissemination and value release of video content on vertical screen terminals.

[0028] In order to solve the above technical problems, an embodiment of the present application provides a video processing method. The basic idea of the embodiment of the present disclosure is: to deeply mine the semantic features of the image through a hybrid deep neural network architecture to analyze the internal logic between the video scene and the object to be identified. This is convenient for accurately identifying and distinguishing primary and secondary targets in complex video scenes, avoiding the defects of the existing technology of not distinguishing between primary and secondary and mistaken deletion of information, and giving vertical screen videos a strong narrative and visual focus.

[0029] For a clearer understanding, this application proposes a video processing method and device, which can be applied to Figure 1 In the system architecture of the exemplary application environment shown.

[0030] like Figure 1 As shown, the system architecture 100 may include a terminal device 101 and a server 102 .

[0031] The terminal device 101 may be, for example, a smartphone, a personal digital assistant (PDA), a laptop computer, a server, a desktop computer, or any other computing device with networking capabilities, but is not limited thereto. The terminal device 101 may be the same device that supports both horizontal and vertical video display, or may be different terminal devices, for example, one terminal device supports horizontal video and another terminal device supports vertical video. The embodiments of the present application do not impose any special restrictions on this.

[0032] The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The network 103 is a medium for providing a communication link between the terminal device 101 and the server 102. The network 103 can include various connection types, such as wired, wireless communication links, or fiber optic cables. It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0033] The video processing method provided in the embodiment of the present application can be executed on the server 102, and accordingly, the video processing device is generally provided in the server 102. The video processing method provided in the embodiment of the present application can also be executed in the terminal device 101, and accordingly, the video processing device can also be provided in the terminal device 101. The video processing method provided in the embodiment of the present application can also be partially executed in the server 102 and partially executed in the terminal device 101, and accordingly, some modules of the video processing device can be provided in the server 102 and some modules can be provided in the terminal device 101.

[0034] For example, in an exemplary embodiment, the server 102 obtains a video to be processed; inputs the video to be processed into a pre-trained hybrid deep neural network, and extracts image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model includes at least a feature map extraction module and a target adaptive attention module, the feature map extraction module is used to extract the corresponding multimodal feature map from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame based on the weight value and correlation value of each modal feature map in the multimodal feature map; the image semantic features of each image sequence frame are input into a region proposal network to determine the target region of each image sequence frame based on the region proposal network; the target region of each image sequence frame is cropped to obtain a target video, so as to convert the target video into a horizontal screen frame and a vertical screen frame. Then, the target video is sent to the terminal device 101 for corresponding vertical screen display.

[0035] The following uses the server 102 as an execution subject and illustrates the application of the video processing method to the server 102 as an example. Figure 2 A flowchart schematically illustrates a video processing method in this exemplary embodiment, see Figure 2 The video processing method provided in the embodiment of the present application includes the following steps 201 to 203:

[0036] Step 201: Obtain the video to be processed.

[0037] Step 202: Input the video to be processed into a pre-trained hybrid deep neural network, and extract the image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model comprises at least a feature map extraction module and a target adaptive attention module, the feature map extraction module is used to extract the corresponding multimodal feature map from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame according to the weight value and correlation value of each modal feature map in the multimodal feature map.

[0038] Step 203: Input the image semantic features of each image sequence frame into a region proposal network to determine the target region of each image sequence frame based on the region proposal network.

[0039] Step 204 : cropping the target area of each image sequence frame to obtain a target video, so as to convert the target video into a horizontal frame and a vertical frame.

[0040] In the above Figure 2 In the technical solution provided, the image semantic features of each image sequence frame in the video to be processed are extracted based on the hybrid deep neural network by inputting the video to be processed. The solution includes a feature map extraction module and a target adaptive attention module. The feature map extraction module is used to extract the corresponding multimodal feature map, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame based on the weight value and correlation value of each modal feature map in the multimodal feature map. The image semantic features are input into the region proposal network to determine the target region of each image sequence frame based on the region proposal network. The target region of each image sequence frame is cropped to convert the target video into a horizontal frame and a vertical frame. This method uses a deep neural network architecture to deeply mine image semantic features, analyze the inherent logic of the scene and target, and accurately identify and distinguish between primary and secondary targets in complex video scenes. In video processing in low-light, occluded or noisy environments, it effectively improves the target recognition rate and positioning accuracy, thereby improving the accuracy of target recognition and ensuring the quality and integrity of vertical screen videos.

[0041] The following will be combined with specific embodiments to Figure 2 The specific implementation of each step in the embodiment shown is described in detail:

[0042] In step 201, a video to be processed is obtained.

[0043] Among them, the video to be processed can be the initial source video captured in real time by the video capture device, or the video file pre-captured and stored by the video capture device and decoded according to the video format, or it can be the video after a series of operations such as decoding, frame rate normalization, and resolution adaptation are performed on the above-mentioned original video or video file according to the video format. The embodiments of the present application do not impose any special restrictions on this.

[0044] For example, a high-speed hardware decoder (e.g., a Field Programmable Gate Array (FPGA) accelerated decoder) can be used to decode the raw video captured by the video capture device according to common video coding standards (e.g., MPEG-2, H.264, H.265) to obtain the processed video. It should be noted that these video coding standards are used to achieve video data compression and efficient transmission.

[0045] For example, the original video captured by the video capture device and the collected video files can be connected to the system using protocols such as the Real-Time Streaming Protocol (RTSP) and the Real-Time Messaging Protocol (RTMP), and then decoded in real time based on the high-speed hardware decoder in the system to obtain the video to be processed.

[0046] Before executing step 201 to obtain the video to be processed, in an optional embodiment of the present disclosure, an initial source video is obtained, and a video quality parameter of the initial source video is determined based on a preset quality assessment index; if the video quality parameter of the initial source video is less than a video quality parameter threshold, the initial source video is preprocessed according to a first preprocessing parameter to obtain the video to be processed; or, if the video quality parameter of the initial source video is less than the video quality parameter threshold, the preprocessing parameter is adjusted to obtain a second preprocessing parameter, so that the initial source video is preprocessed based on the second preprocessing parameter to obtain the video to be processed.

[0047] Among them, the initial source video is the original video captured in real time by the video capture device or the initial video obtained by decoding the video file pre-captured and stored by the video capture device, or the initial video obtained after a series of operations such as decoding, frame rate normalization, and resolution adaptation are performed on the above-mentioned original video or video file according to the video format.

[0048] The preset quality assessment indicators can be full-reference (FR) quality assessment indicators and / or no-reference (NR) quality assessment indicators; for example, FR quality assessment indicators can be: mean square error (MSE), mean absolute error (MAE), signal-to-noise ratio (SNR), peak signal-to-noise ratio (PSNR), etc.; NR quality assessment indicators can be evaluation indicators such as mean, standard deviation, and entropy. Correspondingly, the video quality parameter is an evaluation parameter determined based on the above-mentioned preset quality assessment indicators. The video quality parameter of the video to be processed is greater than or equal to the video quality parameter threshold. The first preprocessing parameter / the second preprocessing parameter can be a filter strength, a contrast enhancement system, etc.

[0049] Exemplarily, after obtaining the video captured by the video capture device, the captured video can be preprocessed first, and the video quality of the initial source video can be evaluated by preset quality evaluation indicators. Specifically, the determined video quality parameters are compared with the video quality parameter threshold. If the video quality parameter threshold is exceeded, the initial source video can be directly used as the video to be processed; conversely, if the video quality parameter threshold is not exceeded, it can be preprocessed according to the first preprocessing parameter, or the adjusted second preprocessing parameter, so that the preprocessed video has better quality, thereby ensuring the video quality of the initial source video, and providing high-quality data for the accuracy of subsequent target area identification and target area cropping processes.

[0050] Furthermore, since not all image regions in each target video frame in the processed video contain key information or objects to be identified, the following will be combined with Figure 3 Provide detailed explanation.

[0051] Figure 3 This is a flow chart of a method for extracting image semantic features provided in one embodiment of the present application. In an optional embodiment of the present disclosure, the method for extracting image semantic features includes steps 301 to 304:

[0052] Step 301: Input the video to be processed into a pre-configured lightweight convolutional neural network.

[0053] The lightweight convolutional neural network may be, for example, a MobileNetV3 variant model.

[0054] Step 302: For each image sequence frame in the video to be processed, determine the confidence level of each image region in each image sequence frame based on a lightweight convolutional neural network.

[0055] Step 303: Determine the image region with a confidence level greater than a confidence threshold as a candidate image region;

[0056] Step 304: Input the candidate target region in each image sequence frame into the hybrid deep neural network, and extract the image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network.

[0057] For example, a lightweight convolutional neural network can quickly pre-screen image regions to exclude low-confidence background areas and identify image regions with confidence levels greater than a confidence threshold as candidate image regions. This process can prioritize excluding regions with less information, allowing image regions with confidence levels greater than the confidence threshold to be input into the hybrid deep neural network for image semantic feature extraction. This significantly reduces the amount of data computation and improves feature extraction efficiency.

[0058] In step 202, the video to be processed is input into a pre-trained hybrid deep neural network, and the image semantic features of each image sequence frame in the video to be processed are extracted based on the hybrid deep neural network; wherein, the hybrid deep neural network model comprises at least a feature map extraction module and a target adaptive attention module, the feature map extraction module is used to extract the corresponding multimodal feature map from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame according to the weight value and correlation value of each modal feature map in the multimodal feature map.

[0059] Exemplarily, after obtaining the video to be processed or the candidate image area, it is input into a pre-trained hybrid deep neural network, and the image semantic features of each image sequence frame in the video to be processed are extracted based on the hybrid deep neural network.

[0060] The overall architecture of the hybrid deep neural network model combines the efficient feature extraction method of EfficientNet with the feature reuse mechanism of DenseNet to construct a hybrid deep neural network.

[0061] In an optional embodiment of the present disclosure, the feature map extraction module includes: a depth-wise separable convolution module and a variant convolution Ghost module; the target adaptive attention module includes: a fused convolution block attention module CBAM module and an attention module of the selection kernel network SKNet, and a lightweight multi-head attention module.

[0062] Among them, the depthwise separable convolution module is an algorithm obtained by improving the standard convolution calculation in convolutional neural networks. It reduces the number of parameters required for convolution calculation by splitting the correlation between spatial dimensions and channel (depth) dimensions, and has been proven in some studies to improve the efficiency of convolution kernel parameters.

[0063] The convolution calculation of the depthwise separable convolution module is divided into two parts. First, spatial convolution (depthwise convolution) is performed on each channel (depth), and the output is concatenated. Then, channel convolution (pointwise convolution) is performed using the unit convolution kernel to obtain the feature map. The variant convolution Ghost module is a model compression method that can generate more feature maps with fewer parameters. In other words, it reduces network parameters and computation while maintaining network accuracy, thereby improving computing speed and reducing latency.

[0064] For example, the front-end of the hybrid deep neural network uses depthwise separable convolution and Ghost modules, which can reduce computational costs while increasing the diversity of feature expression. The middle part of the hybrid deep neural network introduces an adaptive attention module that integrates CBAM and SKNet. It dynamically weights features based on their importance to highlight key targets. A lightweight multi-head attention mechanism is designed on the back-end to balance accuracy and efficiency in small target detection and large scene understanding. The essence of the multi-head self-attention mechanism is to capture rich contextual information in the input data by computing the attention of different subspaces in parallel through multiple different attention heads. Its design mainly addresses the following issues: Long-range dependency problem: It enables the model to pay attention to both global and local information when processing long sequences. Diverse attention modes: Different features and relationships are learned through different heads. Better gradient propagation: After separate processing, information can be transmitted more evenly, alleviating the gradient vanishing problem.

[0065] In an optional embodiment, the network topology can be optimized through NAS-FPN search to further improve the module collaboration performance and meet the target recognition requirements of different scale scenarios. Among them, Neural Architecture Search (NAS) and a feature pyramid network (FPN) are used to train the controller by selecting the best model structure in a given search space. This process uses reinforcement learning technology, and the controller uses the accuracy of the sub-model in the search space as a reward signal to update the parameters. Therefore, through repeated trials, the controller continues to learn and will eventually generate a better structure.

[0066] Furthermore, to overcome the drawback of semantic understanding deficiency in related technical solutions, namely the lack of semantic understanding and target association analysis of video scenes, and the inability to optimize vertical screen composition layout based on video themes, character relationships, and event logic, when processing videos with multi-subject interactions or strong narratives, vertical screen images have difficulty effectively conveying core information and directing audience attention, hindering the efficient dissemination and value release of video content on vertical screens. Multimodal data can be combined with video scenes to generate data.

[0067] In an optional embodiment of the present disclosure, auxiliary modal information is determined; and multimodal data is constructed based on the auxiliary modal information and image visual data of the video to be processed, so as to construct multiple candidate recognition weight files.

[0068] Among them, the auxiliary modal information includes at least one of the following: a depth image captured by a depth camera and audio data associated with the video to be processed. The depth image is used to enhance the spatial layout weight, and the audio data is used to enhance the semantic weight of the image area.

[0069] For example, image visual data and auxiliary modal information are integrated, such as depth images (collected by a depth camera) to construct an RGBD data space, and audio features (audio event location, timbre and emotion analysis) to assist in scene semantic understanding. A multimodal fusion network layer is designed, and a bidirectional cross-modal attention mechanism is used to achieve adaptive feature fusion. For example, the semantic weight of the corresponding image region is enhanced based on audio event location, and depth information is used to optimize the target space layout analysis to improve the robustness of complex scene recognition. Model parameters are optimized through joint training on multimodal datasets (including simulated synthesis and real-world multimodal samples) to enhance cross-modal feature learning capabilities.

[0070] In addition, a scene perception module can be introduced into the system to classify the first frame and key frame scenes of the input video (such as indoor / outdoor, static / dynamic, single-person / multi-person scenes) using a lightweight CNN classification network (such as the improved ShuffleNet).

[0071] In an optional embodiment of the present disclosure, key image frames of a video to be processed are obtained; scene classification is performed on the key image frames based on a preconfigured lightweight convolutional neural classification network to obtain a target scene category of the video to be processed; and according to the target scene category, a target recognition weight file that matches the target scene category is determined from a plurality of candidate recognition weight files, so as to determine the weight values of each modal feature map based on the target recognition weight file.

[0072] Among them, the weight values of the modal feature maps represented by multiple candidate recognition weight files are different.

[0073] Exemplarily, based on the classification result of the target scene, a target recognition weight file that matches the target scene category is determined from multiple candidate recognition weight files to provide prior guidance for subsequent target recognition.

[0074] In step 203, the image semantic features of each image sequence frame are input into a region proposal network to determine the target region of each image sequence frame based on the region proposal network.

[0075] For each key area of each image sequence frame, determine the correlation score between the key area and the adjacent key areas; based on the pre-configured video production template and the target scene category, determine the correlation coefficient of each key area; according to the weight value of each modal feature map, the correlation score and the correlation coefficient of each key area, determine the importance score of each key area; based on the importance score of each key area, determine the target area of each image sequence frame.

[0076] For example, during the target recognition phase, the AI optimal recognition instance generates an image semantic graph through a semantic segmentation network, and dynamically calculates the semantic importance score of the target, i.e., the importance score of each key area, based on the target context. Specifically, the semantic importance score formula can be as follows:

[0077]

[0078] In formula (1), S represents the total semantic importance score of the target / semantic importance score, which is used to screen key targets; A j : The correlation score between the target and the jth surrounding key area (such as action focus, interaction center), calculated by semantic graph analysis (the value range is 0-10-1); W aj : Correlation coefficient, dynamically configured according to the video production intention. For example, W of the action focus area in an action scene a1 =1.5, other areas W a2 =0.8;γ class : Global adjustment coefficient, which is uniformly set according to the target type (such as people, landmark buildings), and is used to balance the overall weight of the association item.

[0079] Based on the above semantic importance scores, the semantic segmentation network outputs the category and location information of the target in the image, and calculates the relevance score A of each target based on the relative position, motion trajectory and contextual logical relationship between the target and the surrounding key areas. j For example, in a multi-person stage performance, the relevance score of the actors’ interaction area close to the center of the screen is significantly higher than that of the actors’ area far from the center of the screen.

[0080] Dynamic coefficient configuration: The system predefines a variety of production intention templates (such as "highlight the subject" and "balanced composition"), and loads the corresponding correlation coefficient W according to user needs or scene types. aj For example, in a character interview video, the W of the larger target area aj Set to 2.0 to give more weight to the core objective.

[0081] For target classification and predefined weight loading: the system first identifies the target category (such as people, buildings) through the semantic segmentation network and loads the predefined semantic weight γ class (such as character γ class =1)

[0082] Finally, total score calculation and target screening: The system calculates the total score S by combining the category weight and relevance items. The system screens targets in descending order of S, ensuring that the vertical screen focuses on elements with high semantic importance.

[0083] It should be understood that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0084] To implement the above video processing method, see Figure 4 An embodiment of the present application provides a video processing device, which may include a video acquisition module 401, a semantic feature extraction module 402, a target area determination module 403, and a video conversion module 404.

[0085] Among them, the video acquisition module 401 is used to acquire the video to be processed; the semantic feature extraction module 402 is used to input the video to be processed into a pre-trained hybrid deep neural network, and extract the image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model comprises at least a feature map extraction module and a target adaptive attention module, the feature map extraction module is used to extract the corresponding multimodal feature map from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame according to the weight value and correlation value of each modal feature map in the multimodal feature map; the target area determination module 403 is used to input the image semantic features of each image sequence frame into the region proposal network, so as to determine the target area of each image sequence frame based on the region proposal network; the video conversion module 404 is used to crop the target area of each image sequence frame to obtain the target video, so as to convert the target video into a horizontal screen frame and a vertical screen frame.

[0086] In an optional embodiment of the present disclosure, the device further includes a determination module, which is used to input the video to be processed into a preconfigured lightweight extreme convolutional neural network; for each image sequence frame in the video to be processed, determine the confidence of each image region in each image sequence frame based on the lightweight extreme convolutional neural network; determine the image region whose confidence is greater than the confidence threshold as a candidate image region; input the candidate target region in each image sequence frame into the hybrid deep neural network, and extract the image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network.

[0087] In an optional embodiment of the present disclosure, the device also includes a scene determination module, which is used to obtain key image frames of the video to be processed; based on a pre-configured lightweight convolutional neural classification network, the key image frames are subjected to scene classification to obtain a target scene category of the video to be processed; according to the target scene category, a target recognition weight file matching the target scene category is determined from multiple candidate recognition weight files, so as to determine the weight values of each modal feature map based on the target recognition weight file; wherein, the weight values of each modal feature map represented by the multiple candidate recognition weight files are different.

[0088] In an optional embodiment of the present disclosure, the device also includes a module for determining auxiliary modal information and a weight file construction module, wherein the auxiliary modal information includes at least one of the following: a depth image captured by a depth camera, and audio data associated with the video to be processed, the depth image is used to enhance the spatial layout weight, and the audio data is used to enhance the semantic weight of the image area; a weight file construction module is used to construct multimodal data based on the auxiliary modal information and the image visual data of the video to be processed, so as to construct the multiple candidate recognition weight files.

[0089] In an optional embodiment of the present disclosure, the target area determination module 403 is used to determine, for each key area of each of the image sequence frames, a correlation score between the key area and the adjacent key areas; determine the correlation coefficient of each key area based on a pre-configured video production template and the target scene category; determine the importance score of each key area according to the weight value of each modal feature map, the correlation score and the correlation coefficient of each key area; and determine the target area of each of the image sequence frames based on the importance score of each key area.

[0090] In an optional embodiment of the present disclosure, the feature map extraction module includes: a depth-separable convolution module and a variant convolution Ghost module; the target adaptive attention module includes: a fused convolution block attention module CBAM module and an attention module of the selection kernel network SKNet, and a lightweight multi-head attention module.

[0091] In an optional embodiment of the present disclosure, the device also includes a preprocessing module, which is used to obtain an initial source video and determine a video quality parameter of the initial source video based on a preset quality evaluation index; if the video quality parameter of the initial source video is less than a video quality parameter threshold, the initial source video is preprocessed according to a first preprocessing parameter to obtain the video to be processed; or the preprocessing parameter is adjusted to obtain a second preprocessing parameter, and the initial source video is preprocessed based on the second preprocessing parameter to obtain the video to be processed; wherein the video quality parameter of the video to be processed is greater than or equal to the video quality parameter threshold.

[0092] For the specific definition of the above-mentioned video processing device, please refer to the definition of the video processing method above, and will not be repeated here. Each module in the above-mentioned video processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0093] In one embodiment, a computer device is provided. The internal structure diagram of the computer device can be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a video processing method as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the video processing method as described above is implemented.

[0094] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above video processing method can be implemented.

[0095] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0099] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0100] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A video processing method, characterized in that: include: Get the video to be processed; Inputting the video to be processed into a pre-trained hybrid deep neural network, and extracting image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model comprises at least a feature map extraction module and a target adaptive attention module, wherein the feature map extraction module is used to extract corresponding multimodal feature maps from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame according to the weight value and correlation value of each modal feature map in the multimodal feature map; inputting the image semantic features of each of the image sequence frames into a region proposal network to determine a target region of each of the image sequence frames based on the region proposal network; The target area of each of the image sequence frames is cropped to obtain a target video, so as to convert the target video into a horizontal screen frame and a vertical screen frame.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the video to be processed into a pre-configured lightweight convolutional neural network; For each image sequence frame in the video to be processed, determining the confidence of each image region in each image sequence frame based on the lightweight convolutional neural network; Determine the image region whose confidence level is greater than the confidence threshold as a candidate image region; The candidate target areas in each image sequence frame are input into the hybrid deep neural network, and the image semantic features of each image sequence frame in the video to be processed are extracted based on the hybrid deep neural network.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining a key image frame of the video to be processed; Based on a pre-configured lightweight convolutional neural classification network, the key image frame is subjected to scene classification to obtain a target scene category of the video to be processed; According to the target scene category, a target recognition weight file that matches the target scene category is determined from multiple candidate recognition weight files, so as to determine the weight values of each modal feature graph based on the target recognition weight file; wherein the weight values of each modal feature graph represented by the multiple candidate recognition weight files are different.

4. The method according to claim 3, characterized in that The method further comprises: Determining auxiliary modal information, where the auxiliary modal information includes at least one of the following: a depth image captured by a depth camera, and audio data associated with the video to be processed, where the depth image is used to enhance the spatial layout weight, and the audio data is used to enhance the semantic weight of the image region; Based on the auxiliary modality information and the image visual data of the video to be processed, multimodal data is constructed to construct the multiple candidate recognition weight files.

5. The method according to claim 3, characterized in that The determining of the target region of each of the image sequence frames based on the region proposal network includes: For each key region of each of the image sequence frames, determining a correlation score between the key region and adjacent key regions; Determining a correlation coefficient of each key area based on a pre-configured video production template and the target scene category; Determining the importance score of each key area based on the weight value, correlation score of each modal feature graph and the correlation coefficient of each key area; Based on the importance scores of the key areas, a target area of each image sequence frame is determined.

6. The method according to claim 1, characterized in that The feature map extraction module includes: a depth-separable convolution module and a variant convolution Ghost module; the target adaptive attention module includes: a fused convolution block attention module CBAM module and an attention module of the selection kernel network SKNet, as well as a lightweight multi-head attention module.

7. The method according to claim 1, characterized in that The method further comprises: Acquire an initial source video, and determine a video quality parameter of the initial source video based on a preset quality assessment indicator; If the video quality parameter of the initial source video is less than the video quality parameter threshold, the initial source video is preprocessed according to the first preprocessing parameter to obtain the video to be processed; or the preprocessing parameter is adjusted to obtain the second preprocessing parameter, and the initial source video is preprocessed based on the second preprocessing parameter to obtain the video to be processed; wherein the video quality parameter of the video to be processed is greater than or equal to the video quality parameter threshold.

8. A video processing device, characterized in that: include: Video acquisition module, used to acquire the video to be processed; A semantic feature extraction module is used to input the video to be processed into a pre-trained hybrid deep neural network, and extract the image semantic features of each image sequence frame in the video to be processed based on the hybrid deep neural network; wherein the hybrid deep neural network model comprises at least a feature map extraction module and a target adaptive attention module, wherein the feature map extraction module is used to extract the corresponding multimodal feature map from each image sequence frame, and the target adaptive attention module is used to determine the image semantic features of each image sequence frame based on the weight value and correlation value of each modal feature map in the multimodal feature map; a target region determination module, configured to input the image semantic features of each of the image sequence frames into a region proposal network, so as to determine the target region of each of the image sequence frames based on the region proposal network; The video conversion module is used to crop the target area of each image sequence frame to obtain a target video, so as to convert the target video into a horizontal screen frame and a vertical screen frame.

9. A computer device comprising: The method comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images