Video semantic segmentation method and device, computer equipment and storage medium

By extracting the global and contextual features of the video keyframe image, the video semantic segmentation results are generated, which solves the problem of insufficient efficiency and accuracy of video semantic segmentation in the prior art, and improves the recognition and response capabilities of the video surveillance system in the financial industry.

CN119091145APending Publication Date: 2024-12-06PING AN TECH (SHENZHEN) CO LTD

Patent Information

Application Number
CN202411403850.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-09
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When existing video semantic segmentation technology deals with complex dynamic scenarios, processing efficiency and accuracy still need to be improved, and it is difficult to meet the high requirements of the financial industry for security and real-time.

Method used

By obtaining the keyframe image of the target video, using the backbone network model to extract global features, input them into the global context network model to generate context features, combining category information for feature fusion, using semantic segmentation model to generate image semantic segmentation results, and integrating all keys The frame result generates the video semantic segmentation result.

Benefits of technology

It significantly improves the efficiency and accuracy of video semantic segmentation, enhances the intelligent identification and response capabilities of the video surveillance system, and improves the security and operational efficiency of financial institutions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091145A_ABST
    Figure CN119091145A_ABST
Patent Text Reader

Abstract

The invention discloses a video semantic segmentation method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology, and the method comprises the steps: obtaining a target video, and recognizing a key frame image of the target video; performing feature extraction on the key frame image by using a backbone network model to obtain a global feature; inputting the global features into a global context network model to generate context features; determining category information of the target video, and fusing the category information with the global feature and the context feature to generate a fusion feature of the key frame image; based on the fusion feature, generating an image semantic segmentation result of the key frame image by using a semantic segmentation model; integrating the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video; therefore, the video semantic segmentation efficiency and accuracy can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a video semantic segmentation method, device, computer equipment and computer-readable storage medium. Background Art

[0002] At present, with the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming to Fintech, and video semantic segmentation technology is no exception. For example, in the physical places of financial institutions such as banks and stock exchanges, video surveillance systems are used to ensure security. Video semantic segmentation technology can enhance the functions of these video surveillance systems, enabling them to more intelligently identify and respond to specific events, so as to improve the security and operational efficiency of financial institutions. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on video semantic segmentation technology.

[0003] Currently, with the rapid development of computer vision and deep learning technologies, video semantic segmentation technology has become a hot topic in research and application. Video semantic segmentation aims to assign pixels of image frames in a video to corresponding category labels, thereby achieving a detailed understanding of dynamic scenes. Although significant progress has been made, the processing efficiency and accuracy of existing technologies when processing complex dynamic scenes still need to be improved.

[0004] Based on this, how to provide a video semantic segmentation method, device, computer equipment and computer-readable storage medium that can effectively improve the efficiency and accuracy of video semantic segmentation is a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a video semantic segmentation method, apparatus, computer device and computer-readable storage medium, aiming to solve the problem of how to effectively improve the efficiency and accuracy of video semantic segmentation.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a video semantic segmentation method, which includes:

[0008] Acquire a target video and identify key frame images of the target video;

[0009] Using a backbone network model to extract features from the key frame image to obtain global features;

[0010] Inputting the global features into a global context network model to generate context features;

[0011] Determining category information of the target video, and fusing the category information with the global feature and the context feature to generate a fusion feature of the key frame image;

[0012] Based on the fusion features, using a semantic segmentation model to generate an image semantic segmentation result of the key frame image;

[0013] The image semantic segmentation results of all the key frame images of the target video are integrated to generate a video semantic segmentation result of the target video.

[0014] In a second aspect, the present invention provides a video semantic segmentation device, comprising:

[0015] An acquisition and recognition module, used to acquire a target video and recognize key frame images of the target video;

[0016] A feature extraction module, used to extract features from the key frame image using a backbone network model to obtain global features;

[0017] A feature input module, used for inputting the global features into a global context network model to generate context features;

[0018] A feature fusion module, used to determine the category information of the target video, and fuse the category information with the global feature and the context feature to generate a fusion feature of the key frame image;

[0019] A first result generating module, configured to generate an image semantic segmentation result of the key frame image based on the fusion feature and using a semantic segmentation model;

[0020] The second result generating module is used to integrate the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video.

[0021] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video semantic segmentation method as described above when executing the computer program.

[0022] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the video semantic segmentation method as described above.

[0023] Compared with the prior art, the present invention provides a video semantic segmentation method, apparatus, computer equipment and computer-readable storage medium, wherein a target video is acquired and key frame images of the target video are identified; features of the key frame images are extracted using a backbone network model to obtain global features; the global features are input into a global context network model to generate context features; category information of the target video is determined, and the category information is fused with the global features and the context features to generate fused features of the key frame images; based on the fused features, an image semantic segmentation result of the key frame images is generated using a semantic segmentation model; the image semantic segmentation results of all the key frame images of the target video are integrated to generate a video semantic segmentation result of the target video; thus, the present invention can effectively improve the efficiency and accuracy of video semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of an application environment of a video semantic segmentation method provided by an embodiment of the present invention.

[0026] Figure 2 A flowchart of a video semantic segmentation method provided by an embodiment of the present invention.

[0027] Figure 3 A schematic diagram of program modules of a video semantic segmentation device provided by an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention.

[0029] Figure 5 Another structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0034] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0036] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0037] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0038] A video semantic segmentation method provided by an embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment shown, the client and the server communicate through the network. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0039] See also Figure 2 An embodiment of the present invention provides a video semantic segmentation method, wherein the method comprises the following steps:

[0040] S100, acquiring a target video, and identifying key frame images of the target video;

[0041] S200, extracting features from the key frame image using a backbone network model to obtain global features;

[0042] S300, inputting the global features into a global context network model to generate context features;

[0043] S400, determining category information of the target video, and fusing the category information with the global feature and the context feature to generate a fusion feature of the key frame image;

[0044] S500, generating an image semantic segmentation result of the key frame image using a semantic segmentation model based on the fusion features;

[0045] S600: Integrate the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video.

[0046] In specific implementation, the video semantic segmentation method of this embodiment achieves efficient and accurate understanding of video content through a series of carefully designed steps. First, through the recognition of key frame images, the method of this embodiment avoids the computationally intensive operation of processing each frame in the target video, thereby improving processing efficiency. Then, the key frame images are extracted using the backbone network model to effectively capture the global features of the key frame images, which contain rich spatial information. Then, the contextual understanding ability of the global features is further enhanced through the global context network model, so that the relationship between the various objects and scenes in the key frame images can be better understood. In addition, the fusion target video The category information of the target video is used to enable the semantic segmentation model to utilize prior knowledge and improve the accuracy of semantic segmentation. The application of the semantic segmentation model combines the powerful ability of deep learning to generate high-quality image semantic segmentation results. Finally, by integrating the image semantic segmentation results of all key frame images of the target video, the video semantic segmentation results of the target video are generated, ensuring the coherence and consistency of the video semantic segmentation results. Overall, the method of this embodiment combines advanced deep learning technology and effective feature fusion strategy to achieve accurate segmentation of the target video content while maintaining high processing efficiency, thereby achieving significant technical effects in improving the processing efficiency and accuracy of video semantic segmentation.

[0047] Furthermore, in one embodiment, the video semantic segmentation method, wherein the step S100, obtaining a target video and identifying a key frame image of the target video, specifically comprises the steps of:

[0048] Get the target video uploaded by the target user;

[0049] Using a key frame detection algorithm to identify the target video and generate an identification result;

[0050] Based on the recognition result, a key frame image is extracted from the target video.

[0051] In specific implementation, this embodiment first obtains the target video uploaded by the target user, then applies the key frame detection algorithm to efficiently identify the main content changes in the target video, generates a recognition result, and finally accurately extracts the key frame image from the target video based on the recognition result. It can quickly and accurately screen out representative video content and provide core data for subsequent video processing tasks, such as semantic segmentation, content analysis or feature extraction, thereby significantly improving processing efficiency and video analysis accuracy.

[0052] The specific implementation process of the steps in this embodiment is roughly as follows:

[0053] 1. Video acquisition:

[0054] Input video: First, receive the target video file uploaded by the target user through the user interface or API interface;

[0055] File processing: Perform necessary checks and processing on video files, including format verification, integrity check, etc.;

[0056] 2. Application of key frame detection algorithm:

[0057] Algorithm selection: Choose a suitable keyframe detection algorithm, which can be a content analysis-based method, such as comparing the similarity between frames, or a deep learning-based method, such as using a pre-trained neural network model;

[0058] Processing the video: Segment the video into individual frames and feed them into a keyframe detection algorithm, which analyzes each frame and identifies the keyframes in it.

[0059] 3. Generate recognition results:

[0060] Marking key frames: After the algorithm analyzes, it generates a list containing the key frame positions and timestamps, or other forms of markings, indicating the importance of these frames in the video;

[0061] Result output: Output the recognition results of the key frames to provide a basis for subsequent extraction steps;

[0062] 4. Extract key frame images:

[0063] Positioning frame: Position the key frame in the video based on the recognition results provided by the key frame detection algorithm;

[0064] Frame extraction: extract these images marked as keyframes from the video and possibly store them as separate files or buffers;

[0065] 5. Post-processing:

[0066] Quality Assessment: Perform quality assessment on the extracted keyframe images to ensure their representativeness and content quality;

[0067] Data sorting: sort the extracted key frame images, such as adjusting the resolution, format conversion, etc., to meet the needs of subsequent applications;

[0068] Through the above process, this embodiment can quickly and effectively identify and extract key frame images from the video uploaded by the user. These key frame images can represent the main content of the video, providing convenience for further analysis and processing of the video content.

[0069] Furthermore, in one embodiment, the video semantic segmentation method, wherein the step S200, extracting features from the key frame image using a backbone network model to obtain global features, specifically comprises the steps of:

[0070] Pre-build the backbone network model;

[0071] Preprocessing the key frame image;

[0072] Inputting the preprocessed key frame image into the backbone network model to extract features and obtain initial features;

[0073] The initial features are normalized to obtain global features of the key frame image.

[0074] In specific implementation, this embodiment pre-builds and utilizes the backbone network model to perform a series of processing on the key frame images, thereby achieving efficient feature extraction and expression; first, by pre-processing the key frame images, it is ensured that the image data is adapted to the input requirements of the backbone network, thereby improving the compatibility and efficiency of the processing; then, the pre-processed key frame images are input into the backbone network model, and the deep structure of the backbone network model is used to automatically learn and extract the rich features of the key frame images to generate initial features; then, the initial features are normalized, and this step not only enhances the numerical stability of the features, but also improves the accuracy and generalization ability of subsequent processing steps; finally, the global features of the key frame images obtained can provide accurate and discriminative feature representations for tasks such as video semantic segmentation, thereby significantly improving the performance of video content analysis and processing.

[0075] The specific implementation process of the steps in this embodiment is roughly as follows:

[0076] 1. Build a backbone network model:

[0077] Select model architecture: Choose a suitable pre-trained backbone network model, such as ResNet, VGG or Inception;

[0078] Load pre-trained weights: Load the model's pre-trained weights on large datasets (such as ImageNet), which helps improve the performance and accuracy of feature extraction;

[0079] 2. Preprocess the key frame images:

[0080] Adjust image size: adjust the key frame image to the input size required by the backbone network model;

[0081] Normalization: Normalize the pixel values ​​of the key frame images, for example, using the mean and standard deviation to make the input data more evenly distributed, which is helpful for model training and reasoning;

[0082] 3. Feature extraction:

[0083] Input image: Input the preprocessed key frame image into the backbone network model;

[0084] Forward propagation: forward propagation through the network, extracting features from each layer; usually extracting features before the last convolutional layer of the network because these features contain rich spatial information;

[0085] 4. Get initial features:

[0086] Extract output: Get the output of the last convolutional layer of the backbone network model as the initial feature;

[0087] Feature processing: may include cropping, splicing or other operations to form a feature representation suitable for the next step of processing;

[0088] 5. Normalization processing:

[0089] Feature normalization: Normalize the initial features, such as L2 normalization, to ensure that the norm of the feature vector is 1;

[0090] Output global features: The processed normalized features are the global features of the key frame image, which can be used for subsequent semantic segmentation or other analysis tasks;

[0091] Through the above process, this embodiment can extract useful global features from key frame images, and these features provide important information for understanding video content.

[0092] Furthermore, in one embodiment, the video semantic segmentation method, wherein the step S300, inputting the global features into a global context network model to generate context features, specifically comprises the steps of:

[0093] Pre-build a global context network model;

[0094] Inputting the global features into the global context network model to generate preliminary features;

[0095] The preliminary features are normalized to obtain context features of the key frame image.

[0096] In specific implementation, this embodiment achieves in-depth processing and understanding of the global features of the key frame image through a pre-built global context network model; first, the global features obtained from the backbone network are input into the global context network model, which is specially designed to capture the contextual information in the image and enhance the semantic expression ability of the features; then, the generated preliminary features are further optimized through normalization processing to ensure the numerical stability and consistency of the features, so that the contextual features of the key frame images obtained are not only rich in the global information of the image, but also have good contextual relevance, which significantly improves the performance and accuracy of subsequent semantic segmentation tasks; this series of processing steps enables more accurate understanding and segmentation of objects and scenes in the image, especially when processing complex dynamic video content, which can effectively improve the accuracy and robustness of segmentation.

[0097] The specific implementation process of the steps in this embodiment is roughly as follows:

[0098] 1. Build a global context network model:

[0099] Design network architecture: Design and build a global context network model that can process the global context information of the image; common architectures include pyramid pooling modules (such as in PSPNet) or other forms of context aggregation layers;

[0100] Initialize the model: Initialize the network parameters and, if possible, load pre-trained weights to improve model performance;

[0101] 2. Input global features:

[0102] Feature adaptation: adjust the global features extracted from the backbone network to a form suitable for the input of the global context network model;

[0103] Input model: The processed global features are sent to the global context network model for further feature extraction;

[0104] 3. Generate preliminary features:

[0105] Context extraction: extract global context features and generate preliminary features through forward propagation of the global context network model;

[0106] Feature fusion: The model may include feature fusion operations, such as merging feature maps at different levels;

[0107] 4. Normalization processing:

[0108] Select normalization method: According to actual requirements and task characteristics, select an appropriate normalization method, such as L2 normalization, BatchNormalization or others;

[0109] Perform normalization: Normalize the preliminary features to ensure that the feature vectors conform to a specific statistical distribution, such as a mean of 0 and a standard deviation of 1;

[0110] 5. Output context features:

[0111] Obtaining normalized features: using the normalized preliminary features as context features of the key frame image;

[0112] Post-processing (optional): If necessary, the context features can be cropped, scaled, or otherwise processed to suit specific application requirements.

[0113] Through the above process, this embodiment can extract features containing rich context information from the key frame images, and these features provide strong support for achieving accurate video semantic segmentation.

[0114] Furthermore, in one embodiment, the video semantic segmentation method, wherein the step S400, determining the category information of the target video, and fusing the category information with the global feature and the context feature to generate the fusion feature of the key frame image, specifically comprises the steps of:

[0115] Based on the preset category label of the target video, obtain category information of the target video, and vectorize the category information into a category embedding vector;

[0116] Fusion the category embedding vector with the global feature and the context feature to generate a primary feature;

[0117] The primary features are normalized to obtain fusion features of the key frame image.

[0118] In specific implementation, this embodiment achieves a richer and more discernible feature representation for the key frame image by vectorizing the category information of the target video into a category embedding vector and fusing it with the extracted global features and context features. This process enhances the ability to understand the semantics of the video content, so that the generated primary features not only contain the visual information directly captured from the image, but also incorporate the semantic context of the video. This embodiment further ensures the consistency and stability of the features by normalizing the primary features, so that the fused features of the key frame images obtained can provide more accurate and robust input for the video semantic segmentation task, significantly improving the quality of the segmentation results and the overall performance of the model.

[0119] The specific implementation process of the steps in this embodiment is roughly as follows:

[0120] 1. Acquisition and vectorization of category information:

[0121] Category information determination: obtaining the category information of the target video according to the preset category label of the target video, such as "sports event", "concert", etc.;

[0122] Embedding vector generation: Through the trained embedding layer, the category information is converted into embedding vectors in a high-dimensional space. These vectors can capture the semantic relationship between categories.

[0123] 2. Feature fusion preparation:

[0124] Global and context feature extraction: Using the backbone network model and global context network model mentioned in the previous step, extract global features and context features from the key frame image;

[0125] Feature adjustment: ensure that the category embedding vector is consistent in dimension with the global and context features for fusion;

[0126] 3. Fusion category embedding vector:

[0127] Feature-level fusion: Fusion of category embedding vectors with global features and contextual features; fusion can be achieved through concatenation, weighted summation, or more complex mechanisms such as attention mechanisms;

[0128] 4. Generate primary features:

[0129] Fusion result: The feature set obtained after the fusion operation is called primary features, which contain the visual information and semantic information of the video;

[0130] 5. Normalization processing:

[0131] Normalization operation: normalize the primary features, such as using L2 normalization, to make the length of the feature vector uniform and reduce the difference in the numerical range between different features;

[0132] Output fusion features: After normalization, the obtained fusion features will be used in the subsequent semantic segmentation model to provide richer and more accurate feature representation for the segmentation task;

[0133] Through the above process, this embodiment can generate fusion features including category, semantics and vision, which are crucial to improving the accuracy and efficiency of video semantic segmentation.

[0134] Furthermore, in one embodiment, the video semantic segmentation method, wherein the step S500, based on the fusion feature, using the semantic segmentation model to generate the image semantic segmentation result of the key frame image, specifically comprises the steps of:

[0135] Pre-built semantic segmentation models;

[0136] Inputting the fused features into the semantic segmentation model to generate an initial semantic segmentation result;

[0137] The initial semantic segmentation result is optimized using a conditional random field to obtain an image semantic segmentation result of the key frame image.

[0138] In specific implementation, this embodiment realizes efficient conversion from fusion features to initial semantic segmentation results through a pre-built semantic segmentation model, and then optimizes these results using conditional random fields (CRF), thereby further improving the accuracy and detail performance of segmentation; this process ensures that the image semantic segmentation results are not only more accurate in categories, but also more natural and smooth in terms of edge and regional coherence; the method of this embodiment can significantly improve the semantic segmentation quality of video key frame images, so that the image semantic segmentation results are more in line with the complexity and diversity of actual scenes, especially when dealing with objects of similar categories or mutually occluded objects, the optimized image semantic segmentation results can better reflect the scene structure of the real world.

[0139] The specific implementation process of the steps in this embodiment is roughly as follows:

[0140] 1. Build a semantic segmentation model:

[0141] Choose an architecture: Choose a suitable semantic segmentation model architecture, such as FCN (fully convolutional network), DeepLab or UNet;

[0142] Pre-trained models: If possible, load model weights pre-trained on standard datasets (such as COCO, PascalVOC) to take advantage of existing general feature extraction capabilities;

[0143] 2. Input fusion features:

[0144] Feature preparation: Ensure that the fused features (including global features, context features, and category embedding vectors) are prepared and meet the input requirements of the semantic segmentation model;

[0145] Data format: Convert the fused features into a data format acceptable to the model, such as resizing, normalizing, etc.

[0146] 3. Generate initial semantic segmentation results:

[0147] Forward propagation: Input the fused features into the semantic segmentation model, perform forward propagation, and the model classifies each pixel to generate the initial semantic segmentation result;

[0148] Temporary result processing: Processing temporary results output by the model, such as converting probability maps into category label maps;

[0149] 4. Conditional Random Field Optimization:

[0150] CRF model: load or build a conditional random field model to optimize the initial semantic segmentation results;

[0151] Post-processing: The initial semantic segmentation results are input into the CRF model for post-processing. The CRF model will consider the relationship between pixels and optimize the consistency of segmentation edges and regions.

[0152] 5. Output image semantic segmentation results:

[0153] Result extraction: Get optimized image semantic segmentation results from the CRF model, which are more visually coherent and accurate;

[0154] Application of results: Use the optimized image semantic segmentation results for subsequent applications, such as video content analysis, object tracking, or other related tasks;

[0155] Through the above process, this embodiment can effectively convert the fusion features into accurate image semantic segmentation results, and significantly improve the quality of segmentation by optimizing the initial semantic segmentation results.

[0156] Furthermore, in one embodiment, the video semantic segmentation method, wherein the step S600, integrating the image semantic segmentation results of all the key frame images of the target video to generate the video semantic segmentation result of the target video, specifically comprises the steps of:

[0157] Collect the image semantic segmentation results of all the key frame images of the target video, and record a timestamp for the image semantic segmentation result corresponding to each key frame image according to the time node at which each key frame image appears in the target video;

[0158] According to the timestamps of the semantic segmentation results of each image, sequentially integrating the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video;

[0159] The video semantic segmentation result is formatted, and the formatted video semantic segmentation result is output or displayed.

[0160] In specific implementation, this embodiment achieves a fine-grained understanding of the video content by accurately collecting and integrating the image semantic segmentation results of all key frame images in the target video and recording a specific timestamp for each result; first, by recording the timestamp for the image semantic segmentation result of each key frame image, the accurate correspondence between the segmentation data and the video timeline is ensured; then, the image semantic segmentation results are sorted and integrated according to the timestamp to generate a coherent and orderly video semantic segmentation result, thereby providing a complete semantic description for the entire video; finally, through formatting processing, the video semantic segmentation results are suitable for different application scenarios and display requirements, thereby improving the availability and practicality of the segmented data; this embodiment not only enhances the accuracy and efficiency of video content analysis, but also provides a high-quality foundation for further processing and analysis of video data, making the application of video semantic segmentation technology in video surveillance, content editing, automatic labeling and other fields more extensive and effective.

[0161] The specific implementation process of the steps in this embodiment is roughly as follows:

[0162] 1. Collect image semantic segmentation results and timestamp records:

[0163] Image semantic segmentation result collection: Apply the semantic segmentation model to the key frame images in the target video and collect all the generated image semantic segmentation results;

[0164] Timestamp recording: While collecting the image semantic segmentation results of key frame images, the timestamp of the corresponding image semantic segmentation results is recorded according to the time node when each key frame image appears in the target video;

[0165] 2. Sort and consolidate by timestamp:

[0166] Sorting: Sort the image semantic segmentation results of all key frame images according to the timestamp to ensure the temporal order of the segmentation results;

[0167] Integration: Integrate the sorted image semantic segmentation results in order into a complete video semantic segmentation result;

[0168] 3. Inter-frame information filling (optional):

[0169] Non-keyframe segmentation: Using the image semantic segmentation results of keyframe images and the temporal continuity between frames, the image semantic segmentation results of non-keyframe images are predicted through interpolation, optical flow method or deep learning technology;

[0170] Information filling: Fill the predicted image semantic segmentation results of non-key frame images into the corresponding non-key frame images to ensure that all frames in the video have complete segmentation information;

[0171] 4. Formatting:

[0172] Data format adjustment: Format the integrated video semantic segmentation results and adjust them to a format suitable for different application requirements;

[0173] Quality optimization: Optimize the visual effects of video semantic segmentation results, such as edge smoothing, regional connectivity processing, etc.

[0174] 5. Output and display:

[0175] Output: Output the formatted video semantic segmentation results, save them as files or send them to other processing systems;

[0176] Display: Display the video semantic segmentation results on the user interface, or overlay the video semantic segmentation results on the original video frame through visualization tools for easy observation and analysis;

[0177] Through the above process, this embodiment can generate a structured and coherent video semantic segmentation result, which provides a reliable foundation for subsequent analysis, editing, automated processing and other applications.

[0178] It can be seen from the above method embodiments that the video semantic segmentation method provided by the present invention includes: obtaining a target video and identifying the key frame images of the target video; using a backbone network model to extract features of the key frame images to obtain global features; inputting the global features into a global context network model to generate context features; determining the category information of the target video, and fusing the category information with the global features and the context features to generate fused features of the key frame images; based on the fused features, using a semantic segmentation model to generate image semantic segmentation results of the key frame images; integrating the image semantic segmentation results of all the key frame images of the target video to generate video semantic segmentation results of the target video. In this way, the efficiency and accuracy of video semantic segmentation can be effectively improved by the method of the present invention.

[0179] It can be understood that the video semantic segmentation method provided by the embodiments of the present invention can be applied to video semantic segmentation scenarios related to the field of financial technology. Specifically, the video semantic segmentation function can be integrated into the corresponding physical locations of financial institutions (such as banks). In the physical locations of financial institutions such as banks and stock exchanges, video surveillance systems are used to ensure security. The video semantic segmentation method provided by the embodiments of the present invention can enhance the functions of these video surveillance systems, enabling them to more intelligently identify and respond to specific events, thereby improving the security and operational efficiency of financial institutions.

[0180] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0181] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a video semantic segmentation device, wherein the device comprises:

[0182] An acquisition and recognition module 11 is used to acquire a target video and recognize key frame images of the target video;

[0183] A feature extraction module 12 is used to extract features from the key frame image using a backbone network model to obtain global features;

[0184] A feature input module 13, used for inputting the global features into a global context network model to generate context features;

[0185] A feature fusion module 14 is used to determine the category information of the target video, and fuse the category information with the global feature and the context feature to generate a fusion feature of the key frame image;

[0186] A first result generating module 15, configured to generate an image semantic segmentation result of the key frame image based on the fusion feature and using a semantic segmentation model;

[0187] The second result generating module 16 is used to integrate the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video.

[0188] Furthermore, in one embodiment, in the video semantic segmentation apparatus, the acquisition and recognition module 11 is specifically used for:

[0189] Get the target video uploaded by the target user;

[0190] Using a key frame detection algorithm to identify the target video and generate an identification result;

[0191] Based on the recognition result, a key frame image is extracted from the target video.

[0192] Furthermore, in one embodiment, in the video semantic segmentation apparatus, the feature extraction module 12 is specifically used for:

[0193] Pre-build the backbone network model;

[0194] Preprocessing the key frame image;

[0195] Inputting the preprocessed key frame image into the backbone network model to extract features and obtain initial features;

[0196] The initial features are normalized to obtain global features of the key frame image.

[0197] Furthermore, in one embodiment, in the video semantic segmentation apparatus, the feature input module 13 is specifically used for:

[0198] Pre-build a global context network model;

[0199] Inputting the global features into the global context network model to generate preliminary features;

[0200] The preliminary features are normalized to obtain context features of the key frame image.

[0201] Furthermore, in one embodiment, in the video semantic segmentation apparatus, the feature fusion module 14 is specifically used for:

[0202] Based on the preset category label of the target video, obtain category information of the target video, and vectorize the category information into a category embedding vector;

[0203] Fusion the category embedding vector with the global feature and the context feature to generate a primary feature;

[0204] The primary features are normalized to obtain fusion features of the key frame image.

[0205] Furthermore, in one embodiment, in the video semantic segmentation apparatus, the first result generating module 15 is specifically used for:

[0206] Pre-built semantic segmentation models;

[0207] Inputting the fused features into the semantic segmentation model to generate an initial semantic segmentation result;

[0208] The initial semantic segmentation result is optimized using a conditional random field to obtain an image semantic segmentation result of the key frame image.

[0209] Furthermore, in one embodiment, in the video semantic segmentation apparatus, the second result generating module 16 is specifically used for:

[0210] Collect the image semantic segmentation results of all the key frame images of the target video, and record a timestamp for the image semantic segmentation result corresponding to each key frame image according to the time node at which each key frame image appears in the target video;

[0211] According to the timestamps of the semantic segmentation results of each image, sequentially integrating the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video;

[0212] The video semantic segmentation result is formatted, and the formatted video semantic segmentation result is output or displayed.

[0213] It should be noted that in the embodiment of the device of the present invention, the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the aforementioned method embodiment part and will not be repeated here.

[0214] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of the video semantic segmentation method in any of the above-mentioned method embodiments.

[0215] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of the video semantic segmentation method in any of the above-mentioned method embodiments.

[0216] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0217] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0218] Among them, the memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of a computer device, and in other embodiments, it can also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory can also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory can also be used to temporarily store data that has been output or is to be output.

[0219] Based on the above method embodiments, another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the video semantic segmentation method in any of the above method embodiments is implemented. The computer-readable storage medium may be non-volatile or volatile.

[0220] It should be noted that the above-mentioned functions or steps that can be implemented by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be found in the relevant descriptions in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.

[0221] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and / or any other suitable types of memory.

[0222] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, in the embodiment of the device of the present invention, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0223] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0224] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0225] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A video semantic segmentation method, It is characterized in that include: Acquire a target video and identify key frame images of the target video; Using a backbone network model to extract features from the key frame image to obtain global features; Inputting the global features into a global context network model to generate context features; Determining category information of the target video, and fusing the category information with the global feature and the context feature to generate a fusion feature of the key frame image; Based on the fusion features, using a semantic segmentation model to generate an image semantic segmentation result of the key frame image; The image semantic segmentation results of all the key frame images of the target video are integrated to generate a video semantic segmentation result of the target video.

2. The video semantic segmentation method according to claim 1, It is characterized in that The step of acquiring a target video and identifying a key frame image of the target video includes: Get the target video uploaded by the target user; Using a key frame detection algorithm to identify the target video and generate an identification result; Based on the recognition result, a key frame image is extracted from the target video.

3. The video semantic segmentation method according to claim 1, It is characterized in that The step of extracting features from the key frame image using the backbone network model to obtain global features includes: Pre-build the backbone network model; Preprocessing the key frame image; Inputting the preprocessed key frame image into the backbone network model to extract features and obtain initial features; The initial features are normalized to obtain global features of the key frame image.

4. The video semantic segmentation method according to claim 1, It is characterized in that The step of inputting the global features into a global context network model to generate context features includes: Pre-build a global context network model; Inputting the global features into the global context network model to generate preliminary features; The preliminary features are normalized to obtain context features of the key frame image.

5. The video semantic segmentation method according to claim 1, It is characterized in that The determining the category information of the target video and fusing the category information with the global feature and the context feature to generate the fused feature of the key frame image includes: Based on the preset category label of the target video, obtain category information of the target video, and vectorize the category information into a category embedding vector; Fusion the category embedding vector with the global feature and the context feature to generate a primary feature; The primary features are normalized to obtain fusion features of the key frame image.

6. The video semantic segmentation method according to claim 1, It is characterized in that The step of generating the image semantic segmentation result of the key frame image by using a semantic segmentation model based on the fusion feature includes: Pre-built semantic segmentation models; Inputting the fused features into the semantic segmentation model to generate an initial semantic segmentation result; The initial semantic segmentation result is optimized using a conditional random field to obtain an image semantic segmentation result of the key frame image.

7. The video semantic segmentation method according to any one of claims 1 to 6, It is characterized in that The step of integrating the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video includes: Collect the image semantic segmentation results of all the key frame images of the target video, and record a timestamp for the image semantic segmentation result corresponding to each key frame image according to the time node at which each key frame image appears in the target video; According to the timestamps of the semantic segmentation results of each image, sequentially integrating the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video; The video semantic segmentation result is formatted, and the formatted video semantic segmentation result is output or displayed.

8. A video semantic segmentation device, It is characterized in that include: An acquisition and recognition module, used to acquire a target video and recognize key frame images of the target video; A feature extraction module, used to extract features from the key frame image using a backbone network model to obtain global features; A feature input module, used for inputting the global features into a global context network model to generate context features; A feature fusion module, used to determine the category information of the target video, and fuse the category information with the global feature and the context feature to generate a fusion feature of the key frame image; A first result generating module, configured to generate an image semantic segmentation result of the key frame image based on the fusion feature and using a semantic segmentation model; The second result generating module is used to integrate the image semantic segmentation results of all the key frame images of the target video to generate a video semantic segmentation result of the target video.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the video semantic segmentation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the video semantic segmentation method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • A semantic segmented convolutional neural network with context information coding

    CN109101972A

  • Unmanned aerial vehicle key component real-time video semantic segmentation method based on deep learning

    CN115497027A

  • Image semantic segmentation method based on relational context aggregation

    CN115512109A

  • Video semantic segmentation method based on depth separable convolution and pyramid pooling

    CN117152436A

  • Ground feature semantic segmentation method and device based on category and edge prior information

    CN117611825A

Cited By

  • System, method and medium for detecting ophthalmic fissure image

    CN119323564A