A Video Object Segmentation Method and Apparatus Based on Semantic Understanding and Self-Perception

Abstract

This invention discloses a video target segmentation method and apparatus based on semantic understanding and self-awareness. The method includes: extracting the image of the current frame of a video using a preset encoder to obtain shallow and deep features; establishing a dense matching relationship between the deep features and feature memory, then enhancing the features using self-awareness relationships, and obtaining the fused feature matching result after feature fusion; forming semantic relationships by densely matching the deep features to obtain a deep semantic output sequence; fusing and decoding the shallow features and the deep semantic output sequence using a semantic decoder to obtain multi-scale semantic features; and finally inputting the fused feature matching result, multi-scale semantic features, and shallow features into a segmentation decoder to obtain the segmentation result. This invention improves target segmentation performance in complex scenes, especially scenes with significant appearance changes, while realistically restoring the edge details of the target and improving the accuracy of segmentation mask boundaries.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision, specifically relating to a video target segmentation method and apparatus based on semantic understanding and self-perception. Background Technology

[0002] With the continuous development of computer vision and image processing, video object segmentation has become an important task that has been widely researched and applied. It aims to accurately segment objects of interest from video sequences, which is of great value to video editing, computer vision applications, and artificial intelligence. However, traditional video object segmentation methods perform poorly in addressing these challenges due to the potential for motion, morphological and appearance changes in video objects, as well as complex backgrounds. Currently, existing video object segmentation methods are mainly divided into two categories: detection-based methods and matching-based methods. Detection-based methods attempt to achieve segmentation by training an object detector, but their performance is limited by the accuracy of the detector, and they perform poorly for objects with significant appearance changes. Matching-based methods achieve segmentation by matching features between the current frame and historical frames. This method is more robust to object motion, but it faces challenges in handling complex backgrounds and semantic understanding, and its segmentation accuracy needs improvement. In summary, existing video object segmentation methods have certain limitations when dealing with fast-moving objects, objects with significant morphological and appearance changes, and complex backgrounds. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a video target segmentation method and apparatus based on semantic understanding and self-perception, which improves the target segmentation performance in complex scenes, especially scenes with large appearance changes, and can realistically restore the edge details of the target and improve the accuracy of the segmentation mask boundary.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a video target segmentation method based on semantic understanding and self-awareness, comprising the following steps:

[0006] The image of the current frame of the video is extracted using a preset encoder to obtain shallow and deep features;

[0007] Deep features are embedded and feature maps are extracted to form feature sequences. Then, dense matching relationships are established between the feature sequences and feature memories to obtain feature matching results. Next, the feature matching results are enhanced using self-perceptive relationships to obtain enhanced feature matching results. Finally, the feature matching results and enhanced feature matching results are fused to obtain fused feature matching results.

[0008] A semantic-aware model is used to extract semantic-aware features from deep features. The semantic-aware model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame of the video image. After multiple enhancements, semantic-aware features are obtained. The semantic decoder is used to fuse and decode the semantic-aware features and shallow features to obtain multi-scale semantic features.

[0009] The fused feature matching results, multi-scale semantic features, and shallow features are input into the segmentation decoder to obtain the target mask.

[0010] As a preferred technical solution, the step of embedding deep features and extracting feature maps to form feature sequences specifically includes:

[0011] A 3×3 sliding window is used to capture local features in the global scope of the feature map obtained by convolution, and the features within the sliding window are expanded into 1×9 scalars as elements of the sequence, so that different elements of the sequence represent different local features, forming a feature sequence.

[0012] As a preferred technical solution, establishing a dense matching relationship between the feature sequence and the feature memory specifically involves:

[0013] A dense matching between each element in the feature sequence and all elements in the feature memory is constructed using a cross-attention mechanism, thereby establishing a dense relationship between the current frame image and historical frame images; the cross-attention mechanism is as follows:

[0014]

[0015] Among them, Q t K represents the depth features of the image in the t-th frame of the video sequence. mem and V mem Let d represent the set of key-value pairs (Keys) and the set of value pairs (Values) from the feature memory, respectively. k The key represents the dimension of the key value. KeyProj represents the convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. (KeyProj(Q) t ),eys) represents the similarity between the query value of the t-th frame of the video and the key value in the feature memory.

[0016] As a preferred technical solution, the feature memory includes a key-value set (keys) and a value set (values); the key-value set (keys) and the value set (values) are obtained by sampling multiple frames of historical video frames, specifically:

[0017] keys = {KeyProj(Q t ),=0,5,…}

[0018] values={ValueEncoder(KeyProj(Q t ), t ),t=0,5,…}

[0019] Among them, Q t This represents the depth features of the image in the t-th frame of the video sequence. KeyProj represents the convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. M t This represents the segmentation mask corresponding to the t-th frame of the video sequence, and ValueEncoder represents the encoder that fuses the segmentation mask with the feature embedding sequence.

[0020] As a preferred technical solution, the enhancement of feature matching results using self-perceived relationships specifically includes:

[0021] A global attention mechanism is used to construct dense relationships between features in the feature sequence. The feature matching results are then input into the global attention mechanism as numerical values ​​to obtain enhanced feature matching results, as shown in the following formula:

[0022]

[0023] Among them, Q t d represents the depth features of the image in the t-th frame of the video sequence. k Representing the dimension of the key value, KeyProj() indicates a convolutional operation that processes the acquired feature map into a feature sequence to perform feature embedding. R mem This represents the feature matching result output by the cross-attention mechanism, while Similarity() calculates the similarity between the query value of the t-th frame of the video and the key value in the feature memory.

[0024] As a preferred technical solution, a feature fusion module is used to fuse the feature matching results and the enhanced feature matching results. The fusion module is designed based on the convolutional attention module and the ResBlock module.

[0025] As a preferred technical solution, the Transformer Encoder is multi-layered. By stacking multiple Transformer Encoders, semantically aware features and deep semantic output sequences are converted into hidden representations, thereby improving the ability to capture complex relationships and semantic information in semantically aware features and deep semantic output sequences.

[0026] As a preferred technical solution, the structure of the segmentation decoder is as follows:

[0027] O i =concat(unsample(O i-1 )+B i ,P i ),O0=R self

[0028] Among them, O i B represents the segmentation feature output by the i-th layer decoding module. i The shallow features represented by P are the encoder output. i This represents the semantic features at the i-th scale.

[0029] Secondly, the present invention also provides a video target segmentation system based on semantic understanding and self-awareness, applied to the aforementioned video target segmentation method based on semantic understanding and self-awareness, including an encoding module, a first feature processing module, a second feature processing module, and a decoding module;

[0030] The encoding module is used to extract the image of the current frame of the video using a preset encoder, and to obtain shallow and deep features;

[0031] The first feature processing module is used to embed deep features and extract feature maps to form a first feature sequence. Then, the first feature sequence is used to establish a dense matching relationship with the feature memory to obtain the feature matching result. Next, the feature matching result is enhanced using a self-perceptive relationship to obtain an enhanced feature matching result. The feature matching result and the enhanced feature matching result are then fused to obtain a fused feature matching result.

[0032] The second feature processing module is used to extract semantic-aware features from deep features using a semantic-aware model. The semantic-aware model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame image of the video, and obtains semantic-aware features after multiple enhancements. The semantic decoder is used to fuse and decode the semantic-aware features and shallow features to obtain multi-scale semantic features.

[0033] The decoding module is used to input the fused feature matching results, multi-scale semantic features, and shallow features into the segmentation decoder to obtain the target mask.

[0034] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0035] At least one processor; and,

[0036] A memory communicatively connected to the at least one processor; wherein,

[0037] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the video object segmentation method based on semantic understanding and self-awareness.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] (1) This invention uses self-perception relationship to enhance feature matching results and defines a self-perception matching mechanism. By utilizing the similarity of the target's own features, the feature memory matching module can be enhanced, enabling the module to extract more effective mask features from the feature memory, thereby improving the segmentation effect.

[0040] (2) This invention uses a semantic perception model to establish semantic relationships for deep features, generates semantic perception features, defines a semantic branch to extract semantic understanding from the query frame, provides accurate semantic guidance for reading and decoding, greatly enhances the expression of semantic objects, eliminates interference from non-semantic noise, and improves the robustness of the segmenter. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a video target segmentation method based on semantic understanding and self-perception according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the enhanced feature matching in an embodiment of the present invention;

[0044] Figure 3 This is a network diagram based on semantic understanding and self-awareness in an embodiment of the present invention;

[0045] Figure 4 This is a network diagram based on semantic understanding and self-awareness in an embodiment of the present invention;

[0046] Figure 5 This is a comparison chart of the segmentation results of the embodiments of the present invention and several advanced video object segmentation algorithms;

[0047] Figure 6 This is a schematic diagram of the structure of the video target segmentation system based on semantic understanding and self-perception according to an embodiment of the present invention;

[0048] Figure 7 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0050] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0051] Please see Figure 1 This embodiment proposes a video target segmentation method based on semantic understanding and self-awareness, which includes the following steps:

[0052] S1. Use a preset encoder to extract the image of the current frame of the video and obtain shallow and deep features;

[0053] This embodiment uses a ResBlock residual module and a downsampled ResNet-50 network as the encoder. When the current frame of the input video is used, the encoder output features are divided into shallow features and deep features according to the number of network layers in the encoder.

[0054] S2. Deep features are embedded and feature maps are extracted to form feature sequences. Then, dense matching relationships are established between the feature sequences and feature memories to obtain feature matching results. Next, the feature matching results are enhanced using self-perceptive relationships to obtain enhanced feature matching results. The feature matching results and enhanced feature matching results are fused to obtain fused feature matching results.

[0055] In step S2, the process of embedding deep features and extracting feature maps to form a feature sequence involves: using a 3×3 sliding window to capture local features within the global scope of the feature map obtained by convolution; and expanding the features within the sliding window into 1×9 scalars as elements of the sequence, so that different elements of the sequence represent different local features, thus forming a feature sequence. This allows different elements of the sequence to represent different local features, enabling a dense matching relationship to be established with the subsequent set of feature memories.

[0056] Therefore, to compensate for the limitations of the fixed receptive field of convolution, this embodiment uses a cross-attention mechanism to construct a dense match between each element in the feature sequence and all elements in the feature memory, thereby building a dense relationship between the current frame image and historical frame images, such as... Figure 2 As shown. Specifically, the cross-attention mechanism is as follows:

[0057]

[0058] Among them, Q t K represents the depth features of the image in the t-th frame of the video sequence. mem and V mem Let d represent the set of key-value pairs (Keys) and the set of value pairs (Values) from the feature memory, respectively. k The key represents the dimension of the key value. KeyProj represents the convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. (KeyProj(Q) t ),eys) represents the similarity between the query value of the t-th frame of the video and the key value in the feature memory, thereby constructing a dense association between each element in sequence Q and the elements in the feature memory keys, and performing matrix multiplication with the corresponding feature memory values.

[0059] It should also be noted that in this embodiment, Similarity() represents the calculation of the similarity between the query value of the t-th frame of the video and the key value in the feature memory. It uses negative squared Euclidean distance similarity, which is better at measuring the similarity between deep features and feature memory in the field of video object segmentation than Manhattan distance similarity.

[0060] Furthermore, the feature memory in this embodiment includes a key-value set (keys) and a value set (values). The key-value set and the value set are obtained by sampling every 5 frames of the video's historical frames, specifically:

[0061] keys = {KeyProj(Q t ),=0,5,…}

[0062] values={ValueEncoder(KeyProj(Q t ), t ),t=0,5,…}

[0063] Among them, Q t This represents the depth features of the image in the t-th frame of the video sequence. KeyProj represents the convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. Mt This represents the segmentation mask corresponding to the t-th frame of the video sequence, and ValueEncoder represents the encoder that fuses the segmentation mask with the feature embedding sequence.

[0064] Subsequently, step S2 defines a self-aware matching mechanism aimed at extracting more effective mask features from the feature memory, thereby improving the segmentation effect. For example... Figure 3 As shown, the enhanced feature matching branch consists of a cross-attention mechanism and a self-aware matching mechanism, with the self-aware matching mechanism enhancing the matching results. When using the self-aware matching mechanism, dependencies within the query frame are constructed. These self-aware relationships enhance the feature matching results output by the cross-attention mechanism, utilizing the image's own feature relationships to construct dense matching between elements within the feature sequence. This compensates for the problem of insufficient feature memory, preventing the acquisition of effective and robust feature matching, thus yielding enhanced feature matching results. For more details, see again... Figure 2 The enhancement of feature matching results using self-perceived relationships specifically includes:

[0065] A global attention mechanism is used to construct dense relationships between features in the feature sequence. The feature matching results are then input into the global attention mechanism as numerical values ​​to obtain enhanced feature matching results, as shown in the following formula:

[0066]

[0067] Among them, Q t d represents the depth features of the image in the t-th frame of the video sequence. k Representing the dimension of the key value, KeyProj() indicates a convolutional operation that processes the acquired feature map into a feature sequence to perform feature embedding. R mem This represents the feature matching result output by the cross-attention mechanism.

[0068] Notice, Figure 3 middle "" indicates matrix multiplication of two matrices. In this embodiment, the establishment of dense relationships is achieved by performing matrix multiplication with the corresponding feature memory values.

[0069] Finally, in order to further enhance the feature representation of the video image, step S2 uses convolutional attention and ResBlock as feature fusion modules during the fusion process, thereby fusing the feature matching result and the enhanced feature matching result to obtain the fused feature matching result.

[0070] S3. Extract semantically perceptual features from deep features using a semantically perceptual model; the semantically perceptual model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame of the video image, and semantically perceptual features are obtained after multiple enhancements; the semantic decoder is used to fuse and decode the semantically perceptual features and shallow features to obtain multi-scale semantic features; the processing procedure is as follows. Figure 4 As shown.

[0071] In addition, see also Figure 3 The semantic perception branch in the middle and Figure 4 This semantic-aware branch structure includes a Transformer Encoder and a semantic decoder, which are used to obtain multi-scale semantic features of the current frame of the video image and guide the segmentation and decoding module to complete a more accurate segmentation result.

[0072] In this embodiment, the Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame of the video image. This is achieved by fusing the semantic decoder with the shallow features to obtain multi-scale semantic features. Next, this embodiment uses a stacked layer of Transformer Encoders to convert the semantically aware features and the deep semantic output sequence into hidden representations. This ensures better capture of the complex relationships and semantic information in the semantically aware features and the deep semantic output sequence, and improves the model's generalization and representation capabilities. Furthermore, since excessive use of the Transformer Encoder can increase model complexity and computational cost, after multiple experiments, this embodiment preferably uses a 3-layer Transformer Encoder as the semantically aware model to establish semantic relationships in the deep features and extract semantic features from them.

[0073] In addition, this embodiment also includes a segmentation decoder VOSDecoder, which processes the fused feature matching results, multi-scale semantic features and shallow features, and segments them into smaller data packets according to specific rules, thereby obtaining segmentation mask results for multiple targets, as in step S4.

[0074] S4. Input the fused feature matching results, multi-scale semantic features and shallow features into the segmentation decoder to obtain the target mask.

[0075] In this embodiment, both the segmentation decoder and the semantic decoder are implemented based on ResBlock and upsampling. The segmentation decoder structure is defined as follows:

[0076] O i=concat(unsample(O i-1 )+B i ,Pi),O0=R self

[0077] Among them, O i B represents the segmentation feature output by the i-th layer decoding module. i The shallow features represented by P are the encoder output. i This represents the semantic features at the i-th scale.

[0078] In practice, the segmentation decoder needs to be pre-trained, specifically through two stages on an RTX3090. The first stage uses the CoCo still image dataset with an initial stride of 0.00001 and 150,000 epochs. The second stage uses a hybrid dataset consisting of the DAVIS and YouTube-VOS datasets, with an initial stride of 0.000005 and 220,000 epochs. After the second stage of training, the prediction results are visible. Figure 5 See Table 1.

[0079] After completing the two phases of training, the trained segmentation decoder is put into use again. In this embodiment, after inputting the fused feature matching result, multi-scale semantic features, and shallow features into the segmentation decoder, a prediction result is generated. Then, the fused feature matching result, multi-scale semantic features, and shallow features are input into the segmentation decoder again, and this process is repeated several times to obtain the segmentation mask results of multiple targets, i.e., the target mask.

[0080] Please see Figure 5 This demonstrates the subjective analysis results of the method of the present invention. For example... Figure 5 As shown, the method provided in this embodiment can better handle target segmentation in complex situations, especially in motion-blurred scenarios where target features are difficult to obtain and vary greatly. In the case of multiple targets, it can better distinguish each target, and the target segmentation boundary is clearer and more defined.

[0081] The objective evaluation results of the method provided in this embodiment are shown in Table 1. This invention uses region accuracy (J), boundary accuracy (F), and average accuracy (J&F) as evaluation indicators. The higher the value, the closer the generated segmentation mask is to the actual segmentation mask. Among them, the average accuracy is the average of the region accuracy and the boundary accuracy.

[0082] Table 1

[0083]

[0084] As shown in Table 1, the prediction results on the DAVIS2016 test dataset indicate that, for boundary accuracy, the method using the SWEM network is equal to the present invention in all other aspects, but lower in all others. However, for region accuracy and average accuracy, the present invention outperforms the other methods with scores of 90.4 and 91.5, respectively. Furthermore, the prediction results on the DAVIS2017 test dataset demonstrate that the method of the present invention exhibits outstanding performance.

[0085] Based on the above subjective and objective comparisons, experiments conducted on challenging benchmark sets (DAVIS2016, DAVIS2017, YouTube-VOS) demonstrate that the semantic understanding and self-awareness-based video object segmentation method provided by this invention outperforms most state-of-the-art video object segmentation methods and achieves leading performance.

[0086] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0087] Based on the same ideas as the video target segmentation method based on semantic understanding and self-awareness in the above embodiments, the present invention also provides a video target segmentation system based on semantic understanding and self-awareness, which can be used to execute the above-described video target segmentation method based on semantic understanding and self-awareness. For ease of explanation, the structural diagram of the embodiment of the video target segmentation system based on semantic understanding and self-awareness only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0088] Please see Figure 6 In another embodiment of this application, a video target segmentation system 10 based on semantic understanding and self-awareness is provided. The system includes an encoding module 11, a first feature processing module 12, a second feature processing module 13, and a decoding module 14.

[0089] Encoding module 11 is used to extract the image of the current frame of the video using a preset encoder and obtain shallow and deep features;

[0090] The first feature processing module 12 is used to embed deep features and extract feature maps to form feature sequences. Then, it establishes a dense matching relationship between the feature sequences and feature memory to obtain feature matching results. Next, it uses self-perceptive relationship to enhance the feature matching results to obtain enhanced feature matching results. Finally, it fuses the feature matching results and the enhanced feature matching results to obtain fused feature matching results.

[0091] The second feature processing module 13 is used to extract semantic-aware features from deep features using a semantic-aware model. The semantic-aware model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame image of the video, and obtains semantic-aware features after multiple enhancements. The semantic decoder is used to fuse and decode the semantic-aware features and shallow features to obtain multi-scale semantic features.

[0092] The decoding module 14 is used to input the fused feature matching results, multi-scale semantic features and shallow features into the segmentation decoder to obtain the target mask.

[0093] It should be noted that the video target segmentation system based on semantic understanding and self-awareness of the present invention corresponds one-to-one with the video target segmentation method based on semantic understanding and self-awareness of the present invention. The technical features and beneficial effects described in the embodiments of the video target segmentation method based on semantic understanding and self-awareness described above are applicable to the embodiments of the video target segmentation method based on semantic understanding and self-awareness. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0094] Furthermore, in the implementation of the video target segmentation system based on semantic understanding and self-awareness in the above embodiments, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the video target segmentation system based on semantic understanding and self-awareness can be divided into different program modules to complete all or part of the functions described above.

[0095] Please see Figure 7 In one embodiment, an electronic device is provided for implementing a video target segmentation method based on semantic understanding and self-awareness. The electronic device 20 may include a first processor 21, a first memory 22 and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a video target segmentation program 23 based on semantic understanding and self-awareness.

[0096] The first memory 22 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as the portable hard drive of the electronic device 20. In other embodiments, the first memory 22 can be an external storage device of the electronic device 20, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 20. Furthermore, the first memory 22 can include both internal storage units and external storage devices of the electronic device 20. The first memory 22 can be used not only to store application software and various types of data installed on the electronic device 20, such as the code of a video target segmentation program 23 based on semantic understanding and self-perception, but also to temporarily store data that has been output or will be output.

[0097] In some embodiments, the first processor 21 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 21 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 22 and calls data stored in the first memory 22 to perform various functions of the electronic device 20 and process data.

[0098] Figure 7 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 7 The structure shown does not constitute a limitation on the electronic device 20, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0099] The video target segmentation program 23 based on semantic understanding and self-awareness stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When run in the first processor 21, it can achieve the following:

[0100] The image of the current frame of the video is extracted using a preset encoder to obtain shallow and deep features;

[0101] Deep features are embedded and feature maps are extracted to form feature sequences. Then, dense matching relationships are established between the feature sequences and feature memories to obtain feature matching results. Next, the feature matching results are enhanced using self-perceptive relationships to obtain enhanced feature matching results. Finally, the feature matching results and enhanced feature matching results are fused to obtain fused feature matching results.

[0102] A semantic-aware model is used to extract semantic-aware features from deep features. The semantic-aware model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame of the video image. After multiple enhancements, semantic-aware features are obtained. The semantic decoder is used to fuse and decode the semantic-aware features and shallow features to obtain multi-scale semantic features.

[0103] The fused feature matching results, multi-scale semantic features, and shallow features are input into the segmentation decoder to obtain the target mask.

[0104] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0105] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A video target segmentation method based on semantic understanding and self-perception, characterized in that, Includes the following steps: The image of the current frame of the video is extracted using a preset encoder to obtain shallow and deep features; Deep features are embedded and feature maps are extracted to form feature sequences. Then, dense matching relationships are established between the feature sequences and feature memories to obtain feature matching results. Next, self-perceptual relationships are used to enhance the feature matching results to obtain enhanced feature matching results. The feature matching results and the enhanced feature matching results are fused to obtain the fused feature matching results. Semantic-aware features are extracted from deep features using a semantic-aware model; The semantic awareness model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame image of the video, and obtains semantic awareness features after multiple enhancements. A semantic decoder is used to fuse and decode semantically perceived features and shallow features to obtain multi-scale semantic features; The fused feature matching results, multi-scale semantic features, and shallow features are input into the segmentation decoder to obtain the target mask. The process of establishing a dense matching relationship between the feature sequence and the feature memory specifically involves: A dense matching between each element in the feature sequence and all elements in the feature memory is constructed using a cross-attention mechanism, thereby establishing a dense relationship between the current frame image and historical frame images; the cross-attention mechanism is as follows: in, This represents the depth features of the image in the t-th frame of the video sequence. and These represent the key-value sets from the feature memory. and the numerical set of feature memory , The dimension representing the key value. This refers to the convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. This represents the query value for retrieving the image of the t-th frame of a video and the key value in the feature memory. The similarity between them; The enhancement of feature matching results using self-perceptive relationships specifically involves: A global attention mechanism is used to construct dense relationships between features in the feature sequence. The feature matching results are then input into the global attention mechanism as numerical values ​​to obtain enhanced feature matching results, as shown in the following formula: in, This represents the depth features of the image in the t-th frame of the video sequence. The dimension representing the key value. () indicates a convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. This refers to the cross-attention mechanism. Output feature matching results () indicates the similarity calculation between the query value of the t-th frame of the video and the key value in the feature memory.

2. The video target segmentation method based on semantic understanding and self-perception according to claim 1, characterized in that, The process of embedding deep features and extracting feature maps to form feature sequences specifically involves: A 3×3 sliding window is used to capture local features in the global scope of the feature map obtained by convolution, and the features within the sliding window are expanded into 1×9 scalars as elements of the sequence, so that different elements of the sequence represent different local features, forming a feature sequence.

3. The video target segmentation method based on semantic understanding and self-perception according to claim 1, characterized in that, The feature memory includes a set of key values. sum of numerical sets The key-value set sum of numerical sets It is obtained by sampling multiple frames of historical video frames, specifically: in, This represents the depth features of the image in the t-th frame of the video sequence. This refers to the convolution operation that processes the acquired feature map into a feature sequence to achieve feature embedding. This represents the segmentation mask corresponding to the t-th frame of the video sequence. This refers to an encoder that fuses segmentation masks with feature embedding sequences.

4. The video target segmentation method based on semantic understanding and self-perception according to claim 1, characterized in that, A feature fusion module is used to fuse the feature matching results and the enhanced feature matching results. The feature fusion module is designed based on the convolutional attention module and the ResBlock module.

5. The video target segmentation method based on semantic understanding and self-perception according to claim 1, characterized in that, The Transformer Encoder is multi-layered. By stacking multiple Transformer Encoders, semantically aware features and deep semantic output sequences are converted into hidden representations, thereby improving the ability to capture complex relationships and semantic information in semantically aware features and deep semantic output sequences.

6. The video target segmentation method based on semantic understanding and self-perception according to claim 1, characterized in that, The structure of the segmentation decoder is as follows: in, This represents the segmentation features output by the i-th layer decoding module. The shallow features represented by the encoder output. This represents the semantic features at the i-th scale.

7. A video target segmentation system based on semantic understanding and self-perception, characterized in that, The video target segmentation method based on semantic understanding and self-awareness, applied to any one of claims 1-6, includes an encoding module, a first feature processing module, a second feature processing module, and a decoding module; The encoding module is used to extract the image of the current frame of the video using a preset encoder, and to obtain shallow and deep features; The first feature processing module is used to embed deep features and extract feature maps to form a first feature sequence. Then, the first feature sequence is used to establish a dense matching relationship with the feature memory to obtain the feature matching result. Next, the feature matching result is enhanced using a self-perceptive relationship to obtain an enhanced feature matching result. The feature matching results and the enhanced feature matching results are fused to obtain the fused feature matching results. The second feature processing module is used to extract semantically aware features from deep features using a semantically aware model. The semantic awareness model includes a Transformer Encoder and a semantic decoder. The Transformer Encoder is used to establish semantic relationships between elements in the deep features of the current frame image of the video, and obtains semantic awareness features after multiple enhancements. A semantic decoder is used to fuse and decode semantically perceived features and shallow features to obtain multi-scale semantic features; The decoding module is used to input the fused feature matching results, multi-scale semantic features, and shallow features into the segmentation decoder to obtain the target mask.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the video object segmentation method based on semantic understanding and self-awareness as described in any one of claims 1-6.

Images