Image segmentation method and device based on semantic features, electronic equipment and medium

Abstract

The embodiment of the invention provides an image segmentation method and device based on semantic features, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: extracting video frame image features from each video frame of a target video containing a target object, and extracting description semantic features from a target description text describing the target object; and coding according to all video frame image features and description semantic features to obtain global joint features. And for each video frame, performing mask processing on the video frame according to the video frame image feature and the global joint feature to obtain a semantic mask region where the target object is located. And performing region propagation prediction on the target object according to all the video frames to obtain a target prediction region of each video frame. And generating a target positioning region according to at least one of the semantic mask region and the target prediction region, and performing image segmentation on the video frame based on the target positioning region. According to the embodiment of the invention, the reliability of video segmentation can be improved.

Description

Image segmentation methods and devices based on semantic features, electronic devices and media Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image segmentation method and apparatus, electronic device and medium based on semantic features. Background Technology

[0002] In related technologies, users typically provide a pre-trained image segmentation model with a video containing a target object and natural language text describing that object. The image segmentation model then aligns the visual information in the video frames with the descriptive text to identify and segment the target in the video frames, ultimately generating a segmentation result. However, when dealing with long video segmentation, because the segmentation of video frames relies solely on individual visual features and semantic matching, lacking global consistency and temporal continuity, the model is prone to incorrect target segmentation. This error is inherited in subsequent frame propagation processes, resulting in low reliability of the final video segmentation result.

[0003] Therefore, improving the reliability of video segmentation has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose an image segmentation method, apparatus, electronic device, and medium based on semantic features, aiming to improve the reliability of video segmentation.

[0005] To achieve the above objectives, a first aspect of this application proposes an image segmentation method based on semantic features. The method includes: acquiring a target video containing a target object and target descriptive text describing the target object; extracting image features from each video frame of the target video to obtain video frame image features, and extracting text features from the target descriptive text to obtain descriptive semantic features; jointly encoding all the video frame image features and the descriptive semantic features to obtain global joint features; for each video frame of the target video, performing masking processing on the video frame based on the video frame image features and the global joint features to obtain a semantic mask region where the target object is located; performing region propagation prediction on the target object based on all the video frames to obtain a target prediction region where the target object is located in each video frame; generating a target localization region based on at least one of the semantic mask region and the target prediction region, and performing image segmentation on the video frame based on the target localization region.

[0006] In some embodiments, generating a target localization region based on at least one of the semantic mask region and the target prediction region includes: performing semantic similarity evaluation based on the semantic mask region and the target prediction region of the same video frame to obtain a semantic consistency score; performing temporal continuity evaluation based on the target prediction regions of two adjacent video frames to obtain a temporal consistency score; using the semantic mask region as a first candidate region based on the semantic consistency score and the temporal consistency score, and using the target prediction region as a second candidate region based on the semantic consistency score and the temporal consistency score; when only the first candidate region exists, determining the first candidate region as the target localization region; when both the first and second candidate regions exist, performing region fusion based on the first and second candidate regions to obtain the target localization region.

[0007] In some embodiments, the step of selecting the semantic mask region as a first candidate region based on the semantic consistency score and the temporal consistency score, and selecting the target predicted region as a second candidate region based on the semantic consistency score and the temporal consistency score, includes: obtaining a temporal score threshold and a semantic score threshold; selecting the semantic mask region as a first candidate region when the temporal consistency score is less than the temporal score threshold and the semantic consistency score is greater than or equal to the semantic score threshold, or when the temporal consistency score is greater than or equal to the temporal score threshold and the semantic consistency score is less than the semantic score threshold; and selecting the target predicted region as a second candidate region when the temporal consistency score is greater than or equal to the temporal score threshold and the semantic consistency score is less than the semantic score threshold.

[0008] In some embodiments, the step of performing region fusion based on the first candidate region and the second candidate region to obtain the target localization region includes: obtaining a semantic lower bound confidence value and a semantic score threshold; calculating weights based on the semantic lower bound confidence value, the semantic score threshold, and the semantic consistency score to obtain weight data; and performing image fusion on the semantic mask region and the target prediction region based on the weight data to obtain the target localization region.

[0009] In some embodiments, the step of masking each video frame of the target video based on the video frame image features and the global joint features to obtain the semantic mask region where the target object is located includes: generating a global query matrix based on the global joint features; generating an original key matrix and an original value matrix based on the image features of each video frame; performing attention fusion based on the global query matrix, the original key matrix, and the original value matrix to obtain the current frame joint features of the video frame; and generating a mask by using a preset mask segmentation model to obtain the semantic mask region.

[0010] In some embodiments, the step of performing attention fusion based on the global query matrix, the original key matrix, and the original value matrix to obtain the current frame joint feature of the video frame includes: performing cross-attention calculation based on the global query matrix, the original key matrix, and the original value matrix to obtain intermediate joint features; integrating the intermediate joint features and the global joint features to obtain intermediate input features; updating the global query matrix, the original key matrix, and the original value matrix based on the intermediate input features, and returning to the step of performing cross-attention calculation based on the global query matrix, the original key matrix, and the original value matrix to obtain intermediate joint features, until the number of returns reaches a preset number, and using the last intermediate joint feature as the current frame joint feature.

[0011] In some embodiments, the step of jointly encoding the image features of all video frames and the descriptive semantic features to obtain global joint features includes: generating a semantic query matrix based on the descriptive semantic features, and generating an original key matrix and an original value matrix based on each video frame image feature; performing attention transformation on the semantic query matrix, the original value matrix, and the original key matrix to obtain a reference query matrix; generating a reference key matrix and a reference value matrix based on the reference query matrix, and performing attention calculation on the reference key matrix, the reference value matrix, and a preset base query matrix to obtain candidate joint features; performing reliability evaluation on all candidate joint features to obtain a reliability score; and selecting the candidate joint feature corresponding to the highest reliability score as the global joint feature.

[0012] To achieve the above objectives, a second aspect of this application proposes an image segmentation apparatus based on semantic features. The apparatus includes: an acquisition module for acquiring a target video containing a target object and target descriptive text describing the target object; a feature extraction module for extracting image features from each video frame of the target video to obtain video frame image features, and extracting text features from the target descriptive text to obtain descriptive semantic features; a joint encoding module for performing joint encoding based on all the video frame image features and the descriptive semantic features to obtain global joint features; a masking module for performing masking processing on each video frame of the target video based on the video frame image features and the global joint features to obtain a semantic mask region where the target object is located; a propagation prediction module for performing region propagation prediction on the target object based on all the video frames to obtain a target prediction region where the target object is located in each video frame; and an image segmentation module for generating a target localization region based on at least one of the semantic mask region and the target prediction region, and performing image segmentation on the video frame based on the target localization region.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0015] This application proposes a semantic feature-based image segmentation method, apparatus, electronic device, and medium. It generates a global joint feature by jointly encoding the semantic features of the target description text with the features of each video image frame. Based on this feature, semantic masking is performed to obtain a semantic mask region aligned with the semantic information, where the target object is located. Subsequently, region propagation prediction is performed on all video frames to obtain the target prediction region in each video frame, ensuring temporal consistency and coherence across multiple frames in a long video. Finally, by combining the semantic mask region and the target prediction region, a target localization region is generated, further improving the accuracy of target image segmentation. The method of this application effectively overcomes the segmentation errors caused by the lack of global consistency and temporal continuity in traditional techniques, significantly improving the reliability and robustness of image segmentation results in long video scenes. Attached Figure Description

[0016] Figure 1 is a flowchart of the image segmentation method based on semantic features provided in the embodiments of this application; Figure 2 is a flowchart of step S103 in Figure 1; Figure 3 is a flowchart of step S104 in Figure 1; Figure 4 is a flowchart of step S303 in Figure 3; Figure 5 is a flowchart of step S106 in Figure 1; Figure 6 is a flowchart of step S505 in Figure 5; Figure 7 is a comparison diagram of the effects of applying existing methods and the image segmentation method based on semantic features provided in the embodiments of this application; Figure 8 is a structural schematic diagram of the image segmentation device based on semantic features provided in the embodiments of this application; Figure 9 is a hardware structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In related technologies, users typically provide a video containing the target object and natural language text describing the target to a pre-trained image segmentation model. The image segmentation model then aligns the visual information in the video frames with the descriptive text to identify and segment the target in the video frames, ultimately generating a segmentation result. Specifically, the steps for target image segmentation of long videos in existing technologies are as follows: 1. First, slice the long video to obtain multiple shorter sub-videos; 2. Run a natural language-driven video target segmentation model to perform segmentation masking on each frame within each sub-video based on the descriptive text; 3. Based on the regional integrity of these segmentation masks or their matching degree with the semantic description, select the frame with the highest confidence score as the starting frame for subsequent propagation; 4. Call mask propagation or pixel-level tracking methods to propagate the segmentation result from the starting frame to the entire long video, i.e., each frame obtains the mask region corresponding to the target object, and finally, image segmentation is performed based on the mask region. However, when dealing with long video segmentation, because the segmentation of video frames relies solely on individual visual features and semantic matching, lacking global consistency and temporal continuity, the model is prone to incorrect target segmentation. Furthermore, this error is inherited during the propagation of subsequent frames, resulting in low reliability of the final video segmentation result. Therefore, improving the reliability of video segmentation has become an urgent technical problem to be solved.

[0021] Based on this, embodiments of this application provide an image segmentation method, apparatus, electronic device, and medium based on semantic features, aiming to improve the reliability of video segmentation.

[0022] The semantic feature-based image segmentation method, apparatus, electronic device, and medium provided in this application are specifically described through the following embodiments. First, the semantic feature-based image segmentation method in this application is described.

[0023] The image segmentation method based on semantic features provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the image segmentation method based on semantic features, but is not limited to the above forms.

[0024] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0025] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0026] Figure 1 is an optional flowchart of an image segmentation method based on semantic features provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S101 to S106.

[0027] Step S101: Obtain the target video containing the target object and the target description text describing the target object.

[0028] Step S102: Extract image features from each video frame of the target video to obtain video frame image features, and extract text features from the target description text to obtain description semantic features.

[0029] Step S103: Perform joint encoding based on the image features and descriptive semantic features of all video frames to obtain global joint features.

[0030] Step S104: For each video frame of the target video, perform masking processing on the video frame based on the video frame image features and global joint features to obtain the semantic mask region where the target object is located.

[0031] Step S105: Perform region propagation prediction on the target object based on all video frames to obtain the target prediction region where the target object is located in each video frame.

[0032] Step S106: Generate a target localization region based on at least one of the semantic mask region and the target prediction region, and perform image segmentation on the video frame based on the target localization region.

[0033] Steps S101 to S106 of this embodiment involve jointly encoding the semantic features of the target description text with the features of each video image frame to generate a global joint feature. Based on this feature, semantic masking is performed to obtain a semantic mask region aligned with the semantic information where the target object is located. Subsequently, region propagation prediction is performed on the target object in all video frames to obtain the target prediction region in each video frame, ensuring temporal consistency and coherence across multiple frames in a long video. Finally, by combining the semantic mask region and the target prediction region, a target localization region is generated, further improving the accuracy of target image segmentation. The method of this embodiment effectively overcomes the segmentation errors caused by the lack of global consistency and temporal continuity in traditional techniques, significantly improving the reliability and robustness of image segmentation results in long video scenes.

[0034] In step S101 of some embodiments, the target object refers to a specific object in the video that needs to be identified and segmented. The target video is a continuous video sequence containing the target object. The target description text is a detailed description of the target object, usually in natural language, explicitly indicating the characteristics, location, behavior, etc. of the target object. For example, if the target object is a moving car, the target description text could be "a moving car ahead," and the target video could be a traffic scene video containing streets and moving vehicles.

[0035] In step S102 of some embodiments, video frame image features refer to high-dimensional features extracted from video frames that represent video content. These features are used to describe the visual information in the video frames. In some embodiments, video frames can be processed using network models such as convolutional neural networks (CNNs) or visual encoders (ViTs) to extract visual features, including information such as edges, colors, and textures, from each frame as image features of the target video. ,in Where T is the total number of video frames in the target video, which can be understood as the video frame image features. N is the number of tokens in each frame (i.e. the number of pixels after encoding; for example, for an 80*100 image, visual encoding with a patch_size of 20 yields (80 / 20, 100 / 20)*d, which is 4*5*d features, so N=4*5=20), and d is the feature dimension.

[0036] Descriptive semantic features refer to the features extracted from the target descriptive text that represent the target information, and are typically used to express the semantic information of the target object. Text encoding models, such as BERT (Bidirectional Encoder Representations from Transformers), can be used to process the target descriptive text, extract its semantic information, and convert it into feature representations to obtain descriptive semantic features. ,in L represents the number of words in the text, and d represents the feature dimension.

[0037] In step S103 of some embodiments, global joint features refer to a fused feature obtained by combining the image features of video frames with the semantic features of the target descriptive text.

[0038] The joint encoding process can be implemented based on a multimodal large language model (MLLM) or a DETR-style model (Detection Transformer).

[0039] For the implementation process based on the DETR model (Detection Transformer), please refer to Figure 2. In some embodiments, step S103 may include, but is not limited to, steps S201 to S205: Step S201, generate a semantic query matrix based on the descriptive semantic features, and generate an original key matrix and an original value matrix based on the image features of each video frame.

[0040] Step S202: Perform attention transformation based on the semantic query matrix, the original value matrix, and the original key matrix to obtain the reference query matrix.

[0041] Step S203: Generate a reference key matrix and a reference value matrix based on the reference query matrix, and perform attention calculation based on the reference key matrix, the reference value matrix and the preset basic query matrix to obtain candidate joint features.

[0042] Step S204: Perform a reliability assessment on all candidate joint features to obtain a reliability score.

[0043] Step S205: The candidate joint feature corresponding to the reliability score with the largest value is taken as the global joint feature.

[0044] In step S201 of some embodiments, the semantic query matrix is ​​generated based on descriptive semantic features. First, a set of learnable tokens of the decoder of the Transformer model, in the form of k*d, is obtained, and then the descriptive semantic features are... Make k copies and add them to the corresponding k tokens to initialize the target query, and use them as the semantic query matrix.

[0045] For each video frame image feature, feature enhancement can be performed on it using an encoder based on the Transformer model. The original key matrix and original value matrix can be obtained by mapping the enhanced video frame image features through a linear transformation (e.g., a fully connected layer or a convolutional layer).

[0046] In step S202 of some embodiments, for each video frame image feature, cross-attention decoding is performed on the original key matrix and the original value matrix according to the semantic query matrix to output the enhanced target query, i.e., the reference query matrix.

[0047] In step S203 of some embodiments, since step S202 is performed on each video frame, the model has not learned the information of the entire target video. Therefore, an additional decoder is added, with the same decoder architecture as in step S201. The target query in the new decoder is used as the new query matrix (in the form of m*d, where m is the number of tokens of the target query in the new decoder and d is the feature dimension). A reference key matrix and a reference value matrix are generated based on the reference query matrix generated in the previous step. Cross-attention decoding is performed based on the reference key matrix, the reference value matrix, and the preset base query matrix, and finally, m*d candidate joint features are output.

[0048] In step S204 of some embodiments, the confidence score of the candidate joint features can be predicted directly by the existing mask prediction network module (SegHead module) of the DETR model, that is, the reliability assessment is realized, and the confidence score is output. The higher the value, the higher the matching degree between the candidate joint features and the target, the higher the accuracy of target localization, and the more reliable the segmentation result.

[0049] In step S205 of some embodiments, the candidate joint feature with the highest score is selected as the final global joint feature. Its form is 1*d, used to refer to "the target object to be segmented in the target description text". Steps S201 to S205 shown in the embodiments of this application, by fusing semantic information with visual information, can obtain a feature representation that combines global visual and semantic alignment, so as to ensure that the target can be stably identified and accurate segmentation results generated in long videos based on the feature representation.

[0050] The implementation process based on a multimodal large language model can be as follows: [The text then abruptly shifts to a different topic:] Video features... The semantic features are concatenated according to a preset template, such as "[Video Features]". Please input the segment [text description, such as "the blackdog"] into the video. The multimodal language model will then predict the output according to a pre-defined template (e.g., "The blackdog is..."). <seg>", in <seg>(These are pre-set special words), take <seg>The feature corresponding to the token is the aforementioned global joint feature. .

[0051] In step S104 of some embodiments, the semantic mask region refers to the region obtained after masking, where each pixel represents a target object.

[0052] Please refer to Figure 3. In some embodiments, step S104 may include, but is not limited to, steps S301 to S304: Step S301, generating a global query matrix based on global joint features.

[0053] Step S302: Generate the original key matrix and the original value matrix based on the image features of each video frame.

[0054] Step S303: Attention fusion is performed based on the global query matrix, the original key matrix, and the original value matrix to obtain the joint features of the current frame of the video frame.

[0055] Step S304: Generate a semantic mask region by using a preset mask segmentation model to combine the joint features of the current frame and the video frame.

[0056] In step S301 of some embodiments, the global joint features are input into the attention calculation model, and a global query matrix is ​​generated after calculation and mapping through the weight matrix in the model. The role of the global query matrix is ​​to provide preliminary target location information for the target segmentation task and to serve as input for subsequent attention calculation.

[0057] In step S302 of some embodiments, this embodiment is the same as the embodiment of step S201, and will not be described again here.

[0058] In step S303 of some embodiments, this embodiment sets up a 6-layer stacked cross-attention network layer structure to emphasize semantically relevant regions layer by layer.

[0059] Please refer to Figure 4. In some embodiments, step S303 may include, but is not limited to, steps S401 to S403: Step S401, perform cross-attention calculation based on the global query matrix, the original key matrix and the original value matrix to obtain intermediate joint features.

[0060] Step S402: Integrate the intermediate joint features and the global joint features to obtain the intermediate input features.

[0061] Step S403: Update the global query matrix, original key matrix, and original value matrix according to the intermediate input features, and return to the step of performing cross-attention calculation based on the global query matrix, original key matrix, and original value matrix to obtain intermediate joint features until the number of returns reaches a preset number, and use the last intermediate joint feature as the joint feature of the current frame.

[0062] In step S401 of some embodiments, the cross-attention computation can refer to the following analytical expression: (1), where, Indicates the intermediate joint feature, and , represents the semantic visual representation of the current semantic target on the current frame t. d represents the feature dimension. softmax() represents the activation function. t represents the identifier of the video frame. This indicates the number of layers across the attention network. This represents the query matrix at level l, performed for the first time in this step. This represents the global query matrix. Represents the original value matrix, Let represent the original bond matrix. D represents the feature dimension of the original bond matrix. This represents the matrix transpose operation.

[0063] In step S402 of some embodiments, residual connections and layer normalization are added after each attention layer. Feature integration can be referred to the following analytical expression: (2), where, This represents the intermediate input features of the current layer. This represents the intermediate input features of the previous layer. When this step is performed for the first time... That is, global joint features . This represents the intermediate joint feature. LayerNorm represents the layer normalization operation.

[0064] In step S403 of some embodiments, the preset number is 6. The following analytical expression can be used to update the global query matrix, the original key matrix, and the original value matrix based on the intermediate input features: (3), (4), (5), among which, This represents the query matrix for the current layer. This represents the intermediate input features of the previous layer. , and This represents the learnable weight parameters of the current attention network layer. This represents the image features of a video frame.

[0065] After updating the above matrix according to the weight parameters of the current layer, return to step S401 to execute the method, which is actually to perform layer-by-layer attention fusion of the global query matrix, the original key matrix and the original value matrix, and use the last generated intermediate joint feature as the joint feature of the current frame.

[0066] Steps S401 to S403, as illustrated in the embodiments of this application, effectively improve the accuracy and stability of target object image segmentation in long videos through the combination of cross-attention mechanisms and feature integration. By iteratively updating the global query matrix, the temporal information between video frames and the semantic information of the target description can be accurately fused, ensuring the consistency and robustness of the segmentation results.

[0067] In step S304 of some embodiments, the mask segmentation model is used to generate a semantic mask region for each frame based on the joint features of the current frame and the image features of the video frame. The mask segmentation model can be Mask R-CNN or U-Net. The image features and global joint features are input into the mask segmentation model to generate a mask matrix of the same size as the video frame, where the value of each pixel is 0 or 1, where 0 indicates that the pixel does not belong to the target and 1 indicates that the pixel belongs to the target object.

[0068] For example, assuming a car in a video frame is the target, after processing by a mask segmentation model, the generated semantic mask region will set the pixel value of each pixel in the pixel region where the car is located to 1, and set the pixel value of other background regions to 0, so as to distinguish the target from other background regions.

[0069] Steps S301 to S304, as illustrated in this embodiment, involve attention fusion based on the global query matrix and video image features to obtain accurate target features for the video frame. Subsequently, a precise semantic mask region is generated using a mask segmentation model, ensuring the segmentation result of the target in the video frame. This technical solution improves the accuracy and stability of the segmentation results when solving long video target segmentation by effectively combining visual and semantic information.

[0070] In step S105 of some embodiments, a DETR-style model or a multimodal large language model can be used to perform preliminary prediction on each video frame to obtain the mask region where the target object is located. Confidence prediction is then performed on the predicted mask region of each video frame to obtain corresponding confidence data. The video frame with the highest confidence data is used as the starting frame for running the mask propagation-based tracking method. Based on the predicted mask region of the starting frame, the tracking model predicts the position of the target object in the next frame, thereby obtaining the target prediction region for that frame.

[0071] In step S106 of some embodiments, the target localization region refers to determining the precise location of the target in each frame of video, and this location region is used for subsequent image segmentation.

[0072] Specifically, please refer to Figure 5. In some embodiments, step S106 may include, but is not limited to, steps S501 to S505: Step S501, semantic similarity is evaluated based on the semantic mask region and the target prediction region of the same video frame to obtain a semantic consistency score.

[0073] Step S502: Evaluate the temporal continuity based on the target prediction regions of two adjacent video frames to obtain a temporal consistency score.

[0074] Step S503: Select the semantic mask region as the first candidate region based on the semantic consistency score and the temporal consistency score, and select the target prediction region as the second candidate region based on the semantic consistency score and the temporal consistency score.

[0075] Step S504: If only the first candidate region exists, the first candidate region is determined as the target location region.

[0076] Step S505: When there is a first candidate region and a second candidate region, perform region fusion based on the first candidate region and the second candidate region to obtain the target positioning region.

[0077] In step S501 of some embodiments, the calculation process of the semantic consistency score can refer to the following analytical expression: (6), where IoU represents the Intersection over Union (IoU) calculation. t represents the identifier of the video frame. This represents the semantic consistency score. This represents the target prediction region in the t-th video frame. This represents the semantic mask region of the t-th video frame.

[0078] In step S502 of some embodiments, the timing consistency score can be calculated with reference to the following analytical expression: (7), where IoU represents the Intersection over Union (IoU) calculation. t represents the identifier of the video frame. This represents the timing consistency score. This represents the target prediction region in the t-th video frame. This represents the target prediction region in the (t-1)th video frame (i.e., the previous video frame).

[0079] Semantic consistency score measures the similarity or overlap between the predicted target region and the semantic mask region. A higher semantic consistency score indicates a better match between the predicted target region and the semantic mask region. Temporal consistency score measures the stability or continuity of the predicted target region in adjacent video frames. A higher temporal consistency score indicates that the target moves smoothly between video frames, and the predicted target region has good temporal stability.

[0080] In step S503 of some embodiments, it is necessary to determine the input for generating the target localization region from the semantic mask region and the target prediction region based on different cases of temporal consistency score and semantic consistency score. The region requiring input is then treated as a candidate region for processing or analysis to obtain the target localization region. Specifically, if generating the target localization region requires the semantic mask region as input, it is determined as the first candidate region. If generating the target localization region requires the target prediction region as input, it is determined as the second candidate region.

[0081] Specifically, the time series score threshold is first obtained. and semantic score threshold In this embodiment, the time-series score threshold and semantic score threshold All values ​​are between [0,1]. For example, the time series score threshold. and semantic score threshold Both can be taken as 0.6.

[0082] When sequence consistency score Less than the time series score threshold And semantic consistency score Greater than or equal to the semantic score threshold At that time, or, at that time sequence consistency score Greater than or equal to the time series score threshold And semantic consistency score Less than the semantic score threshold When the semantic mask region is selected, it is used as the first candidate region.

[0083] When sequence consistency score Greater than or equal to the time series score threshold And semantic consistency score Less than the semantic score threshold If so, the target prediction region will be used as the second candidate region.

[0084] In step S504 of some embodiments, only the first candidate region exists, i.e., when the time sequence consistency score is... Time series score threshold And semantic consistency score Semantic score threshold When the propagation result has shifted spatially, the semantic segmentation model can still accurately identify the target. In this case, the semantic segmentation result (i.e., the semantic mask region) takes precedence, and the semantic mask region is directly used as the target localization region.

[0085] In step S505 of some embodiments, the situation where both the first candidate region and the second candidate region exist simultaneously is: when the timing consistency score Time series score threshold And semantic consistency score Semantic score threshold Simultaneously, the semantic mask region is used as the first candidate region, and the target prediction region is used as the second candidate region. In this scenario, the propagation region is stable in time but has semantic deviations.

[0086] Please refer to Figure 6. In some embodiments, step S505 includes, but is not limited to, steps S601 to S603: Step S601, obtaining the semantic lower bound confidence value and the semantic score threshold.

[0087] Step S602: Calculate the weights based on the semantic lower bound confidence value, semantic score threshold, and semantic consistency score to obtain the weight data.

[0088] Step S603: Perform image fusion on the semantic mask region and the target prediction region according to the weight data to obtain the target localization region.

[0089] In step S601 of some embodiments, the semantic lower bound confidence value is used to normalize the lower bound value of semantic consistency, reflecting the lowest confidence level of the semantic mask region, and its value range is [0,1], which can usually be 0.1.

[0090] In step S602 of some embodiments, the weight data can be calculated with reference to the following analytical expression: (8), among which, This represents the weighted data, with values ​​ranging from [0,1]. This represents the semantic score threshold. This represents the semantic lower bound confidence value. This represents the semantic consistency score.

[0091] In step S603 of some embodiments, the image fusion calculation process can refer to the following analytical expression: (9), among which, Indicates the target location area. This represents weighted data. This represents the target prediction region in the t-th video frame. This represents the semantic mask region of the t-th video frame. · represents a multiplication operation.

[0092] Steps S601 to S603, as illustrated in this embodiment, ensure the accuracy and reliability of the target localization region by combining the semantic lower bound confidence value, the semantic score threshold, and the semantic consistency score in a weighted calculation. Through image fusion technology, not only semantic information is combined, but also the location information of the target prediction region is fused, providing a more stable target localization result.

[0093] It should be noted that when the timing consistency score is met... And semantic consistency score When the propagation chain is stable and semantically consistent, the predicted target region obtained based on the pixel tracking method can be considered reliable. No repair operation is triggered, and the predicted target region is directly used as the target localization region.

[0094] When both timing consistency scores are met And semantic consistency score This indicates that the propagation results are both unstable and deviate from the semantic goal, and when Below When the threshold is reached, it indicates that the segmented region is in a high-risk state. A forced re-initialization operation is performed, which calls the semantic segmentation inference on the most recent video slice of the current video frame to generate a new reference mask and reset the propagation start frame. At this time, the inference process is actively restarted to prevent error accumulation and ensure that the target image segmentation can operate with high security and high robustness in complex scenarios.

[0095] Steps S501 to S505, as illustrated in the embodiments of this application, combine semantic consistency and temporal consistency evaluations to provide more accurate and robust target localization results in long video target segmentation tasks. Through comprehensive evaluation and fusion of candidate regions, the system can ensure the consistency and accuracy of target localization regions, especially maintaining high localization accuracy when facing target occlusion, dynamic changes, and complex backgrounds.

[0096] Please refer to Figure 7, which is a comparison chart of the effects of applying existing methods and the semantic feature-based image segmentation method provided in the embodiments of this application. The figure contains three rows of images, where the area selected by the blue box in the first row shows the real target (i.e., the actual target object in the target video). The second row shows the segmentation results of the existing technology, and the third row shows the segmentation results after using the semantic feature-based image segmentation method provided in the embodiments of this application.

[0097] Assuming the target description text is "a black and white zebra grazing on the right side of the image," the comparison of the images shows that while the segmentation result in the second row roughly locates the zebra's position, it contains obvious errors. Particularly in the first and second images, the zebra region on the left is incorrectly segmented, and the region outline in the second image is blurred, leading to misidentification and incorrect segmentation. In contrast, the segmentation result in the third row is significantly better than the results of existing technical solutions. By using the image segmentation method based on semantic features provided in this application, the target segmentation is more accurate, the zebra's outline is clearer, the background is effectively removed, and the non-target zebra on the left is not identified as the target. The boundary between the target region and the background is more distinct, demonstrating higher segmentation accuracy and robustness.

[0098] Please refer to Figure 8. This application embodiment also provides an image segmentation device based on semantic features, which can implement the above-mentioned image segmentation method based on semantic features. The device includes: an acquisition module 801, which is used to acquire a target video containing a target object and a target description text describing the target object.

[0099] The feature extraction module 802 is used to extract image features from each video frame of the target video to obtain video frame image features, and to extract text features from the target description text to obtain description semantic features.

[0100] The joint coding module 803 is used to perform joint coding based on the image features and descriptive semantic features of all video frames to obtain global joint features.

[0101] The masking module 804 is used to perform masking processing on each video frame of the target video based on the video frame image features and global joint features to obtain the semantic mask region where the target object is located.

[0102] The propagation prediction module 805 is used to perform region propagation prediction on the target object based on all video frames, so as to obtain the target prediction region where the target object is located in each video frame.

[0103] Image segmentation module 806 is used to generate a target localization region based on at least one of a semantic mask region and a target prediction region, and to perform image segmentation on video frames based on the target localization region.

[0104] The specific implementation of this semantic feature-based image segmentation device is basically the same as the specific implementation of the semantic feature-based image segmentation method described above, and will not be repeated here.

[0105] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image segmentation method based on semantic features. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0106] Please refer to Figure 9, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 901, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 902, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the semantic feature-based image segmentation method of the embodiments of this application. The input / output interface 903 is used to implement information input and output. The communication interface 904 is used to realize communication interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 905 transmits information between the various components of the device (such as the processor 901, memory 902, input / output interface 903 and communication interface 904). The processor 901, memory 902, input / output interface 903 and communication interface 904 realize communication connection between each other within the device through the bus 905.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described semantic feature-based image segmentation method.

[0108] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The semantic feature-based image segmentation method, apparatus, electronic device, and storage medium provided in this application jointly encode the semantic features of the target description text with the features of each frame of video image to generate a global joint feature. Based on this feature, semantic masking is performed to obtain a semantic mask region where the target object is located, aligned with the semantic information. Subsequently, region propagation prediction is performed on the target object in all video frames to obtain the target prediction region where the target object is located in each video frame, ensuring temporal consistency and coherence across multiple frames in a long video. Finally, by combining the semantic mask region and the target prediction region, a target localization region is generated, further improving the accuracy of target image segmentation. The method in this application effectively overcomes the erroneous segmentation caused by the lack of global consistency and temporal sequence continuity in traditional techniques, significantly improving the reliability and robustness of image segmentation results in long video scenes.

[0110] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0114] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.< / seg> < / seg> < / seg>

Claims

1. An image segmentation method based on semantic features, characterized in that, The method includes: acquiring a target video containing a target object and target descriptive text describing the target object; extracting image features from each video frame of the target video to obtain video frame image features, and extracting text features from the target descriptive text to obtain descriptive semantic features; jointly encoding all the video frame image features and the descriptive semantic features to obtain global joint features; for each video frame of the target video, performing masking processing on the video frame based on the video frame image features and the global joint features to obtain a semantic mask region where the target object is located; performing region propagation prediction on the target object based on all the video frames to obtain a target prediction region where the target object is located in each video frame; generating a target localization region based on at least one of the semantic mask region and the target prediction region, and performing image segmentation on the video frame based on the target localization region.

2. The method according to claim 1, characterized in that, The step of generating a target localization region based on at least one of the semantic mask region and the target prediction region includes: performing semantic similarity evaluation based on the semantic mask region and the target prediction region of the same video frame to obtain a semantic consistency score; performing temporal continuity evaluation based on the target prediction regions of two adjacent video frames to obtain a temporal consistency score; using the semantic mask region as a first candidate region based on the semantic consistency score and the temporal consistency score, and using the target prediction region as a second candidate region based on the semantic consistency score and the temporal consistency score; when only the first candidate region exists, determining the first candidate region as the target localization region; when both the first candidate region and the second candidate region exist, performing region fusion based on the first candidate region and the second candidate region to obtain the target localization region.

3. The method according to claim 2, characterized in that, The step of selecting the semantic mask region as a first candidate region based on the semantic consistency score and the temporal consistency score, and selecting the target prediction region as a second candidate region based on the semantic consistency score and the temporal consistency score, includes: obtaining a temporal score threshold and a semantic score threshold; selecting the semantic mask region as a first candidate region when the temporal consistency score is less than the temporal score threshold and the semantic consistency score is greater than or equal to the semantic score threshold, or when the temporal consistency score is greater than or equal to the temporal score threshold and the semantic consistency score is less than the semantic score threshold; and selecting the target prediction region as a second candidate region when the temporal consistency score is greater than or equal to the temporal score threshold and the semantic consistency score is less than the semantic score threshold.

4. The method according to claim 2, characterized in that, The step of fusing regions based on the first candidate region and the second candidate region to obtain the target localization region includes: obtaining a semantic lower bound confidence value and a semantic score threshold; calculating weights based on the semantic lower bound confidence value, the semantic score threshold, and the semantic consistency score to obtain weight data; and fusing the semantic mask region and the target prediction region based on the weight data to obtain the target localization region.

5. The method according to claim 1, characterized in that, The step of masking each video frame of the target video based on the video frame image features and the global joint features to obtain the semantic mask region where the target object is located includes: generating a global query matrix based on the global joint features; generating an original key matrix and an original value matrix based on the image features of each video frame; performing attention fusion based on the global query matrix, the original key matrix, and the original value matrix to obtain the current frame joint features of the video frame; and generating a mask by using a preset mask segmentation model to obtain the semantic mask region.

6. The method according to claim 5, characterized in that, The step of performing attention fusion based on the global query matrix, the original key matrix, and the original value matrix to obtain the current frame joint feature of the video frame includes: performing cross-attention calculation based on the global query matrix, the original key matrix, and the original value matrix to obtain intermediate joint features; integrating the intermediate joint features and the global joint features to obtain intermediate input features; updating the global query matrix, the original key matrix, and the original value matrix based on the intermediate input features, and returning to the step of performing cross-attention calculation based on the global query matrix, the original key matrix, and the original value matrix to obtain intermediate joint features, until the number of returns reaches a preset number, and using the last intermediate joint feature as the current frame joint feature.

7. The method according to claim 1, characterized in that, The step of jointly encoding the image features of all video frames and the descriptive semantic features to obtain global joint features includes: generating a semantic query matrix based on the descriptive semantic features, and generating an original key matrix and an original value matrix based on each video frame image feature; performing attention transformation on the semantic query matrix, the original value matrix, and the original key matrix to obtain a reference query matrix; generating a reference key matrix and a reference value matrix based on the reference query matrix, and performing attention calculation on the reference key matrix, the reference value matrix, and a preset base query matrix to obtain candidate joint features; performing reliability evaluation on all candidate joint features to obtain a reliability score; and selecting the candidate joint feature corresponding to the highest reliability score as the global joint feature.

8. An image segmentation device based on semantic features, characterized in that, The apparatus includes: an acquisition module for acquiring a target video containing a target object and target descriptive text describing the target object; a feature extraction module for extracting image features from each video frame of the target video to obtain video frame image features, and extracting text features from the target descriptive text to obtain descriptive semantic features; a joint encoding module for performing joint encoding based on all the video frame image features and the descriptive semantic features to obtain global joint features; a masking module for performing masking processing on each video frame of the target video based on the video frame image features and the global joint features to obtain a semantic mask region where the target object is located; a propagation prediction module for performing region propagation prediction on the target object based on all the video frames to obtain a target prediction region where the target object is located in each video frame; and an image segmentation module for generating a target localization region based on at least one of the semantic mask region and the target prediction region, and performing image segmentation on the video frame based on the target localization region.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images