Visual segmentation and understanding method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a vision segmentation and understanding method and device, electronic equipment and a storage medium, which are used for realizing unified processing of vision and semantics. The method comprises the following steps: inputting to-be-processed visual data and prompt information corresponding to the to-be-processed visual data into a multi-modal coding network to obtain an intermediate mask feature and an intermediate visual feature; inputting the intermediate mask feature and the intermediate visual feature into a visual decoding network to generate a segmentation result for the target area; and inputting the intermediate mask features, the intermediate visual features and preset learnable semantic features into a semantic perception network for cross-modal feature fusion, and generating visual feature representation and semantic feature representation of the to-be-processed visual data, and inputting the visual feature representation, the semantic feature representation and the prompt information into a semantic decoding network to generate a semantic understanding result for the target area.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a visual segmentation and understanding method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of multimodal artificial intelligence, the collaborative understanding of vision and language is becoming increasingly important in scenarios such as intelligent monitoring, autonomous driving, and medical imaging. However, in practical applications, how to achieve unified processing of pixel-level visual understanding and high-level semantic reasoning remains a major challenge for multimodal systems.

[0003] Currently, mainstream visual models primarily rely on segmentation networks or image encoders to acquire visual features, while language models generate semantic descriptions through training on large-scale corpora. Although this approach can achieve good results in their respective fields, there is a lack of close cross-modal correlation between vision and language, making it difficult to achieve region-level semantic alignment.

[0004] Therefore, how to achieve unified processing of vision and semantics is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a visual segmentation and understanding method, apparatus, electronic device, and storage medium to achieve unified visual and semantic processing.

[0006] This application provides a visual segmentation and understanding method, including:

[0007] The visual data to be processed and the corresponding prompt information are input into a multimodal coding network to obtain intermediate mask features and intermediate visual features. The prompt information is used to indicate the target region that needs to be segmented in the visual data to be processed, the intermediate mask features are used to describe the visual range of the target region, and the intermediate visual features are used to describe the visual content in the visual data to be processed.

[0008] The intermediate mask features and the intermediate visual features are input into a visual decoding network to generate a segmentation result for the target region.

[0009] Furthermore, the intermediate mask features, the intermediate visual features, and the preset learnable semantic features are input into a semantic perception network for cross-modal feature fusion to generate a visual feature representation and a semantic feature representation of the visual data to be processed; the visual feature representation, the semantic feature representation, and the prompt information are input into a semantic decoding network to generate a semantic understanding result for the target region; the visual feature representation is a visual expression of the target region that the semantic decoding network can understand, and the semantic feature representation is used to provide the semantic decoding network with semantic cues related to the target region.

[0010] This application provides a visual segmentation and understanding device, comprising:

[0011] A multimodal coding unit is used to input the visual data to be processed and the prompt information corresponding to the visual data to be processed into a multimodal coding network to obtain intermediate mask features and intermediate visual features; the prompt information is used to indicate the target region that needs to be segmented in the visual data to be processed, the intermediate mask features are used to describe the visual range of the target region, and the intermediate visual features are used to describe the visual content in the visual data to be processed.

[0012] A segmentation unit is used to input the intermediate mask features and the intermediate visual features into a visual decoding network to generate a segmentation result for the target region;

[0013] The understanding unit is used to input the intermediate mask features, the intermediate visual features, and the preset learnable semantic features into a semantic perception network for cross-modal feature fusion to generate a visual feature representation and a semantic feature representation of the visual data to be processed; input the visual feature representation, the semantic feature representation, and the prompt information into a semantic decoding network to generate a semantic understanding result for the target region; the visual feature representation is a visual expression of the target region that the semantic decoding network can understand, and the semantic feature representation is used to provide the semantic decoding network with semantic cues related to the target region.

[0014] Optionally, the device further includes:

[0015] The prompt generation unit is used to input the visual data to be processed and the prompt information corresponding to the visual data into the multimodal coding network before the multimodal coding unit inputs the visual data to be processed and the prompt information corresponding to the visual data into the prompt parsing network to generate the prompt information corresponding to the visual data to be processed.

[0016] The prompt information includes at least one of the following: first prompt information: a geometric prompt indicating the shape of the target region; second prompt information: a pseudo-mask prompt indicating the shape of the target region; and third prompt information: a semantic prompt indicating the semantic range of the target region.

[0017] Optionally, the prompt information includes third prompt information; the understanding unit is used for:

[0018] Extract the third prompt information from the prompt information;

[0019] Based on the third prompt information, semantic reasoning instructions for the semantic decoding network are generated;

[0020] The semantic decoding network performs semantic reasoning on the visual feature representation and the semantic feature representation based on the semantic reasoning instructions, and generates a semantic understanding result for the target region.

[0021] Optionally, the visual data to be processed is at least one video frame to be processed in the video data, and the video data includes multiple video frames; the apparatus further includes:

[0022] The video processing unit is configured to, before the multimodal coding unit inputs the visual data to be processed and the corresponding prompt information into the multimodal coding network to obtain intermediate mask features and intermediate visual features, group the multiple video frames and perform the following operations for each group of video frames:

[0023] Based on the scene type of the current group of video frames, determine the frame interval for selecting the video frames to be processed;

[0024] According to the frame interval, at least one video frame to be processed is extracted from the current group of video frames;

[0025] The scene type is used to describe the scene changes in the current group of video frames; if the scene type is a static scene, then the frame interval is the second frame interval; the first frame interval is less than the second frame interval.

[0026] Optionally, the scene type of the current group of video frames is determined in the following way:

[0027] For each video frame in the current group of video frames, the pixel mean square error of the video frame is determined based on the pixel value distribution of the video frame;

[0028] Based on the determined mean square errors of multiple pixels, the mean and standard deviation of the mean square errors of the multiple pixels are generated;

[0029] If the mean is greater than a preset mean threshold and the standard deviation is greater than a preset standard deviation threshold, then the scene is determined to be a dynamic scene; otherwise, the scene is determined to be a static scene.

[0030] Optionally, the understanding unit is used for:

[0031] Perform self-attention computation on the intermediate mask features to obtain the first updated mask features;

[0032] Based on the intermediate visual features and the first updated mask features, perform cross-attention computation to obtain the second updated mask features;

[0033] The learnable semantic features are fused with the second update mask features to obtain the updated semantic features;

[0034] By performing a first feature mapping operation on the second updated mask feature, the second updated mask feature is converted into a visual feature representation that the semantic decoding network can understand;

[0035] Furthermore, by performing a second feature mapping operation on the updated semantic features, the updated semantic features are converted into a semantic feature representation that the semantic decoding network can understand.

[0036] Optionally, the understanding unit is used for:

[0037] Based on a preset scoring mechanism, the importance score of each feature unit in the intermediate mask features is calculated respectively; wherein, the scoring mechanism is used to characterize the importance of each feature unit in the attention update process;

[0038] Based on the calculated importance scores, feature units whose importance scores meet preset conditions are selected as global attention units, and the rest are selected as local attention units.

[0039] Perform global attention calculation on each global attention unit to obtain the global attention calculation result; perform local sparse attention calculation on each local attention unit to obtain the local sparse attention calculation result.

[0040] Based on the global attention calculation results and the local sparse attention calculation results, the intermediate mask features are updated to obtain the first updated mask features.

[0041] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described visual segmentation and understanding methods.

[0042] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described visual segmentation and understanding methods.

[0043] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described visual segmentation and understanding methods.

[0044] The beneficial effects of this application are as follows:

[0045] This application provides a visual segmentation and understanding method, apparatus, electronic device, and storage medium. By inputting prompting information and visual data into a multimodal coding network, the model can focus on the target region based on the prompting information during the feature extraction stage. This yields intermediate mask features that accurately describe the spatial range of the target region and intermediate visual features that fully express the visual content. Based on these, an accurate segmentation result is generated in the visual decoding network. Furthermore, a semantic perception network performs cross-modal fusion of intermediate mask features, intermediate visual features, and learnable semantic features. This allows the generated visual feature representation to characterize the visual information of the target region in an expression form that the semantic decoding network can process. Simultaneously, the generated semantic feature representation provides supplementary semantic cues for the target region. Semantic reasoning is performed simultaneously using visual feature representation, semantic feature representation, and prompting information in the semantic decoding network. This ensures that the obtained semantic understanding result maintains segmentation accuracy while better aligning with the semantic meaning of the target region, thereby achieving integrated processing of segmentation and semantic understanding and improving the accuracy and intelligence of visual understanding tasks.

[0046] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this application;

[0049] Figure 2 A schematic diagram of a multimodal vision-language unified model provided in this application embodiment;

[0050] Figure 3 A flowchart illustrating the implementation of a visual segmentation and understanding method provided in this application embodiment;

[0051] Figure 4 This is a schematic diagram illustrating the generation of prompt information provided in an embodiment of this application;

[0052] Figure 5 A schematic diagram of a semantic awareness network provided in an embodiment of this application;

[0053] Figure 6 A general schematic diagram of a visual segmentation and understanding method provided in an embodiment of this application;

[0054] Figure 7 This is a schematic diagram of the composition structure of a visual segmentation and understanding device provided in an embodiment of this application;

[0055] Figure 8 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0057] The following describes some of the concepts involved in the embodiments of this application.

[0058] 1. Visual Segmentation and Understanding: This refers to the process of dividing visual data into pixel-level target regions and then further inferring the semantic information of these regions. Visual segmentation is used to determine the spatial location and shape of the target region; semantic understanding is used to determine the category, attributes, relationships, or semantic content of the target region that meets task requirements.

[0059] 2. Intermediate Mask Feature: This is a feature representation generated by a multimodal coding network based on visual data and cue information, used to reflect the spatial extent or salient structure of the target region. This feature typically contains encoded information sensitive to the location of the target region and can be used as input to visual decoding networks and semantic perception networks to support subsequent segmentation and semantic processing.

[0060] 3. Intermediate visual features: These are feature representations output by multimodal coding networks used to characterize the overall content of visual data. They contain semantic, texture, and contextual information from the visual data, providing fundamental visual understanding capabilities for segmentation and semantic reasoning networks.

[0061] 4. Segmentation Result: This is the output generated by the visual decoding network based on intermediate mask features and intermediate visual features, describing the pixel range of the target region. This result is typically represented as a mask map, used to accurately indicate the shape and coverage of the target region, and can be further used for semantic understanding tasks.

[0062] 5. Cross-modal feature fusion: This refers to the process of jointly modeling and interactively computing intermediate mask features, intermediate visual features, and learnable semantic features in a semantic perception network. Through feature fusion, the correlation between spatial information, visual information, and semantic information is realized to generate feature representations with more semantic expressive power.

[0063] 6. Visual Feature Representation: This is a feature representation primarily based on visual information obtained after cross-modal feature fusion. It includes the structural features, location features, and contextual information of the target region, and can be understood by semantic decoding networks and used to perform subsequent semantic reasoning tasks.

[0064] 7. Semantic Feature Representation: This is a semantic expression generated by the interaction of learnable semantic features and visually relevant features during cross-modal feature fusion. This representation provides semantic cues related to the target region to the semantic decoding network, thereby assisting the network in generating semantic understanding results for the target region.

[0065] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0066] like Figure 1 The diagram shown illustrates an application scenario of an embodiment of this application. This application scenario includes a terminal device 110 and a server 120.

[0067] Terminal device 110 can be, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, in-vehicle terminals, and other devices. A client for performing visual segmentation and understanding tasks can be installed on the terminal device. This client can realize functions such as visual data uploading, task prompt input, and result display. The client can be a standalone application, embedded software, a webpage, a mini-program, etc.

[0068] Server 120 is used to perform visual segmentation and semantic understanding processing on visual data and prompts uploaded by the terminal. Server 120 can be a regular backend server, a dedicated inference server, an AI acceleration server, etc. This application does not limit the specific configuration of the server. Server 120 can be a single physical server, or a server cluster, distributed system, or cloud server composed of multiple physical servers. The server can provide functions including cloud computing, cloud storage, model inference services, and artificial intelligence platform, and is used to run multimodal coding networks, visual decoding networks, semantic perception networks, and semantic decoding networks.

[0069] It should be noted that the visual segmentation and understanding method in this application can be executed by the terminal device 110, the server 120, or by a combination of both. In one optional implementation, the terminal device 110 is used to acquire visual data to be processed, receive task text prompts input by the user, and send them to the server 120; the server 120 performs multimodal coding, cross-modal feature fusion, segmentation result generation, and semantic understanding processing; the terminal device 110 is ultimately used to display the segmentation results and semantic understanding results returned by the server 120.

[0070] In one alternative implementation, the terminal device 110 and the server 120 communicate via a communication network. The communication network can be a wired network or a wireless network, such as a cellular mobile communication network, a wireless local area network, a Bluetooth network, or any other form of communication network.

[0071] It should be noted that, Figure 1 This is merely an illustrative structure, and the embodiments of this application do not limit the number of terminal devices or servers. In practical applications, multiple terminal devices and multiple servers can be deployed, wherein the multiple servers can be deployed in a distributed manner to improve the processing efficiency of visual segmentation and semantic understanding tasks.

[0072] In one alternative implementation, when there are multiple servers, these servers can form a blockchain network, with each server acting as a blockchain node. Key task prompts, feature summaries, and semantic understanding results involved in visual data processing can be stored on the blockchain to achieve data consistency, immutability, and result traceability among multiple nodes.

[0073] Furthermore, the visual segmentation and understanding method of this application can be applied to a variety of scenarios, including not only basic image analysis scenarios, but also scenarios such as intelligent traffic monitoring, assisted driving environment understanding, security monitoring image analysis, smart home sensing, industrial defect detection, medical image-assisted diagnosis, and robot vision.

[0074] Before introducing the visual segmentation and understanding method of this application, we will first give an overall description of the multimodal vision-language unified model used.

[0075] See Figure 2 As shown, this is an overall schematic diagram of a multimodal vision-language unified model provided in an embodiment of this application. The multimodal vision-language unified model 200 includes a multimodal coding network 201, a visual decoding network 202, a semantic perception network 203, and a semantic decoding network 204. The networks work together through feature transfer and cross-modal coupling mechanisms to achieve integrated processing of visual segmentation and semantic understanding.

[0076] The multimodal coding network 201 receives visual data to be processed and corresponding prompts, and outputs intermediate mask features to describe the visual range of the target region and intermediate visual features to describe the visual content. The visual decoding network 202 generates segmentation results for the target region based on the intermediate mask features and intermediate visual features. The semantic perception network 203 performs cross-modal feature fusion between the visual and semantic spaces to generate visual feature representations and semantic feature representations required for semantic reasoning. The semantic decoding network 204 generates semantic understanding results for the target region based on the visual feature representations, semantic feature representations, and prompts.

[0077] The visual segmentation and understanding method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0078] See Figure 3 The diagram shown is an implementation flowchart of a visual segmentation and understanding method provided in this application embodiment, applied to a server. The specific implementation flow of this method is as follows:

[0079] S31: Input the visual data to be processed and the prompt information corresponding to the visual data to be processed into the multimodal coding network to obtain intermediate mask features and intermediate visual features.

[0080] The prompt information is used to indicate the target region that needs to be segmented in the visual data to be processed, the intermediate mask feature is used to describe the visual range of the target region, and the intermediate visual feature is used to describe the visual content in the visual data to be processed.

[0081] In an alternative implementation, S21 is used to extract basic visual features from the input visual data and combine them with prompt information to generate intermediate mask features and intermediate visual features required for subsequent segmentation and semantic understanding.

[0082] The visual data to be processed and the prompts are simultaneously input into the multimodal coding network of the unified vision-language model. The multimodal coding network performs visual feature extraction on the visual data to obtain basic visual features used to describe the visual content; and encodes the prompts so that they can participate in the overall feature fusion as features. Through this fusion process, the multimodal coding network can adjust the expression of visual features according to the prompts, so that the output features can reflect the target area or task intent indicated by the prompts.

[0083] Through the above encoding and fusion processes, the multimodal coding network finally outputs intermediate mask features and intermediate visual features.

[0084] In one alternative implementation, the multimodal coding network can be based on existing visual segmentation models, such as multimodal visual coding structures like SAM2 and SAM3. Visual features are extracted by a visual encoder, and cue encoders encode cue information, fusing the two within a unified feature space to generate intermediate mask features and intermediate visual features. The aforementioned SAM-like models typically include an image encoding module, a cue encoding module, and a feature fusion module for interactive processing, effectively improving the correlation between cue information and visual content.

[0085] In another alternative implementation, if the basic visual model does not have cue parsing capabilities, a cue parsing subnetwork can be additionally set up in the multimodal coding network to independently encode the cue information and fuse it with the visual features output by the visual encoder to generate intermediate mask features and intermediate visual features.

[0086] This application does not limit the specific implementation of the multimodal coding network. Regardless of whether the network is based on SAM2, SAM3 or other multimodal coding architectures that can process visual data and prompt information simultaneously, the purpose of this application can be achieved, namely, to generate intermediate mask features and intermediate visual features that can accurately reflect the target area range and visual content.

[0087] In complex visual tasks, users typically describe task requirements in natural language. Directly inputting natural language prompts into the visual segmentation model can lead to ambiguous prompts and unclear semantic boundaries, thus affecting segmentation performance and semantic understanding accuracy. Therefore, it is necessary to convert task text prompts into structured prompt information that the visual model can directly utilize.

[0088] Based on this, this application proposes a prompt generation mechanism.

[0089] Optionally, before inputting the visual data to be processed and the corresponding prompt information into the multimodal coding network, the visual data to be processed and the corresponding task text prompts into the prompt parsing network to generate the prompt information corresponding to the visual data to be processed.

[0090] The prompt information includes at least one of the following: first prompt information: geometric prompts for indicating the shape of the target region; second prompt information: pseudo-mask prompts for indicating the shape of the target region; third prompt information: semantic prompts for indicating the semantic range of the target region.

[0091] In this embodiment, the multimodal vision-language unified model may further include a cue parsing network, which parses the task text cue and transforms the target expression implicit in the natural language of the task text cue into a cue expression that matches the visual segmentation model. This includes geometric cue for indicating the location or shape of the target region, pseudo-mask cue for representing candidate regions, and semantic cue for limiting the semantic range.

[0092] See Figure 4 The diagram illustrates a method for generating prompt information according to an embodiment of this application. First, an image or video stream is received as visual input, along with task text prompts input by the user. Then, a prompt parsing network performs semantic understanding and information extraction on the task text prompts, identifying key task information implicit in the text, such as target category, spatial location description, attribute requirements, and operational intent. Based on the parsing results, the prompt parsing network further maps natural language expressions into prompt forms recognizable by the visual segmentation model, including prompt point prompts indicating the specific location of the target region, prompt box prompts defining the scope of the target region, segmentation mask prompts providing a rough region prior, and text prompts defining the semantic scope of the task.

[0093] After generating the above prompt information, the prompt parsing network inputs the prompt points, prompt boxes, pseudo-mask prompts, and text prompts into the multimodal coding network, enabling the latter to effectively utilize these prompts during the segmentation and understanding process, focus on the target region specified by the task, and complete subsequent feature encoding, segmentation prediction, and semantic reasoning.

[0094] By using a prompt generation mechanism, text prompts for different types of tasks can be standardized into a unified prompt format, thereby improving the accuracy of visual segmentation and providing clear target constraints for subsequent semantic reasoning.

[0095] In one alternative implementation, a cue parsing network is used to convert task text cue into structured cue information that can be processed by a multimodal coding network.

[0096] Optionally, the input task text prompts are first semantically understood through the semantic understanding subnetwork in the prompt parsing network. This process parses the semantic components in the text and identifies semantic units such as object categories, object attributes, state descriptions, action requirements, and spatial scope qualifiers involved in the task. Through this semantic decomposition process, semantic points related to the target region can be extracted from the original natural language prompts.

[0097] After semantic parsing is completed, the target localization inference subnetwork in the prompt parsing network infers the target region described in the text based on the aforementioned semantic units, obtaining preliminary localization information of the target object or region that may correspond to it in the visual data. This localization information may include the approximate location, structural shape, semantic anchor points, or range description of the target, providing a basis for the generation of subsequent prompts.

[0098] Subsequently, the cue parsing network invokes the cue generation subnetwork to generate geometric cue information based on the target region localization inference results. These geometric cuees can include point cuees to represent the target center or key locations, line cuees to indicate boundary directions, or bounding box cuees to roughly enclose the target region. Geometric cuees provide explicit spatial constraints for the segmentation model, enabling it to quickly focus on task-relevant regions in visual data.

[0099] While generating geometric cues, the pseudo-mask construction subnetwork in the cue parsing network generates pseudo-mask cues based on the target region inference results. These pseudo-mask cues are constructed using a mask estimation method based on region confidence, which can roughly depict the spatial extent of the target region. This allows the segmentation model to obtain approximate region guidance in advance during subsequent processing, thereby improving segmentation quality.

[0100] Furthermore, the cue parsing network includes a semantic cue encoding subnetwork, which encodes the semantic elements extracted from the task text cue to form semantic cues. Semantic cues can help the multimodal vision-language unified model filter candidate regions under specific semantic conditions, thereby improving understanding accuracy.

[0101] After the above-mentioned prompts are generated, the prompt parsing network outputs geometric prompts, pseudo-mask prompts, and semantic prompts to the multimodal coding network, enabling it to effectively utilize these prompts in subsequent segmentation and understanding processes, thereby achieving accurate detection, segmentation, and semantic reasoning of the target region.

[0102] For example, in a specific application scenario, a user inputs a street view image into the system and provides a task prompt in text form, such as "Find all stationary bicycles in the image and determine whether their kickstands are down." After receiving the task text prompt, the prompt parsing network can first perform semantic parsing on the text content, extracting core semantic elements such as "bicycle," "stationary," and "kidney stand." It can then further identify several levels of information requirements contained in the task, including target category recognition (e.g., "bicycle"), target state constraints (e.g., "stationary"), and local attribute determination (e.g., "whether the kickstands are down").

[0103] Subsequently, the cue parsing network converts the aforementioned semantic elements into semantic cues to indicate the semantic range that the subsequent model needs to focus on. At the same time, based on the possible positional relationships of objects in the visual content, the cue parsing network can automatically generate geometric cues to guide the model to search for targets near suspected locations. When necessary, it can also generate coarse pseudo-mask cues to provide initial shape clues for candidate regions.

[0104] Through the aforementioned prompt generation process, the prompt parsing network can decompose complex natural language tasks into structured prompt information and output geometric prompts, pseudo-mask prompts, and semantic prompts to a multimodal coding network. When fusing this prompt information, the multimodal coding network can more accurately locate the bicycle region in the image and provide an effective feature base for subsequent determinations of whether the vehicle is stationary and the status of its kickstand.

[0105] Considering that the cue parsing network can generate cue information including geometric cue, pseudo-mask cue, and semantic cue, in order to enable the multimodal coding network to make full use of different types of cue information, this application sets up multiple sub-network structures in the multimodal coding network, such as a cue feature extraction sub-network, a cue interaction sub-network, and a mask enhancement sub-network, for co-modeling of cue information and visual features.

[0106] In one optional implementation, the multimodal coding network first performs visual feature extraction on the input visual data to obtain initial visual features representing the overall visual content. Simultaneously, it generates initial mask features based on a pre-defined learnable mask representation, providing a foundational representation for subsequent segmentation and inference. Furthermore, for the cue information output by the cue parsing network, the multimodal coding network can also encode it through a cue feature extraction sub-network to obtain cue features that reflect the cue's geometric structure, shape features, and semantic range. It should be noted that visual features can be considered image embeddings, mask features can be understood as mask tokens, and cue features can be understood as cue tokens.

[0107] Furthermore, considering that the target region indicated by the prompt information usually has clear semantic constraints, in order to improve the model's ability to focus on the target region, the multimodal coding network can jointly enhance the prompt features and the initial mask features.

[0108] Optionally, the multimodal coding network can use a quality evaluation network to perform quality discrimination on the initial mask features to obtain an initial quality confidence index for measuring the reliability of the features; and based on the cue features and the initial quality confidence index, the feature units in the initial mask features are weighted and enhanced, thereby highlighting visual patterns related to the target region in the mask features.

[0109] For example, when the task prompt indicates "red car", the enhancement module will assign higher weights to feature units related to "red area" and "vehicle structure" to enhance the response of that area in the mask features.

[0110] By employing the joint enhancement strategy based on prompt information and mask quality confidence, the initial mask features gain stronger discriminative power in both the visual and semantic spaces, thereby improving the final segmentation and semantic understanding results.

[0111] Subsequently, the multimodal coding network performs a progressive fusion process on the enhanced mask features and the initial visual features. Specifically, the multimodal coding network can use a feature modulation subnetwork to parse the target region cues contained in the enhanced mask features and generate modulation parameters for modulating the initial visual features. Then, based on the modulation parameters, weighting, filtering, or reconstruction is performed on each feature channel or feature unit in the initial visual features, so that the initial visual features focus on the target region indicated by the cues in terms of representational ability. Furthermore, the multimodal coding network can also perform multi-level interactive fusion of the modulated visual features and the enhanced mask features. Through layer-by-layer association, feature alignment, and semantic enhancement, the fused features simultaneously possess the accuracy of the local target range and the contextual consistency of the global visual content.

[0112] Through the aforementioned encoding and fusion process, the multimodal coding network ultimately outputs intermediate mask features and intermediate visual features. The intermediate mask features describe the visual range of the target region, while the intermediate visual features describe the overall content of the visual data, providing effective input for subsequent visual decoding networks to achieve high-precision segmentation and for semantic decoding networks to achieve semantic reasoning. This application does not limit the specific structure and implementation of each subnetwork in the multimodal coding network; its form can be an attention interaction structure, feature fusion structure, subspace mapping structure, region cue injection structure, etc., as long as it can achieve the above functions, it can be used in this application.

[0113] Through the above process, the multimodal coding network can generate intermediate mask features to characterize the region pointed to by the cue and intermediate visual features to describe the overall visual content, providing a unified and structured visual representation for subsequent segmentation reasoning and semantic understanding stages. The intermediate mask features have target region alignment capabilities, while the intermediate visual features retain global contextual information; together, they constitute the key inputs to subsequent visual decoding and semantic decoding networks.

[0114] However, when the visual data is video data, feature extraction and segmentation need to be performed on multiple consecutive video frames separately. In practical applications, there may be significant scene changes between video frames, or they may remain basically consistent for a long time. If all video frames are processed frame by frame directly, it is easy to lead to redundant calculations, which will significantly reduce the system's processing efficiency. Considering the temporal redundancy and uneven differences between frames in video data, this application proposes a scene change-aware video frame selection mechanism. By dynamically adjusting the frame interval according to the degree of change in the video frame content, representative key video frames are selected for processing to improve overall efficiency and ensure the accuracy of segmentation and understanding results.

[0115] Optionally, the visual data to be processed is at least one video frame from the video data, and the video data includes multiple video frames; before inputting the visual data to be processed and the corresponding cue information into the multimodal coding network to obtain intermediate mask features and intermediate visual features:

[0116] After grouping multiple video frames, perform the following operations for each group of video frames:

[0117] Based on the scene type of the current group of video frames, determine the frame interval for selecting video frames to be processed; according to the frame interval, extract at least one video frame to be processed from the current group of video frames; wherein, the scene type is used to describe the scene changes in the current group of video frames; if the scene type is a static scene, then the frame interval is the second frame interval; the first frame interval is less than the second frame interval.

[0118] In one alternative implementation, an adaptive frame skipping mechanism is introduced to dynamically adjust the number of frames based on scene complexity. More frames are skipped in static scenes (such as fixed backgrounds), while keyframes are retained in dynamic scenes (such as scenes with frequent moving objects), ensuring the continuity of segmentation. For example, in traffic monitoring, static road areas can have reduced processing frequency, while areas with moving vehicles are segmented at a higher frequency to capture dynamic changes.

[0119] The above implementation method can effectively reduce the repeated processing of redundant frames in visual data, thereby reducing the computational cost in the segmentation and semantic understanding process and improving the overall processing efficiency.

[0120] To avoid redundant processing of video data, reduce computational overhead, and improve overall processing efficiency while ensuring segmentation and semantic understanding accuracy, this application proposes a frame filtering mechanism based on the degree of scene change to determine the video frames to be processed in each group of video frames.

[0121] Optionally, the scene type of the current group of video frames is determined as follows: For each video frame in the current group of video frames, the pixel mean square error of the video frame is determined based on the pixel value distribution of the video frame; the mean and standard deviation of the multiple pixel mean square errors are generated based on the determined multiple pixel mean square errors; if the mean is greater than a preset mean threshold and the standard deviation is greater than a preset standard deviation threshold, the scene is determined to be a dynamic scene; otherwise, the scene is determined to be a static scene.

[0122] In one optional implementation, a sliding window is used to statistically distinguish between static and dynamic scenes, and the trend of inter-frame changes is statistically analyzed over time to avoid misjudgment due to instantaneous noise. Specifically, the formula for calculating the mean square error (MSE) of each pixel in each video frame of each group of video frames is as follows:

[0123]

[0124] Among them, K t (m, n) represents the pixel value at pixel coordinates (m, n) of the current video frame; K t-1 (m, n) represents the pixel value at pixel coordinates (m, n) of the previous video frame that is temporally adjacent to the current frame; H and W represent the height and width of the video frame, respectively; H × W represents the total number of pixels in the video frame; ((K t (m, n) - K t-1 (m, n)) 2 This represents the squared pixel error at the same pixel position between two adjacent frames; Formula 1 represents the sum of the squared errors of all pixels in a video frame; it is used to represent the mean square error of pixels between adjacent video frames, and is used to characterize the overall degree of change between the current frame and the next frame.

[0125] In one optional implementation, to distinguish between static and dynamic scenes in video data and thus adaptively adjust the frame processing frequency, this application measures the degree of content variation in video frames. Specifically, for N consecutive video frames within a sliding window, the mean square error (MSE) is calculated based on the pixel value distribution of each video frame. For each video frame, the MSE between adjacent frames can be calculated to form a sequence of mean square errors {MSE1, MSE2, ..., MSE} within the sliding window. N}

[0126] After obtaining the above error sequence, the mean μ and standard deviation σ are calculated for the sequence. If the mean μ is greater than the preset mean threshold τ1 and the standard deviation σ is greater than the preset standard deviation threshold τ2, it indicates that the pixel changes within the current frame group are large and the fluctuations are significant, and it can be determined as a dynamic scene; otherwise, it is determined as a static scene. Through this mechanism, an appropriate frame interval can be automatically selected according to the changing trend of video content, enabling the system to maintain high responsiveness in dynamic scenes and reduce redundant frame processing in static scenes, thereby effectively improving the overall performance of segmentation and semantic understanding tasks.

[0127] In video scenarios, multimodal coding networks can generate intermediate mask features to characterize the spatial extent of each target region based on visual features and cue information. Each intermediate mask feature can be represented in vector form, and its feature vector contains representation information for reconstructing the pixel-level mask of the target region, and can carry a mask confidence index (e.g., Intersection over Union) to quantify the mask quality of the target region. When processing N consecutive video frames, the multimodal coding network can output a sequence of intermediate mask features with a dimension of P×N, allowing the temporal features of each target in different frames to be recorded.

[0128] For example, in a scenario involving the identification of a red car, intermediate mask features can reflect information such as the vehicle's outline and local structural regions, along with a mask confidence index, which is used by subsequent semantic perception networks to evaluate the segmentation quality. In video scenarios, intermediate mask features can also capture inter-frame changes, enabling continuous analysis of the target in dynamic regions.

[0129] In order to accurately maintain the consistency of each target in different frames in a video scene, this application can set up a target history feature storage module (hereinafter referred to as "repository") in the segmentation model to save the intermediate mask features and visual embedding features of several historical video frames.

[0130] The repository can employ a sliding window strategy, retaining only the target feature representations from the most recent N frames, and determining whether to update the historical features of the corresponding target in the repository based on the quality confidence of the intermediate mask features.

[0131] Optionally, if the visual data is video data, and the video data includes multiple video frames, the method further includes:

[0132] For each video frame to be processed out of multiple video frames, perform the following operations:

[0133] The current video frame to be processed is input into a multimodal coding network to generate intermediate mask features and mask confidence indices for each segmented target in the current video frame.

[0134] Based on each mask confidence index, determine the mask quality status of each segmented target in the current frame, including high confidence status, medium confidence status and low confidence status;

[0135] If the target is in a high confidence state, the corresponding intermediate mask features are stored in the storage repository as historical reference features for subsequent frames.

[0136] If the target is in a medium confidence state, the historical reference features in the repository are kept unchanged, and the target association operation is performed to improve the mask quality of the segmented target in the current frame;

[0137] If a target is in a low-confidence state, a counting operation is performed on the target; when the target is in a low-confidence state for multiple consecutive video frames, the historical reference feature corresponding to the target is deleted from the repository.

[0138] In one optional implementation, the mask quality state is divided into the following three types:

[0139] High confidence state: If the mask quality confidence is high, the mask features in the middle of the current frame are stored in the storage repository to enhance the target consistency inference of subsequent frames;

[0140] Uncertain (Medium Confidence): If the mask quality is moderate, the repository will not be updated, but target association will be performed later to improve the mask quality of the target by using candidate regions.

[0141] Low confidence state: If the mask quality is poor, the target is counted; if it is in a low confidence state for multiple consecutive frames, the target's history can be cleared from the repository to terminate the tracking of the target.

[0142]

[0143] Among them, S x，y τ represents the confidence index of the y-th target in the x-th video frame, used to characterize the segmentation mask quality of that target in the current frame; h As a high confidence threshold, when S x，y >τ h When τ1 is the low confidence threshold, it indicates that the segmentation mask has high reliability and can be used to update the historical reference features of the target; τ1 is the low confidence threshold, when S x，y When the confidence index is ≤τ1, it indicates that the reliability of the segmentation mask is low, and positioning offset or boundary error may occur; when the confidence index is τ1<S x，y ≤τ h If the segmentation quality is within a certain range, the target is considered to be in an uncertain state, indicating that its segmentation quality is between high and low, and the target's true location needs to be further confirmed through target association strategies.

[0144] Formula (2) is used to dynamically determine the status type of the target based on the numerical range of the confidence index, thereby guiding subsequent tracking and management strategies, such as whether to update the historical features in the repository, whether to enter the target association process, or whether to start the target clearing counter.

[0145] To avoid ID switching during video tracking, this application proposes the following association strategy:

[0146] Obtain the target candidate region of the current video frame and calculate the overlap between the candidate region and the segmentation result of the previous video frame. If the overlap is greater than a preset threshold, the candidate region is determined as the associated candidate region of the old target; otherwise, it is determined as a new target and a new target identifier is assigned to it. For the old target in the medium confidence state, a distance cost-based matching algorithm is used to match it with the associated candidate region to update the segmentation prediction of the target in the current frame.

[0147] In one optional implementation, for the candidate target region detected in the current frame (e.g., the bounding box output by the detector), the overlap (e.g., IoU) between it and the mask region corresponding to the segmentation result of the previous frame is calculated; if the overlap exceeds a threshold, the candidate region is regarded as a candidate trajectory of a historical target, otherwise it is regarded as a new target and its ID is initialized.

[0148] For trajectories that belong to "old targets" but are in a medium confidence state, target association operations (such as the Hungarian algorithm) can be used to associate them with candidate regions of the current frame. The association cost can be the distance to the target center point or the distance in the feature space.

[0149] For targets that have been successfully associated, the prompt message can be regenerated and segmentation can be performed to improve the quality of the mask.

[0150] The above mechanism can solve problems such as missing IDs and ID switching in multi-target tracking.

[0151] S32: Input the intermediate mask features and intermediate visual features into the visual decoding network to generate segmentation results for the target region;

[0152] In one alternative implementation, the visual decoding network can perform mask reconstruction processing on the input intermediate mask features. Through methods such as layer-by-layer upsampling, region aggregation, or attention decoding, the mask representation in the feature space is gradually restored to a pixel-level segmentation mask. Simultaneously, the visual decoding network can refine the mask edges and suppress responses from non-target regions based on the contextual information provided by the intermediate visual features, thereby improving the accuracy and completeness of the segmentation results.

[0153] In one optional embodiment, the visual decoding network may include a mask prediction subnetwork for generating an initial mask based on intermediate mask features; and may include a mask quality evaluation subnetwork for outputting a confidence index corresponding to each predicted mask to characterize the reliability of the segmentation result. This confidence index can be used for semantic understanding processing in subsequent steps, or for maintaining consistency between segmentation and tracking in a video scene.

[0154] Through the above processing, the visual decoding network generates the final segmentation result for the target region indicated by the prompt information. This segmentation result can identify the spatial range of the target region in pixel-level form, providing a clear target region input for the subsequent semantic understanding network.

[0155] S33: Input the intermediate mask features, intermediate visual features, and preset learnable semantic features into the semantic perception network to perform cross-modal feature fusion, and generate visual feature representation and semantic feature representation of the visual data to be processed; input the visual feature representation, semantic feature representation, and prompt information into the semantic decoding network to generate semantic understanding results for the target region.

[0156] Visual features are represented as a visual representation of the target region that the semantic decoding network can understand. Semantic features are used to provide the semantic decoding network with semantic cues related to the target region.

[0157] In one alternative implementation, step S23 is divided into two parts:

[0158] Part 1: Cross-modal feature fusion.

[0159] Considering that multimodal data composed of visual and semantic features suffers from inconsistencies in representation space, heterogeneous feature dimensions, and different regions of interest, directly inputting it into a semantic decoding network may lead to the model's inability to accurately capture the semantic range indicated by the prompts, thus affecting the semantic understanding effect. Therefore, this application proposes a cross-modal feature fusion mechanism, which uses a semantic perception network to structurally align and interact with visual and semantic modalities, enabling features from different modalities to be expressed and work collaboratively in a unified space.

[0160] Optionally, intermediate mask features, intermediate visual features, and pre-defined learnable semantic features are input into a semantic perception network for cross-modal feature fusion to generate visual and semantic feature representations of the visual data to be processed, including:

[0161] Self-attention computation is performed on the intermediate mask features to obtain the first updated mask features; cross-attention computation is performed based on the intermediate visual features and the first updated mask features to obtain the second updated mask features; learnable semantic features and the second updated mask features are fused to obtain updated semantic features; the second updated mask features are converted into a visual feature representation that can be understood by the semantic decoding network by performing a first feature mapping operation on the second updated mask features; and the updated semantic features are converted into a semantic feature representation that can be understood by the semantic decoding network by performing a second feature mapping operation on the updated semantic features.

[0162] In one alternative implementation, the semantic awareness network can employ a lightweight cross-modal feature fusion structure, comprising a self-attention layer, a cross-attention layer, and a pointwise MLP layer. This structure can establish a correlation between the visual modality and the semantic modality, enabling the coordinated fusion of mask information, image context information, and semantic cue information within the feature space.

[0163] See Figure 5 As shown, this is a schematic diagram of a semantic awareness network provided in an embodiment of this application. Figure 5 As shown, the semantic awareness network receives two types of input features from the multimodal coding network: intermediate mask features and intermediate visual features.

[0164] Subsequently, the semantic awareness network performs the following operations in sequence to achieve cross-modal fusion between the visual and semantic modalities:

[0165] Self-attention computation is performed on the intermediate mask features. The self-attention layer is used to establish feature dependencies within the intermediate mask features, enabling the semantic perception network to adaptively enhance the mask features based on the internal structural differences of the relevant regions, thereby improving the expressive power of the target region.

[0166] Cross-attention computation is performed based on intermediate visual features and the first updated mask features to obtain the second updated mask features. In the cross-attention layer, the first updated mask features serve as the query, while the intermediate visual features participate in the attention interaction as keys and values. This process enables the mask features to selectively absorb visual information related to the target from the overall image context, such as the object's appearance texture, semantic boundaries, and locally relevant background, thereby further enhancing the semantic expressive power of the region features.

[0167] The learnable semantic features are fused with the second updated mask features to obtain the updated semantic features. The learnable semantic features consist of several learnable semantic sub-features and are used to express the target semantic space structure required by the semantic decoding network. During the fusion process, the learnable semantic features and the second updated mask features are concatenated and interact through an attention mechanism, enabling the learnable semantic features to capture the appearance attributes, semantic cues, and task-related information of the target region, thus forming the updated semantic features.

[0168] Perform the first feature mapping operation on the second updated mask features to generate a visual feature representation. Perform the second feature mapping operation on the updated semantic features to generate a semantic feature representation.

[0169] The first feature mapping operation, which may include linear transformations, pointwise MLP, or low-dimensional subspace mapping, is used to convert the updated second-updated mask features into a visual feature representation that the semantic decoding network can understand. This visual feature representation accurately reflects the visual boundaries, regional structure, and attribute features of the target region, providing visual support for subsequent semantic reasoning. The second feature mapping operation is used to map the fused semantic features onto a semantic space that the semantic decoding network can accept, enabling the semantic feature representation to reflect the semantic scope, semantic attributes, and task requirements expressed by the prompt text.

[0170] Through the above steps, the semantic perception network can simultaneously output visual feature representations and semantic feature representations, and provide them to the semantic decoding network to generate the final semantic understanding result.

[0171] Considering that intermediate mask features typically have long feature sequence structures during cross-modal feature fusion, directly employing a global attention mechanism would result in high computational cost and low efficiency. This application further proposes an attention optimization mechanism based on feature importance scoring to reduce computational overhead while maintaining feature fusion performance.

[0172] Optionally, based on a preset scoring mechanism, the importance score of each feature unit in the intermediate mask features is calculated respectively; wherein, the scoring mechanism is used to characterize the importance of each feature unit in the attention update process; according to the calculated importance scores, feature units whose importance scores meet the preset conditions are selected as global attention units, and the rest are selected as local attention units; global attention calculation is performed on each global attention unit to obtain the global attention calculation result; local sparse attention calculation is performed on each local attention unit to obtain the local sparse attention calculation result; based on the global attention calculation result and the local sparse attention calculation result, the intermediate mask features are updated to obtain the first updated mask features.

[0173] In one optional implementation, it is assumed that the intermediate mask features consist of multiple feature units, each corresponding to a local visual representation related to the target region. For a target feature unit that needs to be updated, the system calculates its importance score for each feature unit based on its correlation with other feature units. In one optional approach, this importance score can be expressed by the following formula:

[0174]

[0175] Among them, I t，s The value of feature unit t to be updated is relative to feature unit s; G is the number of relevance channels used in the scoring mechanism; f t (i) The i-th channel correlation vector generated by feature unit t; is the i-channel correlation vector generated by feature unit s; wi is the learnable weight parameter for each channel; ReLU is the linear rectified activation function.

[0176] Based on the above importance scores, the importance sequence of the current target feature unit to all feature units can be obtained: {It,1,It,2,...,It,n}

[0177] Subsequently, based on the importance scores, a number of top-ranked feature units (e.g., Top-K) are selected as global attention units, and a complete global attention calculation is performed on them to fully extract their semantic dependencies with the target feature units.

[0178] For the remaining feature units that do not enter the Top-K, they are treated as local attention units, and local sparse attention computation is performed on them to reduce the overall computational complexity.

[0179] Finally, the intermediate mask features are updated based on the combined result of global attention and local sparse attention to obtain the first updated mask features.

[0180] For any feature unit t to be updated, the importance scores of this feature unit and the other feature units s can be calculated to form an importance sequence. It should be noted that the feature unit t itself does not participate in the importance score I. t,s While the importance score is calculated, it still participates in the subsequent attention update process. This is because the role of the importance score is to evaluate the "contribution of other feature units to t", while the contribution of t itself to itself does not need to be judged through a scoring mechanism, so it is not necessary to calculate I. t,s However, in actual global attention and local attention updates, feature unit t will act as a query unit and interact with other selected feature units to complete its own feature update.

[0181] For example, suppose the current intermediate mask features contain six feature units {s1,s2,s3,s4,s5,s6}, where s1 is the feature unit to be updated. By calculating the relevance between s1 and the other feature units using a scoring mechanism, the following importance scores can be obtained: s2 has an importance score of 0.92, s3 has an importance score of 0.88, s4 has an importance score of 0.31, s5 has an importance score of 0.27, and s6 has an importance score of 0.05. Based on these importance scores, the feature units with the highest scores can be selected as global attention units. For example, s2 and s3 can be selected for full global attention computation, allowing the feature unit to absorb high-value information from the most relevant regions. For the remaining feature units s4, s5, and s6, they are treated as local attention units, and only local sparse attention computation is performed, thus significantly reducing the overall computational cost. Subsequently, the global attention result and the local sparse attention result are fused to update the original intermediate mask features, so that the feature unit s1 to be updated has both overall semantic relevance and local structural details. The same applies to other feature units to be updated, thereby obtaining the first updated mask features.

[0182] The above implementation method can effectively select the most contributing feature units to participate in the full attention calculation, while avoiding wasting computing resources on low-relevance features, thereby reducing the computational cost of feature updates while ensuring representational capability.

[0183] In an alternative implementation, to enhance the semantic perception network's ability to understand prompts and enable visual features to respond more accurately to prompt content, this application proposes a fusion coding structure based on cross-modal conditionalization. This structure can be used as an optional component of the semantic perception network to perform feature modulation of visual input to guide prompts before generating visual and semantic feature representations.

[0184] Optionally, the fusion encoder can combine the text cues, geometric cues, and sample cues output by the cue parsing network to form a cue vector, and input it along with the frame embeddings corresponding to the visual data into the fusion encoder. The fusion encoder can include multiple feature fusion units with self-attention and cross-attention mechanisms: the self-attention mechanism is used to model the semantic dependencies within the frame embeddings to maintain the integrity of the visual features; the cross-attention mechanism is used to adjust the region of interest of the frame embeddings using the cue vectors, so that the visual features can be focused on the target area indicated by the cue information.

[0185] Subsequently, the fusion encoder can further conditionalize the cue vector through a pointwise nonlinear mapping structure to enhance the influence of the cue signal on visual features. After the above processing, the output of the fusion encoder is the conditional frame embedding affected by the cue information, which can be directly used as input to the visual feature representation generation module, enabling the semantic decoding network to perform cue-based semantic reasoning more accurately.

[0186] Through the above methods, this embodiment enables the semantic perception network to make fuller use of prompting information, achieving dynamic guidance and semantic modulation of visual features. This application does not limit the attention structure, number of layers, or feature mapping method of the fusion encoder; any method that can effectively modulate visual features through prompting is acceptable.

[0187] In another alternative implementation, to reduce the computational overhead of the semantic decoding network when processing large-scale visual features and to enable efficient alignment of visual and semantic features in the same feature space, this application proposes a feature projection structure based on pixel rearrangement and multi-layer mapping projection. This structure can be used as an optional component of a semantic perception network to perform dimensionality reduction, integration, and spatial mapping on the input features before generating visual and semantic feature representations.

[0188] Optionally, for image-type visual data, the feature projector can perform pixel rearrangement operations in adjacent pixel regions of visual features, reorganizing local high-density visual information into more compact visual features, thereby reducing the number of visual features and lowering the processing cost of subsequent semantic decoding networks. For video-type data, the feature projector can perform the same rearrangement method on cue frames as on images, while using a more aggressive rearrangement mode on non-cue frames to further compress the number of visual features and improve the overall processing efficiency in video scenarios.

[0189] The visual features, after the above rearrangement process, can be non-linearly mapped through a visual projection sub-network to convert them into a visual feature representation that the semantic decoding network can accept. Simultaneously, learnable semantic features can also be mapped through an independent semantic projection sub-network, ensuring that semantic features are aligned with visual features in the same semantic space, providing a consistent input representation format for the semantic decoding network.

[0190] By introducing the aforementioned feature projector structure, this embodiment not only effectively reduces the number of visual features and improves cross-modal interaction efficiency, but also enables the efficient fusion of visual feature representation and semantic feature representation within a unified space, thereby improving the accuracy and stability of the final semantic understanding result. This application does not specifically limit the rearrangement mode, mapping method, or number of layers of this feature projection structure.

[0191] After introducing the process of inputting intermediate mask features, intermediate visual features, and learnable semantic features into the semantic perception network and completing cross-modal fusion, we can proceed to the semantic understanding stage. This application proposes a semantic understanding mechanism in the semantic decoding network that can perform reasoning based on the semantic goals of the task, thereby realizing semantic-level judgment and interpretation of the target region.

[0192] Part Two: Semantic Understanding.

[0193] Considering that relying solely on visual features is insufficient to fully satisfy complex semantic tasks (such as attribute recognition, state judgment, and behavior understanding), this application introduces semantic elements from prompts. By further parsing the prompts, the semantic decoding network obtains a clear reasoning direction, thereby improving the accuracy and controllability of semantic understanding. Therefore, this application proposes a prompt-driven semantic reasoning method to enable the semantic decoding network to execute the reasoning process according to the semantic requirements of the task.

[0194] Optionally, a third prompt information is extracted from the prompt information; based on the third prompt information, a semantic reasoning instruction for the semantic decoding network is generated; the semantic decoding network performs semantic reasoning on the visual feature representation and semantic feature representation based on the semantic reasoning instruction, and generates a semantic understanding result for the target region.

[0195] In one optional implementation, the semantic reasoning mechanism may include the following operational steps:

[0196] First, third-party cue information is extracted from the cue information output by the cue parsing network. The third-party cue information is used to describe the semantic scope that the task needs to focus on, such as target type, expected attributes, state requirements, relationship judgments, etc.

[0197] Subsequently, based on the third prompt information, semantic reasoning instructions are generated for the semantic decoding network, enabling the network to clearly define the semantic constraints and inference objectives to be followed in this reasoning process. These instructions can be used to control the semantic decoding network to focus on specific semantic categories, limit the scope of reasoning, or execute specific semantic analysis strategies.

[0198] Finally, guided by the aforementioned semantic reasoning instructions, the semantic decoding network performs semantic reasoning on the visual and semantic feature representations generated by the semantic perception network, producing a final semantic understanding result for the target region. This semantic understanding result may include, but is not limited to, semantic information such as target category, target attributes, target state, semantic relationships, and behavioral descriptions.

[0199] By introducing the aforementioned prompt-driven semantic reasoning mechanism, the semantic decoding network can perform refined reasoning based on visual features and task semantic requirements, thereby obtaining more accurate and interpretable semantic understanding results in complex task scenarios. This application does not limit the specific encoding form of semantic reasoning instructions or the specific structure of the semantic decoding network; any method that enables task-related semantic reasoning based on third-party prompt information is applicable to this application.

[0200] To enable the final semantic output generated by the semantic decoding network to adapt to the semantic requirements of different task scenarios, this application further proposes a dynamic weighting mechanism during the semantic decoding process. This mechanism adaptively adjusts the weight relationship between visual and semantic features when generating visual and semantic feature representations. The dynamic weighting mechanism generates output features by weighted fusion of visual feature representation V and semantic feature representation S, and its calculation method can be expressed as follows:

[0201] T = α·V + β·S (Formula 4)

[0202] Here, α and β are weighting coefficients that are automatically adjusted according to the task type. When the task is biased towards scenarios that rely on visual cues such as object classification and attribute recognition, visual features are more important, and the value of α can be increased accordingly. When the task is a scenario that requires stronger semantic inference capabilities, such as semantic interpretation, behavioral reasoning, or region description, semantic features are more important, and the value of β can be increased to enhance the semantic expressiveness of the output.

[0203] Through this dynamic weighting strategy, the semantic decoding network can adaptively adjust the fusion method of visual and semantic information under different task requirements, so that the final output token retains sufficient visual details and has consistent semantic expression ability, thereby significantly improving the accuracy and adaptability of multimodal semantic understanding results.

[0204] In an alternative implementation, to further enhance semantic understanding capabilities, this application proposes a multi-granularity semantic output generation mechanism for generating structured outputs with different semantic levels such as classification, definition, description, and explanation based on task instructions. This mechanism leverages the decoding capabilities of a multimodal visual-language model (UMM) and achieves unified processing of multiple task types through instruction-driven fine-tuning and context-aware decoding strategies.

[0205] Specifically, during the input phase, the semantic decoding network receives the visual and semantic feature representations generated by the semantic perception network and fuses them with the instruction embedding vectors corresponding to third-party cue information (such as classifying, describing, or interpreting the region). The fused multimodal token and instruction embeddings are then input into the semantic decoding network, driving the model to generate outputs that are semantically consistent with the task instructions.

[0206] Through this instruction-driven mechanism, the semantic decoding network can adjust its decoding strategy according to different task types. For example, when the task instruction is to classify the region, the semantic decoding network outputs a classification result with a clear semantic label (e.g., a car); when the task instruction is to "explain the region," the semantic decoding network generates defining semantics (e.g., a vehicle used for carrying passengers); when the task is to describe the region, the semantic decoding network combines the image context to output a descriptive statement (e.g., a red car entering an intersection). During the decoding process, the weights of visual tokens and semantic tokens are dynamically adjusted through a context-aware attention interaction mechanism to ensure that the generated content is highly consistent with the task instruction.

[0207] Furthermore, with the support of an instruction-driven fine-tuning strategy, the semantic decoding network can learn the output patterns of different instructions (e.g., classification instruction → label output, description instruction → contextual description, explanation instruction → conceptual definition) through a small number of labeled samples, thereby improving the semantic decoding network's responsiveness and generalization performance to task instructions. For example, in a description task, the semantic decoding network can generate an output indicating that the region is a red taxi driving in the rain; in an explanation task, the semantic decoding network can output that the region is a vehicle used for carrying passengers.

[0208] During the output generation stage, the semantic decoding network can generate structured semantic content through an autoregressive approach and supports multilingual output, including Chinese and English. For example, when the input is a Chinese instruction describing the region, the output could be "This region is a red taxi driving in the rain"; when the input is an English instruction "Describe this region", the output could be "This region is a red taxi driving in the rain". Furthermore, this application introduces a cross-frame semantic caching mechanism, which reduces redundant computation, improves temporal consistency, and lowers overall latency by reusing semantic information generated in the previous video frame (e.g., "red car" as a consistent cross-frame description).

[0209] Based on the same inventive concept as the aforementioned visual segmentation and understanding methods, this application also proposes a training method for a multimodal vision-language unified model (hereinafter referred to as the model), which enables the model to acquire structured, controllable, and high-precision reasoning capabilities in tasks such as region-level visual perception, cross-modal feature fusion, and instruction-driven semantic understanding. The entire training process includes two levels: a pre-training alignment stage and an instruction fine-tuning stage, used to gradually build the model's visual understanding and language generation capabilities.

[0210] In the initial training phase, the model first undergoes a pre-training alignment operation. The goal of this alignment is to establish a consistent mapping between visual features, semantic features, and the embedding space of the language model, enabling the model to recognize the semantic categories or descriptive information corresponding to different local regions in an image. To achieve this goal, this application employs supervised learning on a large-scale region-level image classification and description dataset. In this phase, only the parameters of the semantic perceptron and projector are updated, while the parameters of the multimodal large language model and the segmentation model (e.g., SAM3) are frozen to ensure that the basic visual and language capabilities remain stable. For region segmentation supervision, a segmentation loss between the ground truth segmentation mask and the model's predicted mask is used for training, such as the Dice loss, which can be expressed as:

[0211] L align =L seg (P seg M gt ) (Formula 5)

[0212] Among them, L align M represents the segmentation loss. gt The label representing the segmentation mask, P seg L represents the predicted segmentation mask. seg This refers to the dice loss.

[0213] After pre-training, the model enters the instruction-driven fine-tuning phase. This phase aims to enable the model to generate semantic outputs of corresponding granularity based on different types of task instructions, such as classification, definition, description, or explanation. This phase uses a region-semantic instruction dataset for training, and the input typically includes visual regions, language instructions, and corresponding structured semantic outputs. This application constructs a hybrid loss function to simultaneously optimize the model's segmentation accuracy, semantic output accuracy, cross-frame contextual continuity, and output structure regularity. The hybrid loss function can be expressed as:

[0214] L=λ1L seg +λ2L sem +λ3L ctx ++λ4L format (Formula 6)

[0215] Where L is the mixture loss function, L seg For the dice loss, L sem For semantic loss (such as cross-entropy), L ctx For context continuity loss (such as contrastive learning), L format Output format loss. Ensure that MLLM output strictly adheres to the expected structure and format specifications.

[0216] The format constraint loss requires MLLM to meet the following format specifications in its output:

[0217] (1) Place the content of the reasoning process in <reasoning> …< / reasoning> Tag alignment;

[0218] (2) Place the positioning or segmentation results in <output>…< / output> Tag alignment;

[0219] If the model output format is correct, a reward value of 1 is provided to the loss function; otherwise, the reward value is 0, in order to supervise the model to maintain format consistency in the long term.

[0220] Through the aforementioned fine-tuning, the model achieves deep integration between vision and language. When the input instruction is "classify this region," the model outputs the corresponding category label based on the visual token; when the input instruction is "describe this region," the model generates a natural language description with scene context; when the input instruction is "explain this region," the model combines visual and semantic features to output a defining explanation or high-level semantic description. Based on the autoregressive decoding mechanism, the output content remains semantically coherent and structurally standardized, and can automatically adjust the weight ratio of visual and semantic features according to the nature of the task, making the generated results more suitable for specific task requirements.

[0221] Through the training methods described above, the multimodal vision-language unified model of this application can achieve consistent, accurate and controllable output capabilities in tasks such as region segmentation, semantic recognition, attribute reasoning, scene interpretation and cross-modal understanding, which greatly improves the model's generalization performance and semantic understanding capabilities in complex visual scenes.

[0222] See Figure 6 As shown, this is a general schematic diagram of a visual segmentation and understanding method provided in an embodiment of this application. Figure 6 As shown, the multimodal vision-language unified model first receives image or video frames as visual input, and simultaneously receives task text prompts input by the user. Subsequently, the visual data and task text prompts are fed into a prompt parsing network for processing. The prompt parsing network performs semantic understanding of the task text prompts and parses various forms of prompt information based on the prompt content, including point prompts and bounding box prompts for locating geometric ranges, pseudo-mask prompts for roughly describing the shape of the target region, and textual semantic prompts for limiting the semantic scope of the task.

[0223] After the prompt is generated, the prompt parsing network transmits the prompt information to the multimodal coding network. Upon receiving the prompt information and visual data, the multimodal coding network generates intermediate mask features that characterize the spatial extent of the target region, as well as intermediate visual features that characterize the overall visual content, through steps such as visual feature extraction, prompt feature fusion, and mask feature enhancement.

[0224] Subsequently, the intermediate mask features are fed into a visual decoding network to perform region-level segmentation inference, obtaining segmentation results for the target region. Simultaneously, the intermediate mask features, intermediate visual features, and learnable semantic features are input into a semantic perception network, which generates visual and semantic feature representations through cross-modal attention interactions.

[0225] After acquiring visual and semantic feature representations, the multimodal vision-language unified model further inputs them into a semantic decoding network, enabling it to perform semantic reasoning based on task-specific semantic cues, thereby generating the final semantic understanding result. The semantic understanding result can include various forms such as classification, attribute recognition, definition generation, scene description, or region interpretation to meet the complex and diverse needs of visual understanding tasks.

[0226] Through the coordinated processing of the above steps, the embodiments of this application can effectively realize prompt-driven visual segmentation and semantic understanding. It can not only accurately locate the target region and generate a high-quality segmentation mask according to the user's task instructions, but also further complete higher-order semantic reasoning based on the segmentation results to support more complex multimodal intelligent application scenarios.

[0227] Based on the same inventive concept, embodiments of this application also provide a visual segmentation and understanding device. For example... Figure 7 As shown, this is a schematic diagram of the structure of the visual segmentation and understanding device 700, which may include:

[0228] The multimodal coding unit 701 is used to input the visual data to be processed and the prompt information corresponding to the visual data to be processed into the multimodal coding network to obtain intermediate mask features and intermediate visual features; the prompt information is used to indicate the target region that needs to be segmented in the visual data to be processed, the intermediate mask features are used to describe the visual range of the target region, and the intermediate visual features are used to describe the visual content in the visual data to be processed.

[0229] The segmentation unit 702 is used to input intermediate mask features and intermediate visual features into the visual decoding network to generate segmentation results for the target region;

[0230] The understanding unit 703 is used to input intermediate mask features, intermediate visual features, and preset learnable semantic features into the semantic perception network for cross-modal feature fusion, generating visual feature representation and semantic feature representation of the visual data to be processed; input the visual feature representation, semantic feature representation, and prompt information into the semantic decoding network to generate semantic understanding results for the target region; the visual feature representation is a visual expression of the target region that the semantic decoding network can understand, and the semantic feature representation is used to provide semantic cues related to the target region to the semantic decoding network.

[0231] Optionally, the device 700 also includes:

[0232] The prompt generation unit 704 is used to input the visual data to be processed and the prompt information corresponding to the visual data into the multimodal coding network before the multimodal coding unit inputs the visual data to be processed and the prompt information corresponding to the visual data into the prompt parsing network to generate the prompt information corresponding to the visual data to be processed.

[0233] The prompt information includes at least one of the following: first prompt information: geometric prompts for indicating the shape of the target region; second prompt information: pseudo-mask prompts for indicating the shape of the target region; third prompt information: semantic prompts for indicating the semantic range of the target region.

[0234] Optionally, the prompt information includes a third prompt information; the understanding unit 703 is used for:

[0235] Extract the third prompt information from the prompt information;

[0236] Based on the third prompt information, semantic reasoning instructions for the semantic decoding network are generated;

[0237] The semantic decoding network performs semantic reasoning on visual feature representations and semantic feature representations based on semantic reasoning instructions, generating semantic understanding results for the target region.

[0238] Optionally, the visual data to be processed is at least one video frame to be processed in the video data, and the video data includes multiple video frames; the apparatus 700 further includes:

[0239] The video processing unit 705 is configured to group multiple video frames and perform the following operations for each group of video frames before the multimodal coding unit inputs the visual data to be processed and the corresponding prompt information into the multimodal coding network to obtain intermediate mask features and intermediate visual features:

[0240] Based on the scene type of the current group of video frames, determine the frame interval for selecting the video frames to be processed;

[0241] According to the frame interval, at least one video frame to be processed is extracted from the current group of video frames;

[0242] The scene type describes the scene changes in the current group of video frames; if the scene type is a static scene, the frame interval is the second frame interval; the first frame interval is less than the second frame interval.

[0243] Optionally, the scene type of the current group of video frames is determined in the following way:

[0244] For each video frame in the current group of video frames, determine the pixel mean square error of the video frame based on the pixel value distribution of the video frame;

[0245] Based on the determined mean square errors of multiple pixels, generate the mean and standard deviation of the mean square errors of multiple pixels;

[0246] If the mean is greater than the preset mean threshold and the standard deviation is greater than the preset standard deviation threshold, then the scene is determined to be a dynamic scene; otherwise, the scene is determined to be a static scene.

[0247] Optionally, the understanding unit 703 is used for:

[0248] Perform self-attention computation on the intermediate mask features to obtain the first updated mask features;

[0249] Cross-attention computation is performed based on intermediate visual features and the first updated mask features to obtain the second updated mask features;

[0250] The learnable semantic features are fused with the second update mask features to obtain the updated semantic features;

[0251] By performing the first feature mapping operation on the second updated mask features, the second updated mask features are converted into a visual feature representation that the semantic decoding network can understand.

[0252] Furthermore, by performing a second feature mapping operation on the updated semantic features, the updated semantic features are transformed into a semantic feature representation that the semantic decoding network can understand.

[0253] Optionally, the understanding unit 703 is used for:

[0254] Based on a pre-defined scoring mechanism, the importance score of each feature unit in the intermediate mask features is calculated; the scoring mechanism is used to characterize the importance of each feature unit in the attention update process.

[0255] Based on the calculated importance scores, feature units whose importance scores meet the preset conditions are selected as global attention units, and the rest are selected as local attention units.

[0256] Perform global attention calculation on each global attention unit to obtain the global attention calculation result; perform local sparse attention calculation on each local attention unit to obtain the local sparse attention calculation result.

[0257] Based on the global attention calculation results and the local sparse attention calculation results, the intermediate mask features are updated to obtain the first updated mask features.

[0258] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0259] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0260] Having introduced the visual segmentation and understanding method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0261] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0262] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The server 120 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 8 As shown, it includes a memory 801, a communication module 803, and one or more processors 802.

[0263] The memory 801 is used to store computer programs executed by the processor 802. The memory 801 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0264] Memory 801 may be volatile memory, such as random-access memory (RAM); memory 801 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 801 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 801 may be a combination of the above-described memories.

[0265] The processor 802 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 802 is used to implement the aforementioned visual segmentation and understanding method when it calls the computer program stored in the memory 801.

[0266] The communication module 803 is used to communicate with terminal devices and other servers.

[0267] This application embodiment does not limit the specific connection medium between the memory 801, communication module 803, and processor 802 described above. This application embodiment... Figure 8 The memory 801 and the processor 802 are connected via a bus 804, and the bus 804 is in Figure 8 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 804 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 8 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0268] The memory 801 stores a computer storage medium, which stores computer-executable instructions for implementing the visual segmentation and understanding method of this application embodiment. The processor 802 is used to execute the above-described visual segmentation and understanding method, such as... Figure 3 As shown.

[0269] In some possible implementations, various aspects of the visual segmentation and understanding method provided in this application can also be implemented as a program product, including a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the visual segmentation and understanding method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 3 The steps are shown in the figure.

[0270] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0271] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0272] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0273] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0274] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0275] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0276] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0277] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0278] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0279] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0280] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0281] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0282] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A visual segmentation and understanding method, characterized in that, The method includes: The visual data to be processed and the corresponding prompt information are input into a multimodal coding network to obtain intermediate mask features and intermediate visual features. The prompt information is used to indicate the target region that needs to be segmented in the visual data to be processed, the intermediate mask features are used to describe the visual range of the target region, and the intermediate visual features are used to describe the visual content in the visual data to be processed. The intermediate mask features and the intermediate visual features are input into a visual decoding network to generate a segmentation result for the target region. Furthermore, the intermediate mask features, the intermediate visual features, and the preset learnable semantic features are input into a semantic perception network for cross-modal feature fusion to generate a visual feature representation and a semantic feature representation of the visual data to be processed; the visual feature representation, the semantic feature representation, and the prompt information are input into a semantic decoding network to generate a semantic understanding result for the target region; the visual feature representation is a visual expression of the target region that the semantic decoding network can understand, and the semantic feature representation is used to provide the semantic decoding network with semantic cues related to the target region.

2. The method as described in claim 1, characterized in that, Before inputting the visual data to be processed and the corresponding prompt information into a multimodal coding network to obtain intermediate mask features and intermediate visual features, the method further includes: The visual data to be processed and the task text prompts corresponding to the visual data to be processed are input into the prompt parsing network to generate prompt information corresponding to the visual data to be processed. The prompt information includes at least one of the following: first prompt information: a geometric prompt indicating the shape of the target region; second prompt information: a pseudo-mask prompt indicating the shape of the target region; and third prompt information: a semantic prompt indicating the semantic range of the target region.

3. The method as described in claim 2, characterized in that, The prompt information includes a third prompt information; the step of inputting the visual feature representation, the semantic feature representation, and the prompt information into a semantic decoding network to generate a semantic understanding result for the target region includes: Extract the third prompt information from the prompt information; Based on the third prompt information, semantic reasoning instructions for the semantic decoding network are generated; The semantic decoding network performs semantic reasoning on the visual feature representation and the semantic feature representation based on the semantic reasoning instructions, and generates a semantic understanding result for the target region.

4. The method as described in claim 1, characterized in that, The visual data to be processed is at least one video frame in the video data, and the video data includes multiple video frames; before inputting the visual data to be processed and the corresponding prompt information into a multimodal coding network to obtain intermediate mask features and intermediate visual features, the method further includes: After grouping the multiple video frames, the following operations are performed on each group of video frames: Based on the scene type of the current group of video frames, determine the frame interval for selecting the video frames to be processed; According to the frame interval, at least one video frame to be processed is extracted from the current group of video frames; The scene type is used to describe the scene changes in the current group of video frames; if the scene type is a static scene, then the frame interval is the second frame interval; the first frame interval is less than the second frame interval.

5. The method as described in claim 4, characterized in that, The scene type of the current group of video frames is determined in the following way: For each video frame in the current group of video frames, the pixel mean square error of the video frame is determined based on the pixel value distribution of the video frame; Based on the determined mean square errors of multiple pixels, the mean and standard deviation of the mean square errors of the multiple pixels are generated; If the mean is greater than a preset mean threshold and the standard deviation is greater than a preset standard deviation threshold, then the scene is determined to be a dynamic scene; otherwise, the scene is determined to be a static scene.

6. The method according to any one of claims 1 to 5, characterized in that, The step of inputting the intermediate mask features, the intermediate visual features, and the preset learnable semantic features into a semantic perception network for cross-modal feature fusion to generate visual feature representations and semantic feature representations of the visual data to be processed includes: Perform self-attention computation on the intermediate mask features to obtain the first updated mask features; Based on the intermediate visual features and the first updated mask features, perform cross-attention computation to obtain the second updated mask features; The learnable semantic features are fused with the second update mask features to obtain the updated semantic features; By performing a first feature mapping operation on the second updated mask feature, the second updated mask feature is converted into a visual feature representation that the semantic decoding network can understand; Furthermore, by performing a second feature mapping operation on the updated semantic features, the updated semantic features are converted into a semantic feature representation that the semantic decoding network can understand.

7. The method as described in claim 6, characterized in that, The step of performing self-attention computation on the intermediate mask features to obtain the first updated mask features includes: Based on a preset scoring mechanism, the importance score of each feature unit in the intermediate mask features is calculated respectively; wherein, the scoring mechanism is used to characterize the importance of each feature unit in the attention update process; Based on the calculated importance scores, feature units whose importance scores meet preset conditions are selected as global attention units, and the rest are selected as local attention units. Perform global attention calculation on each global attention unit to obtain the global attention calculation result; perform local sparse attention calculation on each local attention unit to obtain the local sparse attention calculation result. Based on the global attention calculation results and the local sparse attention calculation results, the intermediate mask features are updated to obtain the first updated mask features.

8. A visual segmentation and understanding device, characterized in that, The device includes: A multimodal coding unit is used to input the visual data to be processed and the prompt information corresponding to the visual data to be processed into a multimodal coding network to obtain intermediate mask features and intermediate visual features; the prompt information is used to indicate the target region that needs to be segmented in the visual data to be processed, the intermediate mask features are used to describe the visual range of the target region, and the intermediate visual features are used to describe the visual content in the visual data to be processed. A segmentation unit is used to input the intermediate mask features and the intermediate visual features into a visual decoding network to generate a segmentation result for the target region; The understanding unit is used to input the intermediate mask features, the intermediate visual features, and the preset learnable semantic features into a semantic perception network for cross-modal feature fusion to generate a visual feature representation and a semantic feature representation of the visual data to be processed; input the visual feature representation, the semantic feature representation, and the prompt information into a semantic decoding network to generate a semantic understanding result for the target region; the visual feature representation is a visual expression of the target region that the semantic decoding network can understand, and the semantic feature representation is used to provide the semantic decoding network with semantic cues related to the target region.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any one of the methods described in claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 7.

Images