A method for training an image segmentation network and related equipment
By constructing an auxiliary loss function with two-layer statistical constraints, the problems of over-aggregation of features and background bias in the visual backbone network are solved, improving the segmentation accuracy and cross-modal feature alignment stability of the image segmentation network, and achieving a plug-and-play effect with zero architectural modification.
Patent Information
- Application Number
- CN202611034879.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-13
- Publication Date
- 2026-11-17
AI Technical Summary
Existing visual backbone networks suffer from over-aggregation of features and background bias in image segmentation tasks, causing foreground information to diffuse into the background region. Furthermore, existing evaluation metrics cannot accurately measure the degree of fine-grained alignment, leading to difficulties in cross-modal alignment and performance bottlenecks.
An auxiliary loss function based on two-layer statistical constraints is constructed, including a first-order statistical constraint term and an extreme value statistical constraint term. The response distribution of the foreground and background regions is constrained by the mean interval and the peak interval, thereby improving the calibration in the feature extraction stage.
It significantly improves the segmentation accuracy and cross-modal feature alignment stability of image segmentation networks, without modifying the network structure during the inference stage, and is plug-and-play.
Smart Images

Figure CN123118658A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the interdisciplinary fields of computer vision, natural language processing, and multimodal models, and in particular to an image segmentation network training method and related equipment. Background Technology
[0002] In current image segmentation tasks, models need to accurately understand arbitrary context lengths based on natural language descriptions and segment target objects from images. Existing visual backbone networks generally suffer from severe feature artifacts and abnormally high-response regions when extracting dense features. The root cause is that, lacking fine-grained pixel-level annotations and a global attention mechanism, models tend to take a shortcut of over-aggregating features: using background features that are unrelated to the target but widely distributed to encode global semantics, causing foreground information to spread uncontrollably into the background. Existing evaluation metrics can only coarsely determine whether the highest response point of a feature falls within the rough bounding box of the target, and cannot accurately measure the fine-grained alignment within complex polygon masks. At the same time, when dealing with misaligned backbone networks, existing methods often only intervene crudely at the decoder end, failing to effectively interrupt this systematic background bias during the feature extraction stage. Due to the lack of fine-grained quantification metrics and explicit constraints on the output behavior of the visual backbone network, the model is easily misled by high-frequency noise in the background during the early stages of training, resulting in severe cross-modal alignment difficulties and performance bottlenecks. Summary of the Invention
[0003] The main objective of this application is to propose an image segmentation network training method, electronic device, storage medium, and program product based on two-layer statistical constraint-assisted loss, aiming to solve the problems of excessive feature aggregation and background bias in the visual backbone network in existing referential segmentation and image segmentation tasks.
[0004] To achieve the above objectives, one aspect of this application proposes an image segmentation network training method, the method comprising: The input image is acquired, and visual block features and global features of the input image are extracted through a visual backbone network. Obtain the real segmentation mask corresponding to the input image, and downsample the real segmentation mask to the grid level to divide the foreground region set and the background region set; Based on the response distribution of visual block features in the foreground region set and the response distribution of visual block features in the background region set, a two-layer auxiliary loss is constructed. The two-layer auxiliary loss includes a first-order statistical constraint term and an extreme value statistical constraint term. The first-order statistical constraint term is used to constrain the difference between the overall mean response of the foreground region set and the background region set, and the extreme value statistical constraint term is used to constrain the difference between the mean local extreme response of the foreground region set and the background region set. The dual-layer auxiliary loss is fused with the main segmentation loss to construct a total loss function. The total loss function is then used to train the visual backbone network to calibrate the feature output distribution of the visual backbone network.
[0005] In some embodiments, prior to constructing the two-layer auxiliary loss, the method further includes a step of generating a situation scoring matrix, the step of generating the situation scoring matrix comprising: Setting the first The image sample contains the first Each visual block feature is , The corresponding global feature is ; The correlation between each visual patch and the global features is calculated using cosine similarity, and a temperature hyperparameter is introduced. Perform smoothing adjustments and calculate the semantic score for each visual block. ; A situational score matrix is constructed from the semantic scores of all visual blocks, and the situational score matrix is used to characterize the spatial distribution situation of global semantics.
[0006] In some embodiments, the method further includes a step of performing a fine-grained quantitative evaluation of the feature posture output by the visual backbone network, wherein the step of performing a fine-grained quantitative evaluation of the feature posture output by the visual backbone network includes: Based on the situation score matrix and the foreground region set and background region set, calculate the spatial precise matching constraint index and the global response distribution constraint index; The spatial precision matching constraint indicators include center point hit rate and coverage hit rate; The global response distribution constraint index is the average maximum local score.
[0007] In some embodiments, the center point hit rate is calculated as follows: obtain the index of the extreme point with the highest score in the sample situation score matrix, map the extreme point index back to the center point coordinates of the original image, and if the coordinates fall within the foreground region of the real segmentation mask, it is determined to be a hit; the center point hit rate is the average of the hit rates of all samples. The coverage hit rate is calculated by calculating the proportion of foreground pixels in the region corresponding to the extreme point. Set coverage threshold The coverage hit rate is the percentage of all samples. The percentage of samples; The average maximum local score is calculated as follows: calculate the peak value of the maximum local response for each sample. The average maximum local score is the average of the peak values of the maximum local responses of all samples.
[0008] In some embodiments, the first-order statistical constraint term is a mean interval constraint term, and the mean interval constraint term is constructed as follows: Calculate the foreground region set separately Overall semantic score mean and background area set Overall semantic score mean :
[0009]
[0010] Based on the difference between the foreground mean and the background mean, and a preset safety threshold for the mean interval. Calculate the single-sample mean loss : .
[0012] In some embodiments, the extreme value statistical constraint term is a peak interval constraint term, and the peak interval constraint term is constructed as follows: Extract the foreground region set separately The top semantic score The average of the scores and background area set The top semantic score The average of the scores :
[0013]
[0014] Based on the difference between the mean high-frequency response of the foreground and the mean high-frequency response of the background and a preset peak interval safety threshold. Calculate the peak loss of a single sample. : .
[0016] In some embodiments, the dual-layer auxiliary loss The calculation method is as follows:
[0017] in, The peak penalty weight, The total number of samples, For single-sample mean loss, For single-sample peak loss; The total loss function The calculation method is as follows:
[0018] in, Main segmentation loss, The strength coefficient for auxiliary loss intervention.
[0019] In some embodiments, when the situation scoring matrix determines that there are local spurious peaks in the background region exceeding a preset threshold, the peak penalty weight... And / or the auxiliary loss intervention strength coefficient It is configured to amplify to enhance the gradient intervention strength of the two-layer auxiliary loss on the visual backbone network.
[0020] In some embodiments, the method is performed only during the training phase of the visual backbone network, and the two-layer auxiliary loss is removed during the inference phase, while the inference structure and inference parameters of the visual backbone network remain unchanged.
[0021] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0022] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.
[0023] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.
[0024] The embodiments of this application include at least the following beneficial effects: This application provides an image segmentation network training method, electronic device, storage medium, and program product. This scheme constructs a two-layer auxiliary loss that includes a first-order statistical constraint term and an extreme value statistical constraint term. It constrains the difference between the overall mean response of the foreground region and the background region from the global distribution level, and constrains the difference between the mean peak response of the two from the local extreme value level. This makes the visual backbone network subject to the dual synergistic constraints of "global water level suppression" and "local peak suppression" during the training process. In this way, it systematically interrupts the shortcut of foreground semantics to spread to the background and accurately smooths out isolated noise points in the background during the feature extraction stage. Finally, it significantly improves the segmentation accuracy and cross-modal feature alignment stability in a plug-and-play manner without modifying any network structure in the inference stage. Attached Figure Description
[0025] Figure 1 This is a flowchart of an image segmentation network training method provided in an embodiment of this application; Figure 2 This is an overall architecture diagram of the segmentation system based on visual backbone situation assessment and joint auxiliary loss in the embodiments of this application; Figure 3 This is a flowchart of spatial precise matching constraints and feature situation quantification in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the principle of mean interval and peak interval calculation and feature calibration reconstruction in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0028] Referencing image segmentation requires models to accurately segment the described target object from an image based on natural language descriptions. Unlike traditional semantic segmentation, this task is not limited to predefined object categories, providing a more flexible and human-like interaction method, and has broad application prospects in practical scenarios such as image interactive editing, human-computer interaction, and robot grasping and detection.
[0029] In recent years, visual backbone networks based on the Transformer architecture have made significant progress in the task of algebraic representation segmentation due to their powerful global modeling capabilities. However, existing visual backbone networks generally suffer from the following technical shortcomings when extracting dense features: (a) Feature over-aggregation and background bias problem When visual backbone networks lack fine-grained pixel-level annotations and possess a global self-attention mechanism, the model tends to take a shortcut of feature over-aggregation: using features from widely distributed background regions unrelated to the target to encode global semantics, causing foreground information to uncontrollably diffuse into the background. The root cause of this phenomenon is that the global self-attention mechanism allows the model to easily "borrow" foreground semantics from a large number of background visual blocks, thereby reducing classification loss without needing precise target localization. This systematic bias is particularly severe in the early stages of training, causing significant difficulties in cross-modal alignment and performance bottlenecks.
[0030] (ii) Existing evaluation indicators are too coarse-grained Existing evaluation methods often only coarsely determine whether the highest response point of a feature falls within the rough bounding box of the target, failing to accurately measure the fine-grained alignment within complex polygonal masks. Bounding box-level evaluation can misclassify a large amount of background area around the target as valid feature regions, thus masking the model's true deficiencies in pixel-level accuracy.
[0031] (iii) Existing auxiliary loss schemes have a single constraint dimension. Existing technologies, when addressing the feature shift problem in visual backbone networks, typically only intervene crudely at the decoder end, failing to effectively interrupt the systematic background bias during the feature extraction stage. Furthermore, existing auxiliary loss schemes are mostly single contrastive or pixel-level losses, failing to jointly constrain feature shift from two different dimensions: "global distribution" and "local extrema." This makes it difficult to comprehensively address the two related but fundamentally different problems: foreground semantic diffusion into the background and local pseudo-peaks in the background.
[0032] In view of this, this application provides an image segmentation network training method, electronic device, storage medium and program product based on two-layer statistical constraint auxiliary loss. This scheme constructs a two-layer auxiliary loss of first-order statistical constraint (mean interval) and extreme value statistical constraint (peak interval), and systematically calibrates the feature distribution of the visual backbone network in the feature extraction stage, thereby improving the segmentation accuracy and cross-modal feature alignment stability of the model.
[0033] The image segmentation network training method provided in this application relates to the interdisciplinary fields of computer vision, natural language processing, and multimodal models. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software may be an application implementing the image segmentation network training method, but is not limited to the above forms.
[0034] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0035] like Figure 1 As shown, this embodiment provides an image segmentation network training method, which specifically includes the following steps: Step S101: Obtain the input image, and extract the visual block features and global features of the input image through the visual backbone network; Step S102: Obtain the real segmentation mask corresponding to the input image, and downsample the real segmentation mask to the grid level to divide the foreground region set and the background region set; Step S103: Based on the response distribution of visual block features in the foreground region set and the response distribution of visual block features in the background region set, a two-layer auxiliary loss is constructed; the two-layer auxiliary loss includes a first-order statistical constraint term and an extreme value statistical constraint term. The first-order statistical constraint term is used to constrain the difference between the overall mean response of the foreground region set and the background region set, and the extreme value statistical constraint term is used to constrain the difference between the mean local extreme response of the foreground region set and the background region set. Step S104: Fuse the dual-layer auxiliary loss with the main segmentation loss to construct a total loss function, and use the total loss function to train the visual backbone network to calibrate the feature output distribution of the visual backbone network.
[0036] Below, in conjunction with Figure 2 , Figure 3 , Figure 4 The present application provides a detailed description and explanation of the solutions in its embodiments, along with specific application examples.
[0037] (1) Image feature perception and cross-modal benchmark scoring 1.1) Dual-modal signal perception extraction: The visual backbone network is used to extract multi-scale visual patch tokens from the input image, and global classification features or cross-modal global features after fusion are extracted.
[0038] 1.2) Local-Global Semantic Feature Scoring: For the first... The nth image sample is set to contain the nth image sample. Each visual block feature is Its corresponding global feature is The system calculates the correlation between each visual patch and global features using cosine similarity and introduces a temperature hyperparameter. Smoothing adjustments are performed, and the semantic score for each visual block is calculated. The calculation formula is as follows:
[0039] This scoring matrix constitutes an initial feature profile that reflects the global semantic space distribution.
[0040] (2) Situation assessment algorithm based on unit level See Figure 3 This embodiment constructs a set of underlying rules for feature generation, defining a constraint architecture that the output features of the backbone network must conform to "foreground response mean dominance + background high-frequency peak suppression," thereby mathematically ensuring the accuracy of feature alignment. This embodiment downsamples the real segmentation mask to the grid level to divide the foreground region set. Collection with background area And the following fine-grained situation quantification indicators were defined: 2.1) Spatial exact matching constraints (center mask hit rate and coverage, PiM metric): Center Point Hit Rate (PiM-center): Retrieves the index of the highest-scoring extreme point in the sample. Map it back to the center point coordinates of the original image. If the coordinates fall within the foreground mask, then a hit is determined. Otherwise, it is 0. The batch evaluation value is: .
[0041] Coverage Hit Rate (PiM-cover@t): Calculates the percentage of foreground pixels within the region corresponding to the highest score. Set coverage threshold The batch evaluation value is: .
[0042] 2.2) Global response distribution constraints (average maximum local score, AvgMaxPatchScore metric): The system calculates the maximum local response peak for each sample. And averaged over the entire evaluation set: .
[0043] This evaluation model is not a simple numerical statistical analysis, but rather addresses the "feature background bias" defect that visual backbone networks are prone to when lacking pixel-level supervision (i.e., the model tends to use a broad background as a shortcut to encode global semantics). It establishes three major feature constraint logics from the bottom up: 1) Spatial Precise Boundary Constraints (for PiM Metrics): Traditional bounding box evaluation often misclassifies a large amount of background noise around the target as valid features. This system forcibly restricts the highest response point or high coverage area strictly within an irregular real polygon mask, thus eliminating interference from the nearby background at its source. 2) Consistency Consistency between Response Strength and Localization (for AvgMaxPatchScore): This constraint stipulates that not only must the absolute peak value of the feature response score (Patch Score) be sufficiently high, but the spatial coordinates of the peak value must also be strictly bound to the foreground hit rate (hit). Mathematically, this prevents the network from exhibiting abnormal activation due to "blind confidence" in local high-frequency background features. 3) Global-local semantic correspondence constraint: By calculating the cosine similarity between local visual blocks and global features and introducing a temperature coefficient τ (e.g., τ=0.05) for smoothing, the absolute inner product value is forced to be transformed into a relative situational distribution, ensuring that the evaluation benchmark of the system will not become invalid with drastic changes in image illumination and scale.
[0044] (3) Joint auxiliary loss parameter calibration of the algorithm engine See Figure 4 Based on the aforementioned quantitative indicators, the system extracts the response distributions of the foreground and background, and constructs a two-layer auxiliary loss function to calibrate the gradient optimization direction: Phase 1: Mean-Margin Calculation Calculate the overall semantic score mean for the foreground set and the background set separately: Foreground mean:
[0045] Background mean:
[0046] Single-sample mean loss: in, This is a set mean interval safety threshold. This calculation mandates that, after removing regional interference, the overall response of the foreground must be higher than the background, and at least maintain... Safety margin.
[0047] Phase Two: Peak Margin Calculation Extract the foreground and background with the strongest response. The average of the scores quantifies the local extremum dispersion: Foreground / Background High-Frequency Response:
[0048]
[0049] Single-sample peak loss:
[0050] This auxiliary loss engine defines the underlying rules for penalizing feature offsets during backpropagation by introducing a dynamic safety margin: 1) Global Situational Blocking Constraint (for the Mean-Margin term): Under conditions with only coarse-grained semantic supervision and a global self-attention mechanism, the backbone network is prone to "feature diffusion shortcuts"—that is, spreading foreground semantics into a large number of background visual blocks to easily reduce classification loss. This rule forces the overall mean of the foreground to be ( Not only is it higher than the background mean ( ), and must be maintained (like The absolute safety water level difference directly cuts off the shortcut for the backbone network to borrow features from the background from a global distribution perspective.
[0051] 2) Local Abnormal Extrema Suppression Constraint (for the Peak-Margin term): As training depth increases, unaligned background regions often abruptly produce isolated high-frequency noise points (i.e., local pseudo-peaks or abnormally high-norm features) due to uneven gradient accumulation. This rule extracts... (like ) Calculate the extreme points independently and establish The physical significance of the penalty barrier lies in its ability to act as a "high-frequency noise filter," accurately identifying and selectively "flattening" background pseudo-peaks that attempt to mislead network attention without affecting the smooth distribution of the overall background.
[0052] (4) Overall goal inference mechanism based on multi-task joint optimization Based on the above parameters, the system uses an adaptive regulator to generate the total auxiliary loss. and the main partition loss The signals are fused together, and the final gradient inference signal is output:
[0053]
[0054] When the system determines that the background pseudo-peak value exceeds the limit, the confidence level regulator... This amplifies the auxiliary loss gradient, directly intervening in the attention distribution of the backbone network to achieve smooth calibration of the feature data stream.
[0055] In one embodiment, the system backend connects to a multimodal pre-trained visual backbone network. Preset hyperparameters are loaded: similarity temperature is set. Mean interval Number of extreme points Peak interval Peak penalty weight and total auxiliary intervention intensity .
[0056] (5) On-site testing and real-time feature mapping During training, the system monitors the background region response distribution in the situational scoring matrix S in real time. When a local spurious peak exceeding a preset threshold is detected in the background region (e.g., a single visual block in the background region scores more than 80% of the maximum score in the foreground region, or the mean of the top-5 values in the background region exceeds the mean of the top-5 values in the foreground region), the system automatically performs the following adaptive adjustment operation: 1) Weight the peak penalty The value is increased from the default value of 0.4 to 0.6-0.8, which strengthens the weight of the peak interval constraint term in the auxiliary loss. 2) Incorporate auxiliary loss intervention strength coefficient Increase the value from the default value of 0.02 to 0.05-0.10 to enhance the contribution of the two-layer auxiliary loss to the total loss function.
[0057] Through the above adaptive adjustment, the system can dynamically respond to different types of feature offsets during training, achieving more flexible and efficient gradient calibration.
[0058] (6) Case-by-case inference mechanism in complex scenarios To address the feature diffusion and erroneous aggregation defects that visual backbone networks are prone to when extracting dense features, the algorithm engine in this embodiment automatically triggers different inference constraint paths based on the situational characteristics of different image backgrounds: Scenario 1: Inference from a complex background with high-frequency local interference Situational characteristics: The image background is a densely textured grassland, forest, or scene with many interfering objects. In such scenes, the visual backbone network is prone to generating abnormally high response extreme points in the high-frequency texture regions of the background (i.e., local abnormal high activation regions appear on the feature map), causing the highest-scoring feature point to deviate from the true target, and the center point hit rate to be zero or significantly low.
[0059] Inference Mechanism: At this point, local spurious peaks are extremely obvious. The system automatically extracts the highest-scoring feature blocks and triggers a strong penalty for the peak interval term. The system's optimization computing power is preferentially tilted towards suppressing local spurious peaks in the background, forcibly smoothing out erroneous high-frequency response points.
[0060] Scenario 2: Inference of a smooth but large-area monotonous scene Characteristic features: The image background consists of large, smooth, and expansive areas such as sky or solid-color walls. In such scenarios, the network tends to over-aggregate features within these expansive areas to reduce classification difficulty, resulting in an abnormally high overall mean value for the background region, and even extensive diffusion of foreground semantics into the background.
[0061] Inference Mechanism: Since there are no prominent isolated extreme points, the system mainly triggers the mean interval term penalty. Relying on a preset safety threshold, the gradient is guided to suppress the feature activation distribution level of the background region from a global perspective, thereby increasing the concentration of the target region.
[0062] (7) Specific application case - Visual feature correction inference based on complex background images 7.1) Scenario Definition This embodiment selects a complex natural scene with a broad background (such as a large area of grass or a wall) as the target scene. In such scenes, the backbone network is highly susceptible to feature bias, causing a large number of high-response points to fall on invalid backgrounds. This embodiment aims to use an auxiliary loss algorithm to accurately infer and correct the network's feature output.
[0063] 7.2) Inference Logic and Calculation Process Phase 1: Characteristic Situation Quantification and Benchmark Scoring After extracting features and calculating the situational score matrix through the visual backbone network, assuming that the highest-scoring feature point does not fall inside the true mask of the target, the PiM-center index is 0, indicating that there is a serious feature background bias.
[0064] Phase 2: Mean and Peak Interval Calculation Calculate the target area separately ( ) and background area ( The response status of the target area. Assuming the average value of the target area is measured... background area mean Substituting into the formula produces the mean loss:
[0065] Extract the top rankings Highest score: Average of Top-5 in the target area The background area contains high-frequency noise. Substituting into the formula yields the peak loss:
[0066] Phase 3: Auxiliary loss backpropagation and feature recalibration: Calculate the total inference penalty gradient:
[0067]
[0068] The system backpropagates the additionally calculated penalty gradient of 0.00444 directly to the visual backbone network. In subsequent iterations, the system forces the high-frequency noise in the background to decrease while simultaneously increasing the overall mean response of the target region, ultimately restoring the PiM metric to 1 and completing the correction.
[0069] (8) Advantages and beneficial effects In summary, compared with the prior art, this application has at least the following advantages and beneficial effects: 1) Construct a mean margin constraint to block feature diffusion shortcuts from a global perspective. By establishing a mean margin constraint, the mean semantic response of the foreground region is forced to be higher than the mean semantic response of the background region, while maintaining a preset safety margin. This constraint directly cuts off the shortcut for the visual backbone network to borrow features from the background region to reduce classification loss from a global distribution perspective, significantly reducing the probability of false activation in the background region in complex scenes.
[0070] 2) Construct peak interval constraints to accurately suppress local spurious peaks in the background. An innovative peak interval constraint term was designed, specifically extracting the Top-K extreme points with the strongest responses in the foreground and background for independent calculation, forcibly widening the distance between the high-response foreground and the high-response background. This constraint can accurately identify and suppress isolated high-frequency noise points (local pseudo-peaks) in the background region caused by uneven gradient accumulation, without affecting the overall smooth distribution of the background region.
[0071] 3) The mean and peak values work together to form a three-dimensional constraint system. The mean interval constraint and the peak interval constraint jointly constrain the feature shift from the two dimensions of "global distribution" and "local extrema" respectively. The two work together to form a three-dimensional containment of the problem of excessive feature aggregation. The mean term solves the problem of foreground semantics spreading to the background over a large area, while the peak term solves the problem of abnormally high response in the background. This makes up for the shortcomings of existing single contrast loss or pixel-level loss in terms of constraint dimensions.
[0072] 4) Plug and play feature with zero architectural modification The auxiliary loss mechanism in this application only takes effect during the training phase and is completely removed during the inference phase, without changing the original network inference structure and parameters. This characteristic makes this application a general training aid that can be easily adapted to various existing visual backbone network architectures, and has broad industrial application value.
[0073] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0074] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0075] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.
[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0077] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0078] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0079] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0080] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.
[0081] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0082] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0084] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0085] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0086] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0087] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0091] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A method for training an image segmentation network, characterized in that, The method includes the following steps: The input image is acquired, and visual block features and global features of the input image are extracted through a visual backbone network. Obtain the real segmentation mask corresponding to the input image, and downsample the real segmentation mask to the grid level to divide the foreground region set and the background region set; Based on the response distribution of visual block features in the foreground region set and the response distribution of visual block features in the background region set, a two-layer auxiliary loss is constructed. The two-layer auxiliary loss includes a first-order statistical constraint term and an extreme value statistical constraint term. The first-order statistical constraint term is used to constrain the difference between the overall mean response of the foreground region set and the background region set, and the extreme value statistical constraint term is used to constrain the difference between the mean local extreme response of the foreground region set and the background region set. The dual-layer auxiliary loss is fused with the main segmentation loss to construct a total loss function. The total loss function is then used to train the visual backbone network to calibrate the feature output distribution of the visual backbone network.
2. The method according to claim 1, characterized in that, Before constructing the two-layer auxiliary loss, the method further includes a step of generating a situation scoring matrix, the step of generating the situation scoring matrix including: Setting the first The image sample contains the first Each visual block feature is , The corresponding global feature is ; The correlation between each visual patch and the global features is calculated using cosine similarity, and a temperature hyperparameter is introduced. Perform smoothing adjustments and calculate the semantic score for each visual block. ; A situational score matrix is constructed from the semantic scores of all visual blocks, and the situational score matrix is used to characterize the spatial distribution situation of global semantics.
3. The method according to claim 1, characterized in that, The method further includes a step of performing a fine-grained quantitative evaluation of the feature posture output by the visual backbone network, wherein the step of performing a fine-grained quantitative evaluation of the feature posture output by the visual backbone network includes: Based on the situation score matrix and the foreground region set and background region set, calculate the spatial precise matching constraint index and the global response distribution constraint index; The spatial precision matching constraint indicators include center point hit rate and coverage hit rate; The global response distribution constraint index is the average maximum local score.
4. The method according to claim 3, characterized in that, The center point hit rate is calculated as follows: obtain the index of the extreme point with the highest score in the sample situation score matrix, map the extreme point index back to the center point coordinates of the original image, and if the coordinates fall within the foreground area of the real segmentation mask, it is determined to be a hit; the center point hit rate is the average of the hit rates of all samples. The coverage hit rate is calculated by: calculating the proportion of foreground pixels in the region corresponding to the extreme point. Set coverage threshold The coverage hit rate is the percentage of all samples. The percentage of samples; The average maximum local score is calculated as follows: calculate the peak value of the maximum local response for each sample. The average maximum local score is the average of the peak values of the maximum local responses of all samples.
5. The method according to claim 1, characterized in that, The first-order statistical constraint term is a mean interval constraint term, and the mean interval constraint term is constructed as follows: Calculate the foreground region set separately Overall semantic score mean and background area set Overall semantic score mean : Based on the difference between the foreground mean and the background mean, and a preset safety threshold for the mean interval. Calculate the single-sample mean loss : 。 6. The method according to claim 1, characterized in that, The extreme value statistical constraint term is a peak interval constraint term, and the peak interval constraint term is constructed as follows: Extract the foreground region set separately The top semantic score The average of the scores and background area set The top semantic score The average of the scores : Based on the difference between the mean high-frequency response of the foreground and the mean high-frequency response of the background and a preset peak interval safety threshold. Calculate the peak loss of a single sample. : 。 7. The method according to claim 1, characterized in that, The dual-layer auxiliary loss The calculation method is as follows: in, The peak penalty weight, The total number of samples, For single-sample mean loss, For single-sample peak loss; The total loss function The calculation method is as follows: in, Main segmentation loss, The strength coefficient for auxiliary loss intervention.
8. The method according to claim 7, characterized in that, When the situational scoring matrix determines that there are local spurious peaks in the background region exceeding a preset threshold, the peak penalty weight... And / or the auxiliary loss intervention strength coefficient It is configured to amplify to enhance the gradient intervention strength of the two-layer auxiliary loss on the visual backbone network.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.
