Image object detection method and training method of image object detection model

Abstract

Disclosed are an image object detection method and a training method of an image object detection model, the image object detection method adopting fusion of Gaussian mixture structured modeling and adversarial transfer learning. Foreground region statistical features and deep features representing spectral distribution of a to-be-detected image are extracted and fused; Gaussian mixture model parameters are regressed to obtain position parameters, spatial scale parameters and component weights corresponding to multiple components of an object, thereby generating an object mixture mask and aggregating to obtain a detection bounding box, and then a class label and an instance segmentation mask are predicted. During training, three-layer structured constraints, two-stage sorting alignment and domain adversarial optimization are combined, thereby improving detection segmentation consistency of multi-component and occluded objects and cross-scene adaptability.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and image processing technology, and in particular to an image object detection method and a training method for an image object detection model, so as to obtain object detection bounding boxes, category labels and instance segmentation masks. Background Technology

[0002] Image object detection and instance segmentation are fundamental techniques in computer vision. Image object detection identifies the regions containing objects in a single static image and outputs detection bounding boxes and category information. Instance segmentation, building upon object detection, further determines the actual pixel regions occupied by each object. These techniques can be applied to scenarios such as industrial quality inspection, security static image recognition, commodity image detection, and remote sensing image analysis.

[0003] Existing detection and segmentation frameworks typically train neural networks using images labeled with object categories, detection bounding boxes, and pixel-level instance masks. Some schemes represent a complete object with a single center location or a single spatial distribution, and obtain the bounding box and pixel mask by the detection task and the segmentation task, respectively. For objects with multiple physical parts, local texture differences, mutual occlusion, or dense distribution, a single center location is insufficient to fully represent the internal spatial structure of the object. This can easily lead to the mixing of response regions of adjacent objects or different parts of the same object, thus affecting the spatial consistency between the bounding box and the pixel-level mask.

[0004] Furthermore, there may be domain shift between training images obtained from one scene and detection images from another scene. Domain shift can be caused by differences in physical imaging conditions such as camera resolution, lens focal length, shooting distance, imaging scale, ambient lighting, infrared illumination, surface texture, background composition, or the condition of objects. If a model trained on the original scene is directly used for a new scene, the detection and segmentation results may be affected. Re-acquiring a large number of target scene images and performing pixel-level instance mask annotation requires significant manual labor.

[0005] Existing cross-domain detection and segmentation schemes can adapt models to target domain images by aligning the distribution of depth features. However, when using only depth features obtained from a single path, some interpretable contour, spatial distribution, and texture statistics may not be fully utilized in scenes with few samples, low illumination, weakened local textures, or significant changes in object scale. Furthermore, using only image-level global alignment may lack fine-grained constraints on the correspondence between individual objects and their categories.

[0006] Therefore, an image object detection and instance segmentation scheme is needed that enables multiple parts of an object to have clear spatial representations, allowing detection bounding boxes and instance segmentation to be built on interrelated part-level spatial information, and improving the model's adaptability to changes in imaging conditions when only a small amount of supervision information is provided in the target domain. Summary of the Invention

[0007] This disclosure provides an image object detection method and an image object detection model training method to address at least one of the following problems: difficulty in representing multiple internal components of an object from a single spatial location; lack of a common spatial basis between the detection bounding box and pixel-level instance segmentation results; and the impact of imaging differences between different image acquisition scenarios on the applicability of the model.

[0008] This disclosure provides an image object detection method. The image object detection method includes: acquiring a target image containing a target object, the target object comprising multiple components; extracting a foreground region from the target image to obtain candidate foreground regions, and extracting foreground region statistical features of the candidate foreground regions according to preset image statistical rules to obtain a first feature; performing a frequency domain transformation on the target image to obtain a frequency domain representation of the target image, and extracting depth features from the frequency domain representation to obtain a second feature characterizing the spectral distribution of the target image; fusing the first feature and the second feature to obtain a fused feature map; and based on the fused feature map, obtaining multiple sets of component spatial parameters corresponding one-to-one with the multiple components using a Gaussian mixture model parameter regression method, each set of component spatial parameters including a positional parameter characterizing the spatial location of the corresponding component, a spatial scale parameter determining the spatial coverage of the corresponding component, and so on. The process involves: assigning component weights to corresponding components; generating a component spatial mask corresponding to each component based on the positional and spatial scale parameters included in each set of component spatial parameters; weighting and summing the values ​​of multiple component spatial masks at the same pixel position according to the component weights corresponding to each component to obtain an object blending mask; aggregating the positional and spatial scale parameters of multiple components based on multiple sets of component spatial parameters and according to the component weights corresponding to each component to determine the overall position and overall coverage of the object to be detected; generating a rectangular detection bounding box based on the overall position and overall coverage; predicting an instance mask based on the object blending mask to obtain an instance segmentation mask; predicting the category of the object to be detected based on the fused feature map to obtain a category label; and outputting the detection bounding box, category label, and instance segmentation mask.

[0009] According to one or more embodiments of this disclosure, the number of multiple sets of component spatial parameters is a preset K, where K is an integer not less than 2; the component weight corresponding to each of the multiple components is a non-negative value, and the sum of the component weights corresponding to the multiple components is 1; the component spatial mask is a two-dimensional Gaussian response map centered on the corresponding position parameter, and the spatial attenuation range is determined by the corresponding spatial scale parameter and a preset scaling parameter.

[0010] According to one or more embodiments of this disclosure, when aggregating position parameters and spatial scale parameters according to component weights, the two coordinate axes of the image to be detected are aggregated according to the following formulas:

[0011] in, , , and These represent the position parameter, spatial scale parameter, and component weight of the k-th component on the u-th coordinate axis, respectively. and These represent the global position parameter and global spatial scale parameter of the object to be detected on the u-th coordinate axis, respectively; the two boundaries of the detection bounding box on the u-th coordinate axis are respectively and , These are preset coefficients.

[0012] According to one or more embodiments of this disclosure, the spatial scale parameter includes scale components set along two coordinate axes of the image to be detected, or includes a covariance matrix for characterizing the spatial coverage of the corresponding component in different directions; the component spatial mask generated according to the spatial scale parameter has different spatial attenuation ranges in at least two directions.

[0013] This disclosure also provides a training method for an image object detection model. The training method includes: acquiring training images and their training annotations, the training annotations including detection bounding box annotations, and at least a portion of the training images also including object category annotations and instance segmentation mask annotations; extracting fusion feature maps from the training images using the model to be trained, and performing Gaussian mixture model parameter regression based on the fusion feature maps to obtain multiple sets of component spatial parameters corresponding one-to-one with multiple components of the same object in the training images, each set of component spatial parameters including positional parameters representing the spatial location of the corresponding component, spatial scale parameters determining the spatial coverage of the corresponding component, and component weights of the corresponding component; generating multiple component spatial masks corresponding one-to-one with the multiple components based on the multiple sets of component spatial parameters, weighting and summing the multiple component spatial masks according to the component weights corresponding to each component in the multiple components to obtain an object fusion mask, and aggregating the multiple sets of component spatial parameters to obtain the object's detection bounding box; obtaining the object's instance segmentation mask based on the object fusion mask, and obtaining the object's category prediction result based on the fusion feature map; and determining the object's category based on the detection bounding box annotations and the detection bounding box... The process involves several steps: determining the bounding box regression loss; determining the instance mask segmentation loss based on the instance mask annotations and the instance mask for training images with instance segmentation mask annotations; determining the spatial coverage of each component based on its positional and spatial scale parameters, and determining the component inclusion constraint loss based on the inclusion relationship between the spatial coverage and the object's detection bounding box; determining the inter-object diversity constraint loss based on the similarity between the object mixing masks of different objects in the training image; determining the intra-object component diversity constraint loss based on the similarity between the spatial masks of multiple components of the same object; updating the model parameters of the model to be trained based on the total loss to obtain the trained image object detection model; the total loss includes the category correlation loss determined by the training annotations and category prediction results, the bounding box regression loss, the instance mask segmentation loss, the component inclusion constraint loss, the inter-object diversity constraint loss, and the intra-object component diversity constraint loss; and finally, the trained image object detection model is used to output the detection bounding box, category label, and instance segmentation mask of the object to be detected based on the image to be detected.

[0014] According to one or more embodiments of this disclosure, the component inclusion constraint loss penalizes the portion of the spatial coverage of each of the multiple components that exceeds the detection bounding box of the object; the inter-object diversity constraint loss penalizes the portion of the spatial coverage of each component that exceeds the detection bounding box of the object; the component diversity constraint loss is determined based on the cosine similarity between the spatial masks of different components in the multiple component spatial masks of the same object, and increases with the increase of the cosine similarity.

[0015] According to one or more embodiments of this disclosure, the training images include source domain training images and target domain training images. The training annotations of the source domain training images include object category annotations, detection bounding box annotations, and instance segmentation mask annotations. The target domain training images use detection bounding box annotations as supervision information. The process of updating the model parameters of the model to be trained includes sequentially executing a first training phase and a second training phase. In the first training phase, cross-image negative samples composed of source domain training images and target domain training images are used to perform image-level relevance ranking between the fusion feature map weighted by the object mixing mask and the object category label. In the second training phase, other objects located in the same training image as the current object are used as local negative samples to perform object-level relevance ranking between the fusion feature map weighted by the object mixing mask of the current object and the category label of the current object. In each of the two training phases, a ranking alignment loss is determined based on the relevance ranking result of that training phase, and the total loss also includes the ranking alignment loss.

[0016] According to one or more embodiments of this disclosure, the model to be trained includes a feature extractor, a detection segmentation head, and a domain discriminator. A fused feature map of a source domain training image and a target domain training image is input into the domain discriminator to obtain a domain discrimination result characterizing the data domain to which the fused feature map belongs, and a domain discrimination loss is determined based on the domain discrimination result. Domain adversarial optimization of the feature extractor is performed by alternately executing a first optimization step and a second optimization step to reduce the domain distribution difference between the fused feature maps of the source domain training image and the target domain training image. In the first optimization step, the model parameters of the domain discriminator are fixed, and the model parameters of the feature extractor and the detection segmentation head are updated based on the total loss and the domain discrimination loss weighted by a preset domain adversarial loss balancing coefficient to reduce the total loss and increase the domain discrimination loss. In the second optimization step, the model parameters of the feature extractor and the detection segmentation head are fixed, and the model parameters of the domain discriminator are updated based on the domain discrimination loss to reduce the domain discrimination loss.

[0017] Another aspect of this disclosure provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-described image object detection method or image object detection model training method.

[0018] Another aspect of this disclosure provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the aforementioned image object detection method or the training method for an image object detection model.

[0019] This disclosure characterizes a target object by using multiple sets of component spatial parameters, each corresponding to a specific component. These parameters are used to form both the object blending mask and to determine the overall position and coverage of the target object, thus providing a common component-level spatial information foundation for both bounding box detection and instance segmentation. Diversity constraints are set for different objects and components within the same object, along with implicit constraints between components and object detection bounding boxes, maintaining the hierarchical spatial relationship between "objects" and components during training. Through a two-stage sorting alignment from cross-image negative samples to local negative samples within the same image, and domain adversarial optimization between the feature extractor and the domain discriminator, constraints can be applied to cross-scene semantic relevance, the distinction between objects and categories within the same image, and feature distribution differences caused by imaging condition variations. This allows for model adaptation even when pixel-level instance mask annotations in the target domain are insufficient. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the main flow of an image object detection method according to an embodiment of the present disclosure.

[0021] Figure 2 This is a schematic diagram of the foreground region statistical features and frequency domain depth features extraction and fusion sub-process according to an embodiment of the present disclosure.

[0022] Figure 3 This is a schematic diagram of the component space mask generation and object mixing mask acquisition sub-process according to an embodiment of the present disclosure.

[0023] Figure 4 This is a schematic diagram of the component space parameter aggregation and bounding box generation sub-process according to an embodiment of the present disclosure.

[0024] Figure 5 This is a schematic diagram of the main flow of an image object detection model training method according to an embodiment of the present disclosure.

[0025] Figure 6 This is a schematic diagram of the sub-process for determining the detection box regression loss, instance mask segmentation loss, and three-layer structured constraint loss according to an embodiment of this disclosure.

[0026] Figure 7 This is a schematic diagram of the two-stage sorting and alignment process in step S1100 according to an embodiment of the present disclosure.

[0027] Figure 8 This is a schematic diagram of a domain adversarial update process according to an embodiment of the present disclosure.

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0029] The embodiments of this disclosure will be further described below with reference to the accompanying drawings. It should be understood that the following embodiments are used to illustrate the technical solutions of this disclosure, and are not intended to limit the scope of protection of this disclosure. Where there is no contradiction, the technical features in the various embodiments and variations can be combined with each other.

[0030] Unless otherwise stated, the "image to be detected" in this disclosure refers to a single static image acquired by an image acquisition device and used for computer vision processing. It can be an RGB image or an image composed of other image channels while maintaining the same spatial coordinate relationship. The "object to be detected" is the object in the image to which detection bounding boxes, category labels, and instance segmentation masks need to be obtained. A "part" is a local physical region or structural region within the object to be detected that has a corresponding spatial location and spatial coverage. An object to be detected includes multiple parts, each of which may correspond to different contour regions, texture regions, functional regions, or vulnerable regions.

[0031] In this disclosure, the "part spatial mask" (single Gaussian spatial mask) is a spatial response map formed based on a set of part spatial parameters; the "object blending mask" (complete object blending mask) is an intermediate spatial representation obtained by weighted summation of multiple part spatial masks at the same pixel location according to part weights; and the "instance segmentation mask" is a pixel-level mask predicted by the model through the instance mask and output as the final result. The object blending mask and the instance segmentation mask serve different purposes: the former is used to aggregate the spatial responses of multiple parts and guide subsequent instance mask prediction, while the latter is used to indicate the pixel region actually occupied by the object to be detected.

[0032] In this disclosure, the "spatial scale parameter" is used to determine the spatial coverage of the corresponding component and control the spatial response scale of the component's spatial mask. "Spatial coverage" refers to the component-level geometric range determined based on the position parameter and the spatial scale parameter; "spatial attenuation range" refers to the range from the center indicated by the position parameter to the area where the response value of the component's spatial mask attenuates outward, and can be determined by a preset scaling parameter in addition to the spatial scale parameter; "overall coverage" refers to the object-level geometric range obtained by aggregating the spatial scale parameters of multiple components.

[0033] Reference Figure 1 The image object detection method of this embodiment can be executed by a server, edge computing device, camera processing device, or other electronic device with image processing capabilities. The method includes steps S100 to S600. The step numbers are for ease of explanation in conjunction with the accompanying drawings and do not necessarily imply that each step must be executed in a unique temporal order without violating data dependencies.

[0034] In step S100, an image of the object to be detected is acquired. The object to be detected comprises multiple components. For example, the object to be detected may be an industrial workpiece, facility enclosure, commodity, remotely sensed feature, or other object requiring object detection and instance segmentation, composed of multiple functional or structural parts. The image to be detected may be acquired by a fixed camera, mobile camera equipment, industrial camera, remote sensing imaging equipment, or other image acquisition device.

[0035] In one embodiment, the input is a single static RGB image. ,in, and These represent the image height and image width, respectively. Preprocessing can be performed on the input image to reduce the impact of differences in image size, brightness, and noise on subsequent feature extraction. Preprocessing may include at least one of the following: uniformly scaling the image to a preset size, performing brightness equalization using histogram equalization, and reducing salt-and-pepper noise and texture noise using Gaussian filtering. Preprocessing is an optional, routine preparation operation, and its output still preserves the two-dimensional spatial relationships in the image to be detected.

[0036] The "multiple parts" of the object to be detected do not need to have completely separate boundaries in the input image. Multiple parts can be adjacent inside the object, partially overlapping, or only partially visible due to occlusion. Each part is characterized by a set of part spatial parameters in subsequent steps. For the same type of object, multiple parts can correspond to predefined structural regions; for different types of objects, redundant parts in a preset number can have their contribution to the object blending mask and the overall position of the object reduced by smaller part weights.

[0037] In step S200, foreground regions are extracted from the image to be detected to obtain candidate foreground regions, and foreground region statistical features of the candidate foreground regions are extracted according to preset image statistical rules to obtain a first feature; frequency domain transformation is performed on the image to be detected to obtain a frequency domain representation of the image to be detected, and depth feature extraction is performed on the frequency domain representation to obtain a second feature used to characterize the spectral distribution of the image to be detected; then the first feature and the second feature are fused to obtain a fused feature map.

[0038] Reference Figure 1 and Figure 2 Step S200 may include steps S201 to S206. Steps S201 to S203 constitute the statistical feature branch of the foreground region, and step S204 constitutes the frequency domain depth feature branch. The outputs of the two branches are merged in step S205.

[0039] In step S201, foreground region extraction is performed on the image to be detected. In one embodiment, a multi-channel gray-level gradient can be calculated on the preprocessed image, and combined with adaptive thresholding to obtain a lightweight, saliency-based coarse extraction result. The multi-channel gray-level gradient is used to reflect the edge changes in the pixel neighborhood, and the adaptive threshold is used to distinguish candidate foreground and background based on the gray-level or gradient distribution of the current image. This operation can quickly filter out large areas of solid color or background regions with little variation, and initially delineate pixel regions containing potential targets.

[0040] In step S202, the results of foreground region extraction are used as candidate foreground regions. Candidate foreground regions are coarse regions used to calculate statistical features. A candidate foreground region may include a portion of the background of the object to be detected, or it may omit a small number of edge pixels of the object; its function is to provide a statistical basis for contour, spatial distribution, and texture independent of subsequent detection results.

[0041] In step S203, foreground region statistical features are extracted from the candidate foreground region according to preset image statistical rules. The preset image statistical rules may include at least one of contour dimension rules, bounding box dimension rules, and texture statistical dimension rules. Contour dimension rules can be used to obtain the Canny edge gradient mean, candidate foreground contour perimeter, and contour curvature; bounding box dimension rules can be used to obtain the aspect ratio of the candidate foreground bounding box, the proportion of candidate foreground pixels to the total number of pixels in the image, and the coordinate offset of the candidate foreground region center relative to the image center; texture statistical dimension rules can be used to obtain the grayscale mean, grayscale variance, and high-frequency texture proportion. These statistical measures together form the first feature, also known as a handcrafted feature, prior feature, or image spatial texture feature.

[0042] For spatial locations within the candidate foreground region, statistics corresponding to that region can be filled in; for spatial locations outside the candidate foreground region, preset background values ​​or corresponding regional statistical rules can be set. Through this spatial broadcasting, the first feature retains a two-dimensional structure that corresponds to each position in the depth feature map.

[0043] In step S204, a two-dimensional frequency domain transformation (e.g., a two-dimensional fast Fourier transform) is performed on the image to be detected. In one specific embodiment, the result of the two-dimensional fast Fourier transform is used to calculate the amplitude and form a two-dimensional frequency domain amplitude map. The two-dimensional frequency domain amplitude map is input into a pre-trained VGG16 network for deep feature extraction. During training, the parameters of the first four layers of VGG16 can be frozen and the parameters of the last two layers can be fine-tuned. Then, a single-layer two-dimensional convolution is used to perform spatial correlation modeling on the deep features to obtain a second feature used to characterize the spectral distribution of the image to be detected. The second feature can be represented as a deep feature map. .

[0044] In other embodiments, the frequency domain transformation can employ at least one of discrete cosine transform, wavelet transform, or Gabor filtering, as long as it can generate a frequency domain or multi-scale representation related to changes in the spatial texture of the image. The deep feature extraction network can also be ResNet, Swin Transformer, or Visual Transformer. When replacing the network, the number of output channels and spatial resolution can be adjusted so that the second feature spatially corresponds to the first feature. The above replacement does not change the data relationship of "performing deep feature extraction on the frequency domain representation to obtain a second feature characterizing the spectral distribution of the image to be detected."

[0045] In step S205, the first feature and the second feature are fused. Specifically, the size can be... The artificial feature map and its size are The depth feature maps are stitched together along the channel dimension to obtain a size of The concatenated feature map is then used to perform channel mapping using a fully convolutional layer, resulting in a size of [size missing]. Fusion feature map (Also known as robust hybrid features). It can be represented as:

[0046] ConvFC is a fully convolutional layer. A fully convolutional layer can be... Convolutional layers. This fusion method preserves [the following] during channel mapping. The spatial dimension is used to preserve spatial positional relationships.

[0047] In step S206, a fused feature map is output. The fused feature map simultaneously contains interpretable contour, spatial, and texture information obtained from preset image statistical rules, as well as spectral distribution information extracted from the frequency domain representation by the deep network. This fused feature map serves as the feature basis for subsequent Gaussian mixture model parameter regression, category prediction, and domain distribution alignment during the training phase.

[0048] For specific application scenarios, preset image statistical rules can also include statistical rules set for the surface material or marking status of the detected object. For example, for outdoor metal enclosures, a metal corrosion texture index reflecting local texture changes in rusted areas and a marking fading gradient index reflecting color or brightness gradient changes in warning marking areas can be added to the texture statistical features. These indices can be broadcast together with statistics from the contour dimension, bounding box dimension, and general texture dimension as a spatial feature map. Alternatively, oriented gradient histograms, local binary patterns, or Gabor filter features can be selected and scaled to the same spatial resolution as the second feature before channel stitching.

[0049] In step S300, based on the fused feature map, multiple sets of component spatial parameters corresponding one-to-one with multiple components of the object to be detected are obtained through Gaussian mixture model parameter regression. Each set of component spatial parameters includes a location parameter characterizing the spatial position of the corresponding component, a spatial scale parameter determining the spatial coverage of the corresponding component, and a component weight of the corresponding component.

[0050] In one embodiment, it can be set Each object query unit (also known as an object localizer, object local header, or query token) interacts with the fused feature map and outputs part space parameters through three sets of independent projection layers or other parametric regression layers. Each independent query unit representing a candidate object outputs a complete set of K part space parameters. Different query units are independent of each other and correspond to different candidate objects in the image. The number of query units... The number of components can be no less than K.

[0051] For the k-th component, the position parameter can be expressed as: The spatial scale parameter is expressed as The component weights are represented as The position parameter indicates the spatial center or corresponding reference position of the k-th component in the two-dimensional coordinate system of the image to be detected. The spatial scale parameter determines the spatial coverage of the k-th component in different coordinate axis directions and controls the response scale of the component spatial mask generated based on this set of parameters. The component weight represents the contribution of the k-th component to the spatial representation of the complete object to be detected.

[0052] The number of components, K, can be a pre-defined, fixed parameter that remains constant during inference. When the number of actually identifiable physical components for a certain type of object is less than K, the trained model can make the component weights corresponding to redundant components approach zero. The contribution of these redundant components to the weighted summation of object mixing masks and the aggregation of the overall position and coverage of the object also approaches zero, thereby achieving equivalent masking of invalid components without the need for additional component merging or pruning post-processing. If K is too small, the component-level expressive power may be insufficient; if K is too large, it may increase the risk of overlap of learnable parameters, spatial masks of different components, and forward computation. Therefore, K can be pre-defined based on the physical components of the object being detected.

[0053] In step S400, a component spatial mask corresponding to each of the multiple components is generated based on the position parameters and spatial scale parameters included in each set of component spatial parameters; according to the component weight corresponding to each component, the values ​​of the multiple component spatial masks at the same pixel position are weighted and summed to obtain the object blending mask.

[0054] Reference Figure 1 and Figure 3 Step S400 may include steps S401 to S404. In step S401, the position parameter of each group of component spatial parameters is used as the spatial response center, and the spatial attenuation range is determined using the corresponding spatial scale parameter and a preset scaling parameter to generate a corresponding two-dimensional Gaussian response map. In step S402, the correspondence between each component spatial mask and the component weights in the same group of component spatial parameters is determined. In step S403, according to the above correspondence, a weighted sum is performed on the values ​​of the K component spatial masks at the same pixel position. In step S404, the object blending mask is obtained.

[0055] In an isotropic embodiment, the space mask of the k-th component can be represented as:

[0056] in, This indicates the pixel position in the image to be detected. This represents the position parameter of the k-th component. This represents the spatial scale parameter of the k-th component. This indicates the preset scaling parameters. It is a scale quantity that controls the rate of decay of the spatial response. The spatial scale parameter is used in this disclosure to characterize the spatial coverage scale and is not expressed as a standard deviation in a probabilistic sense.

[0057] The component weights are non-negative, and the sum of the component weights corresponding to the K components is 1, that is:

[0058] in, Let be the component weight (weight coefficient) of the k-th component within the object. Component weights can be obtained through softmax normalization, or through ReLU plus L1 normalization, or Sigmoid plus L1 normalization. The core principle is that the component weights are non-negative and their sum equals 1. The object blending mask can be represented as:

[0059] Object blending mask It is a continuous spatial weight formed by the spatial responses of multiple parts of an object. For an image to be detected containing multiple objects, each object query unit generates K part spatial masks and one object blending mask for the corresponding object.

[0060] In an anisotropic embodiment, the spatial scale parameter includes scale components set along the horizontal and vertical axes of the image to be detected, and the component spatial mask can be represented as:

[0061] when and At different times, the spatial mask of a component has different spatial attenuation ranges along the horizontal and vertical axes, thus enabling it to characterize components that are elongated, flat, or other non-equiaxial shapes in the image.

[0062] In another anisotropic embodiment, the spatial scale parameter may include a covariance matrix. The corresponding component space mask can be represented in matrix form as follows:

[0063] in, , The covariance matrix is ​​used to characterize the spatial coverage of corresponding components along different directions, enabling the two-dimensional Gaussian response map to be stretched or rotated according to the corresponding directions. Here, the covariance matrix is ​​used as an implementation of the spatial scale parameter, and its physical meaning is still the spatial scale and direction in the image plane, without being limited to a statistical probability meaning.

[0064] In step S500, based on multiple sets of component spatial parameters, the position parameters and spatial scale parameters of multiple components are aggregated according to the component weights corresponding to each component, the overall position and overall coverage of the object to be detected are determined, and a rectangular detection bounding box is generated according to the overall position and overall coverage.

[0065] Reference Figure 4 Step S500 may include steps S501 to S504. In step S501, the position parameters of multiple components are aggregated according to component weights. In step S502, the spatial scale parameters of multiple components are aggregated according to component weights. Steps S501 and S502 may be executed separately or in the same parameter aggregation operation. In step S503, the overall position and overall coverage of the object to be detected are determined based on the aggregation result. In step S504, a rectangular detection bounding box is generated based on the overall position and overall coverage.

[0066] In one embodiment, based on the multiple sets of component spatial parameters, the positional parameters and spatial scale parameters of the multiple components are aggregated according to the component weights corresponding to each component. This includes aggregating the multiple sets of component spatial parameters according to the first-order moment and second-order rectangular form of the mixture model (also known as the expectation-variance aggregation algorithm). For the two coordinate axes of the image to be detected, the following formulas are used for calculation:

[0067] In the formula, Indicates the coordinate axis labels of the image to be detected. ; and These represent the mutually perpendicular horizontal and vertical axes in the two-dimensional image coordinate system containing the image to be detected. , and These represent the position parameter, spatial scale parameter, and component weight of the k-th component on the u-axis, respectively. and These represent the overall position parameter and overall spatial scale parameter of the object to be detected on the u-th coordinate axis, respectively. The overall position parameter is used to determine the overall position of the object to be detected, and the overall spatial scale parameter is used to determine the overall coverage area of ​​the object to be detected.

[0068] The above formula holds true for both the x-axis and the y-axis, and is calculated once for each axis, yielding the following result: and Two sets of results. The two boundaries of the detection bounding box on the u-th coordinate axis are as follows:

[0069] in, This is a preset coefficient. In one specific embodiment, A value of 0.5 can be used. This value is a preset parameter that can be adjusted manually. In other embodiments, other values ​​can be preset according to the imaging scale of the object to be detected and the required boundary coverage. (x-axis) , and on the y-axis , The four boundaries together form a rectangular detection bounding box.

[0070] Therefore, the processes of generating object blending masks and generating detection bounding boxes use the same set of component spatial parameters as data sources: position parameters and spatial scale parameters are used to form the spatial response of the corresponding components, and component weights are used to synthesize object blending masks; the same position parameters, spatial scale parameters, and component weights are also used to obtain the overall position and overall coverage of the object. This association allows the detection and segmentation processes to share component-level spatial information.

[0071] In an alternative implementation, the third-order central moment or other higher-order moments can be used to estimate the overall coverage of the asymmetric object. In another alternative implementation, while retaining the K sets of component spatial parameters and the object blending mask generation path, an additional branch can be set for directly regressing the detection bounding box coordinates. This additional branch only changes the calculation path for obtaining the detection bounding box from the component spatial parameters, without changing the path for generating multiple component spatial masks from multiple sets of component spatial parameters to form the object blending mask.

[0072] In step S600, instance mask prediction is performed based on the object fusion mask to obtain the instance segmentation mask; the category prediction is performed on the object to be detected based on the fused feature map to obtain the category label; and the detection bounding box, category label and instance segmentation mask are output.

[0073] In one embodiment, an object blending mask is used to spatially weight the fused feature map. Before spatial weighting, the pixel positions of the object blending mask are made to correspond one-to-one with the spatial positions in the fused feature map. Element-wise weighting is then performed on the corresponding spatial positions, preserving or highlighting the features of the region corresponding to the object to be detected, while weakening the features of the background region. The features weighted by the object blending mask are used to constrain the model to generate instance segmentation masks within the spatial region corresponding to the object to be detected. This process can be performed during both model training and deployment inference.

[0074] The detection segmentation head performs instance mask prediction to obtain an instance segmentation mask representing the actual pixel region of the object to be detected; it also performs category prediction based on the fused feature map to obtain a category label representing the semantic category of the object to be detected. The detection segmentation head can provide category prediction results and instance mask prediction results in parallel. This disclosure does not necessarily limit the specific candidate region cropping method, classification branch structure, or mask branch structure, as long as the category label and instance segmentation mask can be obtained according to the above input-output relationship.

[0075] The final output includes: a rectangular detection bounding box indicating the approximate location and overall extent of the object to be detected; a category label indicating the semantic category of the object to be detected; and an instance segmentation mask indicating the actual contour region of the object to be detected pixel by pixel.

[0076] The following description illustrates the application of image object detection methods in specific physical scenarios. This embodiment is for illustrative purposes only and should not be limited to the sole use of the methods disclosed herein.

[0077] In this embodiment, the object to be detected is a metal fire emergency box installed on the ground of the park. The fire emergency box may include three physical parts: a warning label area on the front of the box, a door latch assembly area, and a rust-prone area on the bottom of the box. K is preset to 3, so that the spatial parameter groups of the three components correspond to the above three physical parts respectively.

[0078] The images to be detected are captured by a fixed camera in the park. A specific configuration is as follows: the fixed camera is a 4-megapixel device with a 6mm lens focal length, installed at a height of 3.5m, and at a shooting distance of 5m from the area to be detected. The camera is equipped with constant infrared illumination and avoids direct sunlight. When a vehicle or pedestrian passes through the corresponding area, a snapshot is triggered, resulting in a single RGB image. The triggering object is only used to initiate image acquisition; in this embodiment, the processing object is a fire emergency box, and no facial recognition, human identity association, or other processing related to personal identity is performed.

[0079] In addition to the fire emergency box, the image to be detected may also contain the ground, walls, green plants, vehicles, and other background objects. First, uniform scaling, brightness equalization, and filtering denoising are performed. Then, multi-channel grayscale gradients and adaptive thresholds are used to initially filter out large areas of solid-color background, obtaining candidate foreground regions that may contain the fire emergency box.

[0080] The contour, spatial distribution, and texture statistics of the enclosure are extracted from the candidate foreground region. Contour statistics reflect the edges and door contours of the enclosure; spatial frame statistics reflect the aspect ratio, foreground proportion, and relative position of the enclosure in a fixed camera shot; texture statistics reflect the grayscale variations and local high-frequency textures of the metal surface. For metal enclosures photographed from a fixed outdoor camera position, metal corrosion texture indicators and warning sign fading gradient indicators can be added, so that the first feature includes image statistical information related to the state of rust-prone areas and warning sign areas.

[0081] Another approach performs a two-dimensional Fast Fourier Transform on the entire image to be detected, obtaining a two-dimensional frequency domain amplitude map. A deep feature extraction network then extracts a second feature reflecting the image's spectral distribution. The first feature is broadcast into an artificial feature map with the same spatial dimensions as the second feature, and these features are then stitched together in the channel dimension and processed... The fully convolutional layer produces a fused feature map.

[0082] Based on the fused feature maps, Gaussian mixture model parametric regression was performed to obtain three sets of component spatial parameters. The first set of component spatial parameters corresponds to the warning label area on the front of the box, the second set corresponds to the door latch assembly area, and the third set corresponds to the rust and damage area on the bottom of the box. The positional parameters in each set reflect the location of the corresponding part in the image, the spatial scale parameters reflect the spatial coverage of the corresponding part, and the component weights reflect the contribution of the corresponding part to the complete representation of the fire emergency box.

[0083] Three component spatial masks are generated from three sets of component spatial parameters, and then weighted and summed according to the weights of the three components to form a mixed object mask for the fire emergency box. Simultaneously, the three sets of positional parameters and spatial scale parameters are aggregated to obtain the overall position and coverage area of ​​the fire emergency box, forming a rectangular detection bounding box surrounding the box. Therefore, the spatial representations of the warning label area on the front of the box, the door latch assembly area, and the rust-prone area on the bottom of the box simultaneously participate in the formation of the intermediate mask and the determination of the overall detection bounding box.

[0084] A rectangular detection bounding box represents the overall area where the fire emergency box is located, but the box may include background pixels such as the ground, walls, or vegetation. An object blending mask is used to highlight the spatial locations corresponding to the three box components and weaken the background in the fused feature map. Based on this, the detection and segmentation head obtains instance segmentation masks distributed along the actual contour of the fire emergency box and outputs the category label for the fire emergency box.

[0085] Detection bounding boxes, category labels, and instance segmentation masks can be used to statistically analyze the presence status of fire emergency boxes within the park, as well as any missing boxes, bottom corrosion, or broken locks. When the instance segmentation result or the corresponding component area indicates an abnormal area that meets preset alarm conditions, a background pop-up message and audible and visual alarm commands can be generated. The corresponding image area or mask result is then archived and pushed to the security inspection terminal. Thus, the detection and segmentation results are used in the practical technical process of park facility inspection.

[0086] For the differences in feature distribution caused by images of fire emergency boxes from different seasons, locations, lighting conditions, or conditions of being new or old, the training method described later can be used to adapt the model to the target domain. The specific division of the source and target domains can be set according to the actual deployment batches; as long as the source domain training images have relatively complete supervision information and the target domain training images only provide a small amount of detection bounding box supervision, and the differences in fused features caused by changes in physical imaging conditions are reduced through domain distribution alignment, it is acceptable.

[0087] Reference Figure 5 This embodiment provides a training method for an image object detection model. The training method includes steps S700 to S1100. The training method is executed during model parameter updates; the trained model is used to execute the image object detection method described above. The three-layer structured constraint loss, two-stage sorting alignment, sorting head, sorting task unit, domain discriminator, and each model parameter update step all belong to the training phase and are not required to be executed during single-image inference after training.

[0088] In step S700, training images and their training annotations are obtained. The training annotations include detection bounding box annotations, and at least some training images also include object category annotations and instance segmentation mask annotations. Therefore, it is not required that all training images have pixel-level instance segmentation mask annotations.

[0089] In a semi-supervised domain adaptation embodiment, the training images include source domain training images and target domain training images. The source domain training images have complete annotations, including object category annotations, detection bounding box annotations, and pixel-level instance segmentation mask annotations. The target domain training images have fewer annotations, using detection bounding box annotations as supervision information, and do not provide pixel-level instance segmentation mask annotations. The number of target domain training images can be 3% to 5% of the number of source domain training images, and can be controlled to within one hundred images. This quantity relationship is a specific embodiment and is not intended to exclude the use of other numbers of target domain training images.

[0090] The source and target domains represent different data distributions. These differences can stem from actual image acquisition conditions, such as different fixed camera positions, different cameras, different lenses, different shooting distances, different imaging scales, different lighting or supplemental lighting conditions, different background compositions, different batches of objects, different surface conditions, or different seasons. By associating these domain differences with the aforementioned physical imaging conditions, the training process aims to reduce the impact of these imaging condition differences on object detection and instance segmentation, rather than simply operating on abstract numerical distributions that lack technical meaning.

[0091] In step S800, a fusion feature map is extracted from the training image using the model to be trained, and Gaussian mixture model parameter regression is performed based on the fusion feature map to obtain multiple sets of component spatial parameters that correspond one-to-one with multiple components of the same object in the training image. Each set of component spatial parameters includes a location parameter for characterizing the spatial position of the corresponding component, a spatial scale parameter for determining the spatial coverage of the corresponding component, and a component weight for the corresponding component.

[0092] The extraction of the fused feature map can adopt the dual-branch approach described in step S200 above. This involves using the statistical features of the candidate foreground region to form the first feature, using the frequency domain representation of the training image to form the second feature, and then forming a fused feature map that preserves the two-dimensional spatial relationships through spatial broadcasting, channel stitching, and fully convolutional mapping. In a broader implementation corresponding to the training weights, other feature extraction structures that can preserve image spatial information can also be used to obtain the fused feature map, provided that subsequent Gaussian mixture model parameter regression can output position parameters, spatial scale parameters, and component weights that correspond one-to-one with multiple parts of the same object based on this fused feature map.

[0093] The model to be trained can be configured with object query units, each of which outputs a set of spatial parameters for K components. The component spatial parameters, component spatial masks, object blending masks, and detection bounding boxes are defined consistently during both the training and inference phases. The component weights learned during training satisfy the constraint that they are non-negative and sum to 1; when the preset K is greater than the number of effective physical components of an object, the weights of redundant components can approach zero.

[0094] In step S900, multiple component spatial masks, each corresponding to a specific component, are generated based on multiple sets of component spatial parameters. These masks are then weighted and summed according to the component weights of each component to obtain an object blending mask. Finally, the object's detection bounding box is obtained by aggregating these masks based on the multiple sets of component spatial parameters. An instance segmentation mask for the object is then obtained based on the object blending mask, and the object's category prediction result is obtained based on the fused feature map.

[0095] The specific methods for generating component spatial masks, object blending masks, and detection bounding boxes during the training phase can be the same as those described in steps S400 and S500. The object blending mask can spatially weight the fused feature map to highlight the spatial region corresponding to the current object. The detection segmentation head obtains the category prediction result and instance segmentation mask based on the corresponding features.

[0096] The detection bounding boxes, instance segmentation masks, and category prediction results output in step S900 are used as predictions for subsequent loss determination processes. The detection bounding box annotations, instance segmentation mask annotations, and other training annotations are input as corresponding supervision quantities to step S1000 or step S1100. For target domain training images without instance segmentation mask annotations, the corresponding instance mask segmentation loss is not calculated using the non-existent instance segmentation mask annotations.

[0097] In step S1000, the detection bounding box regression loss is determined based on the detection bounding box annotations and the detection bounding boxes; for training images with instance segmentation mask annotations, the instance segmentation mask segmentation loss is determined based on the instance segmentation mask annotations and the instance segmentation mask; the spatial coverage of each component is determined based on the position parameters and spatial scale parameters corresponding to each component among multiple components, and the component inclusion constraint loss is determined based on the inclusion relationship between the spatial coverage and the detection bounding boxes of the object; the inter-object diversity constraint loss is determined based on the similarity between the object mixing masks of different objects in the training image; and the intra-object component diversity constraint loss is determined based on the similarity between the spatial masks of multiple components of the same object.

[0098] Reference Figure 6 In step S1001, the detection bounding box regression loss is determined based on the detection bounding box annotations and the detection bounding boxes generated by the model. The detection bounding box regression loss is used to make the detection bounding boxes obtained by aggregating the part spatial parameters close to the corresponding detection bounding box annotations.

[0099] In step S1002, for training images with instance segmentation mask annotations, the instance segmentation loss is determined based on the instance segmentation mask annotations and the instance segmentation mask generated by the model. For target domain training images without instance segmentation mask annotations, the instance mask segmentation loss can be excluded from the loss calculation for the corresponding samples.

[0100] In step S1003, the three-layer structured constraint loss is determined. The three constraints apply to the longitudinal inclusion relationship between objects and components, the lateral separation relationship between different objects, and the lateral separation relationship between components within the same object.

[0101] The first type of structured constraint loss determined in step S1003 is the component-inclusive constraint loss. The component-inclusive constraint loss penalizes the portion of the spatial coverage of each of the multiple components that exceeds the detection bounding box of the object.

[0102] In this embodiment, the component inclusion constraint loss is determined based on the inclusion relationship between the spatial coverage of each component and the object detection bounding box. For the u-th coordinate axis, the spatial coverage of the k-th component can be defined as:

[0103] in, This represents the preset scale coefficient used to determine the spatial coverage of a component. It can be used with preset coefficients of the detection bounding box. The same value can be used, or different values ​​can be preset based on the scale relationship between component coverage and overall object coverage. This "spatial coverage area" does not include preset scaling parameters that control the component spatial mask decay rate. This distinguishes it from the "spatial attenuation range" of the component space mask.

[0104] If the lower and upper boundaries of the object detection bounding box on the u-th coordinate axis are respectively and The constraint loss within a component can then be written as:

[0105] When the spatial coverage of the k-th component is completely within the detection bounding box, the penalties for that component on the corresponding coordinate axis are zero; when the spatial coverage of a component exceeds either side of the detection bounding box, a positive penalty is applied for the excess. Component weights ensure that components with different relative contributions participate in the containment constraint to varying degrees. This constraint is used to keep the spatial representation of each component within the geometry of the corresponding complete object.

[0106] The second type of structured constraint loss determined in step S1003 is the inter-object diversity constraint loss. The inter-object diversity constraint loss imposes a penalty when the cosine similarity between the object mixing masks of different objects exceeds a preset overlap threshold.

[0107] In this embodiment, the loss is determined based on the similarity between the object mixing masks of different objects. Assume there are N objects in the training image, and the... The object blending mask for each object is: The preset overlap threshold is Then the diversity constraint loss between objects can be expressed as:

[0108] When the cosine similarity between the object blending masks of different objects does not exceed a preset overlap threshold When the similarity exceeds a threshold, no additional penalty is imposed, thus allowing a certain degree of spatial overlap caused by reasonable occlusion; when the similarity exceeds a threshold, the penalty increases with the amount of excess, thereby inhibiting the merging of spatial responses of different objects.

[0109] The third type of structured constraint loss determined in step S1003 is the object-internal component diversity constraint loss. The object-internal component diversity constraint loss is determined based on the cosine similarity between pairwise different component space masks in multiple component space masks of the same object, and increases with the increase of cosine similarity.

[0110] In this embodiment, the loss is determined based on the cosine similarity between the spatial masks of two different components within the same object. This loss can be expressed as:

[0111] The loss increases with the cosine similarity between the spatial masks of different components within the same object, and is used to prevent the spatial responses of multiple components from concentrating at the same location. Component inclusion constraints maintain the containment relationship between objects and components, inter-object diversity constraints suppress the adhesion of responses between different objects, and intra-object component diversity constraints suppress the collapse of responses of components within the same object. These three constraints together constitute a three-layer structured constraint.

[0112] In step S1004, the detection box regression loss determined in step S1001, the instance mask segmentation loss determined in step S1002, and the three-layer structured constraint loss determined in step S1003 are combined to obtain five types of loss terms used to constitute the total loss in step S1100.

[0113] In step S1100, the model parameters of the model to be trained are updated according to the total loss to obtain the trained image object detection model. The total loss is determined based on the relevant information from the training annotations, category prediction results, category correlation loss, detection box regression loss, instance mask segmentation loss, part inclusion constraint loss, inter-object diversity constraint loss, and intra-object part diversity constraint loss. The trained image object detection model is used to output the detection bounding box, category label, and instance segmentation mask of the object to be detected based on the image to be detected.

[0114] In one embodiment, the model to be trained includes a shared feature extractor G, a detection segmentation head H, and a domain discriminator D. The model may also include an object query unit, a parametric regression layer for Gaussian mixture model parameter regression, and a two-stage sorting alignment head used during the training phase. The shared feature extractor G generates a fused feature map or performs further feature extraction on the fused feature map; the detection segmentation head H outputs prediction results for the detection and segmentation tasks; and the domain discriminator D outputs data domain discrimination results.

[0115] The task portion of the total loss can be represented as:

[0116] in, This represents the regression loss of the detection box. This represents the instance mask segmentation loss. Indicates the sorting alignment loss. This indicates that the component contains constraint losses. This represents the loss due to the diversity constraints between objects. This represents the loss due to the diversity constraints of the internal components of an object. and These represent the weights of the sorting alignment loss and the component inclusion constraint loss, respectively. When the category supervision loss is set separately, the corresponding category supervision loss can be added to... When category supervision is included in the detection task loss, it will no longer be listed separately.

[0117] Reference Figure 7 The two-stage sorting and alignment sub-process in step S1100 can sequentially include a first training stage and a second training stage. The two training stages respectively change the source of negative samples and the alignment granularity of the relevance sorting.

[0118] In the first training phase corresponding to step S1101, cross-image negative samples, composed of source domain training images and target domain training images, are used to perform image-level relevance ranking between the fused feature map weighted by the object mixing mask and the object category label. The cross-image negative samples can be constructed by changing the pairing relationship between images and object category labels within the training batch, so that the negative samples simultaneously include training images from both the source and target domains. This phase, through cross-image and cross-data domain negative samples, guides the model to prioritize learning object-related semantic features, reducing feature shifts caused by differences in imaging conditions such as illumination and scale.

[0119] In the second training phase corresponding to step S1102, other objects located in the same training image as the current object are used as local negative samples. The fusion feature map weighted by the object mixing mask of the current object is then ranked according to object-level relevance with the current object's category label. Other objects within the same image have the same global imaging conditions but do not correspond to the current object; therefore, they can be used as local hard negative samples to improve the discriminative power between different objects and their category labels within the same image. In the scenario of a fire emergency box in a park, green plants, vehicles, or other facilities in the same frame can be used as local negative samples that do not correspond to the current fire emergency box.

[0120] Image category relevance scoring functions can be used. Image features and category labels weighted by object blending mask Scoring is performed (e.g., using a text-image relevance scoring function or an image category relevance scoring function). For positive category labels... and negative category labels The sorting alignment loss can be expressed as:

[0121] In step S1103, in each of the first and second training phases, a ranking alignment loss is determined based on the relevance ranking results of that training phase. In step S1104, the model parameters of the model to be trained are updated based on the ranking alignment loss of each training phase; specifically, the total loss also includes the ranking alignment loss. For example, in one embodiment, the corresponding ranking alignment loss is used as a component of the total loss for model parameter updates. In one embodiment, a detection task unit can be set. and sorting task unit ,in, Used for detection and instance mask generation. Used for relevance scoring and ranking supervision. The ranking task unit and ranking head are only used during the training phase.

[0122] The first training phase applies cross-domain semantic alignment by mixing negative samples from the source and target domains across images; the second training phase applies object-class differentiation within a single image by applying local negative samples within the same image.

[0123] Reference Figure 8 The model to be trained includes a feature extractor G, a detection and segmentation head H, and a domain discriminator D. The source domain training image and the target domain training image are processed by the feature extractor G to generate a fused feature map or corresponding shared features. Steps S1105 to S1109 perform domain discrimination, total loss determination, and alternating updates of the two optimization steps based on the fused feature map.

[0124] In step S1105, the fused feature map of the source domain training image and the target domain training image is input into the domain discriminator D to obtain the data domain discrimination result used to characterize the data domain to which the fused feature map belongs. In step S1106, the domain discrimination loss is determined based on the data domain discrimination result. In step S1107, the total loss is determined based on the training annotations, category prediction results, the detection box regression loss, instance mask segmentation loss, and three-layer structured constraint loss obtained in step S1000, and the sorting alignment loss obtained in step S1104.

[0125] In the first optimization step corresponding to step S1108, the model parameters of the fixed domain discriminator D are calculated based on the total loss and the preset domain adversarial loss balance coefficient. A weighted domain-discriminative loss is used to update the model parameters of the feature extractor G and the detection segmentation head H, thereby reducing the total loss and increasing the domain-discriminative loss. The optimization objectives for the feature extractor G and the detection segmentation head H can be expressed as:

[0126] The negative sign causes the feature extractor G to tend to produce features that make it difficult for the domain discriminator D to distinguish between the source and target domains, thereby reducing the domain distribution difference between the fused feature maps of the source and target domain training images.

[0127] In the second optimization step corresponding to step S1109, the model parameters of the feature extractor G and the detection segmentation head H are fixed, and the model parameters of the domain discriminator D are updated according to the domain discriminant loss to reduce the domain discriminant loss. After completing this second optimization step, the process returns to step S1108 to alternately execute the first optimization step and the second optimization step. The second optimization step can be expressed as:

[0128] By alternately executing the two optimization steps described above, the domain discriminator D learns to distinguish the data domain to which features belong, while the feature extractor G learns to weaken domain-discriminative information and retain features relevant to the object to be detected. After training, the feature extractor G and the detection segmentation head H are used to process new images to be detected, and the domain discriminator D does not participate in deployment inference.

[0129] When using a unified expression, the joint optimization objective including domain adversarial terms can be written as:

[0130] in, This represents the predefined domain adversarial loss balance coefficient. Here, we use... This serves as a domain adversarial loss balancing coefficient, distinct from the preset coefficient in the detection bounding box boundary formula. When training the feature extractor G and the detection segmentation head H, the above joint optimization objective is minimized; when training the domain discriminator D, the domain discrimination loss is minimized separately.

[0131] Domain distribution alignment is not limited to adversarial approaches. In alternative implementations, maximum mean difference loss or optimal transmission loss can be used to measure and reduce the difference in feature distributions between the source and target domains. When using non-adversarial distribution alignment, a domain discriminator D may not be required, but fused feature maps, Gaussian mixture model parameter regression, three-layer structured constraints, and two-stage ranking alignment can still be retained.

[0132] The image object detection model disclosed herein can be functionally divided into a preprocessing module, a candidate foreground region extraction module, a first feature extraction module, a second feature extraction module, a feature fusion module, a part parameter regression module, a part spatial mask generation module, an object hybrid mask generation module, a detection bounding box generation module, and a detection segmentation head.

[0133] The preprocessing module performs at least one of the following operations on the input single static image: size unification, brightness equalization, and image filtering. The candidate foreground region extraction module obtains candidate foreground regions based on multi-channel grayscale gradients and adaptive thresholds before precise detection. The first feature extraction module obtains the foreground region statistical features of the candidate foreground regions according to preset image statistical rules and broadcasts the corresponding statistics as a spatial feature map. The second feature extraction module performs depth feature extraction on the frequency domain representation of the image to be detected, obtaining a depth feature map characterizing the image's spectral distribution. The feature fusion module concatenates the first and second features along the channel dimension and obtains a fused feature map that preserves the two-dimensional spatial relationship through fully convolutional mapping.

[0134] The component parameter regression module includes an object query unit and a parameter regression layer, used to perform Gaussian mixture model parameter regression based on the fused feature map, outputting K sets of component spatial parameters for each candidate object. The component spatial mask generation module generates a 2D Gaussian response map for the corresponding component based on each set of positional parameters and spatial scale parameters. The object mixture mask generation module performs a weighted summation of the spatial masks of K components of the same object based on component weights. The detection bounding box generation module determines the overall position and coverage of the object based on the same K sets of positional parameters, spatial scale parameters, and component weights, and outputs a rectangular detection bounding box. The detection segmentation head performs class prediction and instance mask prediction.

[0135] The training model can also include a three-layer structured constraint module, a two-stage ranking and alignment module, and a domain discriminator. The three-layer structured constraint module obtains the corresponding loss based on the inclusion relationship between parts and objects, the similarity between mixed masks of different objects, and the similarity between spatial masks of parts within the same object. The two-stage ranking and alignment module performs image-level relevance ranking in the first training phase and object-level relevance ranking in the second training phase. The domain discriminator determines the data domain discrimination result of the fused feature map. These three training modules do not need to be deployed to the inference device after the model training is completed.

[0136] Each module can be implemented by one or more processors executing program instructions stored in memory, or by application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processors (GPUs), neural network accelerators, or a combination of the above hardware. Module division is a functional division; multiple modules can be integrated into the same program component or hardware unit, or a single module can be distributed across multiple program components or hardware units.

[0137] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. (Refer to...) Figure 9 The electronic device 900 includes a processor 910 and a memory 920. The processor 910 is communicatively connected to the memory 920 to read and execute computer programs stored in the memory 920.

[0138] Processor 910 may include a central processing unit, a digital signal processor, a microcomputer, a field-programmable gate array, a system-on-a-chip, a microprocessor, an application-specific integrated circuit, a graphics processor, a neural network processor, or at least one of the above. Processor 910 may also include other hardware circuits or programmable logic devices that implement processor functions.

[0139] The memory 920 is configured to store computer programs executed by the processor 910, as well as model parameters, image data, training annotations, component spatial parameters, detection bounding boxes, category labels, instance segmentation masks or other intermediate data required for the operation of the programs. The memory 920 may include high-speed random access memory, non-volatile memory, or a combination of both.

[0140] When the processor 910 executes the computer program stored in the memory 920, the electronic device 900 implements the image object detection method as described above, or implements the training method for the image object detection model as described above. For example, the electronic device 900 may perform corresponding operations in acquiring an image to be detected, extracting candidate foreground regions, extracting and fusing first features and second features, regressing Gaussian mixture model parameters, generating component spatial masks and object mixture masks, generating detection bounding boxes, performing category prediction, and performing instance mask prediction; the electronic device 900 may also perform corresponding operations in acquiring training images, determining detection box regression loss, instance mask segmentation loss and three-layer structured constraint loss, two-stage sorting alignment, and domain adversarial optimization.

[0141] The electronic device 900 may be a server, a workstation, a desktop computer, a portable computer, an edge computing device, a supporting processing device for an image acquisition apparatus, or other devices having image computing and program execution capabilities. The image object detection method and the model training method may be executed by the same electronic device, or may be respectively executed by a training device and a deployment inference device. When different devices are used for execution, the trained model parameters and programs may be transmitted through a wired communication network, a wireless communication network, or a removable storage medium.

[0142] According to an embodiment of the present disclosure, there is further provided a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image object detection method as described above is implemented, or the training method for the image object detection model as described above is implemented.

[0143] The computer-readable storage medium may include read-only memory, programmable read-only memory, electrically erasable programmable read-only memory, random access memory, dynamic random access memory, static random access memory, flash memory, or other semiconductor memories. The computer-readable storage medium may further include a compact disc read-only memory, a recordable compact disc, a rewritable compact disc, a digital versatile disc, a Blu-ray disc, or other optical storage media; a hard disk drive, a solid-state drive, a memory card, a magnetic tape, a floppy disk, a magneto-optical data storage device, or other magnetic storage media.

[0144] Computer-readable storage media store computer programs and associated data, data files, or data structures in a non-transitory manner, and provide the computer program to a processor or computer, causing the processor or computer to execute corresponding methods. The computer program and associated data, data files, or data structures may also be distributed across multiple networked computer devices, enabling the computer program to be stored, accessed, and executed in a distributed manner via one or more processors.

[0145] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. An image object detection method, characterized in that, include: Acquire an image of a target object, including the target object, wherein the target object includes multiple components; Foreground regions are extracted from the image to be detected to obtain candidate foreground regions. Statistical features of the candidate foreground regions are then extracted according to preset image statistical rules to obtain a first feature. A frequency domain transformation is performed on the image to be detected to obtain its frequency domain representation. Depth features are extracted from the frequency domain representation to obtain a second feature characterizing the spectral distribution of the image to be detected. The first feature and the second feature are then fused to obtain a fused feature map. Based on the fused feature map, a number of sets of component spatial parameters corresponding to the multiple components are obtained by using the Gaussian mixture model parameter regression method. Each set of component spatial parameters includes a location parameter for characterizing the spatial position of the corresponding component, a spatial scale parameter for determining the spatial coverage of the corresponding component, and a component weight of the corresponding component. Based on the position parameters and spatial scale parameters included in each set of component spatial parameters, a component spatial mask corresponding to each of the multiple components is generated. Then, according to the component weight corresponding to each component, the values ​​of the multiple component spatial masks at the same pixel position are weighted and summed to obtain the object blending mask. Based on the multiple sets of component spatial parameters, the position parameters and spatial scale parameters of the multiple components are aggregated according to the component weight corresponding to each component, the overall position and overall coverage of the object to be detected are determined, and a rectangular detection bounding box is generated according to the overall position and the overall coverage. Instance mask prediction is performed based on the object fusion mask to obtain an instance segmentation mask; category prediction is performed on the object to be detected based on the fused feature map to obtain a category label; the detection bounding box, the category label, and the instance segmentation mask are output.

2. The image object detection method according to claim 1, characterized in that, The number of spatial parameters of the multiple sets of components is a preset K, where K is an integer not less than 2; the component weight corresponding to each of the multiple components is a non-negative value, and the sum of the component weights corresponding to the multiple components is 1; The component spatial mask is a two-dimensional Gaussian response map centered on the corresponding position parameters, with the spatial attenuation range determined by the corresponding spatial scale parameters and preset scaling parameters.

3. The image object detection method according to claim 2, characterized in that, Based on the multiple sets of component spatial parameters, the position parameters and spatial scale parameters of the multiple components are aggregated according to the component weight corresponding to each component, including: Aggregation is performed according to the following formula. In the formula, , , and These represent the position parameter, spatial scale parameter, and component weight of the k-th component on the u-th coordinate axis, respectively. and These represent the global position parameter and global spatial scale parameter of the object to be detected on the u-axis, respectively; the two boundaries of the detection bounding box on the u-axis are respectively and , These are preset coefficients.

4. The image object detection method according to claim 2, characterized in that, The spatial scale parameter includes scale components set along the two coordinate axes of the image to be detected, or includes a covariance matrix used to characterize the spatial coverage of the corresponding component in different directions; the component spatial mask generated according to the spatial scale parameter has different spatial attenuation ranges in at least two directions.

5. A training method for an image object detection model, characterized in that, include: Acquire training images and their training annotations, wherein the training annotations include detection bounding box annotations, and at least some of the training images also include object category annotations and instance segmentation mask annotations; The fusion feature map is extracted from the training image using the model to be trained, and Gaussian mixture model parameter regression is performed based on the fusion feature map to obtain multiple sets of component spatial parameters that correspond one-to-one with multiple parts of the same object in the training image. Each set of component spatial parameters includes a position parameter for characterizing the spatial position of the corresponding part, a spatial scale parameter for determining the spatial coverage of the corresponding part, and a component weight of the corresponding part. Multiple component spatial masks, each corresponding to a component, are generated based on the multiple sets of component spatial parameters. The multiple component spatial masks are then weighted and summed according to the component weights of each component to obtain an object blending mask. The object's detection bounding box is then aggregated based on the multiple sets of component spatial parameters. An instance segmentation mask for the object is obtained based on the object blending mask, and the object's category prediction result is obtained based on the fused feature map. A detection bounding box regression loss is determined based on the detection bounding box annotations and the detection bounding boxes. For training images with instance segmentation mask annotations, an instance segmentation loss is determined based on the instance segmentation mask annotations and the instance segmentation mask itself. The spatial coverage of each component is determined based on its position and spatial scale parameters. The inclusion constraint loss within a component is determined based on the inclusion relationship between the spatial coverage and the detected bounding box of the object. The diversity constraint loss between objects is determined based on the similarity between the object mixing masks of different objects in the training image. The diversity constraint loss within an object is determined based on the similarity between the spatial masks of multiple components of the same object. The model parameters of the model to be trained are updated according to the total loss to obtain the trained image object detection model; wherein, the total loss includes the training annotation, the category correlation loss determined by the category prediction result, the detection box regression loss, the instance mask segmentation loss, the part inclusion constraint loss, the inter-object diversity constraint loss and the object part diversity constraint loss, and the trained image object detection model is used to output the detection bounding box, category label and instance segmentation mask of the object to be detected according to the image to be detected.

6. The method according to claim 5, characterized in that, The component-in-component constraint loss penalizes the portion of the spatial coverage of each of the plurality of components that exceeds the detection bounding box of the object; the inter-object diversity constraint loss penalizes the portion of the object mixing mask of different objects that exceeds a preset overlap threshold; the component diversity constraint loss within an object is determined based on the cosine similarity between the spatial masks of two different components in the spatial masks of the same object, and increases as the cosine similarity increases.

7. The method according to claim 5, characterized in that, The training images include source domain training images and target domain training images. The training annotations of the source domain training images include object category annotations, detection bounding box annotations, and instance segmentation mask annotations. The target domain training images use detection bounding box annotations as supervision information. The process of updating the model parameters of the model to be trained includes sequentially executing a first training phase and a second training phase. In the first training phase, cross-image negative samples composed of the source domain training images and the target domain training images are used to perform image-level correlation ranking between the fused feature map weighted by the object mixing mask and the object category label. In the second training phase, other objects located in the same training image as the current object are used as local negative samples, and the fusion feature map weighted by the object mixing mask of the current object and the category label of the current object are sorted by object-level relevance. In each of the first and second training phases, a sorting alignment loss is determined based on the relevance ranking results of that training phase, and the total loss also includes the sorting alignment loss.

8. The method according to claim 7, characterized in that, The model to be trained includes a feature extractor, a detection segmentation head, and a domain discriminator; The fused feature map of the source domain training image and the target domain training image is input into the domain discriminator to obtain the domain discrimination result that characterizes the data domain to which the fused feature map belongs, and the domain discrimination loss is determined based on the domain discrimination result. Domain adversarial optimization is performed on the feature extractor by alternately executing the first optimization step and the second optimization step to reduce the domain distribution difference between the fused feature maps of the source domain training image and the target domain training image. In the first optimization step, the model parameters of the domain discriminator are fixed, and the model parameters of the feature extractor and the detection segmentation head are updated according to the total loss and the domain discrimination loss weighted by a preset domain adversarial loss balancing coefficient to reduce the total loss and increase the domain discrimination loss. In the second optimization step, the model parameters of the feature extractor and the detection segmentation head are fixed, and the model parameters of the domain discriminator are updated according to the domain discrimination loss to reduce the domain discrimination loss.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image object detection method according to any one of claims 1 to 4 and the training method for the image object detection model according to any one of claims 5 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image object detection method according to any one of claims 1 to 4 and the training method for the image object detection model according to any one of claims 5 to 8.

Images