A multi-feature fusion-based instance segmentation and target detection hybrid recognition method

Abstract

The application discloses a kind of instance segmentation and target detection hybrid identification method based on multi-feature fusion, belong to computer vision technical field.The method constructs multi-task fusion network;Through the feature extraction of image to be measured of main network, shared fusion feature map M1 is obtained by multi-scale feature pyramid fusion;It is respectively input into target detection and instance segmentation module, and detection result and segmentation result are obtained;Then feature map M1, target detection result and instance segmentation result are jointly input into posture estimation module, and posture estimation result is obtained;Three kinds of results are uniformly mapped and aligned in space, are spliced and weightedly fused in channel dimension, are input into fully connected layer learning task weight combination, and the identification result containing target class, spatial position, pixel-level segmentation mask and key point coordinate is output.The application is collaboratively designed by feature sharing, multi-task fusion and cross-frame matching mechanism, to improve system operation efficiency while ensuring identification accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and more specifically, relates to a hybrid recognition method for instance segmentation and target detection based on multi-feature fusion. Background Technology

[0002] Visual recognition technology is an important component of artificial intelligence, primarily focusing on enabling computers to recognize and understand content in images and videos in a manner similar to humans. This technology integrates knowledge from multiple fields such as computer vision, image processing, and pattern recognition, aiming to simulate the human visual system to analyze and understand images and videos.

[0003] Early visual recognition research often employed traditional methods based on manually designed features, such as SIFT and HOG features combined with support vector machines or decision trees for classification and detection. These methods rely on human experience for feature construction, have limited feature representation capabilities, struggle to adapt to complex scene changes, and exhibit poor recognition stability in occlusion, lighting variations, and multi-target environments.

[0004] With the development of deep learning technology, object detection and instance segmentation algorithms based on convolutional neural networks have significantly improved recognition accuracy. However, most existing methods are optimized for single tasks, and the detection and segmentation models are usually independent of each other, lacking a unified multi-feature fusion and multi-task collaboration mechanism. In complex application scenarios, they still suffer from problems such as high computational resource consumption, feature redundancy, and insufficient fusion efficiency.

[0005] Therefore, how to effectively integrate target detection and instance segmentation tasks while ensuring detection accuracy, and improve the overall performance and generalization ability of the model, has become a pressing technical problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a hybrid recognition method for instance segmentation and target detection based on multi-feature fusion. By constructing a shared backbone feature extraction and multi-task branch fusion, the method achieves collaborative processing of target detection, instance segmentation and pose estimation tasks under a unified network framework, thereby solving the problems of redundant multi-task features, large computational load and poor recognition consistency in existing technologies.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a hybrid recognition method for instance segmentation and target detection based on multi-feature fusion. Step S1: Construct a multi-task fusion network that includes a backbone feature extraction network, a multi-scale feature pyramid structure, an object detection processing module, an instance segmentation processing module, and a pose estimation processing module. Use the backbone feature extraction network to extract features from the input static image to be tested, and use the multi-scale feature pyramid structure to fuse them to obtain a shared fusion feature map M1. Step S2: Input the fused feature map M1 obtained in step S1 into the target detection processing module and the instance segmentation processing module for processing, respectively, to obtain the target detection result and the instance segmentation result; then send the fused feature map M1, the target detection result and the instance segmentation result together to the pose estimation processing module for processing, to obtain the pose estimation result; Step S3: Perform unified mapping processing on the target detection results, instance segmentation results, and pose estimation results from Step S2. After spatial alignment, the results are concatenated along the channel dimension and fused using a weighted summation method to obtain a unified fused feature representation. Input the fused feature representation into a fully connected layer, and use learned weights to weight and combine features from different tasks to output the recognition result. The output recognition result includes target category information, target spatial location, target pixel-level segmentation mask, and target key point coordinate information.

[0008] As a possible implementation of the first aspect of the present invention, the process of the backbone feature extraction network extracting features of the image to be tested in step S1 is as follows:

[0009] In the formula, The first feature map obtained after convolution represents the first feature map. i Line number j Column pixel values, This represents the convolution operation between the input feature map X and the convolution kernel K. This indicates that the summation is performed by iterating over all elements of the convolution kernel K. This represents the nth element in the input feature map X corresponding to the current position of the convolution kernel. line, number The sampled pixel values ​​of the column, where Indicates the convolution stride. Indicates the filling parameter. and Indicates the index of an element within the convolution kernel. b Indicates the bias term. Denotes the first convolution kernel K. m Line number n The parameter values ​​of the column.

[0010] As a possible implementation of the first aspect of the present invention, the processing flow of the instance segmentation processing module in step S2 is as follows: A parallel ResNet and EfficientNet backbone network is used to extract features from the fused feature map M1, and then the FPN feature pyramid network is used for feature fusion to obtain a fused feature map M2. The obtained fused feature map M2 is input into the detection sub-branch for processing, and the fused feature map M2 and the target position output by the detection sub-branch are used together for the segmentation sub-branch processing. During the processing of the segmentation sub-branch, the segmentation sub-branch generates a mask prototype based on the fused feature map M2, and extracts the features of the corresponding target region from the fused feature map M2 according to the target position obtained by the detection sub-branch. Then, by performing pixel-by-pixel prediction on the features of each target region, the segmentation mask corresponding to each target is output, and finally the instance segmentation result is output.

[0011] As a possible implementation of the first aspect of the present invention, in step S2, the target detection processing module uses the YOLOv8 Anchor-Free detection head to directly regress the center point coordinates and width and height parameters of the target box by performing convolutional prediction on the fused feature map M1, and finally outputs the target detection result containing category information, bounding box position parameters and confidence score.

[0012] As a possible implementation of the first aspect of the present invention, the process of the attitude estimation processing module in step S2 is as follows: Step 1: Based on the target detection results in step S2, obtain the bounding box location information and category information of each target object, and determine the target candidate region in the fused feature map M1 accordingly; Step 2: Combining the target mask information in the instance segmentation result in step S2, foreground constraints are applied to the target candidate regions determined in step 1. The pixel regions belonging to the target subject within the bounding box are retained, and the background regions are suppressed or weakened, thereby obtaining the constraint region features corresponding to the target instance. Step 3: Use the attitude estimation processing module to predict key points of the constrained region features described in Step 2, and output the predicted coordinates and confidence information of a preset number of key points for each target. Step 4: During the training phase, the predicted coordinates of the key points output in Step 3 are compared with the actual coordinates of the key points labeled in the corresponding training samples to calculate the error. The error calculation includes the difference in the three-dimensional coordinates of the key points and the difference in the rotation angle.

[0013] As one possible implementation of the first aspect of the present invention, the key point prediction uses a key point regression prediction method or a key point heatmap prediction method.

[0014] As one possible implementation of the first aspect of the present invention, the calculation of the rotation angle difference uses Euler angles or quaternions, and the specific selection rules are as follows: When the pitch angle is not ±90°, it indicates that the yaw angle and roll angle are not in the same plane. The Euler angle calculation formula is used to calculate the rotation angle information. When the pitch angle is ±90°, it indicates that the yaw angle and roll angle are in the same plane, and the quaternion calculation formula is used to calculate the rotation angle information.

[0015] As one possible implementation of the first aspect of the present invention, when the input of step S1 is a series of video frames, cross-frame target matching and tracking are performed on the final output of step S3. The specific process is as follows: A target state library is established based on the fusion features of the target in the previous frame. The target state library includes the category information, bounding box position parameters, pixel-level segmentation mask, key point coordinate information and time sequence identity of each target. Then, for each target in the current frame, calculate its comprehensive matching similarity score with each target in the previous frame; when the target's motion speed is low and the pose change is small, the bounding box overlap is high, and the IoU matching strategy is adopted; when the target undergoes rapid motion, occlusion, or deformation, which leads to a decrease in the bounding box overlap, the spatial distribution similarity of key points is introduced to enhance the matching robustness. Finally, the Hungarian algorithm or greedy matching strategy is used to perform the optimal matching between the target in the current frame and the target in the previous frame, and the target identity is transferred across frames. The successfully matched target inherits the temporal identity of the previous frame, and the recognition result includes target category information, bounding box position parameters, pixel-level segmentation mask, key point coordinate information and temporal identity.

[0016] In a second aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the above-described hybrid recognition method for instance segmentation and target detection based on multi-feature fusion.

[0017] A third aspect of the present invention also provides a computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform the above-described hybrid recognition method for instance segmentation and target detection based on multi-feature fusion.

[0018] Compared with existing technical solutions that employ independent target detection models, instance segmentation models, and pose estimation models respectively, the advantages of this invention are: (1) This invention constructs a multi-task fusion network to achieve collaborative processing of target detection, instance segmentation and pose estimation tasks under a unified network framework. By sharing the multi-layer feature map output by the backbone feature extraction network N1 and combining it with the feature pyramid fusion network F1, the top-down fusion of semantic information at different scales is achieved, which significantly improves the recognition accuracy of small targets and targets in complex scenarios. At the same time, it avoids the repeated feature calculation caused by the serialization of multiple models, reduces the overall computational complexity, and improves the inference efficiency.

[0019] (2) This invention adopts an Anchor-Free detection head structure to directly perform regression prediction on the target center position and scale parameters, reducing the scale matching error and parameter redundancy problems caused by traditional anchor frame design, thereby improving positioning stability and detection robustness. In terms of instance segmentation, an instance mask is generated by a separate structure of mask prototype image and mask coefficients. By sharing the prototype representation and combining it with the mask coefficients corresponding to the target for linear combination, the computational overhead is reduced while ensuring segmentation accuracy, thus improving segmentation efficiency. (3) In this invention, a pose estimation module is set up in the unified network framework of the multi-task fusion network. Based on the fusion feature map M1, the key point coordinates and their confidence information are output. The spatial distribution features of the key points are introduced into the cross-frame target matching process. By combining the bounding box position, category information and key point features for comprehensive matching, the stable tracking of targets between consecutive frames is realized, and the continuity and stability of target recognition results in video scenes are improved. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of the instance segmentation and target detection hybrid recognition method based on multi-feature fusion of the present invention; Figure 2 This is a schematic diagram of the traditional identification process; Figure 3 This is a flowchart illustrating the segmentation processing module of this invention. Figure 4 This is a schematic diagram illustrating the multi-frame recognition results of the present invention applied to a tennis match video stream. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] Reference Figure 2 , Figure 2This illustrates a traditional recognition process. Traditional solutions typically process object detection, instance segmentation, and pose estimation separately using independent models. Each branch completes feature extraction and outputs results independently, which are then combined directly, simply summed, or serially integrated at the backend to form the final recognition result. The "+" in the figure represents a result-level merging operation with no deep collaboration between the outputs of different recognition branches. This type of process lacks a shared backbone feature extraction and unified feature representation mechanism, making it difficult to achieve deep collaboration between object detection results, instance segmentation results, and pose estimation results. This easily leads to redundant feature extraction, high computational resource consumption, and insufficient semantic consistency between tasks.

[0023] Based on this, this embodiment provides a hybrid recognition method based on multi-feature fusion for instance segmentation and target detection, the overall recognition process of which is as follows: Figure 1 As shown, by sharing the multi-layer feature map output by the backbone feature extraction network N1 and fusing it with the multi-scale feature pyramid structure F1, collaborative processing of target detection, instance segmentation, and pose estimation tasks is achieved within a unified network framework. Simultaneously, a multi-task feature-level fusion mechanism is constructed through unified mapping, spatial alignment, channel stitching, and weighted summation fusion, and the spatial distribution features of key points are introduced into the cross-frame target matching process. This improves the completeness, continuity, robustness, and computational efficiency of target recognition results while ensuring recognition accuracy. Specifically, the following steps are included: Step S1: Shared feature extraction and multi-scale fusion; This step forms the foundation of the entire network, enabling feature extraction once and sharing across multiple tasks, thus avoiding feature redundancy at its source. Input data consists of static images or continuous video frames. This embodiment uses a set of images as input data for illustration. First, a multi-task fusion network is constructed based on a convolutional neural network, including a backbone feature extraction network N1, a multi-scale feature pyramid structure F1, an object detection processing module, an instance segmentation processing module, and a pose estimation processing module. The backbone feature extraction network N1 performs layer-by-layer convolution operations and nonlinear transformations on the input image to output multiple feature maps at different levels, with at least three feature maps having different spatial resolutions. These different level feature maps are then input into the multi-scale feature pyramid structure F1 to extract candidate target region features under different receptive fields. Feature concatenation or weighted fusion is used to connect features, outputting a fused feature map M1. The output fused feature map M1 serves as a shared feature map, providing a common feature basis for the subsequent object detection processing module, instance segmentation processing module, and pose estimation processing module.

[0024] In step S1, the specific process of extracting image features using the backbone feature extraction network N1 is as follows: Assuming the input image is X and the convolution kernel is K, we improve upon the traditional convolution operation by adding the concepts of stride and padding. Let the stride be... S Fill with P The improved formula is as follows:

[0025] In the formula, The first feature map obtained after convolution represents the first feature map. i Line number j Column pixel values, This represents the convolution operation between the input feature map X and the convolution kernel K. This indicates that the summation is performed by iterating over all elements of the convolution kernel K. This represents the nth element in the input feature map X corresponding to the current position of the convolution kernel. line, number The sampled pixel values ​​of the column, where Indicates the convolution stride. Indicates the filling parameter. and Indicates the index of an element within the convolution kernel. b Indicates the bias term. Denotes the first convolution kernel K. m Line number n The parameter values ​​of the column.

[0026] In the convolution operation of this invention, a stride is introduced. S ,filling P and bias terms b Among them, step size S Used to control the sliding interval of the convolution kernel on the input feature map, thereby adjusting the spatial resolution of the output feature map and improving the feature density of the collected data; padding P Used to expand the boundaries of the input feature map to maintain the feature map size, thereby reducing the loss of edge information; bias term b This parameter is used to linearly shift the convolution result, improving the model's expressive and fitting capabilities. By introducing these parameters, the convolution operation can maintain feature representation while balancing computational efficiency and feature integrity.

[0027] Furthermore, during the process of performing layer-by-layer convolution operations on the input image using the backbone feature extraction network N1 at multiple depths, multiple feature maps at different levels are obtained. During the generation of each feature map layer, the spatial size and number of channels of each feature map layer are determined based on the input-output shape calculation formula of the convolution operation, which describes the influence of the convolution kernel size, stride, and padding parameters on the feature map size variation. Specifically, the formula for calculating the output shape is as follows: Output height = (Input height - Convolution kernel height + 2 * padding) / stride + 1; Output width = (input width - kernel width + 2 * padding) / stride + 1; Number of output channels = Number of convolution kernels; Enter the formula for calculating the shape: Input height = (output height - 1) * stride - 2 * padding + convolution kernel height; Input width = (output width - 1) * stride - 2 * padding + convolution kernel width; Number of input channels = number of convolution kernels.

[0028] By using the above formulas for calculating input and output shapes, the spatial resolution and number of channels of the feature map can be precisely controlled according to the kernel size, stride, and padding parameters. Precise control of the feature map size ensures spatial alignment of features at different levels, providing a consistent foundation for upsampling, downsampling, and lateral connections in the multi-scale feature pyramid F1 structure, guaranteeing accurate spatial correspondences of features during fusion. Simultaneously, it provides a unified feature representation basis for subsequent processing such as object detection, instance segmentation, and pose estimation.

[0029] Step S2: Multi-task parallel branch processing; In this step, the fused feature map M1 obtained in step S1 is shared. The three task branches are executed in parallel without being chained together to avoid error accumulation. The fused feature map M1 obtained in step S1 is used as input and fed into the object detection processing module and the instance segmentation processing module for processing, specifically as follows: Combination Figure 1 The target detection processing module uses the YOLOv8 Anchor-Free detection head to directly regress the center point coordinates and width and height parameters of the target box by performing convolutional prediction on the fused feature map M1. Finally, it outputs the target detection result containing category information, bounding box position parameters and confidence score.

[0030] Combination Figure 1 and Figure 3 The processing flow of the instance segmentation module is as follows: Step 1: Extracting core features; In this step, the original YOLOv5 backbone network is replaced with a parallel ResNet and EfficientNet (denoted as the backbone feature extraction network N2) to extract features from the input image (i.e., the fused feature map M1), gradually transforming the image from low-level edge and texture information into high-level semantic feature representations. The feature maps output at different stages have different resolutions and semantic levels, providing a foundation for subsequent multi-scale feature fusion.

[0031] Step 2: Multi-scale feature pyramid fusion; The Feature Pyramid Network (FPN) is used to fuse and laterally connect the feature maps from different levels output in Step 1 in a top-down manner, resulting in a fused feature map M2 that simultaneously contains detailed and semantic information. This provides feature input for subsequent detection and segmentation sub-branches. This step uses the FPN to fuse high-level semantic information with low-level spatial detail information, enhancing the semantic representation of small targets in high-resolution feature maps and improving the overall semantic consistency of large targets in low-resolution feature maps. This addresses the problem that single-scale features are difficult to adapt to targets of different sizes simultaneously. Furthermore, the fused multi-scale feature map provides unified and scale-adaptive feature input for subsequent detection, segmentation, and pose estimation sub-branches, improving the consistency and stability of multi-task recognition.

[0032] Step 3: Detect and process sub-branches; The detection sub-branch mainly comprises a classification prediction part, a bounding box regression part, and a mask coefficient prediction part. The classification prediction part outputs target category information, the bounding box regression part outputs the target center position and width / height parameters, and the mask coefficient prediction part outputs the k-dimensional mask coefficients for the corresponding target. Based on the bounding box position and category information from the target detection results output by the target detection processing module, the detection sub-branch performs auxiliary feature extraction and position refinement on the target region in the fused feature map M2. For each target, it predicts the category information and bounding box parameters, simultaneously outputting a corresponding k-dimensional mask coefficient vector, thus generating target localization information and instance-level segmentation information within the same feature representation framework. The mask coefficients are continuous real values, used for linear combination with the k mask prototype maps generated by the segmentation branch to generate the instance mask for the corresponding target. This structure avoids the repetitive computation problem caused by building a separate segmentation network for each target, achieving shared basic mask representations and weighted expression of instance differences, reducing computational complexity while ensuring segmentation accuracy. Meanwhile, the mask coefficients output by the detection sub-branch are related to the target category and spatial location features, which makes the generated instance mask semantically consistent with the detection results, improving the stability and accuracy of target segmentation in complex scenes.

[0033] Step 4: Process sub-branches; The segmentation sub-branch adopts an instance segmentation structure based on mask prototype generation. On the one hand, it generates a mask prototype map based on the fused feature map M2. On the other hand, it combines the target position and mask coefficients output by the detection sub-branch to perform instance mask decoding on the corresponding target region. The segmentation sub-branch no longer performs independent segmentation of the entire image for each target. Instead, it extracts the features of the corresponding target region from the fused feature map M2 based on the target position obtained by the detection sub-branch. Finally, by linearly combining the mask coefficients corresponding to the target with the mask prototype map and performing pixel-by-pixel prediction on the features of each target region, the segmentation mask corresponding to each target is output to obtain the target instance segmentation result.

[0034] It is worth noting that by generating k mask prototype images through segmentation sub-branches, a shared spatial segmentation foundation representation for the entire image is constructed. The prototype images encode the basic shape and boundary structure information of different regions in the image, providing a unified representation basis for subsequent instance mask generation. Specifically, the segmentation sub-branch uses the fused feature map M2 as the prototype feature, inputs it to several convolutional layers for feature transformation, and generates k mask prototype images through channel mapping. Simultaneously, the target location output by the detection sub-branch is used as the basis for instance region selection, and instance mask decoding is performed by combining the k-dimensional mask coefficients of the corresponding target. Each mask prototype image corresponds to a basic mask representation, used to describe common mask features at different spatial locations, such as the target's contour shape, edge direction, and regional texture pattern.

[0035] This invention employs a collaborative mechanism between a detection sub-branch and a segmentation sub-branch: the detection sub-branch predicts target category information and bounding box parameters while outputting a k-dimensional mask coefficient vector for each target, where each mask coefficient represents the degree of matching between the corresponding target and k prototype images; the segmentation sub-branch focuses on generating k globally shared mask prototype images without performing independent predictions for specific targets. Finally, the mask coefficients output by the detection sub-branch are linearly combined with the k prototype images generated by the segmentation sub-branch, and an instance mask for each target is obtained through a weighted superposition, achieving the transformation from a shared basic representation to target-specific segmentation results.

[0036] Compared to schemes that predict the complete mask separately for each target, this invention generates instance masks by sharing prototype images and combining them linearly with the mask coefficients output from the detection sub-branch. This significantly reduces the computational load and parameter redundancy, improving computational efficiency. On one hand, the segmentation sub-branch only needs to generate k prototype images once to serve all targets in the image, avoiding repeated execution of the complete convolutional forward propagation for each target. On the other hand, the low-dimensional vector representation of the mask coefficients (usually k = 32 or 64) is much smaller than the high-dimensional pixel matrix of the complete mask (resolution is typically hundreds by hundreds), greatly reducing the number of model parameters and memory usage. Simultaneously, the shared prototype representation enhances the consistency of segmentation boundaries between different targets. Since the masks for all targets are generated based on the same set of prototype images, the segmentation boundaries of adjacent targets have spatial coordination at the prototype image level, reducing boundary breaks or overlapping conflicts caused by independent predictions and improving the stability and accuracy of instance segmentation in complex scenes.

[0037] Furthermore, the mask coefficients output by the detection sub-branch share the feature extraction process with the target category and bounding box parameters, ensuring semantic consistency between the generated instance mask and the detection result, thus avoiding the disconnect between detection and segmentation results in traditional methods. For each target object, its corresponding k mask coefficients are multiplied by k prototype images, and then all results are summed to obtain the instance segmentation result of that target object. By performing element-wise weighted summation and linear superposition of the k-dimensional mask coefficients corresponding to each target with the k mask prototype images generated by the segmentation sub-branch, the transformation from shared space representation to instance-level mask representation is achieved. This method transfers instance difference information from high-dimensional space representation to low-dimensional coefficient representation, reducing computational complexity and parameter redundancy while ensuring segmentation accuracy. Compared to the scheme of independently predicting the complete mask for each target, this invention only needs to generate a prototype image once to serve all targets, significantly reducing repeated convolution operations and improving segmentation efficiency. At the same time, the mask coefficients output by the detection sub-branch are related to the target category and spatial features, ensuring semantic consistency between the generated instance mask and the detection result, and enhancing the stability of segmentation results in complex scenes.

[0038] Step 5: Attitude estimation processing; The fused feature map M1, the output of the target detection module, and the output of the instance segmentation module are sent to the pose estimation module for processing. By combining the bounding box position information in the target detection results and the target mask information in the instance segmentation results, the corresponding target regions in the fused feature map M1 are constrained and features are extracted. Finally, the coordinate information and confidence information of a preset number of key points corresponding to each target are output.

[0039] The pose estimation processing module includes several convolutional and regression layers. The convolutional layers are used to extract keypoint features, and the regression layers output the coordinate information and confidence level of the keypoints to obtain the target's pose estimation result. These coordinates are either two-dimensional coordinates [x, y] or three-dimensional representations [x, y, visible], where visible represents the visibility of the keypoint and is used to represent the confidence level of the keypoint prediction output by the pose estimation module. The specific processing flow of step 5 is as follows: Step 5.1: Based on the target detection results in step S2, obtain the bounding box location information and category information of each target object, and determine the target candidate region in the fused feature map M1 accordingly; Step 5.2: Combining the target mask information in the instance segmentation result in step S2, foreground constraints are applied to the target candidate regions determined in step 5.1. The pixel regions belonging to the target subject within the bounding box are retained, and the background regions are suppressed or weakened, thereby obtaining the constraint region features corresponding to the target instance.

[0040] By jointly constraining the bounding box location information and the target mask information, the pose estimation process is focused on the target body region, reducing interference from background noise, occluded areas, and adjacent targets on key point localization. After completing the region constraint, local feature extraction is performed on the constrained region features. Multi-layer feature encoding is performed on the target region through convolutional layers, normalization layers, and nonlinear activation layers to extract edge information, structural information, and local semantic information related to the target pose.

[0041] For targets of different sizes, region alignment or scale normalization can be used to map different target regions to a uniform feature size, so as to ensure the consistency of the feature distribution of each target in the subsequent key point prediction process and improve the adaptability of the attitude estimation module to targets of different scales.

[0042] Step 5.3: Use the attitude estimation processing module to perform keypoint prediction regression calculation on the constrained region features described in Step 5.2, and output the predicted coordinates and confidence information of a preset number of keypoints corresponding to each target. Alternatively, keypoint prediction can be achieved using a keypoint heatmap prediction method. This involves first outputting the response heatmap of each keypoint in the target region, and then obtaining the keypoint coordinates through peak localization or weighted decoding.

[0043] Step 5.4: During the training phase, the predicted coordinates of the key points output in Step 5.3 are compared with the actual coordinates of the key points labeled in the corresponding training samples to calculate the error. The error calculation includes the difference in the three-dimensional coordinates of the key points and the difference in the rotation angle, wherein the difference in the rotation angle is represented by Euler angles or quaternions.

[0044] When the pitch angle is not ±90°, the yaw angle and roll angle are not in the same plane, and all rotational degrees of freedom can be confirmed. In this case, Euler angles are the optimal calculation method. The specific mathematical formula for Euler angles is as follows:

[0045] Where α is the pitch angle about the X-axis, β is the yaw angle about the Y-axis, and γ is the roll angle about the Z-axis. Euler angles are used, with each angle corresponding to the target's rotation on one of the three coordinate axes, facilitating manual interpretation and adjustment.

[0046] When the pitch angle is ±90°, the yaw and roll angles lie in the same plane, resulting in the loss of one rotational degree of freedom. This makes it impossible to uniquely determine the target attitude, and Euler angles are no longer applicable. Quaternions are used, setting the actual and predicted coordinates as four parameters to describe the target's rotational attitude. The mathematical expression of quaternions is:

[0047] in, For the real part, , , The virtual part, For rotation angle, The unit vector is the axis of rotation; The formula for converting a quaternion into a rotation matrix is:

[0048] Quaternions describe three-dimensional rotations in four-dimensional space, eliminating the gimbal lock problem and uniquely representing any rotational posture. They enable smoother rotational transitions during the iterative optimization of posture estimation. Quaternion multiplication requires only 16 multiplications and 12 additions, which is less computationally intensive than the 27 multiplications required for rotation matrices, making them suitable for real-time posture estimation tasks.

[0049] During task execution, the sequence of real and estimated poses is recorded. Each time step corresponds to a pose. For each time step, the deviation between the estimated pose and the real pose is calculated, involving differences in 3D coordinates and rotation angles. At the end of the task, the deviation of the final position, i.e., the endpoint, is calculated. This deviation is the Euclidean distance in 3D space, which is also the difference in rotation angles, to obtain a more general performance metric. The average EPE across multiple tasks or multiple time steps is calculated. The specific formula for calculating the 3D spatial distance between the estimated pose and the real pose in pose estimation is as follows:

[0050] in, , and It is the coordinates for estimating the attitude, and , and That is the true attitude.

[0051] Step S3: Multi-task information fusion and post-processing; The final output of step S2 is mapped to the output of the object detection module and the output of the instance segmentation module. After spatial alignment, the results are concatenated along the channel dimension and fused using a weighted summation method to obtain a unified fused feature representation. The fused feature representation is then input into a fully connected layer, where different task features are weighted and combined using learned weights to achieve collaborative expression of multi-task information. This results in target category information, target spatial location (bounding box), target pixel-level segmentation mask, and target key point coordinate information.

[0052] Specifically, in combination Figure 1 Step S3 includes the following steps: Step S3.1, Unified Data Mapping: The target detection results, instance segmentation results, and pose estimation results obtained in step S2 are subjected to unified mapping processing, which converts the category information, bounding box position parameters, target mask information, and key point coordinate information into a unified feature representation; then, similarity is calculated based on the target bounding box position, category features, and key point features. Step S3.2, Feature Alignment Processing: Based on the bounding box position information in the target detection results, spatial alignment is performed on the target mask in the instance segmentation results and the key point coordinates in the pose estimation results to ensure that the features of the same target corresponding to different tasks remain consistent in spatial position. Step S3.3, Feature fusion processing: The aligned multi-task features are concatenated along the channel dimension and fused using a weighted summation method to obtain a unified fused feature representation; Step S3.4, Output: Input the fused features into the fully connected layer, and combine the features of different tasks by learning weights to output the target category information, target bounding box position, target segmentation mask and target key point coordinate information.

[0053] Additionally, it should be noted that when the input data is a set of images, the output directly includes target category information, target spatial location (bounding box), target pixel-level segmentation mask, and target key point coordinate information. When the input data is a series of video frames, based on the unified fusion features, the spatial distribution features of key points in the pose estimation results are introduced into the cross-frame target matching process, associated with multi-dimensional targets for matching, and the matching results are combined with the fusion features of the current frame to output target category information with temporal identity, target spatial location (bounding box), target pixel-level segmentation mask, target key point coordinate information and target tracking ID.

[0054] Specifically, in combination Figure 1 When the input is a video frame, the specific process of result output matching and tracking of cross-frame targets is as follows: First, a target state library is established based on the fusion features of the target in the previous frame. The target state library includes the category information, bounding box position parameters, pixel-level segmentation mask, key point coordinate information and time sequence identity of each target. Then, for each target in the current frame, calculate its overall matching similarity score with all targets in the previous frame:

[0055] in, The bounding box location similarity is calculated based on IoU (Intersection over Union). , , The target of the current frame target in the previous frame The bounding box; To ensure consistency of category information, 1 is assigned to the same category and 0 to different categories; Similarity of spatial distribution of key points; These are the weighting coefficients for the similarity of bounding box positions. The weighting coefficient for consistency of category information. is the weighting coefficient for the spatial distribution similarity of key points, and , and The value is adaptively adjusted or manually forced to update based on the matching effect of the validation set.

[0056] When the target's movement speed is low and its attitude change is small, the bounding box overlap is high. Based on the optimal matching criteria, the IoU matching strategy is directly adopted; When the target undergoes rapid movement, occlusion, or deformation, resulting in a decrease in bounding box overlap... Reliability decreases, so key point spatial distribution similarity is introduced. The formula for enhancing matching robustness is as follows:

[0057] in, , The target of the current frame target in the previous frame The Coordinates of key points , The visibility confidence level for the corresponding key point. The total number of key points. This is the distance attenuation coefficient. The Gaussian kernel function maps the Euclidean distance to the interval [0,1]. The role of the keypoint spatial distribution features is as follows: when target occlusion causes bounding box overlap or deformation, the keypoint spatial distribution provides topological structure information independent of the box position, maintaining matching stability through the relative positional relationship of keypoints; when similar targets rotate, scale, or change pose in the video, the keypoint distribution features can better characterize the target's internal geometric structure than simple bounding boxes, reducing mismatches of targets with similar appearances; the keypoint visibility confidence score identifies occluded keypoints, automatically reducing the weight of occluded keypoints during matching calculation, avoiding matching drift caused by occlusion; Finally, based on this, the Hungarian algorithm or a greedy matching strategy is used to perform the optimal matching of the target in the current frame with the target in the previous frame, and the target identity (ID) is transmitted across frames. The target that is successfully matched inherits the temporal identity of the previous frame.

[0058] More specifically, combined Figure 3 This embodiment uses a single frame image from a tennis match video with a resolution of 1280×720 as an example to illustrate the main recognition process: 1. Shared feature extraction; A backbone feature extraction network N1 is used to extract multi-layer feature maps, which are then fused using a multi-scale feature pyramid structure F1 to generate a shared fused feature map M1. 2. Multi-task parallel branch processing; Object detection: Using the YOLOv8 Anchor-Free detection head, output the bounding boxes, categories, and confidence scores for "person" and "racket". Instance segmentation: Generate 32 mask prototype images, output 32-dimensional coefficients for each target, and assemble them to obtain pixel-level instance masks; Pose estimation: Based on bounding box constraints, target features are extracted, and the coordinates and confidence scores of 17 human body key points are output; 3. Multi-task information fusion and post-processing; Multi-task feature fusion: The outputs of each module are uniformly mapped and processed. After spatial alignment, they are concatenated in the channel dimension and fused using a weighted summation method to obtain a unified fused feature representation. The fused feature representation is then input into a fully connected layer, and different task features are weighted and combined by learning weights.

[0059] Output: For a single frame image, the recognition result is directly output, completing the collaborative recognition of object detection, instance segmentation and pose estimation. The output content includes category, bounding box, segmentation mask and key point information.

[0060] In addition, combined Figure 4 In another implementation of this embodiment, the input data is replaced with continuous video frames of a tennis match with a resolution of 1280×720. The main recognition process is as follows: 1. Multi-task recognition and processing in the current frame; Perform shared feature extraction, object detection, instance segmentation, and pose estimation on the current video frame to obtain the category information, bounding box, segmentation mask, and key point information of the athlete and racket in the current frame; 2. Cross-frame target association and tracking; Combining the target state library of the previous frame, the comprehensive matching similarity is calculated based on the bounding box overlap, category consistency and key point spatial distribution. The Hungarian algorithm is used for cross-frame association. The tracking ID of the previous frame is inherited for the successfully matched target, and a new temporal identity is assigned to the newly appearing target. 3. Output video frame results; The output includes video frame recognition results containing category, bounding box, segmentation mask, key points, and tracking ID.

[0061] Compared with traditional methods that directly stitch together target detection, instance segmentation, and pose estimation, the method of this invention achieves a comprehensive description of the target's "category-location-contour-pose" through multi-task feature-level fusion. This not only improves the completeness of the recognition results but also effectively enhances the overall recognition accuracy and robustness through multi-task information complementarity. Furthermore, a comparative experiment was conducted between the method of this embodiment and single-algorithm recognition; the experimental results are shown in Table 1. Table 1

[0062] As can be seen from the data in the table, the hybrid recognition method based on multi-feature fusion proposed in this invention outperforms existing mainstream methods in terms of overall performance, achieving an F1 score of 91.8%.

[0063] Compared to the current best-performing fusion method (feature fusion method based on attention mechanism), the method of this invention improves the F1 score by about 1.0 percentage point; compared to the weighted fusion method (Weighted Sum), it improves by about 1.6 percentage points; and compared to the simple feature concatenation method (Concat), it improves by about 2.3 percentage points.

[0064] Compared with traditional object detection and instance segmentation methods, the method of this invention has more significant advantages: it improves performance by 4.3 percentage points compared with YOLOv8 (object detection only), 3.8 percentage points compared with Mask R-CNN, 4.6 percentage points compared with Faster R-CNN, and 5.2 percentage points compared with RetinaNet.

[0065] Furthermore, compared to the serial structure (instance segmentation + detection) without fusion optimization, the method of this invention improves the F1 score by 3.5 percentage points, indicating that the multi-feature fusion and hybrid detection mechanism can effectively avoid the problem of information fragmentation and significantly improve the ability of target recognition and region understanding.

[0066] In terms of the three core metrics of accuracy, precision, and recall, the method of this invention has achieved optimal or near-optimal levels. The recall rate reached 91.5%, indicating that the method has a stronger ability to detect potential risk areas in complex scenarios and a lower false negative rate.

[0067] In terms of inference efficiency, the inference latency of the method of this invention is 32.6ms, which is slightly higher than that of the lightweight single-stage detection method, but significantly lower than that of the traditional two-stage method and serial structure. At the same time, it achieves a good balance between accuracy and efficiency, meeting the dual requirements of real-time performance and accuracy in practical applications.

[0068] In summary, the method of the present invention achieves high-precision identification of target objects and risk areas in complex scenes while ensuring high inference efficiency, and can be effectively used to track the same target in video streams, thus having good engineering application value.

[0069] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A hybrid recognition method for instance segmentation and target detection based on multi-feature fusion, characterized in that: Includes the following steps: Step S1: Construct a multi-task fusion network that includes a backbone feature extraction network, a multi-scale feature pyramid structure, an object detection processing module, an instance segmentation processing module, and a pose estimation processing module. Use the backbone feature extraction network to extract features from the input static image to be tested, and use the multi-scale feature pyramid structure to fuse them to obtain a shared fusion feature map M1. Step S2: Input the fused feature map M1 obtained in step S1 into the target detection processing module and the instance segmentation processing module for processing, respectively, to obtain the target detection result and the instance segmentation result; then send the fused feature map M1, the target detection result and the instance segmentation result together to the pose estimation processing module for processing, to obtain the pose estimation result; Step S3: Perform unified mapping processing on the target detection results, instance segmentation results, and pose estimation results from Step S2. After spatial alignment, the results are concatenated along the channel dimension and fused using a weighted summation method to obtain a unified fused feature representation. Input the fused feature representation into a fully connected layer, and use learned weights to weight and combine features from different tasks to output the recognition result. The output recognition result includes target category information, target spatial location, target pixel-level segmentation mask, and target key point coordinate information.

2. The hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to claim 1, characterized in that, In step S1, the process of the backbone feature extraction network extracting features from the image to be tested is as follows: In the formula, The first feature map obtained after convolution represents the first feature map. i Line number j Column pixel values, This represents the convolution operation between the input feature map X and the convolution kernel K. This indicates that the summation is performed by iterating over all elements of the convolution kernel K. This represents the nth element in the input feature map X corresponding to the current position of the convolution kernel. line, number The sampled pixel values ​​of the column, where Indicates the convolution stride. Indicates the filling parameter. and Indicates the index of an element within the convolution kernel. b Indicates the bias term. Denotes the first convolution kernel K. m Line number n The parameter values ​​of the column.

3. The hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to claim 1, characterized in that, In step S2, the processing flow of the instance segmentation module is as follows: ResNet and EfficientNet, configured in parallel, are used as the backbone extraction networks to extract features from the fused feature map M1, and the FPN feature pyramid network is used to fuse the features to obtain the fused feature map M2. The resulting fused feature map M2 is input into the detection sub-branch for processing, and the fused feature map M2 and the target position output by the detection sub-branch are used together for the segmentation sub-branch processing. During the processing of the segmentation sub-branch, the segmentation sub-branch generates a shared mask prototype based on the fused feature map M2, and extracts the features of the corresponding target region from the fused feature map M2 according to the target position obtained by the detection sub-branch. Then, by performing pixel-by-pixel prediction on the features of each target region, the segmentation mask corresponding to each target is output, and finally the instance segmentation result is output.

4. A hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to any one of claims 1-3, characterized in that, In step S2, the object detection processing module uses the YOLOv8 Anchor-Free detection head to perform convolutional prediction on the fused feature map M1, directly regressing the center point coordinates and width and height parameters of the target box, and finally outputs the object detection result containing category information, bounding box position parameters and confidence score.

5. The hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to claim 4, characterized in that, In step S2, the attitude estimation processing module operates as follows: Step 1: Based on the target detection results in step S2, obtain the bounding box location information and category information of each target object, and determine the target candidate region in the fused feature map M1 accordingly; Step 2: Combining the target mask information in the instance segmentation result in step S2, foreground constraints are applied to the target candidate regions determined in step 1. The pixel regions belonging to the target subject within the bounding box are retained, and the background regions are suppressed or weakened, thereby obtaining the constraint region features corresponding to the target instance. Step 3: Use the attitude estimation processing module to predict key points of the constrained region features described in Step 2, and output the predicted coordinates and confidence information of a preset number of key points for each target. Step 4: During the training phase, the predicted coordinates of the key points output in Step 3 are compared with the actual coordinates of the key points labeled in the corresponding training samples to calculate the error. The error calculation includes the difference in the three-dimensional coordinates of the key points and the difference in the rotation angle.

6. The hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to claim 5, characterized in that, The key point prediction uses either the key point regression prediction method or the key point heatmap prediction method.

7. The hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to claim 5, characterized in that, The calculation of the rotation angle difference uses Euler angles or quaternions, and the specific selection rules are as follows: When the pitch angle is not ±90°, it indicates that the yaw angle and roll angle are not in the same plane. The Euler angle calculation formula is used to calculate the rotation angle information. When the pitch angle is ±90°, it indicates that the yaw angle and roll angle are in the same plane, and the quaternion calculation formula is used to calculate the rotation angle information.

8. The hybrid recognition method for instance segmentation and target detection based on multi-feature fusion according to claim 4, characterized in that, When the input to step S1 is a series of video frames, cross-frame target matching and tracking are performed on the final output of step S3. The specific process is as follows: A target state library is established based on the fusion features of the target in the previous frame. The target state library includes the category information, bounding box position parameters, pixel-level segmentation mask, key point coordinate information and time sequence identity of each target. Then, for each target in the current frame, calculate its comprehensive matching similarity score with each target in the previous frame; when the target's motion speed is low and the pose change is small, the bounding box overlap is high, and the IoU matching strategy is adopted; when the target undergoes rapid motion, occlusion, or deformation, which leads to a decrease in the bounding box overlap, the spatial distribution similarity of key points is introduced to enhance the matching robustness. Finally, the Hungarian algorithm or greedy matching strategy is used to perform the optimal matching between the target in the current frame and the target in the previous frame, and the target identity is transferred across frames. The successfully matched target inherits the temporal identity of the previous frame, and the recognition result includes target category information, bounding box position parameters, pixel-level segmentation mask, key point coordinate information and temporal identity.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it enables the electronic device to implement the hybrid recognition method for instance segmentation and target detection based on multi-feature fusion as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the instance segmentation and target detection hybrid recognition method based on multi-feature fusion as described in any one of claims 1-8.

Images