Semi-supervised training method of instance segmentation model, instance segmentation method and related device

Abstract

Embodiments of the present application disclose an instance segmentation model training method. The method comprises: obtaining a labeled original high-view data set and an unlabeled target high-view data set; converting the depth information in the labeled original high-view data set to a target high-view to generate a projection data set; training a pre-constructed teacher model using the labeled original high-view data set and the projection data set to obtain a target teacher model; and training an initial student model based on the unlabeled target high-view data set, the projection data set and the target teacher model to finally obtain an instance image segmentation model. In the whole process, the labeled original high-view data set is converted to a target high-view by using depth information to obtain a projection data set, and the unlabeled target high-view data set is used to enable the instance segmentation model obtained by training to adapt to changes in different views and heights, thereby enhancing the robustness of the navigation system.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a semi-supervised training method for instance segmentation models, instance segmentation methods, and related equipment. Background Technology

[0002] With the development of artificial intelligence technology, object navigation has received widespread attention in the field of robotics. Object navigation tasks typically rely on instance segmentation models to construct semantic maps, and then achieve path planning and navigation through the detection and localization of target objects.

[0003] However, in practical applications, even if the robot operates in the same environment as the model training environment, the classification results of the same target object may change significantly due to differences in the height and direction of the viewpoint, which directly affects the navigation accuracy and may even lead to the failure of the navigation task. Summary of the Invention

[0004] Based on the above problems, this application provides a semi-supervised training method for instance segmentation models, an instance segmentation method, and related equipment, with the aim of improving the navigation success rate of various modular object target navigation algorithms at different target heights and perspectives.

[0005] In a first aspect, embodiments of this application provide a semi-supervised training method for an instance segmentation model, including:

[0006] Obtain labeled raw height view datasets and unlabeled target height view datasets; the labeled raw height view dataset includes first images of multiple raw height views and label information and depth information of each first image; the unlabeled target height view dataset includes second images of multiple target height views and depth information.

[0007] Using the depth information in the labeled original height viewpoint dataset, the first images under each original height viewpoint are converted to the target height viewpoint to generate a projection dataset composed of the first images under the target height viewpoint; the label information of the first image under the target height viewpoint automatically inherits the label information of the corresponding first image under the original height viewpoint.

[0008] The target teacher model is obtained by training a pre-built teacher model using the labeled original height-view dataset and the projection dataset;

[0009] Based on the parameters of the target teacher model, an initial student model is constructed;

[0010] The initial student model is trained based on the unlabeled target height viewpoint dataset, the projection dataset, and the pre-trained teacher model to obtain the target student model;

[0011] Based on the parameters of the target student model, an instance segmentation model is determined.

[0012] In one embodiment, the step of using the depth information in the labeled original height-view dataset to convert the first images under each original height-view to a target height-view, generating a projection dataset composed of the first images under the target height-view, includes:

[0013] Using the depth information in the labeled raw height-view dataset and the camera's internal parameters, each pixel of the first image is mapped to three-dimensional space to form point cloud data corresponding to the first image under the raw height-view.

[0014] Based on the preset viewpoint migration height, a transformation matrix is ​​established between the original height viewpoint and the target height viewpoint;

[0015] The point cloud data at the original height viewpoint is transformed using the transformation matrix to obtain the point cloud data at the target height viewpoint.

[0016] Using the camera's internal parameters, the point cloud data at each target height viewpoint is converted into a two-dimensional first image at the target height viewpoint, resulting in a projection dataset composed of the first images at each target height viewpoint.

[0017] In one embodiment, training the initial student model based on the unlabeled target height viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model includes:

[0018] The target teacher model is used to generate pseudo-labels for each second image in the unlabeled target height viewpoint dataset at the target height viewpoint.

[0019] Clustering is performed on the sample dataset of the current training batch, the average cluster center of each cluster is calculated, and the cluster centers are fused with those of the historical training batches to update the cluster centers of different categories of clusters; wherein, the sample dataset includes at least one second image in the unlabeled target height viewpoint dataset and at least one first image in the projected dataset under the target height viewpoint;

[0020] Based on the projection dataset, the unlabeled target height view dataset, and the cluster centers of the different categories of clusters, a target loss function is constructed. The initial student model is trained with the target loss function as the optimization objective to obtain the target student model.

[0021] In one embodiment, generating pseudo-labels for each second image in the unlabeled target height viewpoint dataset at the target height viewpoint using the target teacher model includes:

[0022] The target teacher model is used to generate initial pseudo-labels for each of the second images in the unlabeled target height viewpoint dataset; the initial pseudo-labels for the second images include prediction results, confidence scores of the prediction results, bounding box location information, and masks;

[0023] The initial pseudo-labels of each second image are filtered to obtain the final pseudo-labels for each second image;

[0024] The filtering process includes: filtering pseudo-labels whose prediction confidence is lower than a first preset threshold, pseudo-labels whose bounding box aspect ratio is lower than a second preset threshold, pseudo-labels whose mask pixel count is lower than a third preset threshold, and pseudo-labels whose ratio of the number of pixels in the bounding box to the number of pixels in the mask is lower than a fourth preset threshold; if the mask intersection-union ratio and the bounding box intersection-union ratio between any two pseudo-labels are both greater than a fifth preset threshold, then the two pseudo-labels are merged.

[0025] In one embodiment, the method further includes:

[0026] The weights of the pseudo-labels are set based on the confidence levels of each pseudo-label after merging.

[0027] The weight of the first image under each target height viewpoint in the labeled target height viewpoint dataset is set to 1.

[0028] In one embodiment, the step of constructing a target loss function based on the projection dataset, the unlabeled target height viewpoint dataset, and the cluster centers of the different categories of clusters, and training the initial student model with the target loss function as the optimization objective to obtain the target student model includes:

[0029] Based on the projection dataset, construct a first loss function;

[0030] For the unlabeled target height viewpoint dataset, calculate the similarity between each cluster center and the similarity between each second image and each cluster center;

[0031] Based on the similarity between the cluster centers and the pseudo-labels of each second image in the unlabeled target height viewpoint dataset, feature soft labels and category soft labels are generated for each second image.

[0032] A second loss function is constructed based on the feature soft label of each second image, the similarity between each cluster center, the category soft label of each second image, the similarity between each second image and each cluster center, and the category result probability vector output by the student model for each second image.

[0033] The target loss function is constructed from the first loss function and the second loss function;

[0034] The initial student model is trained using the target loss function as the objective to obtain the target student model.

[0035] Secondly, embodiments of this application also provide a method for training an instance segmentation model, including:

[0036] Obtain the target image of the object to be segmented;

[0037] The target image is input into the instance segmentation model to obtain the segmentation result output by the instance segmentation model; the instance segmentation model is trained by a semi-supervised training method based on the instance segmentation model described in the first aspect or any specific implementation of the first aspect.

[0038] Thirdly, embodiments of this application also provide a semi-supervised training device for an instance segmentation model, comprising:

[0039] The first module is used to acquire a labeled raw height viewpoint dataset and an unlabeled target height viewpoint dataset; the labeled raw height viewpoint dataset includes multiple first images of raw height viewpoints and label information and depth information of each first image; the unlabeled target height viewpoint dataset includes multiple second images of target height viewpoints and depth information.

[0040] The second module is used to utilize the depth information in the labeled original height viewpoint dataset to convert the first image under each original height viewpoint to the target height viewpoint, generating a projection dataset composed of the first image under the target height viewpoint; the label information of the first image under the target height viewpoint automatically inherits the label information of the corresponding first image under the original height viewpoint.

[0041] The third module is used to train a pre-built teacher model using the labeled original height-view dataset and the projection dataset to obtain the target teacher model;

[0042] The fourth module is used to construct an initial student model based on the parameters of the target teacher model;

[0043] The fifth module is used to train the initial student model based on the unlabeled target height viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model;

[0044] The sixth module is used to determine the instance segmentation model based on the parameters of the target student model.

[0045] Fourthly, embodiments of this application also provide a computer device, including:

[0046] Central processing unit, memory, input / output interfaces;

[0047] The memory is either a short-term storage memory or a persistent storage memory;

[0048] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in any of the above.

[0049] Fifthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs the method described in any of the above-mentioned embodiments.

[0050] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0051] This application embodiment obtains a labeled original height-view dataset and an unlabeled target height-view dataset. The labeled original height-view dataset includes multiple first images from the original height-view and the label and depth information of each first image. The unlabeled target height-view dataset includes multiple second images from the target height-view. Using the depth information in the labeled original height-view dataset, the first images from each original height-view are transformed to the target height-view, generating a projection dataset composed of the first images from the target height-view. The label information of each first image from the target height-view automatically inherits the label information of the corresponding first image from the original height-view. A pre-built teacher model is trained using the labeled original height-view dataset and the projection dataset to obtain a target teacher model. An initial student model is constructed based on the parameters of the target teacher model. The initial student model is trained based on the unlabeled target height-view dataset, the projection dataset, and the target teacher model to obtain a target student model. An instance segmentation model is determined based on the parameters of the target student model. Throughout the process, by utilizing labeled raw height-view datasets and unlabeled target height-view datasets, combined with depth information, the first image from the raw height-view is projected onto the target height-view. The projected dataset inherits the label information from the raw height-view dataset, enabling automatic image annotation at the target height-view. This improves the model's segmentation capability across different viewpoints, thereby effectively enhancing the accuracy of target detection and localization, reducing navigation deviations caused by changes in category recognition, and improving the applicability and robustness of various modular object target navigation algorithms in complex environments. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 This application provides a schematic diagram of a semi-supervised training method for an instance segmentation model, as illustrated in an embodiment of the present application.

[0054] Figure 2 A flowchart illustrating a semi-supervised height-view knowledge transfer algorithm provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram comparing the detection effect and navigation effect of a target object provided in an embodiment of this application;

[0056] Figure 4A schematic diagram of a semi-supervised training device for an instance segmentation model provided in this application embodiment;

[0057] Figure 5 This is a schematic diagram of a computer device structure provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Object navigation tasks require an agent to start from a random initial position in an unknown environment, find a target object of a specific category, and execute a stopping action. Current object navigation methods are mainly divided into two types: end-to-end and modular. End-to-end object navigation methods typically use a single network to encode visual and localization information, and then this network directly outputs the agent's action commands, encoding visual RGB images, depth maps, navigation targets, and localization information, using the model to output the agent's actions. Modular object navigation methods divide the entire navigation system into different modules responsible for different functions, such as a semantic map module, a navigation target point decision module, and a path planning module, and then achieve path planning and navigation through the detection and localization of target objects.

[0060] However, when using robots to perform actual target navigation, even if the robot and the model are trained in the same environment, the category recognition results of the same target object can vary greatly from different heights and directions, which can lead to navigation errors or failures.

[0061] Based on this, in the various embodiments of this application, when there is an existing labeled original height viewpoint dataset, unlabeled data is collected only under the target height viewpoint for semi-supervised training. In each iteration of the semi-supervised training process, the data of each batch is compared with the cluster center repository, and a training loss function is designed through contrastive learning method, so that the output features of the student model gradually migrate to the target height viewpoint domain, enabling the object target navigation algorithm to achieve a similar navigation effect under the target height viewpoint as under the original height viewpoint.

[0062] The various embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0063] This application provides a semi-supervised training method for an instance segmentation model, such as... Figure 1 As shown, the method includes steps S101-S106.

[0064] S101: Obtain the labeled raw height-view dataset and the unlabeled target height-view dataset;

[0065] The labeled raw height viewpoint dataset includes multiple first images of the raw height viewpoint and the label and depth information for each first image; the unlabeled target height viewpoint dataset includes multiple second images of the target height viewpoint. It should be noted that the raw height viewpoint refers to the original height viewpoint used during the training of the navigation model. The target height viewpoint refers to the new height viewpoint that the navigation model needs to adapt to during inference or practical applications, i.e., a height viewpoint different from the raw height viewpoint used during training.

[0066] The labeled raw height-view dataset is a manually labeled training dataset with category labels and location information. The unlabeled target height-view dataset is an unlabeled dataset containing only images from the target height perspective. The label information for each first image includes at least the category label of the target object in each image (e.g., "sofa," "bed," etc.) and the location label (e.g., bounding box or pixel-level segmentation mask). Depth information includes the depth value of each pixel in each image, which can usually be obtained through a depth sensor (e.g., LiDAR, stereo vision RGB-D camera), representing the distance of the object from the camera.

[0067] The labeled raw height-view dataset provides direct supervision signals (i.e., category and location labels) for training the teacher / student model. The unlabeled target height-view dataset is used for semi-supervised learning. Pseudo-labels are generated by the trained target teacher model and combined with the labeled raw height-view dataset to optimize the student model.

[0068] S102: Using the depth information in the labeled original height viewpoint dataset, the first images under each original height viewpoint are converted to the target height viewpoint to generate a projection dataset composed of the first images under the target height viewpoint.

[0069] In this embodiment of the application, the projection dataset is a dataset composed of the first images under each target height viewpoint. During the viewpoint transformation process, the object category and relative position relationship of the identified objects will not change or will change very little (although the position may change visually due to the viewpoint change). Therefore, the label information of the first image under each target height viewpoint automatically inherits the label information of the first image under its corresponding original height viewpoint. This label inheritance avoids the need to re-label the data under the target height viewpoint and reduces the labeling cost.

[0070] S103: Train the pre-built teacher model using the labeled original height viewpoint dataset and the projection dataset to obtain the target teacher model;

[0071] By training a pre-built teacher model using labeled raw height viewpoint datasets and projected datasets, the trained target teacher model can not only have good object detection and segmentation capabilities in the raw height viewpoint, but also have preliminary recognition capabilities in the target height viewpoint, providing support for subsequent semi-supervised learning to generate pseudo-labels.

[0072] S104: Construct an initial student model based on the parameters of the target teacher model;

[0073] The parameters of the target teacher model are obtained by training the teacher model on labeled raw height-view datasets and projected datasets, and specifically include network weights, learning rate, bias parameters, and parameters of the classification head.

[0074] By copying the parameters of the target teacher model into the pre-built student model, the cross-perspective knowledge learned in the teacher model is transferred to the student model, thus completing the initialization of the student model. This enables the initial student model to have the basic feature extraction and preliminary classification capabilities of the target teacher model, thereby providing basic support for subsequent semi-supervised training. At the same time, it improves the adaptability and recognition performance of the final instance segmentation model in the target height perspective domain.

[0075] S105: Train the initial student model based on the unlabeled target height viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model;

[0076] S106: Determine the instance segmentation model based on the parameters of the target student model.

[0077] This application embodiment inherits the cross-perspective knowledge of the target teacher model and further optimizes the student model through semi-supervised learning. The resulting instance segmentation model, combined with the unlabeled target height perspective dataset and the training signals of the real labels, has a certain object detection and classification capability. At the same time, it can adapt to the characteristics of the target height perspective, and thus can perform object recognition and navigation tasks in the target domain.

[0078] In summary, this embodiment of the application obtains a labeled original height-view dataset and an unlabeled target height-view dataset. The labeled original height-view dataset includes multiple first images from the original height-view and the label and depth information of each first image. The unlabeled target height-view dataset includes multiple second images from the target height-view. Using the depth information in the labeled original height-view dataset, the first images from each original height-view are transformed to the target height-view, generating a projection dataset composed of the first images from the target height-view. The label information of each first image from the target height-view automatically inherits the label information of the corresponding first image from the original height-view. A pre-built teacher model is trained using the labeled original height-view dataset and the projection dataset to obtain a target teacher model. An initial student model is constructed based on the parameters of the target teacher model. The initial student model is trained based on the unlabeled target height-view dataset, the projection dataset, and the target teacher model to obtain a target student model. An instance segmentation model is determined based on the parameters of the target student model. Throughout the process, by utilizing labeled raw height-view datasets and unlabeled target height-view datasets, combined with depth information, the first image from the raw height-view is projected onto the target height-view. The projected dataset inherits the label information from the raw height-view dataset, enabling automatic image annotation at the target height-view. This improves the model's segmentation capability across different viewpoints, thereby effectively enhancing the accuracy of target detection and localization, reducing navigation deviations caused by changes in category recognition, and improving the applicability and robustness of various modular object target navigation algorithms in complex environments.

[0079] In one embodiment, using depth information from a labeled raw height-view dataset, the first images under each raw height view are transformed to a target height view, generating a projection dataset composed of the first images under the target height view. This includes: using depth information from the labeled raw height-view dataset and camera intrinsic parameters, mapping the pixels of each first image to three-dimensional space to form point cloud data corresponding to the first image under the raw height view; establishing a transformation matrix between the raw height view and the target height view based on a preset viewpoint migration height; applying the transformation matrix to the point cloud data under the raw height view to obtain point cloud data under the target height view; and using camera intrinsic parameters to convert the point cloud data under each target height view into a two-dimensional first image under the target height view, thus obtaining a projection dataset composed of the first images under each target height view.

[0080] Camera internal parameters include focal length, principal point, intrinsic parameter matrix, distortion coefficient, etc. In the embodiments of this application, the method of generating 3D point cloud data from a 2D image using the depth information of the image and the camera internal parameters, and the method of converting point cloud data from the target height viewpoint into 2D image data using the camera internal parameters are both existing technologies and will not be described here.

[0081] The transformation matrix between the original height viewpoint and the target height viewpoint can be established using the following formula:

[0082]

[0083] Among them, P p Represents a 3D projection point with semantic annotation (i.e., "labeled") in the target height-view camera coordinate system; P represents the transformation matrix between the original height-view coordinate system and the target height-view camera coordinate system; s This represents a semantically labeled (i.e., "tagged") 3D projection point in the original height-view camera coordinate system. It is important to note that... That is, P p P s Both belong to three-dimensional real space A three-dimensional coordinate point in the system, with coordinates (x, y, z); That is, it represents a real matrix with 3 rows and 4 columns.

[0084] This application's embodiments, through the combination of depth information and transformation matrices, utilize a mathematical model to achieve the transformation from the original height viewpoint to the target height viewpoint. This avoids the tedious process of re-collecting data at different height viewpoints, significantly improving data preparation efficiency and effectively expanding the original dataset. It provides more diverse training samples for model training, contributing to improved model generalization ability. Furthermore, by utilizing the image's depth information, the two-dimensional image is mapped to three-dimensional point cloud data, and then rotation, translation, and projection are performed to finally obtain the first two-dimensional image at the target height viewpoint. This improves the accuracy of viewpoint transformation, ensures image quality at the target height viewpoint, and provides high-quality input data for model training at the target height viewpoint. This allows the trained model to adapt to different height viewpoints, improving the applicability and robustness of the instance segmentation model in practical applications.

[0085] Please combine Figure 2 understand, Figure 2 This is a flowchart illustrating a semi-supervised height-view knowledge transfer algorithm. In one embodiment, an initial student model is trained based on an unlabeled target height-view dataset, a projection dataset, and a target teacher model to obtain a target student model. The process includes: generating pseudo-labels for each second image in the unlabeled target height-view dataset at the target height view using the target teacher model; clustering the sample dataset of the current training batch, calculating the average cluster center of each cluster, and fusing it with the cluster centers of historical training batches to update the cluster centers of different categories of clusters; wherein the sample dataset includes at least one second image in the unlabeled target height-view dataset and at least one first image at the target height view in the projection dataset; and constructing a target loss function based on the projection dataset, the unlabeled target height-view dataset, and the cluster centers of different categories of clusters, using the target loss function as the optimization objective to train the initial student model to obtain the target student model.

[0086] The target teacher model uses its trained model parameters to infer and predict all candidate regions in each second image of the unlabeled target height viewpoint dataset, and assigns a pseudo-label to each candidate region based on the model's output. The generation of candidate regions is a prior art technique and will not be elaborated upon here. Each pseudo-label includes the category label of the target object in the candidate region (e.g., "sofa," "bed," etc.) and its location label (e.g., the top-left corner coordinates + width and height of the bounding box, or the top-left corner coordinates + bottom-right corner coordinates of the bounding box, pixel-level segmentation mask binary or multi-valued map), thus filling the gaps in the unlabeled data and allowing the unlabeled data to function similarly to labeled data during training.

[0087] Then, during training, clustering is performed on the sample dataset of the current batch in each training iteration, grouping sample data with similar characteristics into the same category, thereby obtaining multiple clusters of different categories. The sample dataset includes at least one second image from the unlabeled target height viewpoint dataset and at least one first image from the target height viewpoint in the projected dataset.

[0088] It should be noted that when using the first or second image as a sample, each candidate region in that image will be treated as a sample. Each candidate region can be a rectangular bounding box, which may or may not contain the object to be detected. Furthermore, candidate regions that do not contain objects or contain only a small portion of objects will be considered as the background category. These candidate regions are the basic units for clustering during training; by analyzing the characteristics of these candidate regions, they can be divided into different clusters.

[0089] In the clustering process, the average cluster center of each cluster (i.e., a set of samples with similar characteristics) is first calculated. This average cluster center is the average value or center value of the characteristics of all samples. Taking a certain category c as an example, the average cluster center of category c can be calculated with reference to the following formula:

[0090]

[0091] Among them, Q c The vector representing the average cluster center of category c is the average of the feature vectors of all samples belonging to category c; q b B represents the candidate feature vector of the b-th sample; B represents the sample q. b The total number; 1[y b =c] is an indicator function used to determine q b Does y have the same category label as category c? b =c, then 1[y b =c]=1; if y b ≠c, then 1[y b =c]=0; q represents the feature vector of all samples belonging to class c. b Accumulated value; This indicates the number of samples belonging to category c.

[0092] To make the clustering process more stable and avoid excessive fluctuations, the cluster centers calculated in the current batch are merged with the cluster centers from historical training batches. The merging method may be a weighted average or other strategies to smooth out fluctuations in the clustering process. For example, the calculation method for merging cluster centers can be referenced in the following formula:

[0093]

[0094] Where, n e This indicates the current training iteration number. It is the current nth e The cluster centers of category c stored in the cluster center repository after the next iteration are the latest cluster centers after fusion. It is the nth e After -1 iterations, the cluster centers of category c are stored in the cluster center repository. α is a constant, used as a smoothing coefficient for updating cluster centers, controlling the fusion ratio between historical clusters and the current batch of clusters. This represents the cluster center of category c calculated in the current mini-batch data. The same formula is used to calculate this. The current mini-batch refers to a small subset of data randomly selected from the entire training dataset during training, used for updating model parameters in one iteration.

[0095] By merging the cluster centers of the current batch and the cluster centers of the historical training batches, we can maintain stable identification of different categories and avoid the instability of clustering results caused by randomness due to unlabeled data.

[0096] Through the above process, the pseudo-labels generated by the target teacher model enable the use of unlabeled data, while the clustering operation further extracts the feature information of the samples and optimizes the category division. Combined with the subsequent training strategy of the target loss function, the segmentation performance of the student model under the target height perspective can be effectively improved.

[0097] In one embodiment, a target teacher model is used to generate pseudo-labels for each second image in the unlabeled target height viewpoint dataset at the target height viewpoint. This includes: generating initial pseudo-labels for each second image in the unlabeled target height viewpoint dataset using the target teacher model; the initial pseudo-labels for the second image include the prediction result, the confidence level of the prediction result, the bounding box position information, and the mask; filtering the initial pseudo-labels for each second image to obtain the final pseudo-labels for each second image; wherein the filtering process includes: filtering pseudo-labels whose prediction result confidence level is lower than a first preset threshold, pseudo-labels whose bounding box aspect ratio is lower than a second preset threshold, pseudo-labels whose mask pixel count is lower than a third preset threshold, and pseudo-labels whose ratio of the number of pixels in the bounding box to the number of pixels in the mask is lower than a fourth preset threshold; if the mask intersection-union ratio and the bounding box intersection-union ratio between any two pseudo-labels are both greater than a fifth preset threshold, then the two pseudo-labels are merged.

[0098] Specifically, the target teacher model infers for each second image in the unlabeled target height viewpoint dataset, generates a prediction result for each candidate region in each image, and generates a confidence score for each prediction result. Furthermore, in order to ensure the accuracy of subsequent target recognition, duplicate prediction results and poor-quality prediction results need to be filtered out. The target teacher model also identifies the bounding boxes and masks of target objects in the image, for example, by calculating the aspect ratio of the bounding box or the number of pixels in the bounding box or the number of pixels in the mask as filtering conditions.

[0099] Specific filtering conditions include:

[0100] (1) Confidence level: If the confidence level of a prediction result is lower than the first preset threshold, the prediction is considered unreliable and should be filtered out. Specifically, the first preset threshold can be set to 0.6.

[0101] (2) Bounding box aspect ratio: If the aspect ratio of the bounding box of a prediction result is too small as the second preset threshold, it means that the shape of the bounding box does not meet expectations, or it can be considered that it is not the target object to be identified and should be filtered out. Specifically, the second preset threshold can be set to 0.15.

[0102] (3) Number of pixels in the mask: If the number of pixels in the mask of a prediction result is less than the third preset threshold, it means that the target object is very small or not obvious and needs to be filtered. Specifically, the third preset threshold can be set to 200.

[0103] (4) Ratio of pixels within the bounding box to pixels in the mask: If the ratio of the number of pixels within the bounding box representing the target object to the number of pixels in the mask is less than the fourth preset threshold, it indicates that the prediction is inaccurate or incomplete, and will be filtered out. Specifically, the fourth preset threshold can be set to 0.15.

[0104] For example, the pseudo-label information of the u-th instance (also called "sample" / "candidate region of each second image") in the unlabeled target height viewpoint dataset includes three components: confidence score, bounding box coordinates, and binary mask data, which can be specifically represented as follows:

[0105]

[0106] in, This represents the pseudo-label of the u-th instance in the current image. It includes the normalized instance prediction confidence score; It is a binary mask after merging multiple overlapping instances. The binary image (i.e., the background is 0 and the target object's area (foreground) is 1) is predicted by the target teacher model, which represents the specific segmentation region of the instance. If the mask regions of multiple overlapping instances conflict, the final mask is the result of merging. The minimum bounding box is used to locate the position of an instance target in an image. It can also be understood as the smallest circumscribed rectangle. It is usually represented by coordinates (x, y, w, h), where x and y are the coordinates of the center of the bounding box, and w and h are the width and height of the bounding box.

[0107] (6) Mask crossover ratio and bounding box crossover ratio between the two prediction results:

[0108] Intersection over Union (IoU) is a metric that measures the degree of overlap between two regions (such as bounding boxes or segmentation masks for object detection). If the IoU between the mask of prediction result A and the mask of prediction result B is higher than a fifth preset threshold, and the IoU between the bounding boxes of prediction result A and prediction result B is also higher than the fifth preset threshold, then prediction results A and B are considered to be predictions of the same object, and can be merged. Specifically, the fifth preset threshold can be set to 0.75.

[0109] After the above steps, the target teacher model obtains a pseudo-label by merging similar prediction results and filtering out reliable results. It should be noted that the first, second, third, fourth, and fifth preset thresholds in this embodiment can be set according to specific application scenarios, and are not limited here. Through the above-described pseudo-label generation and filtering methods, even without manually labeled tags, the student model during training can learn using the pseudo-labels generated by the target teacher model.

[0110] In one embodiment, the method of this application further includes: setting the weight of the pseudo-labels according to the confidence of each pseudo-label after merging; and setting the weight of the first image under each target height viewpoint in the labeled target height viewpoint dataset to 1.

[0111] For example, the confidence level of the pseudo-label can be directly set as the weight. Alternatively, the results obtained after normalizing or nonlinearly transforming the confidence scores can be used as weights. This can be adapted to different training needs. For example, a threshold can be set to ignore low-confidence pseudo-labels, or the influence of high-confidence pseudo-labels can be increased by weighting the confidence levels. Determining the weights of pseudo-labels can be used to calculate the loss function, ensuring that the model prioritizes high-quality pseudo-label samples while making full use of the supervision information from the labeled data.

[0112] For the first image at the target height (i.e., the projection dataset), its weight can be set to 1. The labels of these projection datasets were previously manually labeled and are automatically inherited after the viewpoint transformation, so their reliability is high. Therefore, during training, it is necessary to ensure that the loss of these data is not weakened by the weight adjustment, and that they can provide stable, high-quality supervision signals for the instance models.

[0113] In one embodiment, a target loss function is constructed based on a projection dataset, an unlabeled target height viewpoint dataset, and cluster centers of different categories of clusters. An initial student model is trained using this target loss function as the optimization objective to obtain the target student model. This process includes: constructing a first loss function based on the projection dataset; calculating the similarity between cluster centers and the similarity between each second image and each cluster center for the unlabeled target height viewpoint dataset; generating feature soft labels and category soft labels for each second image based on the similarity between cluster centers and the pseudo-labels of each second image in the unlabeled target height viewpoint dataset; constructing a second loss function based on the feature soft labels of each second image, the similarity between cluster centers, the category soft labels of each second image, the similarity between each second image and each cluster center, and the category result probability vector output by the student model for each second image; constructing a target loss function from the first and second loss functions; and training the initial student model with the target loss function as the objective to obtain the target student model.

[0114] It's important to note that the initial student model refers to the student model at the start of training, whose parameters can be initialized using the pre-trained target teacher model. During subsequent training, the student model's parameters are continuously updated with each optimization step (e.g., gradient descent or other optimization algorithms), thus the student model's output changes dynamically. It's worth noting that the parameters of the student model during training generally differ from those of the initial student model. Therefore, the student model mentioned below can be understood as the current version of the student model after several optimization steps during training.

[0115] Specifically, in each iteration, the student model updates its parameters based on the sample data of the current training batch, thereby adjusting the prediction results output by the student model. Therefore, the current version of the student model can be understood as a temporary state at a certain stage of training, and its parameters and output results will continue to optimize as training progresses, gradually approaching the performance of the final target model.

[0116] In this embodiment of the application, for the projected dataset, a feature loss function (infoNCE loss) can be added to the standard cross-entropy loss calculation method in the detectron2 library to obtain a first loss function. This first loss function is used to measure the similarity between sample features and cluster centers, causing sample features to move closer to the corresponding cluster center while moving away from the cluster centers of other classes.

[0117] The first loss function can be specifically expressed as:

[0118]

[0119] in, This represents the label loss, specifically the cross-entropy loss of the class labels; L infoNCE Indicates infoNCE loss; L box L represents the bounding box position loss during instance segmentation of an image, used to measure the positional difference between the predicted target bounding box and the ground truth bounding box; mask The Dice loss represents the overlap between the mask and the real mask generated during instance segmentation. The Dice loss can be determined by measuring the degree of overlap between the mask generated during instance segmentation and the real mask. This represents the loss in determining whether an object exists in the candidate region generated by the Region Proposal Network (RPN). λ represents the candidate region localization loss, which measures the geometric positional error between the predicted bounding box of the candidate region and the true target bounding box; λ is a constant that can be adjusted according to the actual situation, and can be 0.25 for example.

[0120] Among them, the cross-entropy loss function of category labels This can be expressed by the following formula:

[0121]

[0122] Where, N i This represents the total number of samples in the dataset. This represents the true label of the i-th sample (where 1 represents the positive class and 0 represents the negative class; that is, if the sample is "bed", the corresponding bed category label is set to 1). It is the probability that the student model predicts the i-th sample as positive.

[0123] The infoNCE loss function can be expressed by the following formula:

[0124]

[0125] Among them, L infoNCEN represents the feature loss function of infoNCE; s This indicates that the current data batch contains the nth labeled data. s The predicted feature vector of each sample The total number; It is the nth s Sample Cluster center of the category; express The cosine similarity between them; τ1 represents the temperature parameter, used to control the smoothness of the similarity distribution, specifically it can be taken as 0.5; This indicates that the current data batch contains the nth labeled data. s The predicted feature vector of each sample Cluster centers of its category Similarity score; This indicates that the current data batch contains the nth labeled data. s The predicted feature vector of each sample Cluster centers Q for each category c The sum of similarities. This formula is used to enhance the sample feature vector. Cluster center of its category The similarity between the sample feature vectors and the cluster centers Q of other categories is weakened. c By comparing the similarity of the samples, the separability of the sample feature vectors in the category space is enhanced through contrastive learning, making the features of different categories more concentrated and easier to distinguish. At the same time, it can also combat noisy samples in the data and ensure that the student model can capture the essential features of the categories.

[0126] Bounding box position loss function L box This can be expressed by the following formula:

[0127]

[0128] sum() represents the summation of errors over all samples, representing the cumulative total loss; The true bounding box coordinates of the i-th sample are represented by a four-dimensional vector (x, y, w, h), which represent the coordinates (x, y) of the top-left corner of the instance bounding box on the image and the width and height (w, h), respectively. In this embodiment, the true bounding box coordinates of the candidate region can be taken from the bounding box coordinates in the label information of the first image at each target height view. Let be the bounding box coordinates predicted by the student model for the i-th sample. It is also a four-dimensional vector. This formula is used to optimize the accuracy of bounding boxes in object detection, reduce the error between the predicted box and the ground truth box, and improve the object detection capability of student models.

[0129] The loss function of the masking Dice loss for instance segmentation can be expressed by the following formula:

[0130]

[0131] Where N m y is the total number of pixels in the mask; m This represents the true mask pixel label of the m-th pixel (1 indicates foreground, 0 indicates background); in this embodiment, the true mask pixel label of the m-th pixel can be taken from the mask pixel label in the label information of the first image at each target height viewpoint; p m This represents the probability that the m-th pixel predicted by the student model is the foreground. By quantifying the matching degree between the predicted mask and the ground truth mask using Dice loss, the quality of subsequent instance segmentation can be optimized, ensuring that the prediction of each pixel in instance segmentation is closer to the true value.

[0132] It's important to note that the foreground refers to the target object in an image that needs to be detected or segmented. For example, in an instance segmentation task, the foreground might be an object of actual interest such as a bed, sofa, or chair; the foreground is the part that the model primarily needs to learn and focus on, representing a meaningful target object. The background, on the other hand, is the region in the image that does not belong to the target object. It is usually an "irrelevant part" of the image, such as the sky, ground, or walls. Background pixels are used to help the model identify the boundaries of the target object, but they are not the region that the model primarily needs to learn.

[0133] The classification loss of RPN can be expressed by the following formula:

[0134]

[0135] Where N r y represents the total number of candidate regions predicted by the student model; r The true label indicates whether the r-th candidate region belongs to the foreground or the background (1 for foreground, 0 for background); p r This represents the probability that the r-th candidate region predicted by the student model is foreground. This formula is used to supervise the classification of candidate regions, ensuring that the region generation module can effectively distinguish between foreground and background.

[0136] The RPN bounding box regression loss can be expressed by the following formula:

[0137]

[0138] Where sum() represents the summation of all errors; The true bounding box coordinates of the r-th candidate region are represented. In this embodiment, the true bounding box coordinates of the candidate region can be taken from the bounding box coordinates in the label information of the first image at each target height view. The vector (x, y, w, h) represents the coordinates and width and height of the top-left corner of the candidate region bounding box on the image, respectively. The bounding box coordinates of the r-th candidate region predicted by the student model. It is also a four-dimensional vector. This formula is used to optimize the bounding boxes of candidate regions, making them closer to the true target boxes.

[0139] For unlabeled target height viewpoint datasets, an unsupervised function (second loss function) needs to be defined, which can be expressed by the following formula:

[0140]

[0141] in, This represents the label loss, specifically the cross-entropy loss between the class labels; L1 contra The loss function representing the soft labels of sample features, also known as the second loss function; L1 box L1 represents the bounding box position loss when performing instance segmentation on the second image, used to measure the positional difference between the predicted target bounding box and the ground truth bounding box; mask The Dice loss represents the overlap between the mask and the real mask generated during instance segmentation. The Dice loss can be determined by measuring the degree of overlap between the mask generated during instance segmentation and the real mask. This indicates the loss in determining whether there is an object in the candidate region generated by RPN (i.e., measuring the degree of matching between the candidate region and the real object region); λ represents the candidate region localization loss, which measures the geometric positional error between the predicted bounding box of the candidate region and the true target bounding box; λ is a constant that can be adjusted according to the actual situation, and can be 0.25 for example.

[0142] For a target height viewpoint dataset labeled with pseudo-labels, first calculate the cosine similarity matrix S between each cluster center in the cluster center repository. The values ​​in the cosine similarity matrix S... ij = i Q j > represents the cluster center Q of cluster i. i The cluster center Q of the cluster of category j j The cosine similarity between them is calculated. The feature soft label and category soft label for each sample in the unlabeled dataset are calculated using the cosine similarity matrix S and one-hot labels (for "unlabeled data at target height," pseudo-labels annotated by the target teacher model are used). This can be expressed by the following formula:

[0143]

[0144] in, It is a feature soft pseudo-label vector; ​This is the instance prediction result of the student model. The corresponding category soft label vector; ∈1 and ∈2 are hyperparameters used to control the weight distribution between the pseudo-labels and the cosine similarity labels; This represents the score of each merged pseudo-label instance. The one-hot label corresponding to the label with the highest confidence score; S represents the cosine similarity matrix, where S is the value of the cosine similarity matrix S. ij = i Q j > represents the cluster center Q of cluster i. i The cluster center Q of the cluster of category j j Cosine similarity between them.

[0145]

[0146] in, Indicates the m-th u The similarity vector of unlabeled samples; Indicates the m-th u The predicted feature vector of each sample Similarity to cluster centers of category 1 (i.e., c=1); Indicates the m-th u The predicted features The similarity to the cluster centers of category C is calculated as follows, taking category C as an example:

[0147]

[0148] Wherein, the temperature coefficient τ2 = 0.8; Indicates the m-th u Each sample feature vector Cluster center Q with category c c Similarity score; This indicates the m-th unlabeled target in the current batch of data. u The predicted feature vector of each sample Cluster centers Q for all categories c The sum of similarities is used for normalization to ensure... The value is between [0, 1].

[0149] ​In this embodiment, two different soft label vectors (feature soft label and classification soft label) are introduced for each sample in the unlabeled dataset to guide the student model's feature and classification learning. The feature soft label captures the relationship between cluster centers using a cosine similarity matrix, helping the student model better optimize sample features during the learning process, making the feature distribution more aggregated, while considering the potential similarity between different categories. Furthermore, the category soft label predicted by the student model instances reflects a more specific classification objective, directly guiding the student model to optimize classification performance. The combination of these two soft labels provides richer supervision signals at both the feature and classification levels, avoiding overfitting or local optima that might result from a single signal.

[0150] Subsequently, using the soft labels of sample features as the optimization objective and the weights corresponding to each sample, a feature loss function was designed to prioritize pseudo-label samples with high confidence and shift towards the cluster centers of the corresponding categories.

[0151] Feature loss function L1 contra Specifically, it can be expressed as:

[0152]

[0153] Among them, M u This represents the total number of samples (instances) in the current mini-batch that are labeled with pseudo-labels and are unlabeled target height view datasets. Indicates sample m u The corresponding weights; Indicates a soft label for a feature; Represents a similarity vector; For similarity vectors Take the logarithm to calculate the soft label of the feature. The differences between them.

[0154] Add pseudo-label weights to unlabeled data and calculate the cross-entropy loss function. Specifically, it can be expressed as:

[0155]

[0156] in, M represents the weighted cross-entropy loss function for the unlabeled target height viewpoint dataset; u This represents the total number of unlabeled data samples (instances) in the current batch that have been marked with pseudo-labels; Indicates sample m u The corresponding weights; This represents the probability vector of the class result output by the student model, indicating the probability of the student model for the m-th class. u The probability that an unlabeled sample belongs to each category.

[0157] Bounding box position loss function L1 box This can be expressed by the following formula:

[0158]

[0159] sum() represents the summation of errors over all unlabeled samples, and represents the cumulative total loss; Indicates the m-th u The ground truth bounding box coordinates of the unlabeled samples are given by a four-dimensional vector (x, y, w, h), representing the coordinates of the top-left corner of the instance bounding box on the image (i.e., (x, y)) and its width and height (i.e., (w, h)), respectively; here, the bounding box loss for unlabeled data is L1. box True bounding box The pseudo-labels generated from the unlabeled target height viewpoint dataset can be taken from the target teacher model. The m-th prediction for the student model u The bounding box coordinates of an unlabeled sample. It is also a four-dimensional vector. This formula is used to optimize the accuracy of bounding boxes in object detection, reduce the error between the predicted box and the ground truth box, and improve the model's object detection capability.

[0160] The mask for instance segmentation, the Dice loss function, and the L1 loss function. mask This can be expressed by the following formula:

[0161]

[0162] Where N1 m y1 represents the total number of pixels in the mask. m Let represent the true mask pixel label of the m-th pixel (1 for foreground, 0 for background); here, the bounding box loss L1 for unlabeled data is used. mask True mask pixel label y1 m The pseudo-labels generated from the unlabeled target height viewpoint dataset can be taken from the target teacher model. p1 m This represents the probability that the m-th pixel predicted by the student model is the foreground. By quantifying the matching degree between the predicted mask and the ground truth mask using Dice loss, the quality of subsequent instance segmentation can be optimized, ensuring that the prediction of each pixel in instance segmentation is closer to the true value.

[0163] RPN classification loss This can be expressed by the following formula:

[0164]

[0165] Where N1 r y1 represents the total number of candidate regions predicted by the student model; rThe true label indicates whether the r-th candidate region belongs to the foreground or background (1 for foreground, 0 for background); p1 r This represents the probability that the r-th candidate region predicted by the student model is foreground. This formula is used to supervise the classification of candidate regions, ensuring that the region generation module can effectively distinguish between foreground and background.

[0166] RPN bounding box regression loss This can be expressed by the following formula:

[0167]

[0168] Where sum() represents the summation of all errors; Let (x, y, w, h) represent the coordinates of the ground truth bounding box of the r-th candidate region, which is a four-dimensional vector (x, y, w, h), representing the coordinates of the top-left corner and the width and height of the candidate region's bounding box on the image, respectively; here, the bounding box loss for unlabeled data is L1. box True bounding box The pseudo-labels generated from the unlabeled target height viewpoint dataset can be taken from the target teacher model. Let be the bounding box coordinates of the r-th candidate region predicted by the student model, also a four-dimensional vector. This formula is used to optimize the bounding box of the candidate region, making it closer to the true target box.

[0169] For the entire semi-supervised phase, the target loss function, composed of the first loss function and the second loss function, is as follows:

[0170] L=λ s L s +λ u L u

[0171] Among them, L s L represents the first loss function for a labeled projected dataset; u λ represents the second loss function for the unlabeled target height viewpoint dataset; s λ represents the parameters of the first loss function for the labeled projected dataset; u λ represents the parameters of the second loss function for the unlabeled target height viewpoint dataset; s and λ u During training, it can be set to 0.5.

[0172] It should be noted that L box L mask , These loss functions can vary depending on the structure of the instance model. For example, they can be replaced with different losses designed for other parameters, or one of the loss functions can be increased or decreased.

[0173] Accordingly, this application also provides an instance segmentation method, which includes steps S201-S202:

[0174] S201: Obtain the target image of the target object to be segmented;

[0175] S202: Input the target image into the instance segmentation model to obtain the segmentation result output by the instance segmentation model; wherein, the instance segmentation model is trained by a semi-supervised training method based on the instance segmentation model described in the first aspect or any specific implementation of the first aspect.

[0176] In this embodiment, the target image is the input image to be segmented, typically originating from a camera or other image acquisition device in a real-world application scenario. The target image can be a single image or multiple images taken from the target's height perspective. The target image contains the target objects to be segmented, such as tables and chairs in an indoor object navigation task. The target image is passed as input to a pre-trained instance segmentation model. The instance segmentation model analyzes pixel-level or region-level information in the target image through forward propagation and outputs the segmentation result. The segmentation result may specifically include the following: 1. Category distribution: the category to which each target object belongs (e.g., chair, table, etc.); 2. Binary mask: the pixel-level region of the target object; 3. Bounding box: the location of the bounding box of the target object.

[0177] The instance segmentation model provided in this application has high-precision target object segmentation capabilities and is applicable to a wide range of practical application scenarios, such as indoor navigation scenarios, identifying target objects to plan paths; or autonomous driving scenarios, for detecting and segmenting traffic targets; or industrial inspection scenarios, for segmenting specific target areas for subsequent processing.

[0178] Please refer to Figure 3 , Figure 3 This diagram illustrates a comparison between the implementation effects of the embodiments of this application and the implementation effects of other target segmentation algorithms. Figure 3The upper part contains four figures, illustrating the performance of different methods in object detection tasks: The first figure shows the detection failure when using the lower bound method (i.e., supervised training using only labeled data from the original height viewpoint). Specifically, the detection results incorrectly identify the target (e.g., a toilet) as another category or fail to detect the target correctly at all. The second figure shows the limitations of using the pseudo-label alignment (PAIS) method in semi-supervised instance segmentation. Target recognition errors still exist, and some target regions are not correctly segmented (missed detections). The third figure shows partial success in object segmentation using the guided distillation method for semi-supervised instance segmentation, but the segmentation mask is incomplete and does not fully cover the target object. The last figure shows the implementation effect of the embodiment of this application, where the target detection result is correct and the mask is complete, and the target object (e.g., a toilet) is clearly marked.

[0179] Figure 3 The lower half of the diagram illustrates the robot's navigation path and target object localization in an indoor scene during the target navigation task: The first image shows the navigation path using the lower bound method, with the red line representing the path and the blue arrow indicating the navigation end position. The robot failed to locate the target object, resulting in navigation failure. The second image shows the navigation result using the PAIS method. Although the navigation path is closer to the target, it still ends at the wrong position (blue arrow). The third image shows the result using the Guided method, where the robot successfully locates the target object (green area indicates the target object's position). The final image demonstrates the effectiveness of the method described in this embodiment. The navigation path (red line) is shorter and more direct, and the robot reaches the target position accurately in fewer steps (steps = 315) than in the previous three images, demonstrating higher navigation efficiency.

[0180] In practical applications and tests, the navigation success rate of the embodiments of this application has been improved to varying degrees under different target height perspectives, and the improvement effect of navigation success rate is more significant when the difference in height perspective is greater.

[0181] Specifically, the performance of different semi-supervised methods in instance segmentation tasks can be evaluated through the following comparative experiments. The instance segmentation models trained by various semi-supervised methods are used to verify their recognition accuracy and performance from different target height perspectives.

[0182] For different semi-supervised methods, the original height viewpoint was set to 0.88 meters. The trained 0.88-meter instance segmentation model was then transferred to target height viewpoints of 0.28 meters, 0.48 meters, and 0.68 meters, and retrained to adapt to the different height viewpoints. Subsequently, the recognition accuracy of the instance segmentation model was tested on the validation set corresponding to each target height viewpoint.

[0183] Please refer to Table 1, which shows the recognition accuracy of instance segmentation models obtained by different semi-supervised methods.

[0184]

[0185] In Table 1, AP represents Average Precision; AP50 represents the average precision when the IOU threshold is 50%; AP75 represents the average precision when the IOU threshold is 75%; AP, AP50, and AP75 are all general test metrics for instance segmentation models.

[0186] D u For unlabeled target height viewpoint datasets; D' u Represented as a labeled target height viewpoint dataset (i.e., a labeled dataset collected and labeled directly at the target height), D s The dataset consists of the original height-viewpoint dataset with labels; MaskR-CNN indicates that the Mark R-CNN model is used.

[0187] The lower bound indicates that the instance segmentation model is trained in a supervised manner using only the labeled raw height viewpoint (0.88 meters) dataset, without any additional adaptation to data from other target height viewpoints. In other words, this training method directly uses the instance segmentation model trained on the raw height viewpoint as a semantic module to process data from different target height viewpoints. The performance of the instance segmentation model trained only on the labeled raw height viewpoint (0.88 meters) dataset represents the lower bound performance of the instance segmentation model without specific optimization or training for target height viewpoints.

[0188] The upper bound indicates that the instance segmentation model used two types of data for supervised training: one is the original height-viewpoint (0.88 meters) labeled dataset D. s Secondly, labeled target height viewpoint dataset D' is directly collected from the target height perspective (e.g., 0.28 meters, 0.48 meters, 0.68 meters). uThe model is trained separately at each target height viewpoint to adapt to the data at these height views. This upper bound training method makes the most of the target height viewpoint data, so its performance represents the upper limit of the instance segmentation model's performance under fully supervised training conditions.

[0189] PAIS (Pseudo-label alignment algorithm in semi-supervised instance segmentation), Guided (Guided distillation algorithm for semi-supervised instance segmentation), and Our Method (semi-supervised training method and instance segmentation method of instance segmentation model provided in the embodiments of this application) will all be retrained when transferred to different target height perspectives (such as 0.28 meters, 0.48 meters, and 0.68 meters) to adapt to the data of each target height perspective.

[0190] Furthermore, the instance segmentation models in Table 1 are used as semantic modules, and the navigation success rate and efficiency are tested in conjunction with the following three modular navigation methods. Please refer to Table 2, which shows the impact of the following three instance segmentation models on the navigation success rate under different target height perspectives. The three instance segmentation models include the instance segmentation models trained using two existing semi-supervised methods (PAIS and Guided Distillation), and the instance segmentation models trained using the semi-supervised training method and instance segmentation method proposed in the embodiments of this application.

[0191] Among them, the three modular object target navigation methods used to test the navigation effect are SemExp, Peanut, and Frontier; specifically, SemExp represents the object target navigation algorithm using Goal-Oriented Semantic Exploration; Peanut represents the navigation algorithm adopted from the paper (Zhai A, Wang S. 2022. PEANUT: Predicting and Navigating to Unseen Targets[J]. 2023 IEEE / CVF International Conference on Computer Vision (ICCV), 10892-10901); Frontier represents the algorithm using Frontier Semantic Exploration for Visual Target Navigation.

[0192] In Table 2, SR (Success Rate) represents the success rate of navigation tasks, specifically the proportion of tasks that the navigation algorithm successfully completes among all tasks, that is, the proportion of tasks in which the navigator can accurately reach the target within a given target range.

[0193] SPL (Success weighted by Path Length) is a metric for measuring the efficiency of navigation algorithms. SPL considers whether the navigator, while successfully completing the task, has taken a shorter and more efficient path. A high SPL value indicates that the algorithm not only successfully completes the task but also completes it using a more efficient path.

[0194] In addition, Ground Truth (GT) was used as a reference when testing the navigation effect. GT does not rely on any instance segmentation model, but directly uses the real object labels (i.e., Ground Truth data) provided by the simulator as input. The navigation results obtained represent the performance under the most ideal conditions.

[0195]

[0196] To implement the semi-supervised training method for the instance segmentation model in this application, this application also provides a semi-supervised training apparatus for the instance segmentation model, such as... Figure 4 As shown, the device includes:

[0197] The first module 401 is used to acquire a labeled raw height viewpoint dataset and an unlabeled target height viewpoint dataset; the labeled raw height viewpoint dataset includes multiple first images of raw height viewpoints and label information and depth information of each first image; the unlabeled target height viewpoint dataset includes multiple second images of target height viewpoints and depth information.

[0198] The second module 402 is used to utilize the depth information in the labeled original height viewpoint dataset to convert the first image under each original height viewpoint to the target height viewpoint, and generate a projection dataset composed of the first image under the target height viewpoint; the label information of the first image under the target height viewpoint automatically inherits the label information of the corresponding first image under the original height viewpoint.

[0199] The third module 403 is used to train a pre-built teacher model using the labeled original height-view dataset and the projection dataset to obtain the target teacher model;

[0200] The fourth module 404 is used to construct an initial student model based on the parameters of the target teacher model;

[0201] The fifth module 405 is used to train the initial student model based on the unlabeled target height viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model;

[0202] The sixth module 406 is used to determine the instance segmentation model based on the parameters of the target student model.

[0203] In one embodiment, the second module is further configured to:

[0204] Using the depth information in the labeled raw height-view dataset and the camera's internal parameters, each pixel of the first image is mapped to three-dimensional space to form point cloud data corresponding to the first image under the raw height-view.

[0205] Based on the preset viewpoint migration height, a transformation matrix is ​​established between the original height viewpoint and the target height viewpoint;

[0206] The point cloud data at the original height viewpoint is transformed using the transformation matrix to obtain the point cloud data at the target height viewpoint.

[0207] Using the camera's internal parameters, the point cloud data at each target height viewpoint is converted into a two-dimensional first image at the target height viewpoint, resulting in a projection dataset composed of the first images at each target height viewpoint.

[0208] In one embodiment, the fifth module is further configured to:

[0209] The target teacher model is used to generate pseudo-labels for each second image in the unlabeled target height viewpoint dataset at the target height viewpoint.

[0210] Clustering is performed on the sample dataset of the current training batch, the average cluster center of each cluster is calculated, and the cluster centers are fused with those of the historical training batches to update the cluster centers of different categories of clusters; wherein, the sample dataset includes at least one second image in the unlabeled target height viewpoint dataset and at least one first image in the projected dataset under the target height viewpoint;

[0211] Based on the projection dataset, the unlabeled target height view dataset, and the cluster centers of the different categories of clusters, a target loss function is constructed. The initial student model is trained with the target loss function as the optimization objective to obtain the target student model.

[0212] In one embodiment, the fifth module is further configured to:

[0213] The target teacher model is used to generate initial pseudo-labels for each of the second images in the unlabeled target height viewpoint dataset; the initial pseudo-labels for the second images include prediction results, confidence scores of the prediction results, bounding box location information, and masks;

[0214] The initial pseudo-labels of each second image are filtered to obtain the final pseudo-labels for each second image;

[0215] The filtering process includes: filtering pseudo-labels whose prediction confidence is lower than a first preset threshold, pseudo-labels whose bounding box aspect ratio is lower than a second preset threshold, pseudo-labels whose mask pixel count is lower than a third preset threshold, and pseudo-labels whose ratio of the number of pixels in the bounding box to the number of pixels in the mask is lower than a fourth preset threshold; if the mask intersection-union ratio and the bounding box intersection-union ratio between any two pseudo-labels are both greater than a fifth preset threshold, then the two pseudo-labels are merged.

[0216] In one embodiment, the fifth module is further configured to:

[0217] The weights of the pseudo-labels are set based on the confidence levels of each pseudo-label after merging.

[0218] The weight of the first image under each target height viewpoint in the labeled target height viewpoint dataset is set to 1.

[0219] In one embodiment, the fifth module is further configured to:

[0220] Based on the projection dataset, construct a first loss function;

[0221] For the unlabeled target height viewpoint dataset, calculate the similarity between each cluster center and the similarity between each second image and each cluster center;

[0222] Based on the similarity between the cluster centers and the pseudo-labels of each second image in the unlabeled target height viewpoint dataset, feature soft labels and category soft labels are generated for each second image.

[0223] A second loss function is constructed based on the feature soft label of each second image, the similarity between each cluster center, the category soft label of each second image, the similarity between each second image and each cluster center, and the category result probability vector output by the student model for each second image.

[0224] The target loss function is constructed from the first loss function and the second loss function;

[0225] The initial student model is trained using the target loss function as the objective to obtain the target student model.

[0226] It should be noted that the semi-supervised training device for instance segmentation models provided in the above embodiments is only illustrated by the division of the above-described program modules when performing semi-supervised training of the instance segmentation model. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the semi-supervised training device for instance segmentation models provided in the above embodiments and the semi-supervised training method embodiments for instance segmentation models belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0227] Based on the hardware implementation of the above program modules, and in order to implement the semi-supervised training method for an instance segmentation model provided in this application embodiment, this application embodiment also provides a computer device, such as... Figure 5 As shown, computer device 500 includes:

[0228] Central processing unit 501, memory 502, and input / output interface 503;

[0229] The memory 502 is a short-term storage memory or a persistent storage memory;

[0230] The central processing unit 501 is configured to communicate with the memory 502 and execute instructions in the memory 502 to perform a semi-supervised training method for any of the above instance segmentation models.

[0231] Of course, in practical applications, the various components in the computer device 500 are coupled together through a bus system 504. It is understood that the bus system 504 is used to realize communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general designated all buses as Bus System 504.

[0232] The memory 502 in this embodiment is used to store various types of data to support the operation of the computer device 500. Examples of such data include any computer program used to operate on the computer device 500.

[0233] It is understood that when the processor in the computer device described above executes the computer program, it can also realize the functions of each unit in the corresponding device embodiments described above, which will not be repeated here. Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in memory and executed by the processor to complete the various embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device. For example, the computer program can be divided into units in the aforementioned computer device, and each unit can implement the specific functions described in the corresponding computer device above.

[0234] Computer equipment can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. Computer equipment may include, but is not limited to, processors and memory. Those skilled in the art will understand that processors and memory are merely examples of computer equipment and do not constitute a limitation on the computer equipment. It may include more or fewer components, or combinations of certain components, or different components. For example, computer equipment may also include input / output devices, network access devices, buses, etc.

[0235] A processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.

[0236] Memory can be used to store computer programs and / or modules. The processor performs various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a given function, etc.; the data storage area can store data created based on terminal usage, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0237] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the semi-supervised training method of the instance segmentation model described above and the instance segmentation method.

[0238] This application also provides a computer program product storing a computer program / instruction, which, when executed by a processor, is used to implement the semi-supervised training method for the instance segmentation model described in the first aspect or any specific implementation of the first aspect of this application, as well as the instance segmentation method described in the second aspect or any specific implementation of the second aspect of this application.

[0239] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0240] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0241] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0242] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0243] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A semi-supervised training method for an instance segmentation model, characterized in that, include: Obtain labeled raw height-view datasets and unlabeled target height-view datasets; the labeled raw height-view datasets include multiple first images of raw height views and label information and depth information of each first image; the unlabeled target height-view datasets include multiple second images of target height views; Using the depth information in the labeled original height viewpoint dataset, the first images under each original height viewpoint are converted to the target height viewpoint to generate a projection dataset composed of the first images under the target height viewpoint; the label information of the first image under the target height viewpoint automatically inherits the label information of the corresponding first image under the original height viewpoint. The target teacher model is obtained by training a pre-built teacher model using the labeled original height-view dataset and the projection dataset; Based on the parameters of the target teacher model, an initial student model is constructed; The initial student model is trained based on the unlabeled target height viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model; Based on the parameters of the target student model, an instance segmentation model is determined.

2. The semi-supervised training method for the instance segmentation model according to claim 1, characterized in that, The step of using the depth information in the labeled original height-view dataset to transform the first images under each original height-view to the target height-view, generating a projection dataset composed of the first images under the target height-view, includes: Using the depth information in the labeled raw height-view dataset and the camera's internal parameters, each pixel of the first image is mapped to three-dimensional space to form point cloud data corresponding to the first image under the raw height-view. Based on the preset viewpoint migration height, a transformation matrix is ​​established between the original height viewpoint and the target height viewpoint; The point cloud data at the original height viewpoint is transformed using the transformation matrix to obtain the point cloud data at the target height viewpoint. Using the camera's internal parameters, the point cloud data at each target height viewpoint is converted into a two-dimensional first image at the target height viewpoint, resulting in a projection dataset composed of the first images at each target height viewpoint.

3. The semi-supervised training method for the instance segmentation model according to claim 1, characterized in that, The process of training the initial student model based on the unlabeled target height-viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model includes: The target teacher model is used to generate pseudo-labels for each second image in the unlabeled target height viewpoint dataset at the target height viewpoint. Clustering is performed on the sample dataset of the current training batch, the average cluster center of each cluster is calculated, and the cluster centers are fused with those of the historical training batches to update the cluster centers of different categories of clusters; wherein, the sample dataset includes at least one second image in the unlabeled target height viewpoint dataset and at least one first image in the projected dataset under the target height viewpoint; Based on the projection dataset, the unlabeled target height view dataset, and the cluster centers of the different categories of clusters, a target loss function is constructed. The initial student model is trained with the target loss function as the optimization objective to obtain the target student model.

4. The semi-supervised training method for the instance segmentation model according to claim 3, characterized in that, The step of generating pseudo-labels for each second image in the unlabeled target height viewpoint dataset at the target height viewpoint using the target teacher model includes: The target teacher model is used to generate initial pseudo-labels for each of the second images in the unlabeled target height viewpoint dataset; the initial pseudo-labels for the second images include prediction results, confidence scores of the prediction results, bounding box location information, and masks; The initial pseudo-labels of each second image are filtered to obtain the final pseudo-labels for each second image; The filtering process includes: filtering pseudo-labels whose prediction confidence is lower than a first preset threshold, pseudo-labels whose bounding box aspect ratio is lower than a second preset threshold, pseudo-labels whose mask pixel count is lower than a third preset threshold, and pseudo-labels whose ratio of the number of pixels in the bounding box to the number of pixels in the mask is lower than a fourth preset threshold; if the mask intersection-union ratio and the bounding box intersection-union ratio between any two pseudo-labels are both greater than a fifth preset threshold, then the two pseudo-labels are merged.

5. The semi-supervised training method for the instance segmentation model according to claim 4, characterized in that, The method further includes: The weights of the pseudo-labels are set based on the confidence levels of each pseudo-label after merging. The weight of the first image at each target height viewpoint in the projection dataset is set to 1.

6. The semi-supervised training method for the instance segmentation model according to claim 3, characterized in that, Based on the projection dataset, the unlabeled target height view dataset, and the cluster centers of the different categories of clusters, a target loss function is constructed. The initial student model is trained using this target loss function as the optimization objective to obtain the target student model, including: Based on the projection dataset, construct a first loss function; For the unlabeled target height viewpoint dataset, calculate the similarity between each cluster center and the similarity between each second image and each cluster center; Based on the similarity between the cluster centers and the pseudo-labels of each second image in the unlabeled target height viewpoint dataset, feature soft labels and category soft labels are generated for each second image. A second loss function is constructed based on the feature soft label of each second image, the similarity between each cluster center, the category soft label of each second image, the similarity between each second image and each cluster center, and the category result probability vector output by the student model for each second image. The target loss function is constructed from the first loss function and the second loss function; The initial student model is trained using the target loss function as the objective to obtain the target student model.

7. An instance segmentation method, characterized in that, include: Obtain the target image of the object to be segmented; The target image is input into the instance segmentation model to obtain the segmentation result output by the instance segmentation model; The instance segmentation model is obtained based on the method described in any one of claims 1-6.

8. A semi-supervised training device for an instance segmentation model, characterized in that, include: The first module is used to acquire a labeled raw height-view dataset and an unlabeled target height-view dataset; the labeled raw height-view dataset includes multiple first images of raw height views and label information and depth information of each first image; the unlabeled target height-view dataset includes multiple second images of target height views. The second module is used to utilize the depth information in the labeled original height viewpoint dataset to convert the first image under each original height viewpoint to the target height viewpoint, generating a projection dataset composed of the first image under the target height viewpoint; the label information of the first image under the target height viewpoint automatically inherits the label information of the corresponding first image under the original height viewpoint. The third module is used to train a pre-built teacher model using the labeled original height-view dataset and the projection dataset to obtain the target teacher model; The fourth module is used to construct an initial student model based on the parameters of the target teacher model; The fifth module is used to train the initial student model based on the unlabeled target height viewpoint dataset, the projection dataset, and the target teacher model to obtain the target student model; The sixth module is used to determine the instance segmentation model based on the parameters of the target student model.

9. A computer device, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1 to 7.

Images