Sample data acquisition method, semantic segmentation model training method and device

Abstract

The present invention provides a sample data acquisition method, a semantic segmentation model training method and a device, and a specific implementation scheme includes: acquiring multiple sample images to be labeled; constructing a first labeling result set based on at least one first sample image among the multiple sample images; for any second sample image other than at least one first sample image among the multiple sample images, labeling the second sample image using the first labeling result set to obtain a second labeling result corresponding to the second sample image, and adding the second labeling result to the first labeling result set to obtain a second labeling result set; updating the first labeling result set based on the second labeling result set, and repeatedly performing the operation of labeling the second sample image using the first labeling result set until the labeling processing of all the second sample images in the multiple sample images is completed to obtain a target labeling result set; obtaining sample data based on the target labeling result set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to technical fields such as deep learning, computer vision, image processing, and more particularly to a sample data acquisition method and device, a semantic segmentation model training method and device, a semantic segmentation method and device, electronic equipment, storage medium, and computer program product. Background Art

[0002] With the rapid development of deep learning technology, deep learning technology is increasingly being applied to the processing and extraction of remote sensing images. It has made great progress in image classification, target recognition, change detection, etc. of remote sensing image big data, and promoted the development of intelligent information extraction of remote sensing images.

[0003] Methods based on deep learning usually rely on a large amount of high-quality sample data. Therefore, how to obtain high-quality sample data is particularly important. Summary of the invention

[0004] The present invention provides a sample data acquisition method and device, a semantic segmentation model training method and device, a semantic segmentation method and device, electronic equipment, storage medium and computer program product.

[0005] According to one aspect of the present invention, a sample data acquisition method is provided, comprising: acquiring a plurality of sample images to be labeled; constructing a first labeling result set based on at least one first sample image among the plurality of sample images, the first labeling result set comprising a first labeling result corresponding to each first sample image; for any second sample image other than at least one first sample image among the plurality of sample images, labeling the second sample image using the first labeling result set to obtain a second labeling result corresponding to the second sample image, and adding the second labeling result to the first labeling result set to obtain a second labeling result set; updating the first labeling result set based on the second labeling result set, and repeatedly performing the operation of labeling the second sample image using the first labeling result set until the labeling processing of all second sample images among the plurality of sample images is completed to obtain a target labeling result set; and obtaining sample data based on the target labeling result set.

[0006] According to another aspect of the present invention, a training method for a semantic segmentation model is provided, comprising: obtaining sample data, the sample data being obtained according to the sample data acquisition method described above; inputting the sample data into a second semantic segmentation model to obtain a third semantic segmentation result; obtaining a sample loss based on the difference between the third semantic segmentation result and the category label of the sample data; and adjusting the parameters of the second semantic segmentation model based on the sample loss.

[0007] According to another aspect of the present invention, a semantic segmentation method is provided, comprising: inputting an image to be processed into a second semantic segmentation model to obtain a semantic segmentation result for the image to be processed; wherein the second semantic segmentation model is trained according to the training method of the semantic segmentation model described above.

[0008] According to another aspect of the present invention, a sample data acquisition device is provided, comprising: a first acquisition module, used to acquire multiple sample images to be labeled; a construction module, used to construct a first labeling result set based on at least one first sample image among the multiple sample images, the first labeling result set including a first labeling result corresponding to each first sample image; a first labeling module, used to label any second sample image other than at least one first sample image among the multiple sample images using the first labeling result set to obtain a second labeling result corresponding to the second sample image, and add the second labeling result to the first labeling result set to obtain a second labeling result set; a second labeling module, used to update the first labeling result set based on the second labeling result set, and repeatedly perform the operation of labeling the second sample image using the first labeling result set until the labeling processing of all second sample images among the multiple sample images is completed to obtain a target labeling result set; and a second acquisition module, used to obtain sample data based on the target labeling result set.

[0009] According to another aspect of the present invention, a training device for a semantic segmentation model is provided, comprising: a third acquisition module, used to acquire sample data, the sample data being obtained according to the sample data acquisition device described above; a first semantic segmentation module, used to input the sample data into a second semantic segmentation model to obtain a third semantic segmentation result; a determination module, used to obtain a sample loss based on the difference between the third semantic segmentation result and the category label of the sample data; and an adjustment module, used to adjust the parameters of the second semantic segmentation model based on the sample loss.

[0010] According to another aspect of the present invention, a semantic segmentation device is provided, comprising: a second semantic segmentation module, used to input the image to be processed into a second semantic segmentation model to obtain a semantic segmentation result for the image to be processed; wherein the second semantic segmentation model is trained according to the training device of the semantic segmentation model described above.

[0011] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method as described above.

[0012] According to another aspect of the present invention, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the method as described above.

[0013] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which implements the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 is a flow chart of a method for obtaining sample data according to an embodiment of the present invention;

[0016] Figure 2 is a flowchart of a method for training a semantic segmentation model according to an embodiment of the present invention;

[0017] Figure 3 is a flowchart of a semantic segmentation method according to an embodiment of the present invention;

[0018] Figure 4 is a block diagram of a sample data acquisition device according to an embodiment of the present invention;

[0019] Figure 5 is a block diagram of a training device for a semantic segmentation model according to an embodiment of the present invention;

[0020] Figure 6 is a block diagram of a semantic segmentation apparatus according to an embodiment of the present invention;

[0021] Figure 7 It is a block diagram of an electronic device suitable for implementing a sample data acquisition method, a semantic segmentation model training method and a semantic segmentation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention and the drawings in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0023] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0024] In addition, in the description of the present invention, the terms “first”, “second”, “third”, etc. (if any) are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (for example, including but not limited to user personal information) shall comply with the relevant laws and regulations of the relevant countries and regions and shall not violate public order and good morals.

[0026] In addition, in the technical solution of the present invention, before obtaining or collecting relevant data, the authorization or consent of the data owner is obtained.

[0027] Figure 1 is a flow chart of a method for acquiring sample data according to an embodiment of the present invention.

[0028] like Figure 1 As shown, the sample data acquisition method 100 includes operations S110 to S150.

[0029] In operation S110 , a plurality of sample images to be labeled are obtained.

[0030] In operation S120 , a first annotation result set is constructed based on at least one first sample image among the plurality of sample images.

[0031] In operation S130, for any second sample image other than at least one first sample image among the multiple sample images, the second sample image is annotated using the first annotation result set to obtain a second annotation result corresponding to the second sample image, and the second annotation result is added to the first annotation result set to obtain a second annotation result set.

[0032] In operation S140, the first annotation result set is updated based on the second annotation result set, and the operation of annotating the second sample image using the first annotation result set is repeatedly performed until the annotation processing of all second sample images in the multiple sample images is completed to obtain a target annotation result set.

[0033] In operation S150 , sample data is obtained based on the target annotation result set.

[0034] According to an embodiment of the present invention, the sample image is a remote sensing image including one or more objects. The objects in the sample image may include, but are not limited to, buildings, farmland, roads, and grasslands.

[0035] According to an embodiment of the present invention, when annotating multiple sample images, at least one first sample image can be firstly selected from the multiple sample images to construct a first annotation result set, wherein the first annotation result set includes a first annotation result corresponding to each first sample image. Then, based on the constructed first annotation result set, at least one second sample image other than the at least one first sample image in the multiple sample images is annotated to obtain a target annotation result set, and sample data is further obtained based on the target annotation result set.

[0036] According to an embodiment of the present invention, the first sample image is a remote sensing image including one or more first objects. In some embodiments, at least one first sample image can be randomly selected from multiple sample images for constructing a first annotation result set. In other embodiments, object recognition can be performed on multiple sample images respectively to obtain an object recognition result for each sample image, and the object recognition result indicates the category of each object in the sample image. Then, based on these object recognition results, at least one first sample image is screened out from the multiple sample images for constructing a first annotation result set. The category of the first object contained in the at least one first sample image can represent the category of the object in the multiple sample images.

[0037] According to an embodiment of the present invention, the first annotation result corresponding to each first sample image is used to describe the first category of each first object in the first sample image and the image area where each first object is located. Abnormal annotation results have been eliminated from the first annotation results corresponding to each first sample image, so each first annotation result in the first annotation result set has a higher quality. The first annotation result set can be used as a priori to assist in the annotation of each second sample image, which is conducive to improving the accuracy of annotating the second sample images, and further conducive to improving the accuracy of the sample data. In an embodiment of the present invention, abnormal annotation results include incorrect category annotation of the first object in the first sample image, incorrect or incomplete annotation of the image area where the first object is located, etc.

[0038] According to an embodiment of the present invention, the second sample image is a remote sensing image including one or more second objects. In some embodiments, in the process of annotating each second sample image using the first annotation result set, a target annotation result set can be obtained in an iterative manner. This process is described below.

[0039] First, any second sample image is annotated auxiliaryily using the first annotation result set to obtain a second annotation result corresponding to the second sample image, and the second annotation result is added to the first annotation result set to obtain a second annotation result set.

[0040] In an embodiment of the present invention, the second sample image is annotated using the first annotation result set, and the second sample image can be matched with the relevant first annotation result in the first annotation result set by using the second sample image, and the second sample image is annotated auxiliaryly based on the matched first annotation result, so as to obtain a second annotation result corresponding to the second sample image. The second annotation result is used to describe the second category of each second object in the second sample image and the image area where each second object is located.

[0041] It can be understood that since the first annotation result set has a higher quality, by using the first annotation result set to assist in the annotation of the second sample image, the obtained second annotation result can also have a higher quality, thereby ensuring the quality of the second annotation result set composed of the first annotation result set and the second annotation result, which provides a basis for the subsequent accurate annotation of other second sample images.

[0042] Next, another second sample image is auxiliary-annotated using the second annotation result set to obtain a second annotation result corresponding to the other second sample image, and the second annotation result is added to the second annotation result set to obtain another second annotation result set.

[0043] Similarly, another second sample image can be used to match the relevant annotation results in the second annotation result set, and the other second sample image can be assisted in annotation based on the matched annotation results, so as to obtain a second annotation result corresponding to the other second sample image, and then another second annotation result set can be obtained according to the second annotation result and the second annotation result set. Accordingly, the other second annotation result set also has a high quality and can be used to assist in annotation of the remaining second sample images later.

[0044] Next, another second annotation result set is used to assist in annotating the new second sample image, and the operation is repeated until the annotation processing of all second sample images in the multiple sample images is completed to obtain a target annotation result set. The target annotation result set is used to describe the category of each object in the multiple sample images and the image area where each object is located.

[0045] By iteratively annotating the second sample image in the above manner, the second sample image can match the relevant annotation results in a larger annotation result set, which is conducive to matching more, more comprehensive and more accurate annotation results, thereby improving the accuracy and comprehensiveness of the matching, and further helping to improve the quality of the target annotation result set.

[0046] In other embodiments, when annotating each second sample image using the first annotation result set, if it is determined that the first annotation result set contains more categories of the first object, for example, the category of the first object contained in the first annotation result set can represent the category of the object in multiple sample images, the first annotation result set can be used to annotate each second sample image to obtain the second annotation result corresponding to each second sample image. Then, the second annotation result corresponding to each second sample image is added to the first annotation result set to obtain the target annotation result set. The target annotation result set is used to describe the category of each object in the multiple sample images and the image area where each object is located.

[0047] In the embodiment of the present invention, the process of labeling each second sample image using the first labeling result set is the same as or similar to the process described above, and will not be described in detail here.

[0048] It can be understood that since the first annotation result set has a higher quality, by using the first annotation result set to assist in the annotation of each second sample image, the obtained second annotation results can also have a higher quality, thereby ensuring the quality of the target annotation result set composed of the first annotation result set and each second annotation result, and then ensuring the quality of the sample data.

[0049] According to an embodiment of the present invention, since the target annotation result set can be used to describe the category of each object in the multiple sample images and the image area where each object is located, the target annotation result set can be used as sample data, or the target annotation results and multiple sample images can be used as sample data.

[0050] According to the technical solution of the present invention, in the process of annotating multiple sample images, a first annotation result set is constructed by utilizing at least one first sample image among the multiple sample images, and the first annotation result set is used as a priori to assist in the annotation of each second sample image. Since the first annotation result set has a high quality, each second annotation result obtained also has a high quality, thereby ensuring the quality of the target annotation result set finally obtained, and further ensuring the quality of the sample data.

[0051] According to an embodiment of the present invention, constructing a first annotation result set based on at least one first sample image may include the following process.

[0052] First, for each first sample image, the first sample image is input into a first semantic segmentation model to obtain a first semantic segmentation result. Then, based on the first semantic segmentation result, an initial annotation result is obtained.

[0053] According to an embodiment of the present invention, the initial annotation results of each first sample image can be obtained based on semantic segmentation and annotation. For example, a first semantic segmentation model can be used to perform semantic segmentation on each first sample image to obtain the semantic segmentation results corresponding to each first sample image. For each first sample image, since the semantic segmentation result marks the category of each first object in the first sample image and the image area where each first object is located, the initial annotation result of the first sample image can be determined based on the first semantic segmentation result. The initial annotation result includes multiple initial mask images corresponding to the first sample image and initial categories corresponding to each initial mask image, and the initial category is used to indicate the category of the first object in the initial mask image.

[0054] In the embodiment of the present invention, the first semantic segmentation model includes, for example, but is not limited to, a SAM (SegmentAnything Model) model, which can be selected according to actual needs.

[0055] Next, for each first sample image, an abnormal mask image among a plurality of initial mask images corresponding to the first sample image is determined, and the abnormal mask image and the initial category corresponding to the abnormal mask image are removed from the initial annotation result to obtain a first annotation result.

[0056] It is understandable that there may be some abnormal annotation results in the initial annotation results obtained using the first semantic segmentation model, such as an incorrect category annotation of a first object in the first sample image, or an incomplete mask image where the first object is located, etc. These abnormal annotation results cannot be used for subsequent annotation tasks, so they need to be removed.

[0057] According to an embodiment of the present invention, abnormal annotation results existing in the initial annotation results can be determined by determining abnormal mask images in multiple initial mask images. If a certain initial mask image is determined to be an abnormal mask image, the initial mask image and the initial category corresponding to the initial mask image are determined as abnormal annotation results, and they are removed from the initial annotation results.

[0058] According to an embodiment of the present invention, an abnormal mask image among a plurality of initial mask images may be determined in the following manner.

[0059] First, the annotation information corresponding to the first sample image is obtained.

[0060] According to an embodiment of the present invention, the annotation information includes a mask label set and a category label set for the first sample image. The mask label set includes mask true values ​​of multiple first objects in the first sample image, and the category label set includes category true values ​​of multiple first objects in the first sample image. The annotation information can be used to determine whether each initial mask image is an abnormal mask image.

[0061] Next, for each initial mask image, the degree of overlap between the initial mask image and the corresponding mask truth value, and the similarity between the features of the initial category corresponding to the initial mask image and the features of the corresponding category truth value are determined. If the degree of overlap corresponding to a certain initial mask image is less than a first threshold and / or the similarity is less than a second threshold, the initial mask image is determined to be an abnormal mask image. If the degree of overlap corresponding to the initial mask image is greater than or equal to the first threshold and the similarity is greater than or equal to the second threshold, it is confirmed that the initial mask image is not an abnormal mask image.

[0062] In an embodiment of the present invention, the degree of overlap between the initial mask image and the corresponding mask true value may be determined by determining an intersection-over-union ratio between the initial mask image and the corresponding mask true value.

[0063] In the embodiment of the present invention, the first threshold and the second threshold can be set according to actual conditions and are not limited here.

[0064] Through the above process, the abnormal annotation results in the initial annotation results (including the abnormal mask image and the initial category corresponding to the abnormal mask image) can be determined, and then the abnormal mask image and the initial category corresponding to the abnormal mask image can be removed from the initial annotation results to obtain the first annotation result. The first annotation result includes multiple first mask images corresponding to the first sample image and the first category corresponding to each first mask image, and the first category is used to indicate the category of the first object in the first mask image.

[0065] In some embodiments of the present invention, before removing abnormal mask images and the initial categories corresponding to the abnormal mask images from the initial annotation results, morphological operations can be performed on each initial mask image, such as corroding and / or dilating each initial mask image to make each initial mask image smoother, thereby achieving the purpose of correcting the shape of each initial mask image.

[0066] Next, based on the first annotation results corresponding to each first sample image, a first annotation result set is obtained.

[0067] Since abnormal annotation results have been eliminated from the first annotation results corresponding to each first sample image, the obtained first annotation result set has a higher quality and can be used to assist in the annotation of each second sample image.

[0068] In some embodiments, the first annotation result set may also be deduplicated.

[0069] According to an embodiment of the present invention, annotating the second sample image by using the first annotation result set to obtain a second annotation result corresponding to the second sample image includes the following operations.

[0070] First, at least one first mask image matching the second sample image in the first annotation result set is determined to obtain a target mask image.

[0071] According to an embodiment of the present invention, based on an instance retrieval algorithm or a pre-trained instance retrieval deep learning model, at least one first mask image that is identical to each second object in the second sample image can be identified from the first annotation result set, and the at least one first mask image can be used as a target mask image for the subsequent annotation process of the second sample image.

[0072] Next, the target mask image and the second sample image are input into the first semantic segmentation model to obtain a second semantic segmentation result.

[0073] According to an embodiment of the present invention, the annotation result of the second sample image can be obtained based on the semantic segmentation annotation method. For example, based on the target mask image, the first semantic segmentation model (such as the SAM model) can be used to perform semantic segmentation on the second sample image to obtain the second semantic segmentation result. In the above process, the target mask image can be used as a mask prompt to assist in the semantic segmentation of the second sample image, thereby improving the accuracy of the second semantic segmentation result, and further facilitating the improvement of the quality of the annotation result of the second sample image.

[0074] Next, based on the second semantic segmentation result, a second labeling result is obtained.

[0075] According to an embodiment of the present invention, the second semantic segmentation result annotates the category of each second object in the second sample image and the image area where each second object is located, so the second annotation result of the second sample image can be determined based on the second semantic segmentation result. The second annotation result includes a plurality of second mask images corresponding to the second sample image and a second category corresponding to each second mask image, and the second category is used to indicate the category of the second object in the second mask image.

[0076] According to an embodiment of the present invention, by using a target mask image associated with the second sample image to assist in labeling the second sample image, the accuracy of the second labeling result can be improved.

[0077] Figure 2 4 is a flowchart of a method for training a semantic segmentation model according to an embodiment of the present invention.

[0078] like Figure 2 As shown, the training method 200 of the semantic segmentation model includes operations S210~S240.

[0079] In operation S210, sample data is acquired.

[0080] According to the embodiment of the present invention, the sample data is obtained according to the sample data acquisition method in the above embodiment, which will not be described again here to save space.

[0081] In operation S220, the sample data is input into the second semantic segmentation model to obtain a third semantic segmentation result.

[0082] According to an embodiment of the present invention, the second semantic segmentation model may be, for example, any image semantic segmentation model capable of performing semantic segmentation on an image. The second semantic segmentation model may be selected according to actual needs, and is not specifically limited.

[0083] In operation S230 , a sample loss is obtained based on a difference between the third semantic segmentation result and the category label of the sample data.

[0084] According to an embodiment of the present invention, the category label of the sample data may be determined according to the category of the mask image described in the above embodiment.

[0085] In operation S240 , parameters of the second semantic segmentation model are adjusted based on the sample loss.

[0086] According to an embodiment of the present invention, by using high-quality sample data to train the second semantic segmentation model, the second semantic segmentation model can achieve better results, thereby improving the accuracy of the model output.

[0087] It should be noted that in addition to the semantic segmentation model, the sample data in the embodiment of the present invention can also be used to train other deep learning models. The type of deep learning model can be determined based on the actual application scenario, which is not limited here.

[0088] Figure 3 is a flowchart of a semantic segmentation method according to an embodiment of the present invention.

[0089] like Figure 3 As shown, the semantic segmentation method 300 includes operation S310.

[0090] In operation S310, the image to be processed is input into a second semantic segmentation model to obtain a semantic segmentation result for the image to be processed.

[0091] According to an embodiment of the present invention, the image to be processed is a remote sensing image including one or more objects. The second semantic segmentation model is trained according to the training method of the semantic segmentation model in the above embodiment. By using the trained second semantic segmentation model to process the image to be processed, a semantic segmentation result corresponding to the image to be processed can be obtained, and the image area where each object in the image to be processed is located and the category corresponding to each object are marked in the semantic segmentation result.

[0092] According to an embodiment of the present invention, by using the second semantic segmentation model trained in the above manner to process the image to be processed, the accuracy of the semantic segmentation result can be improved.

[0093] Figure 4 is a block diagram of a sample data acquisition device according to an embodiment of the present invention.

[0094] like Figure 4 As shown, the sample data acquisition device 400 includes a first acquisition module 410 , a construction module 420 , a first labeling module 430 , a second labeling module 440 and a second acquisition module 450 .

[0095] The first acquisition module 410 is used to acquire a plurality of sample images to be labeled.

[0096] The construction module 420 is used to construct a first annotation result set based on at least one first sample image among the multiple sample images, where the first annotation result set includes a first annotation result corresponding to each first sample image.

[0097] The first annotation module 430 is used to annotate any second sample image other than at least one first sample image among the multiple sample images using the first annotation result set to obtain a second annotation result corresponding to the second sample image, and add the second annotation result to the first annotation result set to obtain a second annotation result set.

[0098] The second annotation module 440 is used to update the first annotation result set based on the second annotation result set, and repeatedly perform the operation of annotating the second sample image using the first annotation result set until the annotation processing of all second sample images in the multiple sample images is completed to obtain a target annotation result set.

[0099] The second acquisition module 450 is used to obtain sample data based on the target annotation result set.

[0100] According to an embodiment of the present invention, the construction module 420 includes: a first semantic segmentation submodule, a first acquisition submodule and a removal submodule. The first semantic segmentation submodule is used to input the first sample image into the first semantic segmentation model for each first sample image to obtain a first semantic segmentation result; the first acquisition submodule is used to obtain an initial annotation result based on the first semantic segmentation result, the initial annotation result includes a plurality of initial mask images corresponding to the first sample image and an initial category corresponding to each initial mask image, and the initial category is used to indicate the category of the first object in the initial mask image; the removal submodule is used to determine an abnormal mask image among the plurality of initial mask images, and remove the abnormal mask image and the initial category corresponding to the abnormal mask image from the initial annotation result to obtain a first annotation result, the first annotation result includes a plurality of first mask images corresponding to the first sample image and a first category corresponding to each first mask image, and the first category is used to indicate the category of the first object in the first mask image.

[0101] According to an embodiment of the present invention, the elimination submodule includes: an acquisition unit and a determination unit. The acquisition unit is used to acquire the annotation information corresponding to the first sample image, the annotation information includes a mask label set and a category label set for the first sample image, the mask label set includes the mask true values ​​of multiple first objects in the first sample image, and the category label set includes the category true values ​​of multiple first objects in the first sample image; the determination unit is used to determine, for each initial mask image, the degree of overlap between the initial mask image and the corresponding mask true value, and the similarity between the features of the initial category corresponding to the initial mask image and the features of the corresponding category true value, and determine the initial mask image as an abnormal mask image when the degree of overlap is less than a first threshold and / or the similarity is less than a second threshold.

[0102] According to an embodiment of the present invention, the determining unit comprises a determining subunit, and the determining subunit is used to determine an intersection-over-union ratio between the initial mask image and the corresponding mask true value.

[0103] According to an embodiment of the present invention, the first annotation module 430 includes: a determination submodule, a second semantic segmentation submodule and a second acquisition submodule. The determination submodule is used to determine at least one first mask image matching the second sample image in the first annotation result set to obtain a target mask image; the second semantic segmentation submodule is used to input the target mask image and the second sample image into the first semantic segmentation model to obtain a second semantic segmentation result, and the target mask image is used as a mask prompt for semantic segmentation of the second sample image; the second acquisition submodule is used to obtain a second annotation result based on the second semantic segmentation result, and the second annotation result includes a plurality of second mask images corresponding to the second sample image and a second category corresponding to each second mask image, and the second category is used to indicate the category of the second object in the second mask image.

[0104] According to an embodiment of the present invention, the sample data acquisition device 400 further includes a correction module, and the correction module is used to perform a morphological operation on each initial mask image to correct the shape of the initial mask image.

[0105] According to an embodiment of the present invention, the first semantic segmentation model includes a SAM model.

[0106] Figure 5 is a block diagram of a training device for a semantic segmentation model according to an embodiment of the present invention.

[0107] like Figure 5 As shown, the training device 500 for the semantic segmentation model includes a third acquisition module 510 , a first semantic segmentation module 520 , a determination module 530 and an adjustment module 540 .

[0108] The third acquisition module 510 is used to acquire sample data. The sample data is obtained according to the sample data acquisition device of the above embodiment.

[0109] The first semantic segmentation module 520 is used to input the sample data into the second semantic segmentation model to obtain a third semantic segmentation result.

[0110] The determination module 530 is used to obtain the sample loss based on the difference between the third semantic segmentation result and the category label of the sample data.

[0111] The adjustment module 540 is used to adjust the parameters of the second semantic segmentation model based on the sample loss.

[0112] Figure 6 is a block diagram of a semantic segmentation apparatus according to an embodiment of the present invention.

[0113] like Figure 6 As shown, the semantic segmentation device 600 includes a second semantic segmentation module 610, which is used to input the image to be processed into the second semantic segmentation model to obtain a semantic segmentation result for the image to be processed. The second semantic segmentation model is trained according to the semantic segmentation model training device of the above embodiment.

[0114] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0115] Figure 7 A block diagram of an electronic device suitable for implementing a sample data acquisition method, a semantic segmentation model training method and a semantic segmentation method according to an embodiment of the present invention is schematically shown.

[0116] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0117] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0118] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0119] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the sample data acquisition method, the semantic segmentation model training method and the semantic segmentation method according to the embodiment of the present invention are implemented.

[0120] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0121] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the sample data acquisition method, semantic segmentation model training method and semantic segmentation method provided in the embodiment of the present invention.

[0122] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 701. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0123] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0124] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0125] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0128] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for obtaining sample data, characterized in that: include: Acquire multiple sample images to be labeled; Based on at least one first sample image among the plurality of sample images, construct a first annotation result set, wherein the first annotation result set includes a first annotation result corresponding to each of the first sample images; the first annotation result includes a plurality of first mask images corresponding to the first sample image and a first category corresponding to each of the first mask images, wherein the first category is used to indicate a category of a first object in the first mask image; For any second sample image among the multiple sample images except at least one of the first sample images, annotate the second sample image using the first annotation result set to obtain a second annotation result corresponding to the second sample image, and add the second annotation result to the first annotation result set to obtain a second annotation result set; updating the first annotation result set based on the second annotation result set, and repeatedly performing the operation of annotating the second sample image using the first annotation result set until the annotation processing of all the second sample images in the multiple sample images is completed to obtain a target annotation result set; Based on the target annotation result set, sample data is obtained; The step of annotating the second sample image using the first annotation result set to obtain a second annotation result corresponding to the second sample image includes: determining at least one first mask image in the first annotation result set that matches the second sample image to obtain a target mask image; The target mask image and the second sample image are input into a first semantic segmentation model to obtain a second semantic segmentation result, wherein the target mask image is used as a mask hint to perform semantic segmentation on the second sample image; and based on the second semantic segmentation result, the second annotation result is obtained.

2. The method according to claim 1, characterized in that The constructing a first annotation result set based on at least one first sample image among the plurality of sample images comprises: for each of the first sample images, Inputting the first sample image into a first semantic segmentation model to obtain a first semantic segmentation result; Based on the first semantic segmentation result, an initial annotation result is obtained, where the initial annotation result includes a plurality of initial mask images corresponding to the first sample image and an initial category corresponding to each initial mask image, where the initial category is used to indicate a category of a first object in the initial mask image; An abnormal mask image among the multiple initial mask images is determined, and the abnormal mask image and the initial category corresponding to the abnormal mask image are removed from the initial annotation result to obtain the first annotation result.

3. The method according to claim 2, characterized in that The determining of an abnormal mask image among the plurality of initial mask images comprises: Acquire annotation information corresponding to the first sample image, the annotation information comprising a mask label set and a category label set for the first sample image, the mask label set comprising mask true values ​​of a plurality of first objects in the first sample image, and the category label set comprising category true values ​​of a plurality of first objects in the first sample image; For each of the initial mask images, determine the degree of overlap between the initial mask image and the corresponding mask true value, as well as the similarity between the features of the initial category corresponding to the initial mask image and the features of the corresponding category true value, and when the degree of overlap is less than a first threshold and / or the similarity is less than a second threshold, determine the initial mask image as the abnormal mask image.

4. The method according to claim 3, characterized in that Determining the degree of overlap between the initial mask image and the corresponding mask true value comprises: An intersection-over-union ratio between the initial mask image and the corresponding mask ground truth is determined.

5. The method according to claim 2, characterized in that: The second annotation result includes a plurality of second mask images corresponding to the second sample image and a second category corresponding to each second mask image, where the second category is used to indicate a category of a second object in the second mask image.

6. The method according to any one of claims 2 to 5, characterized in that Before removing the abnormal mask image and the initial category corresponding to the abnormal mask image from the initial annotation result, the method further includes: For each of the initial mask images, a morphological operation is performed on the initial mask image to correct the shape of the initial mask image.

7. The method according to any one of claims 2 to 5, characterized in that The first semantic segmentation model includes a SAM model.

8. A method for training a semantic segmentation model, characterized in that: include: Acquire sample data, wherein the sample data is obtained by the method according to any one of claims 1 to 7; Inputting the sample data into a second semantic segmentation model to obtain a third semantic segmentation result; Obtaining a sample loss based on a difference between the third semantic segmentation result and the category label of the sample data; The parameters of the second semantic segmentation model are adjusted based on the sample loss.

9. A semantic segmentation method, characterized in that: include: Inputting the image to be processed into the second semantic segmentation model to obtain a semantic segmentation result for the image to be processed; The second semantic segmentation model is trained according to the method according to claim 8.

10. A sample data acquisition device, characterized in that: include: A first acquisition module, used to acquire a plurality of sample images to be labeled; A construction module, configured to construct a first annotation result set based on at least one first sample image among the plurality of sample images, wherein the first annotation result set includes a first annotation result corresponding to each of the first sample images; the first annotation result includes a plurality of first mask images corresponding to the first sample image and a first category corresponding to each of the first mask images, wherein the first category is used to indicate a category of a first object in the first mask image; a first annotation module, configured to annotate any second sample image among the plurality of sample images except at least one of the first sample images by using the first annotation result set to obtain a second annotation result corresponding to the second sample image, and add the second annotation result to the first annotation result set to obtain a second annotation result set; a second annotation module, configured to update the first annotation result set based on the second annotation result set, and repeatedly perform the operation of annotating the second sample image using the first annotation result set until the annotation processing of all the second sample images in the plurality of sample images is completed, thereby obtaining a target annotation result set; A second acquisition module, used to obtain sample data based on the target annotation result set; The first labeling module includes: a determination submodule, used to determine at least one first mask image in the first labeling result set that matches the second sample image to obtain a target mask image; a second semantic segmentation submodule, used to input the target mask image and the second sample image into a first semantic segmentation model to obtain a second semantic segmentation result, wherein the target mask image is used as a mask hint to perform semantic segmentation on the second sample image; and a second acquisition submodule, used to acquire the second labeling result based on the second semantic segmentation result.

11. A training device for a semantic segmentation model, characterized in that: include: A third acquisition module, configured to acquire sample data, wherein the sample data is obtained by the sample data acquisition device according to claim 10; A first semantic segmentation module, used for inputting the sample data into a second semantic segmentation model to obtain a third semantic segmentation result; A determination module, configured to obtain a sample loss based on a difference between the third semantic segmentation result and the category label of the sample data; An adjustment module is used to adjust the parameters of the second semantic segmentation model based on the sample loss.

12. A semantic segmentation device, characterized in that: include: The second semantic segmentation module is used to input the image to be processed into the second semantic segmentation model to obtain a semantic segmentation result for the image to be processed; wherein the second semantic segmentation model is trained by the training device of the semantic segmentation model according to claim 11.

13. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 9.

15. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images