Image semantic segmentation method and device based on multiple modes, equipment and medium

Abstract

The invention discloses an image semantic segmentation method and device based on multiple modes, equipment and a medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining a target data set comprising a target image, label text information and first label frame information; obtaining a target semantic segmentation request, determining a to-be-segmented image corresponding to the target image from the target image, and generating first positive and negative sample prompt points and second positive and negative sample prompt points for the to-be-segmented image; determining a first target grayscale image based on the first positive and negative sample prompt points, and determining a second target grayscale image based on the second positive and negative sample prompt points; determining second label frame information based on the target image and the label text information, and determining a third target grayscale image based on the second label frame information; determining a fourth target grayscale image based on the first label frame information; and determining a final target grayscale image according to the four different target grayscale images, and rendering the target grayscale image to obtain a target image after semantic segmentation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and particularly to a multi-modal based image semantic segmentation method, device, equipment and medium. Background Art

[0002] With the development of deep learning technology, multi-modal large models have demonstrated powerful capabilities in many fields such as natural language, computer vision, and speech. In the field of computer vision, classic large models such as Segment Anything Model (SAM, a deep learning model for image segmentation) and Bootstrapping Language - Image Pre - training (BLIP) have emerged continuously, and they have achieved remarkable results in tasks such as image scene classification, object detection, and semantic segmentation. Among them, the general large model SAM for semantic segmentation tasks has been widely adopted and improved in many studies due to its excellent performance.

[0003] Semantic segmentation, as a key task in the field of computer vision, aims to classify each pixel in an image to achieve fine-grained understanding and segmentation of the image. However, semantic segmentation annotation of large-scale image datasets is a labor-intensive and time-consuming task. To alleviate this burden, existing studies have developed interactive semi-automatic image segmentation annotation tools by integrating the SAM model, such as Segment Anything Labelling Tool (SALT) and Interactive Semi-automatic Annotation Tool (ISAT). However, there are still some challenges in the current semantic segmentation annotation methods. On the one hand, the segmentation performance of existing semantic segmentation models for specific domain data that has not been trained remains to be improved, or there are problems of low accuracy and high annotation cost when dealing with complex images. On the other hand, some fast annotation tools still require a large amount of manual discrimination and correction during the annotation process, with problems such as strong subjectivity, time-consuming and laborious annotation, thus restricting the rapid development and application of semantic segmentation models. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multi-modal based image semantic segmentation method, device, equipment and medium, which can improve the conversion efficiency from the target dataset to the semantic segmentation dataset, reduce the annotation cost, and achieve an efficient, accurate and automated semantic segmentation method. The specific solutions are as follows: In the first aspect, this application provides a multi-modal based image semantic segmentation method, including: Obtain a target dataset; the target dataset includes target images, label text information corresponding to each target object in the target images, and first label border information; Obtain the target semantic segmentation request sent by the client, determine the image to be segmented corresponding to the target semantic segmentation request from the target image, use the HC saliency detection technology to generate the first positive sample hint points and the first negative sample hint points for the image to be segmented, and use the SLIC superpixel segmentation technology to generate the second positive sample hint points and the second negative sample hint points for the image to be segmented; Determine the first target grayscale image based on the first positive sample hint points and the first negative sample hint points through a preset image segmentation model, and determine the second target grayscale image based on the second positive sample hint points and the second negative sample hint points; Determine the second label bounding box information corresponding to the target semantic segmentation request based on the target image and the label text information through a preset object detection model, and use the preset image segmentation model to determine the third target grayscale image based on the second label bounding box information; Determine the fourth target grayscale image corresponding to the target semantic segmentation request based on the first label bounding box information through the preset image segmentation model; Determine the final target grayscale image according to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained by different methods, and perform visual rendering of different colors on the objects in the target grayscale image to obtain the target image after semantic segmentation.

[0005] Optionally, after determining the image to be segmented corresponding to the target semantic segmentation request from the target image, it further includes: Determine the target local image corresponding to the target semantic segmentation request from the image to be segmented based on the label bounding box coordinates in the first label bounding box information.

[0006] Optionally, the using the HC saliency detection technology to generate the first positive sample hint points and the first negative sample hint points for the image to be segmented includes: Use the HC saliency detection technology to determine the core target area in the target local image, generate multiple groups of first positive sample hint points inside the core target area, and generate multiple groups of first negative sample hint points outside the core target area; wherein, different groups correspond to different numbers of first positive sample hint points or first negative sample hint points; Correspondingly, the using the SLIC superpixel segmentation technology to generate the second positive sample hint points and the second negative sample hint points for the image to be segmented includes: Use the SLIC superpixel segmentation technology to determine the core target area in the target local image, generate multiple groups of second positive sample hint points inside the core target area, and generate multiple groups of second negative sample hint points outside the core target area.

[0007] Optionally, the step of determining the first target grayscale image based on the first positive sample cue points and the first negative sample cue points by the preset image segmentation model, and determining the second target grayscale image based on the second positive sample cue points and the second negative sample cue points includes: Inputting the multiple groups of first positive sample cue points and the multiple groups of first negative sample cue points into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on different combinations of the first positive and negative sample cue points, obtains first image semantic segmentation results respectively corresponding to the different combinations of the first positive and negative sample cue points, and takes the first image semantic segmentation result with the optimal model performance as the final first target image semantic segmentation result, and saves the first target image semantic segmentation result as the first target grayscale image; Inputting the multiple groups of second positive sample cue points and the multiple groups of second negative sample cue points into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on different combinations of the second positive and negative sample cue points, obtains second image semantic segmentation results respectively corresponding to the different combinations of the second positive and negative sample cue points, and takes the second image semantic segmentation result with the optimal model performance as the final second target image semantic segmentation result, and saves the second target image semantic segmentation result as the second target grayscale image.

[0008] Optionally, the step of determining the second label bounding box information corresponding to the target semantic segmentation request by the preset object detection model based on the target image and the label text information, and determining the third target grayscale image by the preset image segmentation model based on the second label bounding box information includes: Inputting the target image and the label text information into the preset object detection model, so that the preset object detection model determines the target region corresponding to the target semantic segmentation request from the target image, and obtains the second label bounding box information respectively corresponding to the target regions; Inputting the second label bounding box information into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on the second label bounding box information, obtains the third target image semantic segmentation result, and saves the third target image semantic segmentation result as the third target grayscale image.

[0009] Optionally, the step of determining the fourth target grayscale image corresponding to the target semantic segmentation request by the preset image segmentation model based on the first label bounding box information includes: Determine the first label border information corresponding to the target semantic segmentation request from the target data set, perform semantic segmentation on the image based on the label border coordinates in the first label border information to obtain the fourth target image semantic segmentation result, and save the fourth target image semantic segmentation result as a fourth target grayscale image.

[0010] Optionally, determining the final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained by different methods, and performing visual rendering of different colors on the objects in the target grayscale image to obtain the target image after semantic segmentation, includes: Determine the pixel categories corresponding to each pixel point in the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image; According to the determination method based on each target grayscale image, assign corresponding weights to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image respectively, and use the fourth target grayscale image as the reference image; For any pixel point in the reference image, determine the pixel category corresponding to the any pixel point in the fourth target grayscale image, determine the pixel categories of the pixel points corresponding to the any pixel point in the first target grayscale image, the second target grayscale image, and the third target grayscale image, and determine the target scores of the pixel categories of the pixel points corresponding to the any pixel point in different target grayscale images based on the weights of each target grayscale image, add the target scores of the same pixel categories, and use the pixel category with the highest target score as the target pixel category. If the pixel category corresponding to the any pixel point in the reference image is different from the target pixel category, then use the target pixel category as the new pixel category corresponding to the any pixel point; Update the reference image based on the new pixel category to obtain the target grayscale image; Set different colors for different pixel categories in the target grayscale image, and perform visual rendering on each pixel point in the target grayscale image based on different colors to obtain the target image after semantic segmentation.

[0011] In a second aspect, the present application provides a multi-modal based image semantic segmentation device, including: A data acquisition module, configured to acquire a target data set; the target data set includes a target image, label text information corresponding to each target object in the target image, and first label border information; A hint point generation module, configured to obtain a target semantic segmentation request sent by a client, determine a to-be-segmented image corresponding to the target semantic segmentation request from the target image, generate first positive sample hint points and first negative sample hint points for the to-be-segmented image by using HC saliency detection technology, and generate second positive sample hint points and second negative sample hint points for the to-be-segmented image by using SLIC superpixel segmentation technology; A first image determination module, configured to determine a first target grayscale image based on the first positive sample hint points and the first negative sample hint points through a preset image segmentation model, and determine a second target grayscale image based on the second positive sample hint points and the second negative sample hint points; A second image determination module, configured to determine second label bounding box information corresponding to the target semantic segmentation request based on the target image and the label text information through a preset object detection model, and determine a third target grayscale image based on the second label bounding box information by using the preset image segmentation model; A third image determination module, configured to determine a fourth target grayscale image corresponding to the target semantic segmentation request based on the first label bounding box information through the preset image segmentation model; A semantic segmentation module, configured to determine a final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained in different ways, and perform visual rendering of different colors on the objects in the target grayscale image to obtain a target image after semantic segmentation.

[0012] In a third aspect, the present application provides an electronic device, including: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the foregoing multi-modal based image semantic segmentation method.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, where the computer program, when executed by a processor, implements the foregoing multi-modal based image semantic segmentation method.

[0014] In this application, a target data set is obtained; the target data set includes target images, label text information corresponding to each target object in the target images, and first label bounding box information; a target semantic segmentation request sent by a client is obtained, a to-be-segmented image corresponding to the target semantic segmentation request is determined from the target images, first positive sample cue points and first negative sample cue points are generated for the to-be-segmented image by using the HC saliency detection technology, and second positive sample cue points and second negative sample cue points are generated for the to-be-segmented image by using the SLIC superpixel segmentation technology; a first target grayscale image is determined by a preset image segmentation model based on the first positive sample cue points and the first negative sample cue points, and a second target grayscale image is determined based on the second positive sample cue points and the second negative sample cue points; a second label bounding box information corresponding to the target semantic segmentation request is determined by a preset object detection model based on the target images and the label text information, and a third target grayscale image is determined by using the preset image segmentation model based on the second label bounding box information; a fourth target grayscale image corresponding to the target semantic segmentation request is determined by the preset image segmentation model based on the first label bounding box information; a final target grayscale image is determined according to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained by different methods, and objects in the target grayscale image are visually rendered in different colors to obtain a target image after semantic segmentation. As can be seen from the above, this application uses the HC saliency detection technology and the SLIC superpixel segmentation technology to generate positive and negative sample cue points for the to-be-segmented image respectively. These cue points provide more target region information for the preset image segmentation model, helping the model to more accurately identify and segment the target; for complex images, the model can focus on key regions according to the cue points, reduce misjudgment, and improve the segmentation accuracy; on the other hand, using the HC saliency detection technology and the SLIC superpixel segmentation technology to automatically generate cue points reduces the workload of manually annotating cue points. In large-scale image annotation scenarios, a large amount of manpower and time are saved, the annotation efficiency is improved, and the labor cost is reduced. By combining the label text information, the first label bounding box information, and the second label bounding box information, multiple target grayscale images are obtained through the preset object detection model and the preset image segmentation model, and these grayscale images are fused to determine the final target grayscale image. Based on the comprehensive use of multi-modal information, the model can understand the image content from different perspectives, make up for the deficiencies of single information, and enhance the segmentation performance for data in specific fields that have not been trained. In this way, through a series of automated processing and multi-modal information fusion, the final semantic segmentation result is obtained, reducing the need for manual discrimination and correction in the annotation process, reducing subjectivity, avoiding the time-consuming and laborious problem of manual annotation, and realizing an efficient, accurate, and automated semantic segmentation method, which promotes the rapid development and application of semantic segmentation models. Brief Description of the Drawings

[0015] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on the provided drawings.

[0016] Figure 1 Flowchart of a multi-modal based image semantic segmentation method disclosed in the present application; Figure 2 Schematic diagram of a method for generating hint points based on HC saliency detection technology disclosed in the present application; Figure 3 Schematic diagram of a specific multi-modal based image semantic segmentation method disclosed in the present application; Figure 4 Schematic diagram for comparing the semantic segmentation results of a typical fire image disclosed in the present application; Figure 5 Schematic diagram of the structure of a multi-modal based image semantic segmentation device disclosed in the present application; Figure 6 Schematic diagram of the structure of an electronic device disclosed in the present application. Detailed implementation manners

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0018] Currently, some studies have developed interactive semi-automatic image segmentation annotation tools by integrating the SAM model. However, there are still some challenges in the current semantic segmentation annotation methods. On the one hand, the segmentation performance of existing semantic segmentation models for specific domain data that has not been trained remains to be improved, or there are problems of low accuracy and high annotation costs when dealing with complex images. On the other hand, some fast annotation tools still require a large amount of manual discrimination and correction during the annotation process, with problems such as strong subjectivity, time-consuming and laborious annotation, thus restricting the rapid development and application of semantic segmentation models. For this reason, the present application provides a multi-modal based image semantic segmentation method, which can improve the conversion efficiency from the target dataset to the semantic segmentation dataset, reduce the annotation cost, and achieve an efficient, accurate and automated semantic segmentation method.

[0019] See Figure 1As shown, an embodiment of the present application discloses a multi-modal based image semantic segmentation method, including: Step S11, obtain a target data set; the target data set includes a target image, label text information corresponding to each target object in the target image, and first label border information.

[0020] In this embodiment, first, a target data set needs to be obtained. The target data set includes, but is not limited to, a target image, label text information corresponding to each target object in the target image, and first label border information. Among them, the target image is the processing object of semantic segmentation, covering rich visual content, and these images provide the original data basis for subsequent analysis and processing. The label text information includes the semantic description of each target object in the target image. Taking a traffic scene image as an example, the label text information will clearly indicate that an object in the image is a "car", a "pedestrian", or a "traffic signal". The first label border information is used to define the position range of each target object in the target image. On the image, it can mark the approximate position of the target in the form of a rectangular border. For example, in an image containing multiple cars, the first label border information will frame the position of each car respectively. These border information can assist the model in quickly locating the position of the target object in the image, narrowing the search range of the model for the target, and improving the segmentation efficiency.

[0021] Step S12, obtain a target semantic segmentation request sent by the user side, determine a to-be-segmented image corresponding to the target semantic segmentation request from the target image, generate first positive sample hint points and first negative sample hint points for the to-be-segmented image by using the HC saliency detection technology, and generate second positive sample hint points and second negative sample hint points for the to-be-segmented image by using the SLIC superpixel segmentation technology.

[0022] In this embodiment, first, obtain a target semantic segmentation request sent by the user side, and screen out an image corresponding to the target semantic segmentation request from the target images in the already obtained target data set as the to-be-segmented image. Then, based on the label border coordinates in the first label border information in the target data set, determine a target local image corresponding to the target semantic segmentation request from the to-be-segmented image; among them, the label border coordinates can be expressed as the coordinates of the upper left corner and the lower right corner of a rectangular box.

[0023] Furthermore, use the HC saliency detection technology (a histogram-based contrast method; HC, i.e., Histogram-based Contrast) to determine the core target area in the target local image, generate multiple groups of first positive sample hint points inside the core target area, and generate multiple groups of first negative sample hint points outside the core target area; among them, different groups correspond to different numbers of first positive sample hint points or first negative sample hint points.

[0024] It is understandable that after obtaining the core target area in the target local image based on the HC saliency detection technology, 1, 4, 8, or 16 random points can be generated respectively within the core target area as the first positive sample cue points of the model, and 0, 2, 4, 8, or 16 random points can be generated outside the core target area as the first negative sample cue points of the model. So that the model can generate different target grayscale images based on different combinations of the first positive and negative sample cue points.

[0025] Correspondingly, similar to the above HC saliency detection technology, the SLIC superpixel segmentation technology (Simple Linear Iterative Clustering; SLIC) is used to determine the core target area in the target local image, and multiple groups of second positive sample cue points are generated for the inside of the core target area, and multiple groups of second negative sample cue points are generated for the outside of the core target area.

[0026] Step S13: Determine a first target grayscale image based on the first positive sample cue points and the first negative sample cue points through a preset image segmentation model, and determine a second target grayscale image based on the second positive sample cue points and the second negative sample cue points.

[0027] In this embodiment, multiple groups of first positive sample cue points and multiple groups of first negative sample cue points can be input into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on different combinations of the first positive and negative sample cue points, obtains the first image semantic segmentation results corresponding to each combination of the first positive and negative sample cue points respectively, and takes the first image semantic segmentation result with the optimal model performance as the final first target image semantic segmentation result, and saves the first target image semantic segmentation result as the first target grayscale image. Among them, the grayscale image can more conveniently represent the semantic segmentation result, and the value of each pixel in the grayscale image can represent the category or segmentation area to which the pixel belongs, etc.

[0028] Correspondingly, multiple groups of second positive sample cue points and multiple groups of second negative sample cue points can be input into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on different combinations of the second positive and negative sample cue points, obtains the second image semantic segmentation results corresponding to each combination of the second positive and negative sample cue points respectively, and takes the second image semantic segmentation result with the optimal model performance as the final second target image semantic segmentation result, and saves the second target image semantic segmentation result as the second target grayscale image.

[0029] It can be understood that the positive and negative sample prompting points generated based on the HC saliency detection technology and the SLIC superpixel segmentation technology can be respectively input into the SAM model to generate a semantic segmentation result. Among them, each group of positive and negative sample prompting points corresponds to a semantic segmentation result, and the semantic segmentation result with the optimal model performance is used as the final target semantic segmentation result. For example, when using the HC saliency detection technology, the model performance is optimal when 8 positive sample points and 0 negative sample points are used as the combined prompting points. Therefore, the semantic segmentation result of the model obtained by combining the positive and negative sample points [8, 0] is the target semantic segmentation result, and the target semantic segmentation result can be saved as a grayscale image for subsequent use, such as Figure 2 shown

[0030] In this way, by respectively using the prompting points generated based on different technologies to perform semantic segmentation on the image and selecting the optimal result, the advantages of different technologies can be fully utilized to improve the accuracy and robustness of image semantic segmentation.

[0031] Step S14: Determine the second label bounding box information corresponding to the target semantic segmentation request based on the target image and the label text information through a preset object detection model, and use the preset image segmentation model to determine a third target grayscale image based on the second label bounding box information.

[0032] In this embodiment, first, the target image and label text information in the target dataset can be input into the preset object detection model, so that the preset object detection model determines the target region corresponding to the target semantic segmentation request from the target image and obtains the second label bounding box information corresponding to each target region; then, the second label bounding box information is input into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on the second label bounding box information, obtains the third target image semantic segmentation result, and saves the third target image semantic segmentation result as a third target grayscale image.

[0033] It should be noted that since the SAM model does not support the semantic segmentation ability for text prompts, the target image and label text information in the target dataset can be first input into the Grounding DINO model (Grounding Language-Image Pre-training with DINO, an innovative vision-language model) so that the Grounding DINO model determines the target region corresponding to the target semantic segmentation request from the target image and obtains the second label bounding box information corresponding to each target region. Then, the second label bounding box information can be input into the SAM model to obtain the semantic segmentation result based on text prompts, and the obtained semantic segmentation result can be directly saved as a grayscale image for subsequent use. This method can be abbreviated as the GSAM method.

[0034] Step S15: Based on the first label bounding box information, determine the fourth target grayscale image corresponding to the target semantic segmentation request through the preset image segmentation model.

[0035] In this embodiment, first, determine the first label bounding box information corresponding to the target semantic segmentation request from the target dataset, then perform semantic segmentation on the image based on the label bounding box coordinates in the first label bounding box information to obtain the semantic segmentation result of the fourth target image, and save the semantic segmentation result of the fourth target image as the fourth target grayscale image.

[0036] For example, after obtaining the first label bounding box information, the upper left corner coordinates and the lower right corner coordinates of the label box in the first label bounding box information can be input into the SAM model to obtain the semantic segmentation result corresponding to the target semantic segmentation request, and the semantic segmentation result can be saved as a grayscale image for subsequent use.

[0037] Step S16: Determine the final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained in different ways, and perform visual rendering of different colors on the objects in the target grayscale image to obtain the target image after semantic segmentation.

[0038] In this embodiment, determining the final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained in different ways can make full use of multi-modal information, make up for the deficiencies of a single method, and obtain a more accurate and complete segmentation result.

[0039] Specifically, first, the pixel categories corresponding to each pixel point in the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image can be determined; then, according to the determination methods based on each target grayscale image, corresponding weights are assigned to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image respectively, and the fourth target grayscale image is used as the reference image; for any pixel point in the reference image, the pixel category corresponding to the any pixel point in the fourth target grayscale image is determined, and the pixel categories of the pixel points corresponding to the any pixel point in the first target grayscale image, the second target grayscale image, and the third target grayscale image are determined, and the target scores of the pixel categories of the pixel points corresponding to the any pixel point in different target grayscale images are determined based on the weights of each target grayscale image. The target scores of the same pixel category are added together, and the pixel category with the highest target score is used as the target pixel category. If the pixel category corresponding to any pixel point in the reference image is different from the target pixel category, then the target pixel category is used as the new pixel category corresponding to the any pixel point; further, the reference image is updated based on the new pixel category to obtain the target grayscale image; different colors are set for different pixel categories in the target grayscale image, and each pixel point in the target grayscale image is visually rendered based on different colors to obtain the target image after semantic segmentation.

[0040] It should be noted that the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image are segmentation results obtained by different methods. Each pixel point in each image corresponds to a specific category, such as "car", "pedestrian", "building", etc. The pixel category of each pixel point can be determined according to the grayscale value of the pixel point and the pre-defined grayscale value-category mapping relationship. Since these four target grayscale images are obtained by different methods, their reliability and importance may be different. Therefore, corresponding weights can be assigned to each target grayscale image according to the determination methods of different target grayscale images. Determining the final pixel category according to the weights of different grayscale images makes the method have a certain adaptability. If a certain grayscale image has a more reliable segmentation result in some areas or for some target objects, a higher weight can be assigned to it, so that the segmentation result can be flexibly adjusted according to the specific situation, and the segmentation quality can be improved.

[0041] By fusing multiple grayscale images, the advantages of multiple grayscale images can be combined, and the target object can be more accurately segmented in various situations. At the same time, using one of the target grayscale images as the reference image provides a stable basis for the entire fusion process. In the fusion process, it is not necessary to perform complex equal comparisons and syntheses on the four grayscale images. Instead, with the reference image as a reference, the pixel categories in the reference image are updated and adjusted, which can reduce the computational complexity and improve the processing efficiency.

[0042] For example Figure 3 As shown, first, obtain a target dataset including a target image, label text information corresponding to each target object in the target image, and first label bounding box information, and a target semantic segmentation request sent by the client. In the first method, determine a to-be-segmented image corresponding to the target semantic segmentation request from the target image, use the HC saliency detection technology to generate first positive sample cue points and first negative sample cue points for the to-be-segmented image, input the first positive sample cue points and the first negative sample cue points into the SAM model to obtain a semantic segmentation result based on point cues, and save the semantic segmentation result as a first target grayscale image. In the second method, use the SLIC superpixel segmentation technology to generate second positive sample cue points and second negative sample cue points for the to-be-segmented image, input the second positive sample cue points and the second negative sample cue points into the SAM model to obtain a semantic segmentation result based on point cues, and save the semantic segmentation result as a second target grayscale image. In the third method, first input the target image and label text information in the target dataset into the Grounding DINO model, so that the Grounding DINO model determines a target region corresponding to the target semantic segmentation request from the target image, and obtains second label bounding box information corresponding to each target region. Then input the second label bounding box information into the SAM model to obtain a semantic segmentation result based on box cues, and save the semantic segmentation result as a third target grayscale image. In the fourth method, determine first label bounding box information corresponding to the target semantic segmentation request from the target dataset, input the upper left coordinate and the lower right coordinate of the label box in the first label bounding box information into the SAM model to obtain a semantic segmentation result corresponding to the target semantic segmentation request, and save the semantic segmentation result as a fourth target grayscale image. Finally, determine the final target grayscale image according to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained by different methods, and perform visual rendering of different colors on the objects in the target grayscale image to obtain the target image after semantic segmentation.

[0043] As can be seen from the above, in this embodiment, the HC saliency detection technology and the SLIC superpixel segmentation technology are used to generate positive and negative sample cue points for the image to be segmented respectively. These cue points provide more target region information for the preset image segmentation model, helping the model to more accurately identify and segment the target. For complex images, the model can focus on the key regions according to the cue points, reduce misjudgment, and improve the segmentation accuracy. On the other hand, the use of the HC saliency detection technology and the SLIC superpixel segmentation technology to automatically generate cue points reduces the workload of manually annotating cue points. In large-scale image annotation scenarios, it saves a large amount of manpower and time, improves the annotation efficiency, and reduces the labor cost. By combining the label text information, the first label border information, and the second label border information, multiple target grayscale images are obtained through the preset object detection model and the preset image segmentation model. By fusing these grayscale images, the final target grayscale image is determined. Based on the comprehensive use of multi-modal information, the model can understand the image content from different perspectives, make up for the deficiencies of single information, enhance the segmentation performance for data in specific fields that have not been trained, and at the same time, by fusing multiple grayscale images, the advantages of multiple grayscale images can be combined to more accurately segment the target object in various situations. In this way, through a series of automated processing and multi-modal information fusion, the final semantic segmentation result is obtained, reducing the need for manual discrimination and correction in the annotation process, reducing subjectivity, avoiding the time-consuming and laborious problem of manual annotation, realizing an efficient, accurate, and automated semantic segmentation method, significantly improving the generation speed of semantic segmentation data, and promoting the rapid development and application of semantic segmentation models.

[0044] See Figure 4 As shown below, taking the comparison chart of the semantic segmentation results of a typical fire image as an example, the technical solution in this application will be described.

[0045] Based on the classic flame and smoke target detection dataset FASDD, in the present invention, typical images in three sub-datasets, namely FASDD_RS, FASDD_UAV, and FASDD_CV, are used to conduct a comparative evaluation of the method performance, and the semantic segmentation effect is as Figure 4 shown. Figure 4 (a) compares the semantic segmentation results of different methods on satellite images. The GSAM method wrongly predicts the smoke target as a flame target and identifies the background area as smoke. The BOX method wrongly predicts some background pixels within the yellow circle area as smoke. In contrast, the HC saliency detection and the SLIC superpixel segmentation can better notice this detail. Figure 4 (b) and Figure 4(c) Compares the semantic segmentation results of different methods on UAV (Unmanned Aerial Vehicle) images. As can be seen from the circled areas, the GSAM method and the SLIC method both show a relatively high degree of false detection and missed detection on these two images. The HC method is slightly better than these two methods, while the method of the present invention obtains semantic segmentation results very close to the ground truth labels. Figure 4 (d) and Figure 4 (e) Compares the semantic segmentation results of different methods on ground sensor images. In Figure 4 (d), the BOX method shows an obvious omission in the yellow circled area, while the method of the present invention combines the correct results of the GSAM, SLIC, and HC methods in this area, effectively compensating for the omission of the BOX method for the flame object. In Figure 4 (e), due to the relatively high weight of the BOX method, the result obtained by the method of the present invention ignores the incorrect segmentation of other methods in the yellow circled area and obtains a result consistent with the ground truth label. In this way, by trusting the BOX method with better performance more, the good inference results of the BOX method are retained. In the areas where the BOX method performs poorly, the incorrectness of the BOX method is corrected by referring to the inference results of other methods, especially the HC method based on saliency detection technology, and finally a semantic segmentation result with better performance is achieved.

[0046] See Figure 5 As shown, an image semantic segmentation device based on multi-modal is also disclosed in an embodiment of the present application, including: A data acquisition module 11, configured to acquire a target data set; the target data set includes a target image, label text information corresponding to each target object in the target image, and first label border information; A hint point generation module 12, configured to acquire a target semantic segmentation request sent by a user terminal, determine a to-be-segmented image corresponding to the target semantic segmentation request from the target image, generate first positive sample hint points and first negative sample hint points for the to-be-segmented image by using HC saliency detection technology, and generate second positive sample hint points and second negative sample hint points for the to-be-segmented image by using SLIC superpixel segmentation technology; A first image determination module 13, configured to determine a first target grayscale image based on the first positive sample hint points and the first negative sample hint points through a preset image segmentation model, and determine a second target grayscale image based on the second positive sample hint points and the second negative sample hint points; The second image determination module 14 is configured to determine the second label bounding box information corresponding to the target semantic segmentation request based on the target image and the label text information through a preset object detection model, and determine a third target grayscale image based on the second label bounding box information by using the preset image segmentation model; The third image determination module 15 is configured to determine a fourth target grayscale image corresponding to the target semantic segmentation request based on the first label bounding box information through the preset image segmentation model; The semantic segmentation module 16 is configured to determine a final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained in different ways, and perform visual rendering of different colors on the objects in the target grayscale image to obtain a target image after semantic segmentation.

[0047] As can be seen from the above, the present application uses the HC saliency detection technology and the SLIC superpixel segmentation technology to generate positive and negative sample hint points for the image to be segmented respectively. These hint points provide more target region information for the preset image segmentation model, helping the model to more accurately identify and segment the target; for complex images, the model can focus on the key regions according to the hint points, reduce misjudgment, and improve the segmentation accuracy; on the other hand, using the HC saliency detection technology and the SLIC superpixel segmentation technology to automatically generate hint points reduces the workload of manually annotating hint points. In large-scale image annotation scenarios, it saves a large amount of manpower and time, improves the annotation efficiency, and reduces the labor cost. Combining the label text information, the first label bounding box information, and the second label bounding box information, multiple target grayscale images are obtained through the preset object detection model and the preset image segmentation model. These grayscale images are fused to determine the final target grayscale image. Based on the comprehensive use of multi-modal information, the model can understand the image content from different angles, make up for the deficiencies of single information, and enhance the segmentation performance of data in specific fields that have not been trained. In this way, through a series of automated processing and multi-modal information fusion, the final semantic segmentation result is obtained, reducing the need for manual discrimination and correction in the annotation process, reducing subjectivity, avoiding the time-consuming and laborious problem of manual annotation, realizing an efficient, accurate, and automated semantic segmentation method, and promoting the rapid development and application of semantic segmentation models.

[0048] In some specific embodiments, the hint point generation module 12 further includes: An image determination unit, configured to determine a target local image corresponding to the target semantic segmentation request from the image to be segmented based on the label bounding box coordinates in the first label bounding box information.

[0049] In some specific embodiments, the hint point generation module 12 includes: The first hint point generation unit is used to determine the core target area in the target local image by using the HC saliency detection technology, generate multiple groups of first positive sample hint points inside the core target area, and generate multiple groups of first negative sample hint points outside the core target area; wherein, different groups correspond to different numbers of first positive sample hint points or first negative sample hint points. The second hint point generation unit is used to determine the core target area in the target local image by using the SLIC superpixel segmentation technology, generate multiple groups of second positive sample hint points inside the core target area, and generate multiple groups of second negative sample hint points outside the core target area.

[0050] In some specific embodiments, the first image determination module 13 includes: The first grayscale image determination unit is used to input the multiple groups of first positive sample hint points and the multiple groups of first negative sample hint points into a preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on different combinations of first positive and negative sample hint points, obtains the first image semantic segmentation results corresponding to the different combinations of first positive and negative sample hint points respectively, and takes the first image semantic segmentation result with the optimal model performance as the final first target image semantic segmentation result, and saves the first target image semantic segmentation result as the first target grayscale image. The second grayscale image determination unit is used to input the multiple groups of second positive sample hint points and the multiple groups of second negative sample hint points into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on different combinations of second positive and negative sample hint points, obtains the second image semantic segmentation results corresponding to the different combinations of second positive and negative sample hint points respectively, and takes the second image semantic segmentation result with the optimal model performance as the final second target image semantic segmentation result, and saves the second target image semantic segmentation result as the second target grayscale image.

[0051] In some specific embodiments, the second image determination module 14 includes: The information determination unit is used to input the target image and the label text information into a preset object detection model, so that the preset object detection model determines the target area corresponding to the target semantic segmentation request from the target image, and obtains the second label bounding box information corresponding to each target area. The third grayscale image determination unit is used to input the second label bounding box information into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on the image based on the second label bounding box information, obtains the third target image semantic segmentation result, and saves the third target image semantic segmentation result as the third target grayscale image.

[0052] In some specific embodiments, the third image determination module 15 includes: A fourth grayscale image determination unit, configured to determine first label border information corresponding to the target semantic segmentation request from the target dataset, perform semantic segmentation on the image based on the label border coordinates in the first label border information to obtain a fourth target image semantic segmentation result, and save the fourth target image semantic segmentation result as a fourth target grayscale image.

[0053] In some specific embodiments, the semantic segmentation module 16 includes: A pixel category determination unit, configured to determine the pixel category corresponding to each pixel point in the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image; A reference image unit, configured to assign corresponding weights to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image respectively according to the determination method based on each target grayscale image, and use the fourth target grayscale image as the reference image; A pixel category update unit, configured to, for any pixel point in the reference image, determine the pixel category corresponding to the any pixel point in the fourth target grayscale image, determine the pixel categories of the pixel points corresponding to the any pixel point in the first target grayscale image, the second target grayscale image, and the third target grayscale image, and determine the target scores of the pixel categories of the pixel points corresponding to the any pixel point in different target grayscale images based on the weights of each target grayscale image, add the target scores of the same pixel categories, and use the pixel category with the highest target score as the target pixel category. If the pixel category corresponding to the any pixel point in the reference image is different from the target pixel category, then use the target pixel category as the new pixel category corresponding to the any pixel point; A target grayscale image determination unit, configured to update the reference image based on the new pixel category to obtain a target grayscale image; A semantic segmentation unit, configured to set different colors for different pixel categories in the target grayscale image, and perform visual rendering on each pixel point in the target grayscale image based on the different colors to obtain a target image after semantic segmentation.

[0054] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 6 which is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be considered as any limitation on the scope of use of the present application.

[0055] Figure 6Schematic diagram of the structure of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the multi-modal based image semantic segmentation method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0056] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and specific limitations are not imposed here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitations are made here.

[0057] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a disk, or an optical disc, etc., and the resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be short-term storage or permanent storage.

[0058] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the multi-modal based image semantic segmentation method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks.

[0059] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the multi-modal based image semantic segmentation method disclosed above is implemented. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

[0060] In the present specification, the various embodiments are described in a progressive manner, and the key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts may be referred to the description of the method part.

[0061] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered as exceeding the scope of this application.

[0062] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0063] Finally, it should also be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0064] The technical solutions provided in this application have been introduced in detail above. Specific examples have been used in this text to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A multi-modal based image semantic segmentation method, characterized in that, Including: Obtain a target data set; the target data set includes target images, label text information corresponding to each target object in the target images, and first label bounding box information; Obtain a target semantic segmentation request sent by a client, determine a to-be-segmented image corresponding to the target semantic segmentation request from the target image, generate first positive sample hint points and first negative sample hint points for the to-be-segmented image by using HC saliency detection technology, and generate second positive sample hint points and second negative sample hint points for the to-be-segmented image by using SLIC superpixel segmentation technology; Determine a first target grayscale image based on the first positive sample hint points and the first negative sample hint points through a preset image segmentation model, and determine a second target grayscale image based on the second positive sample hint points and the second negative sample hint points; Determine second label bounding box information corresponding to the target semantic segmentation request based on the target image and the label text information through a preset object detection model, and determine a third target grayscale image based on the second label bounding box information by using the preset image segmentation model; Determine a fourth target grayscale image corresponding to the target semantic segmentation request based on the first label bounding box information through the preset image segmentation model; Determine a final target grayscale image according to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained in different ways, and perform visual rendering of different colors on the objects in the target grayscale image to obtain a target image after semantic segmentation.

2. The multimodal-based image semantic segmentation method according to claim 1, wherein After determining the to-be-segmented image corresponding to the target semantic segmentation request from the target image, it further includes: Determine a target local image corresponding to the target semantic segmentation request from the to-be-segmented image based on the label bounding box coordinates in the first label bounding box information.

3. The multimodal-based image semantic segmentation method according to claim 2, wherein The step of generating first positive sample hint points and first negative sample hint points for the to-be-segmented image by using HC saliency detection technology includes: Use HC saliency detection technology to determine a core target area in the target local image, generate multiple groups of first positive sample hint points inside the core target area, and generate multiple groups of first negative sample hint points outside the core target area; wherein, different groups correspond to different numbers of first positive sample hint points or first negative sample hint points; Correspondingly, the step of generating second positive sample hint points and second negative sample hint points for the to-be-segmented image by using SLIC superpixel segmentation technology includes: Use SLIC superpixel segmentation technology to determine a core target area in the target local image, generate multiple groups of second positive sample hint points inside the core target area, and generate multiple groups of second negative sample hint points outside the core target area.

4. The multimodal-based image semantic segmentation method according to claim 3, wherein The step of determining a first target grayscale image based on the first positive sample hint points and the first negative sample hint points through a preset image segmentation model, and determining a second target grayscale image based on the second positive sample hint points and the second negative sample hint points includes: Input the multiple groups of first positive sample cue points and the multiple groups of first negative sample cue points into a preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on an image based on different combinations of the first positive and negative sample cue points, obtains first image semantic segmentation results corresponding to the respective combinations of the first positive and negative sample cue points, and takes the first image semantic segmentation result with the optimal model performance as the final first target image semantic segmentation result, and saves the first target image semantic segmentation result as a first target grayscale image; Input the multiple groups of second positive sample cue points and the multiple groups of second negative sample cue points into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on an image based on different combinations of the second positive and negative sample cue points, obtains second image semantic segmentation results corresponding to the respective combinations of the second positive and negative sample cue points, and takes the second image semantic segmentation result with the optimal model performance as the final second target image semantic segmentation result, and saves the second target image semantic segmentation result as a second target grayscale image.

5. The multimodal-based image semantic segmentation method according to claim 1, wherein The step of determining, by a preset object detection model, second label bounding box information corresponding to the target semantic segmentation request based on the target image and the label text information, and using the preset image segmentation model to determine a third target grayscale image based on the second label bounding box information includes: Input the target image and the label text information into a preset object detection model, so that the preset object detection model determines a target region corresponding to the target semantic segmentation request from the target image, and obtains second label bounding box information corresponding to the respective target regions; Input the second label bounding box information into the preset image segmentation model, so that the preset image segmentation model performs semantic segmentation on an image based on the second label bounding box information, obtains a third target image semantic segmentation result, and saves the third target image semantic segmentation result as a third target grayscale image.

6. The multimodal-based image semantic segmentation method according to claim 1, wherein The step of determining, by the preset image segmentation model, a fourth target grayscale image corresponding to the target semantic segmentation request based on the first label bounding box information includes: Determine first label bounding box information corresponding to the target semantic segmentation request from the target dataset, perform semantic segmentation on an image based on the label bounding box coordinates in the first label bounding box information, obtain a fourth target image semantic segmentation result, and save the fourth target image semantic segmentation result as a fourth target grayscale image.

7. The multi-modal based image semantic segmentation method according to any one of claims 1 to 6, characterized in that, The step of determining a final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained by different methods, and performing visual rendering of objects in the target grayscale image with different colors to obtain a target image after semantic segmentation includes: Determine the pixel categories corresponding to the respective pixel points in the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image; According to the determination method based on each target grayscale image, assign corresponding weights to the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image respectively, and use the fourth target grayscale image as the reference image; For any pixel point in the reference image, determine the pixel category corresponding to the any pixel point in the fourth target grayscale image, determine the pixel categories of the pixel points corresponding to the any pixel point in the first target grayscale image, the second target grayscale image, and the third target grayscale image, and determine the target scores of the pixel categories of the pixel points corresponding to the any pixel point in different target grayscale images based on the weights of the target grayscale images. Add up the target scores of the same pixel categories, and use the pixel category with the highest target score as the target pixel category. If the pixel category corresponding to the any pixel point in the reference image is different from the target pixel category, use the target pixel category as the new pixel category corresponding to the any pixel point; Update the reference image based on the new pixel category to obtain the target grayscale image; Set different colors for different pixel categories in the target grayscale image, and perform visual rendering on each pixel point in the target grayscale image based on different colors to obtain the target image after semantic segmentation.

8. An image semantic segmentation device based on multi-modalities, characterized in that, Including: A data acquisition module for acquiring a target data set; The target data set includes a target image, label text information corresponding to each target object in the target image, and first label border information; A hint point generation module for obtaining a target semantic segmentation request sent by a client, determining a to-be-segmented image corresponding to the target semantic segmentation request from the target image, generating a first positive sample hint point and a first negative sample hint point for the to-be-segmented image using the HC saliency detection technique, and generating a second positive sample hint point and a second negative sample hint point for the to-be-segmented image using the SLIC superpixel segmentation technique; A first image determination module for determining a first target grayscale image based on the first positive sample hint point and the first negative sample hint point through a preset image segmentation model, and determining a second target grayscale image based on the second positive sample hint point and the second negative sample hint point; A second image determination module for determining second label border information corresponding to the target semantic segmentation request based on the target image and the label text information through a preset object detection model, and determining a third target grayscale image based on the second label border information using the preset image segmentation model; A third image determination module for determining a fourth target grayscale image corresponding to the target semantic segmentation request based on the first label border information through the preset image segmentation model; A semantic segmentation module, configured to determine a final target grayscale image based on the first target grayscale image, the second target grayscale image, the third target grayscale image, and the fourth target grayscale image obtained in different ways, and perform visual rendering of objects in the target grayscale image with different colors to obtain a target image after semantic segmentation.

9. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor for executing the computer program to implement the multi-modal based image semantic segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program, which when executed by a processor implements the multi-modal based image semantic segmentation method according to any one of claims 1 to 7.

Images