Scene sketch semantic segmentation system, method and device based on open vocabularies
Through the scene sketch semantic segmentation system based on open vocabulary, the CLIP model is used for label-free training, which solves the problem of relying on pixel-level annotation in sketch semantic segmentation, and realizes high-precision semantic segmentation, and significantly improves the accuracy.
Patent Information
- Application Number
- CN202411976365.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
The prior art relies on pixel-level annotation in sketch semantic segmentation, which is difficult to generalize to new categories and sketches, and is time-consuming and requires a lot of labeling labor costs.
A scene sketch semantic segmentation system based on open vocabulary is adopted, and the generalization ability of the image big model CLIP is used to realize the unmarked semantic segmentation model training through text analysis and sketch image encoding, thereby capturing the information of the overall scene and local objects.
High-precision open vocabulary scene sketch semantic segmentation is achieved, with pixel-level segmentation accuracy reaching 85%, and the accuracy of segmentation directly using CLIP model is increased by nearly 57%, solving the problem of difficulty in generalization and time-consuming of traditional methods.
Smart Images
Figure CN119919657A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and in particular relates to a scene sketch semantic segmentation system, method and device based on open vocabulary. Background Art
[0002] Hand-drawn sketches are a common tool in industrial, interactive, architectural and other design fields. Scene sketches contain rich information such as scene layout and object form. With the development of touch screen electronic devices, hand-drawn sketches can be more easily obtained and participated in the next stage of drawing and modeling tasks, helping users to better realize their ideas and carry out innovative designs.
[0003] However, current open vocabulary segmentation methods focus on processing and segmenting photos, while there is little work on understanding sketches, especially semantic segmentation. Traditional sketch segmentation methods often rely on pixel-level annotation, which is not only labor-intensive and difficult to generalize to new categories and sketches, but also requires a lot of annotation labor costs. Therefore, there is an urgent need for an open vocabulary-based semantic scene sketch segmentation method that can generalize to new categories. Summary of the invention
[0004] The present invention aims to provide a method and device for semantic segmentation of scene sketches based on open vocabulary to solve the above-mentioned technical problems.
[0005] In order to solve the above technical problems, the specific technical solutions of the scene sketch semantic segmentation system, method and device based on open vocabulary of the present invention are as follows:
[0006] A scene sketch semantic segmentation system based on an open vocabulary comprises: a user terminal, and a model reasoning device and an image segmentation device located at a server;
[0007] The model inference device is used to perform real-time inference on the image generation model online;
[0008] The user terminal exchanges information with the image segmentation device at the server end through the network;
[0009] The image segmentation device provides a service interface to the user terminal via a network, and the user terminal displays the service interface to the user. The user draws a sketch and inputs text via the service interface, and the user terminal sends the sketch drawn by the user and the input text to the image segmentation device on the server side via the network; the image segmentation device uses an image segmentation model to perform semantic segmentation on the sketch according to the input hand-drawn sketch and description text, and sends the segmented result image to the user terminal via the network for display by the user terminal.
[0010] Furthermore, the image segmentation model includes a text parsing module, a scene image parsing module and a category image parsing module. The text parsing module obtains a sketch category text prompt through text parsing; encodes the scene text description and the category text; the scene image parsing module encodes the overall scene sketch through a sketch image encoder to capture global scene information; the category image parsing module encodes the category sketch through a sketch image encoder to capture local object information.
[0011] The present invention also discloses a scene sketch semantic segmentation method based on an open vocabulary scene sketch semantic segmentation system, comprising the following steps:
[0012] Step 1: Obtain a hand-drawn sketch image and scene text description;
[0013] Step 2: Obtain the sketch category text prompt through text parsing; encode the scene text description and category text;
[0014] Step 3: Encode the overall scene sketch through the sketch image encoder to capture the global scene information; Step 4: Calculate the cross-modal similarity between the sketch image and the category text to obtain the category sketch segmentation map;
[0015] Step 5: Encode the category sketch through the sketch image encoder to capture local object information;
[0016] Step 6: Calculate the image-text cross loss of the global scene and local objects, perform training, and obtain the semantic segmentation model.
[0017] Furthermore, the hand-drawn sketch in step 1 is a user's description of the image to be generated using simple strokes, and the scene text description is a simple description of the scene represented by the scene sketch;
[0018] The image segmentation device at the server end provides a service interface to the user terminal, and the user terminal displays the service interface to the user. The user inputs sketches and text through the service interface. The service interface includes a component for inputting sketches. The user draws and inputs sketches in the canvas area where sketches are drawn on the service interface, or browses files to upload drawn sketches. The service interface includes a component for inputting text. Further, step 2 is performed using a text parsing module, and the scene prompt text is input into a large language model. The large language model first analyzes the scene description to obtain the object information contained in the text, and outputs a category text description. The scene text description and the category text description are respectively encoded by the pre-trained CLIP text encoder to obtain scene text encoding and category text encoding features.
[0019] Furthermore, the step 3 is performed using a scene image parsing module, and the sketch image features are encoded based on the SC-CLIP image encoder to obtain scene image block features and scene graph global features;
[0020] The SC-CLIP image encoder is fine-tuned based on the pre-trained CLIP image encoder. First, the input sketch is segmented into non-overlapping blocks, which are flattened and linearly projected into the feature space, and a label is assigned to each block, denoted as P k ; Then, an additional set of learnable markers is added, denoted as V s ; Finally, add a special marker to encode the global sketch feature, denoted as VST, and get the marker X of the input visual encoder SC-VIT, which is expressed as follows:
[0021] X=[VST,P 1 ,…,P k ,V 1 ,…,V s ]
[0022] The visual encoder SC-VIT is modified based on the VIT model, and the qk attention in the VIT model is changed to qq attention, denoted as Attn qq At the same time, the residual connection is removed and the output of the last self-attention layer is directly used as the final image encoding output X visual , the calculation formula is as follows:
[0023] X visual =X attn =Proj(Att qq ·v).
[0024] Furthermore, step 4 includes the following steps:
[0025] First, calculate the cosine similarity between the category text encoding obtained in step 2 and the image encoding block obtained in step 3 to obtain the category similarity probability matrix of the image block; then, through interpolation mapping, map the similarity matrix size to the image size and multiply it with the sketch to obtain the similarity of the image pixels; finally, binarize the pixels according to the similarity to obtain a single sketch category segmentation map.
[0026] Furthermore, the step 5 is performed using the category image parsing module, and the category sketch image obtained by segmentation is encoded based on the SC-CLIP image encoder to obtain the category sketch global feature [VCT 1 ,…,VCT N ], where N is the number of object categories contained in the scene.
[0027] Furthermore, the cross loss in step 6 By global scene sketch-text triplet loss and local object sketch-text triplet loss The calculation formula is as follows:
[0028]
[0029] Furthermore, all weights except the linear layer are frozen during training, and the linear layer weights and the learnable markers introduced in the image encoder are updated by updating the global and object losses [V 1 ,…,V s ]. The present invention also discloses a scene sketch semantic segmentation device based on open vocabulary, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the steps of the scene sketch semantic segmentation method described in any one of claims 3-9.
[0030] The scene sketch semantic segmentation system, method and device based on open vocabulary of the present invention have the following advantages:
[0031] The method and device for semantic segmentation of scene sketches based on open vocabulary of the present invention utilize the generalization ability of the large image model CLIP, fine-tune on the sketch dataset, capture and extract information from the scene sketch at the overall scene and local object level, and implement unlabeled semantic segmentation model training through image-text cross-modal alignment. This method solves the problems of difficult generalization and long time consumption of traditional supervised learning methods that rely on pixel-level annotations. It achieves high-precision semantic segmentation of scene sketches with open vocabulary, with a pixel-level segmentation accuracy of 85%, while the accuracy of segmentation using the CLIP model directly is 28.8%, an improvement of nearly 57%. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a system architecture diagram applicable to the embodiment of the present invention;
[0033] Figure 2 is a flow chart of a scene sketch semantic segmentation method based on open vocabulary provided by an embodiment of the present invention;
[0034] Figure 3 It is a schematic diagram of the principle of the model architecture of an embodiment of the present invention;
[0035] Figure 4 Schematic diagram of the structure of the device in the embodiment of the present invention. DETAILED DESCRIPTION
[0036] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a scene sketch semantic segmentation system, method and device based on open vocabulary in conjunction with the accompanying drawings.
[0037] Figure 1 An exemplary system architecture to which the present invention can be applied is shown. Figure 1 As shown in , the system architecture may include: a user terminal, and a model reasoning device and an image segmentation device located at the server end.
[0038] Among them, the model inference device on the server side can adopt the method provided by the embodiment of the present invention to perform real-time inference on the image generation model online.
[0039] The user terminal can exchange information with the image segmentation device at the server end through the network.
[0040] As one of the achievable methods, the image segmentation device at the server end can provide a service interface to the user terminal through the network, and the user terminal displays the service interface to the user. The user can draw a sketch and input text through the service interface. The user terminal sends the sketch drawn by the user and the input text to the image segmentation device at the server end through the network. The image segmentation device can use the image segmentation model to perform semantic segmentation on the sketch based on the input hand-drawn sketch and description text, and send the segmented result image to the user terminal through the network for display by the user terminal.
[0041] Figure 1 The number of user terminals, image segmentation devices, and image segmentation models in the embodiment is only for illustration. Any number of user terminals, image segmentation devices, and image segmentation models may be provided according to implementation requirements.
[0042] like Figure 3 As shown in the figure, the image segmentation model mainly includes a text parsing module, a scene image parsing module and a category image parsing module. The text parsing module obtains the sketch category text prompt through text parsing; encodes the scene text description and category text; the scene image parsing module encodes the overall scene sketch through the sketch image encoder to capture the global scene information; the category image parsing module encodes the category sketch through the sketch image encoder to capture the local object information.
[0043] like Figure 2 As shown, a scene sketch semantic segmentation method based on an open vocabulary of the present invention comprises the following steps:
[0044] Step 1: Obtain a hand-drawn sketch image and scene text description;
[0045] A hand-drawn sketch is a user's description of the image he wants to generate using simple strokes, and a scene text description is a simple description of the scene represented by the scene sketch.
[0046] As one of the feasible ways, the image segmentation device at the server end can provide a service interface to the user terminal, and the user terminal displays the service interface to the user. The user can input sketches and text through the service interface. The service interface includes a component for inputting sketches. For example, the service interface can be in the form of a canvas, or in the form of a specific page. The user draws and inputs the sketch in the canvas area for sketching on the service interface, or browses the file to upload the drawn sketch. The service interface includes a component for inputting text. For example, the service interface can be in the form of a text box, or in the form of a specific page.
[0047] Step 2: Obtain the sketch category text prompt through text parsing; encode the scene text description and category text;
[0048] This step 2 is performed using a text parsing module.
[0049] like Figure 3 As shown, the scene prompt text is input into the large language model, and the large language model first analyzes the scene description, obtains the object information contained in the text, and outputs the category text description.
[0050] The scene text description and category text description are encoded separately through the pre-trained CLIP text encoder to obtain the scene text encoding and category text encoding features.
[0051] Step 3: Encode the overall scene sketch through the sketch image encoder to capture the global scene information;
[0052] like Figure 3 As shown, step 3 is performed using the scene image parsing module. The sketch image features are encoded based on the SC-CLIP image encoder to obtain scene image block features and scene graph global features.
[0053] The SC-CLIP image encoder is fine-tuned based on the pre-trained CLIP image encoder. First, the input sketch is segmented into non-overlapping blocks, which are flattened and linearly projected into the feature space, and each block is assigned a label, denoted by P. k ; Then, an additional set of learnable markers is added, denoted as V s ; Finally, add a special marker to encode the global sketch feature, denoted as VST. The marker X of the input visual encoder SC-VIT is obtained, and the expression is as follows:
[0054] X=[VST,P 1 ,…,P k ,V 1 ,…,V s ]
[0055] The visual encoder SC-VIT is modified based on the VIT model, and the qk attention in the VIT model is changed to qq attention, denoted as Attn qq At the same time, the residual connection is removed and the output of the last self-attention layer is directly used as the final image encoding output X visual , the calculation formula is as follows:
[0056] X visual =X attn =Proj(Attn qq ·v).
[0057] Step 4: Calculate the cross-modal similarity between the sketch image and the category text to obtain the category sketch segmentation map;
[0058] First, the cosine similarity between the category text encoding obtained in step 2 and the image encoding block obtained in step 3 is calculated to obtain the category similarity probability matrix of the image block; then, the similarity matrix size is mapped to the image size through interpolation mapping, and multiplied with the sketch to obtain the similarity of the image pixels. Finally, the pixels are binarized according to the similarity to obtain a single sketch category segmentation map.
[0059] Step 5: Encode the category sketch through the sketch image encoder to capture local object information;
[0060] like Figure 3 As shown, step 5 is performed using the category image parsing module. Based on the SC-CLIP image encoder, the category sketch image obtained by segmentation is encoded to obtain the category sketch global feature [VCT 1 ,…,VCT N ]. Where N is the number of object categories contained in the scene.
[0061] Step 6: Calculate the image-text cross loss of the global scene and local objects, perform training, and obtain the semantic segmentation model.
[0062] like Figure 3 As shown, the cross loss By global scene sketch-text triplet loss and local object sketch-text triplet loss The calculation formula is as follows:
[0063]
[0064] Furthermore, in step 6, all weights except the linear layer are frozen during training, and the linear layer weights and the learnable markers introduced in the image encoder are updated by updating the global and object losses. 1 ,…,V s ].
[0065] Corresponding to the aforementioned embodiment of the scene sketch semantic segmentation method based on open vocabulary, the present invention also provides an embodiment of a scene sketch semantic segmentation method device based on open vocabulary.
[0066] See also Figure 4 The open vocabulary-based scene sketch semantic segmentation method and device provided in an embodiment of the present invention include a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the steps of the scene sketch semantic segmentation method in the above embodiment.
[0067] The embodiments of the scene sketch semantic segmentation method and device based on open vocabulary of the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory and running them. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability where the scene sketch semantic segmentation method and device based on open vocabulary of the present invention is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0068] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0069] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.
[0070] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A scene sketch semantic segmentation system based on open vocabulary, characterized by: include: A user terminal, and a model reasoning device and an image segmentation device located at the server end; The model inference device is used to perform real-time inference on the image generation model online; The user terminal exchanges information with the image segmentation device at the server end through the network; The image segmentation device provides a service interface to the user terminal via a network, and the user terminal displays the service interface to the user. The user draws a sketch and inputs text via the service interface, and the user terminal sends the sketch drawn by the user and the input text to the image segmentation device on the server side via the network; the image segmentation device uses an image segmentation model to perform semantic segmentation on the sketch according to the input hand-drawn sketch and description text, and sends the segmented result image to the user terminal via the network for display by the user terminal.
2. The open vocabulary based scene sketch semantic segmentation system according to claim 1, characterized in that: The image segmentation model includes a text parsing module, a scene image parsing module and a category image parsing module. The text parsing module obtains a sketch category text prompt through text parsing; encodes the scene text description and the category text; the scene image parsing module encodes the overall scene sketch through a sketch image encoder to capture global scene information; the category image parsing module encodes the category sketch through a sketch image encoder to capture local object information.
3. A scene sketch semantic segmentation method according to claim 1 or 2, characterized in that: The steps include: Step 1: Obtain a hand-drawn sketch image and scene text description; Step 2: Obtain the sketch category text prompt through text parsing; encode the scene text description and category text; Step 3: Encode the overall scene sketch through the sketch image encoder to capture the global scene information; Step 4: Calculate the cross-modal similarity between the sketch image and the category text to obtain the category sketch segmentation map; Step 5: Encode the category sketch through the sketch image encoder to capture local object information; Step 6: Calculate the image-text cross loss of the global scene and local objects, perform training, and obtain the semantic segmentation model.
4. The scene sketch semantic segmentation method according to claim 3, characterized in that: The hand-drawn sketch in step 1 is a user's description of the image to be generated using simple strokes, and the scene text description is a simple description of the scene represented by the scene sketch; The image segmentation device at the server end provides a service interface to the user terminal, and the user terminal displays the service interface to the user. The user inputs sketches and text through the service interface. The service interface includes a component for inputting sketches. The user draws the input sketch in the sketch drawing canvas area on the service interface, and can also browse the file to upload the drawn sketch. The service interface includes a component for inputting text.
5. The scene sketch semantic segmentation method according to claim 3, characterized in that: The step 2 is performed using a text parsing module, and the scene prompt text is input into a large language model. The large language model first analyzes the scene description to obtain object information contained in the text, and outputs a category text description. The scene text description and the category text description are respectively encoded using a pre-trained CLIP text encoder to obtain scene text encoding and category text encoding features.
6. The scene sketch semantic segmentation method according to claim 3, characterized in that: The step 3 is performed by using a scene image parsing module, encoding the sketch image features based on the SC-CLIP image encoder to obtain scene image block features and scene graph global features; The SC-CLIP image encoder is fine-tuned based on the pre-trained CLIP image encoder. First, the input sketch is segmented into non-overlapping blocks, which are flattened and linearly projected into the feature space, and a label is assigned to each block, denoted as P k ; Then, an additional set of learnable markers is added, denoted as V s ; Finally, add a special marker to encode the global sketch feature, denoted as VST, and get the marker X of the input visual encoder SC-VIT, which is expressed as follows: X=[VST,P1,…,P k ,V1,…,V s ] The visual encoder SC-VIT is modified based on the VIT model, and the qk attention in the VIT model is changed to qq attention, denoted as Attn qq At the same time, the residual connection is removed and the output of the last self-attention layer is directly used as the final image encoding output X visual , the calculation formula is as follows: X visual =X attn =Proj(Attn qq ·v)。 7. The scene sketch semantic segmentation method according to claim 3, characterized in that: The step 4 comprises the following steps: First, calculate the cosine similarity between the category text encoding obtained in step 2 and the image encoding block obtained in step 3 to obtain the category similarity probability matrix of the image block; then, through interpolation mapping, map the similarity matrix size to the image size and multiply it with the sketch to obtain the similarity of the image pixels; finally, binarize the pixels according to the similarity to obtain a single sketch category segmentation map.
8. The scene sketch semantic segmentation method according to claim 3, characterized in that: The step 5 is performed using the category image parsing module, and the category sketch image obtained by segmentation is encoded based on the SC-CLIP image encoder to obtain the category sketch global feature [VCT1,…,VCT N ], where N is the number of object categories contained in the scene.
9. The scene sketch semantic segmentation method according to claim 3, characterized in that: The cross loss of step 6 By global scene sketch-text triplet loss and local object sketch-text triplet loss The calculation formula is as follows: Furthermore, all weights except the linear layer are frozen during training, and the linear layer weights and the learnable labels [V1,…,V s ].
10. A scene sketch semantic segmentation device based on open vocabulary, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the steps of the scene sketch semantic segmentation method described in any one of claims 3 to 9.