Semantic annotation and retrieval method based on image segmentation

Abstract

The invention discloses a semantic annotation and retrieval method based on image segmentation. The specific process is as follows: formulating an image classification standard system; importing an image data set; using Dify to build a process type application of image segmentation and semantic annotation; selecting multiple pieces of data from the image data set to create a test set, calling an application API to label the test set, and obtaining a labeling result and a score; application parameters are adjusted, and the cases with low scores are optimized; calling an application API to carry out image recognition and semantic annotation operation on all the images, and recording annotation results and scores; an annotation result is rechecked in a manual intervention mode, and the annotation data quality is guaranteed; based on the rechecked data, converting the semantic tag of each region into a vector, associating the vector with the image ID and the labeling region ID, and constructing an index database; and executing semantic retrieval, matching a result from the index database, and returning the result to the user. According to the semantic annotation and retrieval method, the subsequent image retrieval efficiency can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image intelligent processing and information retrieval technology, and in particular to a semantic annotation and retrieval method based on image segmentation. BACKGROUND

[0002] With the popularity of terminals such as cameras, drones, and medical equipment, global image data is growing exponentially. Traditional manual retrieval or simple retrieval based on file names and tags cannot quickly locate target information from massive data. Image retrieval technology matches visual features (color, texture, semantic regions, etc.) to directly cross the limit of text description, realizing the transformation from data accumulation to information extraction, and making massive images from useless data to useful resources.

[0003] The semantic annotation and retrieval technology based on image segmentation is an important branch of computer vision, and its core is to obtain semantic information of an image through pixel-level image segmentation technology, and then realize image retrieval based on semantic content. The development of this technology is due to the "semantic gap" problem faced by traditional content-based image retrieval (CBIR) technology, which refers to the huge difference between low-level visual features and high-level semantic concepts.

[0004] In recent years, with the rapid development of deep learning technology, the performance of this technology has been improved, but there are still many problems in data, segmentation effect, and retrieval matching. For example: in terms of data, semantic annotation requires pixel-level fine annotation, which relies on professional personnel to mark the category pixel by pixel. For images containing complex scenes and numerous objects, the annotation process is time-consuming and labor-intensive, and the cost is extremely high; in terms of image segmentation, if an image contains multiple semantic information, some segmentation algorithms may not accurately capture the region of interest corresponding to each semantic; in terms of retrieval matching, when an image involves more than two semantic concepts, the retrieval method based on the organization mode of concept co-occurrence has significantly reduced efficiency. Although deep learning has narrowed the gap between low-dimensional features of images and high-level semantics, semantic matching still has defects. SUMMARY

[0005] To solve the above technical problems, the present application provides a semantic annotation and retrieval method based on image segmentation. This semantic annotation and retrieval method establishes a mapping relationship between image visual regions and semantic concepts, extracts deep features of images and annotated regions with semantic annotation, forms a structured semantic feature representation, stores the semantic features in a feature library, and constructs an index, which can effectively improve the efficiency of subsequent image retrieval.

[0006] A semantic annotation and retrieval method based on image segmentation, the specific process is as follows:

[0007] S1, a system of image classification standards is formulated, including core categories and subcategories obtained by multidimensional disassembly of the core categories;

[0008] S2, import the image dataset, extract file information, and classify the image data based on the classification system defined in step S1;

[0009] S3 utilizes Dify to build a workflow application for image segmentation and semantic annotation;

[0010] S4. Select multiple data points from the image dataset to create a test set. The test set data covers all categories under the image classification system defined in step S1. Use the test set data to test the application created in step S3 and generate annotation results and scores.

[0011] S5 calls the application API to perform image recognition and semantic annotation operations on all image data, and records the annotation results and scores;

[0012] S6. The annotation results generated in step S5 are reviewed by manual intervention to ensure the quality of the annotation data;

[0013] S7. Based on the data reviewed in step S6, the semantic labels of each region are converted into vectors, associated with the image ID and the labeled region ID, and an index library is constructed.

[0014] S8 performs semantic retrieval, matches results from the index, and returns them to the user.

[0015] As a preferred embodiment of the above technical solution, in step S1, the core categories include five subcategories: natural scenes, man-made objects, organisms and life, human activities, and professional fields.

[0016] As a preferred embodiment of the above technical solution, in step S3, the application process design includes five nodes: user input, image preprocessing, model invocation, automatic scoring, and result return. The specific process is as follows:

[0017] S31, receive the image to be recognized based on user input;

[0018] S32 performs preprocessing operations on the image, including denoising and contrast enhancement, so that the model can receive a standardized image.

[0019] S33 calls an open-source model to segment the processed image, parses the segmentation results, and generates labeled region location information and semantic label information;

[0020] S34 selects several automatically calculated metrics to meet the core quality requirements of image segmentation and semantic annotation.

[0021] As a preferred embodiment of the above technical solution, in step S34, the selected calculation indicators include semantic accuracy, region integrity, label consistency, and boundary precision.

[0022] As a preferred embodiment of the above technical solution, in step S4, the formula for calculating the comprehensive score is as follows:

[0023] SA×40% + CI×30% + LC×20% + BA×10%

[0024] Among them, SA stands for semantic accuracy, CI for region integrity, LC for label consistency, and BA for boundary precision.

[0025] As a preferred embodiment of the above technical solution, in step S6, the verification standard is:

[0026] A score of >80 indicates a light review and rapid release.

[0027] Rating 60-80 points, key corrections and optimizations to details;

[0028] If the score is less than 60, investigate the root cause. If it is an image quality issue, discard the image; otherwise, re-label it.

[0029] As a preferred embodiment of the above technical solution, in step S8, semantic retrieval supports two retrieval methods: text and image search.

[0030] As a preferred technical solution, the core of semantic retrieval is the similarity comparison between query features and index features;

[0031] During text retrieval, it is necessary to perform semantic parsing on the query text, extract key tags, and use the same encoding model as the annotation stage to transform the text tags into query feature vectors.

[0032] When performing image retrieval, it is necessary to perform automatic semantic segmentation on the query image, obtain the annotation results, and convert the extracted semantic labels into query feature vectors.

[0033] The beneficial effects of this invention are as follows:

[0034] 1. This invention proposes a relatively complete image classification system that can basically cover all types of sample data.

[0035] 2. This invention proposes a scoring mechanism for machine annotation results. Through objective indicators, it solves the core problems of lack of standards for annotation effectiveness, difficulty in locating problems, and lack of direction for application iteration, enabling applications to be upgraded from simply running to being implemented and optimized.

[0036] 3. This invention proposes a review scheme for annotation results at different scoring levels. By adopting a differentiated review strategy, it reduces the amount of ineffective work and shortens the overall review cycle by allocating review resources according to the scoring level.

[0037] 4. This invention proposes a semantic image retrieval method that simultaneously supports text retrieval and image search, adapting to different user habits. Non-professional users can quickly search using text (e.g., "Korean folk dance mask"), while professional users can accurately match details using image search (e.g., uploading a partial image of the mask pattern). Text clearly defines the category / topic, while images clearly define visual features (e.g., texture, color, composition). Combining these methods enhances retrieval flexibility. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention.

[0039] Figure 2 Image classification system of this invention Figure 1 .

[0040] Figure 3 Image classification system of this invention Figure 2 .

[0041] Figure 4 This is a schematic diagram of the annotation result scoring mechanism in an embodiment of the present invention.

[0042] Figure 5 This is a schematic diagram of the data structure returned by the annotation results in an embodiment of the present invention.

[0043] Figure 6 This is a flowchart of the image semantic annotation process according to an embodiment of the present invention.

[0044] Figure 7 This is a flowchart of the image semantic retrieval process according to an embodiment of the present invention. Detailed Implementation

[0045] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0046] like Figure 1 The semantic annotation and retrieval method based on image segmentation is shown below, and the specific process is as follows:

[0047] S1, Establish an image classification standard system, including core categories and subcategories that are broken down into multiple dimensions based on the core categories;

[0048] S2, import the image dataset, extract file information, and classify the image data based on the classification system defined in step S1;

[0049] S3 utilizes Dify to build a workflow application for image segmentation and semantic annotation;

[0050] S4. Select multiple data points from the image dataset to create a test set. The test set data covers all categories under the image classification system defined in step S1. Use the test set data to test the application created in step S3 and generate annotation results and scores.

[0051] S5 calls the application API to perform image recognition and semantic annotation operations on all image data, and records the annotation results and scores;

[0052] S6. The annotation results generated in step S5 are reviewed by manual intervention to ensure the quality of the annotation data;

[0053] S7. Based on the data reviewed in step S6, the semantic labels of each region are converted into vectors, associated with the image ID and the labeled region ID, and an index library is constructed.

[0054] S8 performs semantic retrieval, matches results from the index, and returns them to the user.

[0055] Specifically:

[0056] S1: Develop a standardized image classification system, starting with five core categories: natural scenes, man-made objects, organisms and life, human activities, and professional fields, and further subdividing them sequentially. Each core category must cover all samples without omission; subcategories should be broken down according to core dimensions to ensure no overlap. The image classification structure is as follows: Figure 2 , Figure 3 As shown.

[0057] S2: Import the image dataset, extract basic image information such as file name, file format (JPG / PNG / TIFF, etc.), file size, and resolution, and perform preliminary classification of the image data based on the image classification system defined in step S1.

[0058] S3: Utilize Dify to build a workflow application for image segmentation and semantic annotation. The application workflow design includes five nodes: user input, image preprocessing, model invocation, automatic scoring, and result return.

[0059] S31: Receive the image to be recognized based on user input;

[0060] S32: Perform preprocessing operations such as denoising and contrast enhancement on the image to allow the model to receive a standardized image;

[0061] S33: Call the open-source model to segment the processed image, parse the segmentation results, and generate labeled region location information (coordinates of the top left and bottom left corners) and semantic label information (type, color, shape, and state).

[0062] S34: Four automatically calculated metrics are selected to address the core quality requirements of image segmentation and semantic annotation:

[0063] Semantic accuracy (SA), region integrity (CI), label consistency (LC), and boundary precision (BA) are used to calculate the overall score of the current annotation result: Overall score = SA × 40% + CI × 30% + LC × 20% + BA × 10%. The definition and calculation of the scoring mechanism are as follows: Figure 4 As shown;

[0064] S35: Organize the annotation results and scoring information into structured data and return it to the user. The returned data format is as follows: Figure 5 As shown.

[0065] S4: Select several data points from the image dataset to create a test set. Input the test set data and test the application created in step 3 to generate annotation results and scores. Perform overall statistical analysis of the quality of the generated annotation results based on the scores, track the causes of low scores, and adjust the application parameters to ensure that the application achieves good recognition and annotation results.

[0066] The selected data must cover all categories defined in step S1; the number of samples for each category must be no less than 100.

[0067] Call the API to perform image segmentation and annotation operations on sample data in batches, and record the annotation results and scores. A score >80 is recorded as "good", a score between 60 and 80 is recorded as "average", and a score <60 is recorded as "unsatisfactory".

[0068] According to the statistical annotation results of the scoring level, if the proportion of data of each type is higher than 80% and the proportion of unqualified data is lower than 5%, then the overall identification and annotation effect is considered to have met expectations, and step S5 can be continued.

[0069] Otherwise, it is necessary to filter low-scoring cases, compare the original image with the annotation results, and analyze the reasons for the low scores. Check for issues such as complete mismatch between annotations and targets, blurred annotation areas, and incorrect label names. Then, adjust the application parameters and re-test the application.

[0070] S5: Call the API to perform image recognition and semantic annotation operations on all image data, and record the annotation results and scores.

[0071] S6: The annotation results generated in step S5 are reviewed manually to ensure the quality of the annotated data. The core principle of manual intervention is to allocate manpower according to scoring priority, focusing on high-value corrections and low-trust reviews. The processing flow, tool selection, manpower allocation, and quality control of the three types of data each have their own emphasis, as detailed in the following solutions:

[0072] Score > 80 ("Good"): Light review and quick release (10%-20% manpower);

[0073] A score of 60-80 ("Average"): Focus on key corrections and optimization of details (60%-7% of manpower required);

[0074] Score < 60 (“Unacceptable”): Investigate the root cause. If it is an image quality issue, discard the image; otherwise, re-label it (10%-20% of the work will be done manually).

[0075] S7: Based on the data reviewed in step S6, convert the semantic labels of each region into vectors, associate them with the image ID and the labeled region ID, and build an index library; the image semantic annotation flowchart is as follows. Figure 6 As shown:

[0076] S8: Perform semantic retrieval, match results from the index, and return them to the user. It supports both "text search" and "image search," with the core being the similarity comparison between query features and index features. Text search and image search can be used independently or in combination. The image semantic retrieval flowchart is shown below. Figure 7 As shown:

[0077] When a user enters text, semantic parsing is performed on the query text to extract key tags; using the same encoding model as the annotation stage, the text tags are transformed into query feature vectors. When a user uploads an image, automatic semantic segmentation is first performed on the query image to obtain the annotation results, and the extracted semantic tags are transformed into query feature vectors.

[0078] Algorithms such as cosine similarity and Euclidean distance are used to compare the query feature vector with the feature vectors of all images in the index. The higher the similarity, the more consistent the semantics. The query results are then sorted from high to low similarity.

[0079] The system combines regional overlap (the proportion of overlap between the segmented regions of the candidate image and the query image) and label completeness (whether the candidate image contains all the semantic labels of the query image) for secondary sorting, returns a list of matching images, and labels them with similarity scores.

[0080] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantic annotation and retrieval method based on image segmentation, characterized in that: The specific process is as follows: S1, Establish an image classification standard system, including core categories and subcategories that are broken down into multiple dimensions based on the core categories; S2, import the image dataset, extract file information, and classify the image data based on the classification system defined in step S1; S3 utilizes Dify to build a workflow application for image segmentation and semantic annotation; S4. Select multiple data points from the image dataset to create a test set. The test set data covers all categories under the image classification system defined in step S1. Use the test set data to test the application created in step S3 and generate annotation results and scores. S5 calls the application API to perform image recognition and semantic annotation operations on all image data, and records the annotation results and scores; S6. The annotation results generated in step S5 are reviewed by manual intervention to ensure the quality of the annotation data; S7. Based on the data reviewed in step S6, the semantic labels of each region are converted into vectors, associated with the image ID and the labeled region ID, and an index library is constructed. S8 performs semantic retrieval, matches results from the index, and returns them to the user.

2. The semantic annotation and retrieval method based on image segmentation according to claim 1, characterized in that: In step S1, the core categories include five subcategories: natural scenes, man-made objects, organisms and life, human activities, and professional fields.

3. The semantic annotation and retrieval method based on image segmentation according to claim 1, characterized in that: In step S3, the application process design includes five nodes: user input, image preprocessing, model invocation, automatic scoring, and result return. The specific process is as follows: S31, receive the image to be recognized based on user input; S32 performs preprocessing operations on the image, including denoising and contrast enhancement, so that the model can receive a standardized image. S33 calls an open-source model to segment the processed image, parses the segmentation results, and generates labeled region location information and semantic label information; S34 selects several automatically calculated metrics to meet the core quality requirements of image segmentation and semantic annotation.

4. The semantic annotation and retrieval method based on image segmentation according to claim 3, characterized in that: In step S34, the selected calculation metrics include semantic accuracy, region integrity, label consistency, and boundary precision.

5. The semantic annotation and retrieval method based on image segmentation according to claim 4, characterized in that: In step S4, the formula for calculating the overall score is: SA×40% + CI×30% + LC×20% + BA×10% Among them, SA stands for semantic accuracy, CI for region integrity, LC for label consistency, and BA for boundary precision.

6. The semantic annotation and retrieval method based on image segmentation according to claim 5, characterized in that: In step S6, the review criteria are as follows: A score of >80 indicates a light review and rapid release. Rating 60-80 points, key corrections and optimizations to details; If the score is less than 60, investigate the root cause. If it is an image quality issue, discard the image; otherwise, re-label it.

7. The semantic annotation and retrieval method based on image segmentation according to claim 1, characterized in that: In step S8, semantic retrieval supports two retrieval methods: text search and image search.

8. The semantic annotation and retrieval method based on image segmentation according to claim 7, characterized in that: The core of semantic retrieval is the similarity comparison between query features and index features; During text retrieval, it is necessary to perform semantic parsing on the query text, extract key tags, and use the same encoding model as the annotation stage to transform the text tags into query feature vectors. When performing image retrieval, it is necessary to perform automatic semantic segmentation on the query image, obtain the annotation results, and convert the extracted semantic labels into query feature vectors.

Images