User-customized intent understanding method and system for planar scanned images

By constructing a user-customized intent understanding method, training the model using base domain and target domain datasets, and combining template matching and box matching modules, the problem of insufficient datasets and limited resources in user intent understanding of planar scanned images is solved, achieving efficient user intent prediction under small sample conditions and improving system performance.

CN117746018BActive Publication Date: 2026-11-10ZHEJIANG UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311718695.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2026-11-10
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

In existing technologies, the task of understanding user-customized intent in planar scanned images faces problems such as sequential and incremental task occurrence, small number of user input samples, limited resources, and insufficient public datasets, making it difficult to accurately understand user intent.

Method used

A user-customized intent understanding method for planar scanned images is constructed. By acquiring base domain and target domain datasets, a base domain segmentation model and an object detection model are trained. Using template matching and bounding box matching modules, combined with a self-boosting module, the user intent is predicted.

Benefits of technology

It alleviates the problem of insufficient datasets, improves the generalization ability of the task domain, realizes the understanding of user-customized intent under small sample conditions, reduces the noise of prediction results, and ensures that the performance of old tasks does not degrade.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117746018B_ABST
    Figure CN117746018B_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for user-customized intent understanding of planar scanned images. The method includes: obtaining an incremental test set from a base domain dataset and a target domain dataset; training a base domain segmentation model on the base domain dataset and performing inference to obtain a basic pixel-level mask; performing inference using the target domain inference model to obtain a target pixel-level mask; and fusing and post-processing the basic pixel-level mask and the target pixel-level mask using a self-boosting module to obtain a final pixel-level mask. The system includes a dataset acquisition module, an incremental test set construction module, a basic pixel-level mask acquisition module, a target pixel-level mask acquisition module, and a final pixel-level mask acquisition module. This method can be used to predict the final pixel-level mask for planar scanned images, achieving user-customized intent understanding. This invention solves the small sample incremental problem with relatively low overhead and can better adapt to user-personalized intent understanding tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a user-customized intent understanding method and system for planar scanned images. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, various intelligent systems have emerged to assist human behavior. Intelligent systems typically possess a degree of autonomy; they can receive input information from the environment, process this information through algorithms and reasoning, and then make corresponding actions or decisions based on the processing results. Based on the characteristics of intelligent systems, they are often used for user intent understanding, with the design goal of accurately understanding the user while reducing the user's memory and behavioral load. Current conventional user intent understanding methods have achieved considerable performance, but customized intent understanding tasks for planar scanned images remain a significant challenge. Customized intent understanding tasks refer to tasks where the image contains multiple regions of interest that the user may have, meaning the image sample has multiple potential intents that require customized user input to complete.

[0003] The main challenges for customized intent understanding tasks of planar scanned images are as follows: (1) The tasks are sequential and incremental, and users have different intents in different tasks; (2) The number of customized samples input by users is small, and most users have limited patience for calibration samples; (3) Users are sensitive to overhead, and storage, computing and other resources are limited; (4) There are currently few publicly available relevant datasets. Summary of the Invention

[0004] The purpose of this invention is to solve the problems existing in the prior art and to provide a user-customized intent understanding method and system for planar scanned images.

[0005] In a first aspect, the present invention provides a user-customized intent understanding method for planar scanned images, the specific technical solution of which is as follows:

[0006] S1. Obtain the base domain dataset and the target domain dataset; the base domain dataset includes several planar scan images and several pixel-level masks corresponding to the planar scan images. The base domain dataset is divided into a base domain test set and a base domain training set; the target domain dataset is divided into N groups of incremental subsets. Each planar scan image in the target domain dataset corresponds to a labeling file. Each labeling file contains the labeling status, the height, width, and rotation angle of the image corresponding to the labeling file;

[0007] S2. In each incremental subset, the planar scan images are divided into unlabeled, partially labeled, and fully labeled categories. In each incremental subset, one planar scan image is selected as a template image from the fully labeled planar scan images. N template images are used to form a template set. The template images must satisfy the condition that the rotation angle in the corresponding annotation file is 0. In each incremental subset, the unlabeled, partially labeled, and template images are removed. The remaining planar scan images in each incremental subset, together with the base domain test set, constitute the incremental test set.

[0008] S3. Train a base domain segmentation model on the base domain training set, and use the trained base domain segmentation model to infer the incremental test set to obtain the basic pixel-level mask for each image in the incremental test set.

[0009] S4. Obtain a trained object detection model and a trained segmentation model, and construct an object domain inference model using the object detection model, the segmentation model, a template matching module, and a box matching module; use the object domain inference model to infer the incremental test set to obtain the target pixel-level mask for each image in the incremental test set;

[0010] S5. Input the basic pixel-level mask and the target pixel-level mask together into the self-boosting module to obtain the final pixel-level mask for each image in the incremental test set. The prediction of the user's customized intent is realized by the final pixel-level mask.

[0011] As a preferred embodiment of the first aspect mentioned above, when the annotation status of each annotation file is 1, it indicates that the image is annotated, and the bounding box coordinate information of the user intent region can be obtained. The user intent region is annotated using quadrilateral bounding boxes. The user intent region with the largest area is defined as the outer frame, and the coordinates of the vertices of the outer frame are used to form the outer frame coordinate set. The user intent region with the largest area is removed, and the remaining user intent regions are defined as the inner frames, and the coordinates of the vertices of the inner frames are used to form the inner frame coordinate set. When the annotation status of each annotation file is 0, it indicates that the image is not annotated, that is, the bounding box coordinate information of the user intent region cannot be obtained.

[0012] As a preferred embodiment of the first aspect mentioned above, the base domain segmentation model employs an image saliency detection network.

[0013] As a preferred embodiment of the first aspect above, the specific process in the target domain reasoning model is as follows:

[0014] S41. Using the template matching module, obtain the template corresponding to each image in the incremental test set and the coordinate information of the target region bounding box of each image;

[0015] S42. The target detection model is used to detect possible user intent regions in each image of the incremental test set. The coordinate information of multiple user intent region detection boxes is obtained for each image, and the coordinate set of user intent region detection boxes for each image is formed.

[0016] S43. The coordinate information of the detection box of each user intent region corresponding to each image and the coordinate information of the target region bounding box of each image are input into the box matching module for matching to obtain the prompt box of each image in the incremental test set.

[0017] S44. Using the segmentation model and the prompt box of each image in the incremental test set, segment each image in the incremental test set to obtain the target pixel-level mask of each image in the incremental test set.

[0018] As a preferred embodiment of the first aspect above, the specific process of the template matching module is as follows:

[0019] First, each template image is reduced to a preset fixed size. The SIFT feature descriptor of each reduced template image is calculated, and the N SIFT feature descriptors are saved to N text files. Next, each image in the incremental test set is reduced to the fixed size, and the SIFT feature descriptor of each image in the reduced incremental test set is calculated. The distance between the SIFT feature descriptor of each image and the SIFT feature descriptor of each template image is calculated sequentially. The distance between each pair of feature descriptors is used as the similarity between each image and each template image. The similarity between each image and all template images constitutes the similarity set of each image. The image with the highest similarity is selected from the similarity set of each image, and the template image corresponding to the highest similarity is used as the template image for that image.

[0020] From the annotation file corresponding to each template image, obtain the bounding box coordinates of the user intent region for each template image; determine the height, width, and rotation angle of each image in the incremental test set relative to its corresponding template image, and based on the determination results, obtain the coordinates of the bounding box of the target region for each image in the incremental test set.

[0021] When the height, width, and rotation angle of each image in the incremental test set are the same as those of the corresponding template image, the bounding box coordinates of the user intent region of each template image are used as the coordinates of the target region bounding box of each image in the incremental test set.

[0022] When the height, width, and rotation angle of each image in the incremental test set are inconsistent with the corresponding template image, the height, width, and rotation angle of the corresponding template image are transformed to be the same as the image. The bounding box coordinates of the user intent area of ​​each transformed template image are used as the coordinates of the target area bounding box of each image in the incremental test set. The target outer frame is obtained using the outer frame, and the target inner frame is obtained using the inner frames.

[0023] As a preferred embodiment of the first aspect above, the specific process of the box matching module is as follows:

[0024] First, the coordinate information of each user intent region detection box is matched with the coordinate information of the target region bounding box of each image using the Hungarian algorithm. The intersection-union ratio between the user intent region detection box and the target region bounding box is used as the distance between the user intent region detection box and the target region bounding box. The intersection-union ratio between the user intent region detection box and the target outer box and the target inner box are calculated in sequence to obtain a distance matrix.

[0025] Then, Hungarian matching is performed using the distance matrix to find the box combination with the smallest distance to both the target outer box and the target inner box within the user intent region detection box, thus obtaining a preliminary box matching matrix; det a The detection bounding box of the user intent region detection box that is closest to the target bounding box position will be det b The detected inner frame is the user intent region detection box that is closest to the target inner frame position; wherein, each row of the preliminary box matching matrix represents a matching pair, the first column of the preliminary box matching matrix represents the index of the detected outer frame and the index of the detected inner frame respectively, and the second column of the preliminary box matching matrix represents the index of the target outer frame and the index of the target inner frame respectively;

[0026] The detection bounding box is determined based on a preset intersection-union ratio (IU) threshold. a The intersection-union ratio with the target outline is used to determine the detection inner frame (det). b The intersection-union ratio with the target inner bounding box is used to determine whether a match is successful, resulting in a prompt box for segmentation.

[0027] When the detection outer frame det a The intersection-union ratio with the target outer bounding box, and the detection inner bounding box det bWhen the intersection-union ratio (IU) of the target inner bounding box is greater than a preset IU threshold, the first row of the preliminary box matching matrix is ​​used as the first row of the final box matching matrix, and the second row of the preliminary box matching matrix is ​​used as the second row of the final box matching matrix, thus obtaining the final box matching matrix. If the actual user target region is the target outer bounding box within the target region boundary, the index of the detected outer bounding box is obtained from the first row of the final box matching matrix, and the detected outer bounding box is det... a As a cue box for each image in the incremental test set; if the actual user target area is the inner bounding box of the target area, the index of the detected outer bounding box is obtained from the second row of the final box matching matrix, and the detected inner bounding box is det b As a prompt box for each image in the incremental test set;

[0028] When the detection outer frame det a The intersection-union ratio (CUB) with the target outer bounding box is greater than a preset CUB threshold, and the detected inner bounding box det b If the intersection-union ratio (IU) of the inner bounding box with the target bounding box is not greater than a preset IU threshold, the first row of the preliminary box matching matrix is ​​used as the final box matching matrix. If the actual user target region is the outer bounding box of the target region, the index of the detected outer bounding box is obtained from the final box matching matrix, and the detected outer bounding box is set to `det`. a As the prompt box for each image in the incremental test set; if the actual user target area is the inner box of the target area bounding box, then select the one with the largest area from all the user intent area detection boxes as the prompt box for each image in the incremental test set.

[0029] When the detection outer frame det a The intersection-union ratio (CUB) with the target outer bounding box is not greater than a preset CUB threshold, and the detected inner bounding box det b When the intersection-union ratio (IU) of the detected inner bounding box with the target inner bounding box is greater than a preset IU threshold, the second row of the preliminary box matching matrix is ​​used as the final box matching matrix. If the actual user target region is the target inner bounding box within the target region boundary, the index of the detected inner bounding box is obtained from the final box matching matrix, and the detected inner bounding box is det... b As the prompt box for each image in the incremental test set; if the actual user target area is the target outer box within the target area boundary box, then select the one with the largest area from all the user intent area detection boxes as the prompt box for each image in the incremental test set.

[0030] When the detection outer frame det a The intersection-union ratio with the target outline and the detection inner frame det b If the intersection-to-union ratios with the target inner frame are all not greater than the preset intersection-to-union ratio threshold, S44 does not need to be executed.

[0031] As a preferred embodiment of the first aspect mentioned above, the specific process of the self-boosting module is as follows:

[0032] First, the basic pixel-level mask and the target pixel-level mask are compared one by one and the same pixels are selected. They are then fused in a bitwise AND manner to obtain the preliminary mask for each image in the incremental test set.

[0033] Next, obtain the hierarchical contours in the preliminary mask of each image in the incremental test set. Select the contour with the largest area in the hierarchical contours as the target contour of each image in the incremental test set. Treat all pixels contained inside the target contour as the foreground part and fill the inside of the target contour to obtain the final pixel-level mask of each image in the incremental test set.

[0034] As a preferred embodiment of the first aspect above, the fixed size is set to 400×300.

[0035] As a preferred embodiment of the first aspect described above, the crossover-union ratio threshold is set to 0.3.

[0036] Secondly, the present invention provides a user-customized intent understanding system for planar scanned images, which is used to predict the final pixel-level mask of the planar scanned image to achieve user-customized intent understanding; the user-customized intent understanding system includes:

[0037] The dataset acquisition module is used to acquire the base domain dataset and the target domain dataset. The base domain dataset includes several planar scan images and several pixel-level masks corresponding to the planar scan images. The base domain dataset is divided into a base domain test set and a base domain training set. The target domain dataset is divided into N groups of incremental subsets. Each planar scan image in the target domain dataset corresponds to a label file. Each label file contains the labeling status, the height, width, and rotation angle of the image corresponding to the label file.

[0038] The incremental test set construction module is used to divide the planar scan images into unlabeled, partially labeled, and fully labeled categories in each incremental subset. For each fully labeled image in each incremental subset, one planar scan image is selected as a template image. A template set is constructed using N template images, where each template image must have a rotation angle of 0 in its corresponding annotation file. In each incremental subset, the unlabeled, partially labeled, and template images are removed, and the remaining planar scan images in each incremental subset, along with the base domain test set, constitute the incremental test set.

[0039] The basic pixel-level mask acquisition module is used to train a base domain segmentation model on the base domain training set, and use the trained base domain segmentation model to infer the incremental test set to obtain the basic pixel-level mask for each image in the incremental test set.

[0040] The target pixel-level mask acquisition module is used to acquire a trained target detection model and a trained segmentation model, and to construct a target domain inference model using the target detection model, the segmentation model, a template matching module and a box matching module; the target domain inference model is used to infer the target pixel-level mask of each image in the incremental test set.

[0041] The final pixel-level mask acquisition module is used to input the basic pixel-level mask and the target pixel-level mask together into the self-lifting module to obtain the final pixel-level mask for each image in the incremental test set.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] (1) This invention constructs a benchmark dataset for customized intent understanding of planar scanned images, which alleviates the problem of insufficient public datasets available for user-customized intent understanding tasks of planar scanned images.

[0044] (2) The present invention proposes a “divide and conquer” approach to ensure that the new and old tasks do not affect each other and the performance of the old tasks does not degrade, thereby improving the generalization ability of the task domain.

[0045] (3) This invention realizes user-customized intent understanding based on small samples without fine-tuning from the perspective of template matching, and customizes the basic model into personalized scenarios with less overhead.

[0046] (4) This invention proposes a self-enhancing module to improve the performance of the user-customized intent understanding system for planar scan images and reduce the noise of the prediction results. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a user-customized intent understanding method for planar scanned images;

[0048] Figure 2 A schematic diagram of a module for a user-customized intent understanding system for planar scanned images. Detailed Implementation

[0049] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0050] In a preferred embodiment of the present invention, such as Figure 1 As shown, a user-customized intent understanding method for planar scanned images is provided, which specifically includes the following steps:

[0051] S1. Obtain the base domain dataset and the target domain dataset; the base domain dataset includes several planar scan images and several pixel-level masks corresponding to the planar scan images. The base domain dataset is divided into a base domain test set and a base domain training set; the target domain dataset is divided into N groups of incremental sub-datasets. Each planar scan image in the target domain dataset corresponds to a label file. Each label file contains the labeling status, the height, width, and rotation angle of the image corresponding to the label file.

[0052] It should be noted that in this invention, the base domain dataset is divided into a base domain test set and a base domain training set. In this embodiment, to verify the performance of the base domain segmentation model, the base domain dataset is further divided into a base domain test set, a base domain training set, and a base domain validation set. The inference performance of the base domain segmentation model is evaluated on the base domain validation set. In the target domain dataset of step S1 in this embodiment, each labeled file contains a labeling status. When the labeling status is 1, it indicates that the image is labeled, and the bounding box coordinate information of the user intent region can be obtained. The user intent region is labeled using a quadrilateral bounding box. The user intent region with the largest area is defined as the outer box. There is only one outer box. The vertex coordinates of the outer box are used to form the outer box coordinate set. The user intent region with the largest area is removed, and the remaining user intent regions are defined as inner boxes. There may be one or more inner boxes. The vertex coordinates of the inner boxes are used to form the inner box coordinate set. When the labeling status is 0, it indicates that the image is unlabeled, that is, the bounding box coordinate information of the user intent region cannot be obtained.

[0053] It should be noted that the aforementioned planar scanned images refer to scanned images obtained by taking a picture of objects such as documents, books, and cards placed on a two-dimensional plane (such as a desktop).

[0054] S2. In each incremental subset, the planar scan images are divided into unlabeled, partially labeled, and fully labeled categories. From the fully labeled images in each incremental subset, one planar scan image is selected as a template image. A template set is constructed using N template images. These template images must satisfy the condition that the rotation angle in the corresponding annotation file is 0. In each incremental subset, the unlabeled, partially labeled, and template images are removed. The remaining planar scan images in each incremental subset, along with the base domain test set, constitute the incremental test set. It should be noted that in this invention, those skilled in the art can reduce each template image to a suitable size according to actual conditions. In this embodiment, each template image is reduced to 400×300.

[0055] It should be noted that "unlabeled" means the labeling status of the labeled file is 0; "incompletely labeled" means that the image actually has K user intent regions, but the corresponding labeled file only has 1 to (K-1) user intent regions. Then, in each incremental subset, unlabeled, incompletely labeled, and template images are removed. The remaining images in the target domain dataset and the base domain test set are used together to form the incremental test set and incremental validation set. In this embodiment, only the base domain segmentation model needs to be trained using the base domain training set, while the object detection model and segmentation model are already trained and do not need to be trained using the images in the target domain dataset. Therefore, only the incremental test set and incremental validation set need to be constructed. The accuracy of the user-customized intent understanding system for planar scanned images is evaluated on the incremental validation set, and the inference process is completed on the incremental test set.

[0056] It should be noted that, in this invention, those skilled in the art can select or set the number K of user intent areas according to the actual situation. In this embodiment, the number of user intent areas is set to 2 to 6.

[0057] Of course, in order to verify the specific technical effects of the present invention, in addition to the above-mentioned validation set, corresponding datasets can also be set. In the present invention, the goal of the final user-customized intent understanding system for planar scanned images is: to generate a prediction mask corresponding to a single test image in the target domain dataset, the intent represented by which is the same as the intent selected by the user in the customized sample, and the intent understanding performance of the user-customized intent understanding system for planar scanned images on the base domain dataset is not affected.

[0058] S3. Train a base domain segmentation model on the base domain dataset, and use the trained base domain segmentation model to infer on the incremental test set to obtain the basic pixel-level mask M for each image in the incremental test set. B .

[0059] It should be noted that in this embodiment of the invention, an image saliency detection network is selected, and the network is trained using images and their masks in the base domain training set to obtain a base domain segmentation model, BaseSeg. Then, the trained base domain segmentation model is used to infer the images in the base domain validation set to obtain pixel-level segmentation results. The mIoU is calculated using the mask label of each image and the pixel-level segmentation results. The average mIoU of all images in the base domain validation set is taken as the accuracy mIoU of the base domain segmentation model on the base domain dataset. base .

[0060] It should be noted that, in this embodiment of the invention, the trained base domain segmentation model BaseSeg is used to perform inference on each image X in the incremental test set. During inference, the same image preprocessing methods and various parameters as those used during the training of the base domain segmentation model are used to obtain the basic pixel-level mask M for each image in the incremental test set. B :

[0061] M B =BaseSeg(X; Θ B )

[0062] Where, Θ B These are the network parameters for the base domain segmentation model.

[0063] S4. Obtain a trained object detection model and a trained segmentation model. Utilize these two models, along with a template matching module and a bounding box matching module, to construct an object domain inference model. Infer the incremental test set using this model to obtain the pixel-level target mask M for each image in the incremental test set. T .

[0064] In this embodiment, step S4 is specifically implemented through the following sub-steps:

[0065] S41. Using the template matching module described above, obtain the template corresponding to each image in the incremental test set and the coordinate information of the bounding box of the target region of each image.

[0066] It should be noted that the specific process of the template matching module mentioned above is as follows:

[0067] Each template image is reduced to a preset fixed size, and the SIFT (Scale-invariant feature transform) feature descriptor of each reduced template image is calculated. The N SIFT feature descriptors are saved to N text files. Each image in the incremental test set is reduced to the above fixed size, and the SIFT feature descriptor of each image in the reduced incremental test set is calculated. The distance between the SIFT feature descriptor of each image and the SIFT feature descriptor of each template image is calculated in turn. The distance between each pair of feature descriptors is used as the similarity between each image and each template image. The similarity between each image and all template images constitutes the similarity set of each image. The image with the highest similarity is selected from the similarity set of each image, and the template image corresponding to the highest similarity is used as the template image corresponding to each image.

[0068] From the annotation file corresponding to each template image, obtain the bounding box coordinates of the user intent region for each template image; determine the height, width, and rotation angle of each image in the incremental test set relative to its corresponding template image, and based on the determination results, obtain the coordinates of the bounding box of the target region for each image in the incremental test set.

[0069] When the height, width, and rotation angle of each image in the incremental test set are consistent with those of the corresponding template image, the bounding box coordinates of the user intent region of each template image are used as the coordinates of the target region bounding box of each image in the incremental test set.

[0070] When the height, width, and rotation angle of each image in the incremental test set are inconsistent with the corresponding template image, the height, width, and rotation angle of the corresponding template image are transformed accordingly to make them the same as each image in the incremental test set. The bounding box coordinates of the user intent region of each transformed template image are used as the coordinates of the target region bounding box of each image in the incremental test set. Among them, the target outer box external is obtained using the outer box, and the target inner box inner is obtained using the inner boxes, i.e., target_box = [external, inner].

[0071] It should be noted that in the preparation stage of step S41, that is, before the inference begins, the N text files saved in the above steps are read into a list L. sift Li, i.e., L siftIt stores the SIFT feature descriptors for each template image. It loads the base domain segmentation model BaseSeg, the pre-trained object detection model ObjDet, and the pre-trained segmentation model ObjSeg.

[0072] S42. Using the above object detection model ObjDet, detect possible user intent regions in each image of the incremental test set. For each image, obtain the coordinate information of multiple user intent region detection boxes, and construct the coordinate set det_box of the user intent region detection boxes for each image:

[0073] det_box = ObjDet(X; Θ1)

[0074] Among them, det_box=[det0, det1,..., det k ], det0, det1, ..., det k All represent the coordinate information of the detection box for each user intent region; X represents each image in the above incremental test set; Θ1 is the network parameter of the object detection model.

[0075] S43. Input the coordinate information of the detection box of each user intent region corresponding to each image and the coordinate information of the bounding box of the target region of each image into the above box matching module for matching, and obtain the prompt box of each image in the incremental test set.

[0076] It should be noted that the specific process of the box matching module in this embodiment is as follows:

[0077] The coordinates of the user intent region detection box and the target region bounding box for each image are matched using the Hungarian algorithm. The intersection-over-union (IoU) ratio between the user intent region detection box and the target region bounding box is used as the distance between them. The IoU ratios between the user intent region detection box and the target outer bounding box and the target inner bounding box are calculated sequentially to obtain a (k+1) row, 2 column distance matrix iou_matrix.

[0078] iou_matrix[i,j]=IoU(det_box[i], target_box[j])i=0,1,...k; j=0,1

[0079] Where, when j=0, target_box[0] corresponds to the outer bounding box of the target; when j=1, target_box[1] corresponds to the inner bounding box of the target; i represents the coordinate information of the i-th user intent region detection box.

[0080] Then, Hungarian matching is performed using the aforementioned distance matrix iou_matrix, which involves finding the combination of boxes with the smallest distances to both the outer and inner bounding boxes of the target within the user intent region detection box, resulting in the initial box matching matrix match_matrix_initial:

[0081] match_matrix_initial=HungarianMatching(-iou_matrix)

[0082] HungarianMatching(·) represents Hungarian matching.

[0083] Then, det a The detection bounding box of the user intent region detection box that is closest to the external position of the target bounding box will be det b The detected inner bounding box is the user intent region detection box that is closest to the inner position of the target inner bounding box. Specifically, the general form of the initial box matching matrix match_matrix_initial is: Each row represents a matching pair. The first column of the preliminary box matching matrix represents the index of the outer bounding box and the inner bounding box of the user intent region detection box, respectively, and the second column represents the index of the outer bounding box and the inner bounding box of the target.

[0084] The detection bounding box det is determined based on the preset intersection-union ratio (IU) threshold β. a The intersection-union ratio (IoU) with the external bounding box of the target is used to determine the inner bounding box (det) of the target. b The intersection-union ratio with the inner bounding box of the target is used to determine whether a match is successful, resulting in a prompt box for segmentation:

[0085] When the above detection outer frame det a The intersection-union comparison with the external bounding box of the target, and the detection inner bounding box det of the target. bWhen the intersection-union ratio (IU) of the target inner bounding box (inner) is greater than a preset IU threshold, the first row of the initial box matching matrix `match_matrix_initial` is used as the first row of the final box matching matrix `match_matrix`, and the second row of the initial box matching matrix `match_matrix_initial` is used as the second row of the final box matching matrix `match_matrix`, thus obtaining the final box matching matrix `match_matrix`. If the actual user target region is the target outer bounding box `external` within the target region bounding box `target_box`, the index of the detected outer bounding box is obtained from the first row of the final box matching matrix `match_matrix`, and the index of the detected outer bounding box `det` is... a As a cue box for each image in the incremental test set; if the actual user target region is the inner bounding box (target_box) within the target region bounding box (target_box), the index of the detected outer bounding box is obtained from the second row of the final box matching matrix (match_matrix), and the index of the detected inner bounding box (det) is set. b This serves as a tooltip for each image in the incremental test set.

[0086] When the above detection outer frame det a The intersection-union ratio with the external bounding box of the target, or the detection inner bounding box det of the target. b If one of the intersection-union ratios (IU) of the target inner bounding box (inner) is greater than a preset IU threshold, the row in the preliminary box matching matrix that is greater than the threshold is taken as the final box matching matrix. If the target bounding box index in the final box matching matrix is ​​the same as the actual user target region, the detection box in the final box matching matrix is ​​taken as the prompt box for each image in the incremental test set. If the target bounding box index in the final box matching matrix is ​​different from the actual user target region, the largest area among all user intent region detection boxes is selected as the prompt box for each image in the incremental test set.

[0087] Specifically, when detecting the outer frame det a The intersection-union ratio (CUB) with the target bounding box is greater than the preset CUB threshold, and the inner bounding box (det) is detected. b If the intersection-union ratio (IU) with the target inner bounding box is not greater than a preset IU threshold, the first row of the preliminary box matching matrix is ​​used as the final box matching matrix. If the actual user target region is the target outer bounding box within the target region's bounding box, the index of the detected outer bounding box is obtained from the final box matching matrix, and the detected outer bounding box is det... a As the cue box for each image in the incremental test set; if the actual user target region is the inner bounding box of the target region, then select the largest area from all user intent region detection boxes as the cue box for each image in the incremental test set; when the outer bounding box is detected... aThe intersection-union ratio (CUB) with the target bounding box is not greater than a preset CUB threshold, and the inner bounding box (det) is detected. b If the intersection-union ratio (IU) with the target inner bounding box is greater than a preset IU threshold, the second row of the preliminary box matching matrix is ​​used as the final box matching matrix. If the actual user target region is the target inner bounding box within the target region's bounding box, the index of the detected inner bounding box is obtained from the final box matching matrix, and the detected inner bounding box is det... b As the prompt box for each image in the incremental test set; if the actual user target area is the target outer box within the target area bounding box, then select the one with the largest area among all user intent area detection boxes as the prompt box for each image in the incremental test set.

[0088] When the above detection outer frame det a The intersection-union ratio with the external bounding box of the target mentioned above, and the detection inner bounding box det mentioned above. b If the intersection-union ratio (IU) of the target inner frame is not greater than the preset IU threshold, there is no need to execute S44.

[0089] It should be noted that, in this invention, the crossover ratio threshold can be selected by those skilled in the art according to the actual situation. In this embodiment, it is set to 0.3.

[0090] It should be noted that when detecting the outer frame det a Intersection over union (IoU) with the target bounding box, and detection of the inner bounding box (det). b A match is considered successful when the intersection-union ratio (IU) with the target inner frame is greater than the preset IU threshold.

[0091] It should be noted that when detecting the outer frame det a Intersection over union (IoU) with the target bounding box, or detection of the inner bounding box (det). b One of the cross-union ratios (IoU) with the target inner bounding box is greater than a preset IoU threshold. If this threshold is greater than the threshold, it means that a matching pair was filtered out due to low IoU, resulting in only one row in the final box matching matrix showing a successful match. Assume the user's intent region is the target inner bounding box (target_box), and the final box matching matrix (match_matrix) only contains the detected outer bounding box (det). a - If only one target bounding box (external) is matched, then the largest area among all user intent region detection boxes is selected as the prompt.

[0092] It should be noted that when detecting the outer frame det a The intersection-union ratio with the target bounding box, and the detection of the inner bounding box (det) bWhen the intersection-union ratio (IoU) of the target bounding box is not greater than the preset IoU threshold, the final box matching matrix match_matrix is ​​considered to be empty. There are no matching pairs that do not reach the IoU threshold, indicating that neither of the two matching pairs in the final box matching matrix is ​​successfully matched. This means that the actual target region of the current image does not match the user intent region in the template image at all. Therefore, this image is considered not to belong to the remaining images in each incremental subset, i.e., it does not belong to the target domain image, and it is determined to be an image in the base domain test set. The reasoning can be completed by using the base domain segmentation model, and the following step S44 is no longer required.

[0093] S44. Using the segmentation model ObjSeg and the cue boxes for each image in the incremental test set, segment each image in the incremental test set to obtain the target pixel-level mask M for each image in the incremental test set. T :

[0094] M T =ObjSeg(X; prompt; Θ2)

[0095] Where Θ2 represents the network parameters of the segmentation model; prompt represents the cue box. S5. Apply the above basic pixel-level mask M B And the aforementioned target pixel-level mask M T These images are input together into the self-boosting module to obtain the final pixel-level mask M for each image in the incremental test set. F The user's customized intent is predicted by the final pixel-level mask.

[0096] It should be noted that the user-customized intent understanding process in the method of this invention refers to the following: for a planar scanned image, a final pixel-level mask of the planar scanned image can be output according to the user's customized intent. This final pixel-level mask is then used as the predicted value of the user's intent region (region of interest) in the planar scanned image. The user's customized intent means that a prediction result matching the user's intent region can be output based on different intent regions (regions of interest). For example, if a planar scanned image contains both a book and a bank card, and the user's intent region is the bank card region in the planar scanned image, the method of this invention can output the predicted result for the bank card region.

[0097] It should be noted that the self-elevation module in step S5 of this embodiment is specifically implemented through the following sub-steps:

[0098] First, apply the above basic pixel-level mask M B And the aforementioned target pixel-level mask M T By comparing and selecting identical pixels one by one, and fusing them using a bitwise AND operation, a preliminary mask M is obtained for each image in the incremental test set. A .

[0099] Obtain the initial mask M for each image in the incremental test set. A The hierarchical contours are selected, and the contour with the largest area is chosen as the target contour for each image in the incremental test set. All pixels contained within the target contour are considered as the foreground portion, and the interior of the target contour is filled to obtain the final pixel-level mask M for each image in the incremental test set. F .

[0100] It should be noted that all pixel values ​​inside the target contour are either 0 or 1. When filling the interior of the target contour, all pixel values ​​inside the target contour need to be set to 1 to obtain the final pixel-level mask M for each image in the incremental test set. F .

[0101] Similarly, based on the same inventive concept, such as Figure 2 As shown, another preferred embodiment of the present invention also provides a user-customized intent understanding system for planar scanned images, corresponding to the user-customized intent understanding method for planar scanned images provided in the above embodiments, which includes:

[0102] The dataset acquisition module is used to acquire the base domain dataset and the target domain dataset. The base domain dataset includes several planar scan images and several pixel-level masks corresponding to the planar scan images. The base domain dataset is divided into a base domain test set and a base domain training set. The target domain dataset is divided into N groups of incremental sub-datasets. Each planar scan image in the target domain dataset corresponds to a label file. Each label file contains the labeling status, the height, width, and rotation angle of the image corresponding to the label file.

[0103] The incremental test set construction module is used to divide the planar scan images into unlabeled, partially labeled, and fully labeled categories in each incremental subset. In each incremental subset, one planar scan image is selected as a template image from the fully labeled images. A template set is constructed using N template images. The template images must satisfy the condition that the rotation angle in the corresponding annotation file is 0. In each incremental subset, the unlabeled, partially labeled, and template images are removed. The remaining planar scan images in each incremental subset, together with the base domain test set, constitute the incremental test set.

[0104] The basic pixel-level mask acquisition module is used to train a base domain segmentation model on the base domain training set, and use the trained base domain segmentation model to infer the incremental test set to obtain the basic pixel-level mask for each image in the incremental test set.

[0105] The target pixel-level mask acquisition module is used to acquire a trained target detection model and a trained segmentation model, and to construct a target domain inference model using the target detection model, the segmentation model, a template matching module and a box matching module; the target domain inference model is used to infer the incremental test set to obtain the target pixel-level mask for each image in the incremental test set.

[0106] The final pixel-level mask acquisition module is used to input the basic pixel-level mask and the target pixel-level mask together into the self-lifting module to obtain the final pixel-level mask for each image in the incremental test set.

[0107] Specifically, in the above-mentioned user-customized intent understanding system for planar scanned images, each module corresponds one-to-one with the aforementioned S1 to S5. Therefore, the specific implementation methods of each module can also be found in the aforementioned S1 to S5.

[0108] The user-customized intent understanding method using planar scanned images, as shown in S1 to S5 above, will be applied to a specific example to demonstrate its technical effects.

[0109] Example

[0110] The implementation method of this embodiment is as described in S1 to S5 above, and the specific steps will not be elaborated in detail again. The following only demonstrates its effect on example data. This invention is implemented on a dataset with truth value annotation:

[0111] Customized Intent Understanding Dataset for Planar Scanned Images: This dataset contains 2241 images, divided into 112 sub-data sets. Each sub-data set contains approximately 20 images, all of which are images of the same object taken from different angles. Each image contains one or more potential intents. The template set contains 112 images, and the test set contains 2031 images.

[0112] The pre-trained object detection model used in this embodiment is GroundingDINO, and the pre-trained segmentation model is Mobile-SAM. Inference is performed on the test set images to obtain prediction results, and the accuracy metrics mIoU (mean Intersection over Union) and mPA (mean Pixel Accuracy) are calculated together with the labels. The overall accuracy of the test set and the accuracy of some sub-groups are shown in the table below:

[0113] mIoU mPA test set 90.47% 94.33% Group 1 98.35% 98.93% Group 2 99.00% 99.34% Group 3 92.62% 95.31% …… …… …… Group 51 86.65% 92.75% Group 52 87.15% 92.72% Group 53 87.35% 93.27% …… …… …… Group 110 98.47% 98.71% Group 111 83.83% 86.59% Group 112 91.73% 92.60%

[0114] As shown in the table above, the user-customized intent understanding method for planar scanned images described in this invention achieved an accuracy of over 90% on the test set, indicating excellent performance. Furthermore, this embodiment of the invention was also tested on the base domain dataset. The method of this embodiment achieved an mIoU of 95.69% on the validation set, while the base domain segmentation model achieved an mIoU of 95.41% on the validation set. This demonstrates that the method of this embodiment, while performing incremental tasks, not only does not cause a decrease in base task performance but can also slightly improve base task performance.

[0115] Through the above technical solutions, this invention combines a pre-trained large model with traditional methods to develop a user-customized intent understanding method and system for planar scanned images. This invention ensures that the base task and target task do not interfere with each other and that the performance of the base task is not degraded through "divide and conquer" domain generalization; it solves the incremental few-sample problem of the task simply and efficiently through template matching from a few-sample angle; and it achieves integrated decision-making based on knowledge from the base domain and target domain through a self-boosting module, reducing noise in the inference results and better adapting to the user-customized intent understanding task of planar scanned images.

[0116] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for understanding user-customized intents in planar scanned images, characterized in that, Includes the following steps: S1. Obtain the base domain dataset and the target domain dataset; the base domain dataset includes several planar scan images and several pixel-level masks corresponding to the planar scan images. The base domain dataset is divided into a base domain test set and a base domain training set; the target domain dataset is divided into N groups of incremental subsets. Each planar scan image in the target domain dataset corresponds to a labeling file. Each labeling file contains the labeling status, the height, width, and rotation angle of the image corresponding to the labeling file; S2. In each incremental subset, the planar scan images are divided into unlabeled, partially labeled, and fully labeled categories. In each incremental subset, one planar scan image is selected as a template image from the fully labeled planar scan images. N template images are used to form a template set. The template images must satisfy the condition that the rotation angle in the corresponding annotation file is 0. In each incremental subset, the unlabeled, partially labeled, and template images are removed. The remaining planar scan images in each incremental subset, together with the base domain test set, constitute the incremental test set. S3. Train a base domain segmentation model on the base domain training set, and use the trained base domain segmentation model to infer the incremental test set to obtain the basic pixel-level mask for each image in the incremental test set. S4. Obtain a trained object detection model and a trained segmentation model, and construct an object domain inference model using the object detection model, the segmentation model, a template matching module, and a box matching module; use the object domain inference model to infer the incremental test set to obtain the target pixel-level mask for each image in the incremental test set; S5. Input the basic pixel-level mask and the target pixel-level mask together into the self-lifting module to obtain the final pixel-level mask for each image in the incremental test set. The prediction of the user's customized intent is realized by the final pixel-level mask. The specific process of the self-elevation module is as follows: First, the basic pixel-level mask and the target pixel-level mask are compared one by one and the same pixels are selected. They are then fused in a bitwise AND manner to obtain the preliminary mask for each image in the incremental test set. Next, obtain the hierarchical contours in the preliminary mask of each image in the incremental test set. Select the contour with the largest area in the hierarchical contours as the target contour of each image in the incremental test set. Treat all pixels contained inside the target contour as the foreground part and fill the inside of the target contour to obtain the final pixel-level mask of each image in the incremental test set.

2. The user-customized intent understanding method for planar scanned images as described in claim 1, characterized in that, When the annotation status of each annotation file is 1, it indicates that the image has annotations, and the bounding box coordinate information of the user intent region can be obtained. The user intent region is annotated using quadrilateral bounding boxes. The user intent region with the largest area is defined as the outer frame, and the coordinates of the vertices of the outer frame are used to form the outer frame coordinate set. The user intent region with the largest area is removed, and the remaining user intent regions are defined as the inner frames, and the coordinates of the vertices of the inner frames are used to form the inner frame coordinate set. When the annotation status of each annotation file is 0, it indicates that the image has no annotations, that is, the bounding box coordinate information of the user intent region cannot be obtained.

3. The user-customized intent understanding method for planar scanned images as described in claim 1, characterized in that, The base domain segmentation model employs an image saliency detection network.

4. The user-customized intent understanding method for planar scanned images as described in claim 2, characterized in that, The specific process in the target domain inference model is as follows: S41. Using the template matching module, obtain the template corresponding to each image in the incremental test set and the coordinate information of the target region bounding box of each image; S42. The target detection model is used to detect possible user intent regions in each image of the incremental test set. The coordinate information of multiple user intent region detection boxes is obtained for each image, and the coordinate set of user intent region detection boxes for each image is formed. S43. The coordinate information of the detection box of each user intent region corresponding to each image and the coordinate information of the target region bounding box of each image are input into the box matching module for matching to obtain the prompt box of each image in the incremental test set. S44. Using the segmentation model and the prompt box of each image in the incremental test set, segment each image in the incremental test set to obtain the target pixel-level mask of each image in the incremental test set.

5. The user-customized intent understanding method for planar scanned images as described in claim 4, characterized in that, The specific process of the template matching module is as follows: First, each template image is reduced to a preset fixed size. The SIFT feature descriptor of each reduced template image is calculated, and the N SIFT feature descriptors are saved to N text files. Next, each image in the incremental test set is reduced to the fixed size, and the SIFT feature descriptor of each image in the reduced incremental test set is calculated. The distance between the SIFT feature descriptor of each image and the SIFT feature descriptor of each template image is calculated sequentially. The distance between each pair of feature descriptors is used as the similarity between each image and each template image. The similarity between each image and all template images constitutes the similarity set of each image. The image with the highest similarity is selected from the similarity set of each image, and the template image corresponding to the highest similarity is used as the template image for that image. From the annotation file corresponding to each template image, obtain the bounding box coordinates of the user intent region for each template image; determine the height, width, and rotation angle of each image in the incremental test set relative to its corresponding template image, and based on the determination results, obtain the coordinates of the bounding box of the target region for each image in the incremental test set. When the height, width, and rotation angle of each image in the incremental test set are the same as those of the corresponding template image, the bounding box coordinates of the user intent region of each template image are used as the coordinates of the target region bounding box of each image in the incremental test set. When the height, width, and rotation angle of each image in the incremental test set are inconsistent with the corresponding template image, the height, width, and rotation angle of the corresponding template image are transformed to be the same as the image. The bounding box coordinates of the user intent area of ​​each transformed template image are used as the coordinates of the target area bounding box of each image in the incremental test set. The target outer frame is obtained using the outer frame, and the target inner frame is obtained using the inner frames.

6. The user-customized intent understanding method for planar scanned images as described in claim 5, characterized in that, The specific process of the box matching module is as follows: First, the coordinate information of each user intent region detection box is matched with the coordinate information of the target region bounding box of each image using the Hungarian algorithm. The intersection-union ratio between the user intent region detection box and the target region bounding box is used as the distance between the user intent region detection box and the target region bounding box. The intersection-union ratio between the user intent region detection box and the target outer box and the target inner box are calculated in sequence to obtain a distance matrix. Then, Hungarian matching is performed using the distance matrix to find the box combination with the smallest distance to both the target outer box and the target inner box within the user intent region detection box, thus obtaining a preliminary box matching matrix; det a The detection bounding box of the user intent region detection box that is closest to the target bounding box position will be det b The detected inner frame is the user intent region detection box that is closest to the target inner frame position; wherein, each row of the preliminary box matching matrix represents a matching pair, the first column of the preliminary box matching matrix represents the index of the detected outer frame and the index of the detected inner frame respectively, and the second column of the preliminary box matching matrix represents the index of the target outer frame and the index of the target inner frame respectively; The detection bounding box is determined based on a preset intersection-union ratio (IU) threshold. a The intersection-union ratio with the target outer bounding box is used to determine the detection inner bounding box (det). b The intersection-union ratio with the target inner bounding box is used to determine whether a match is successful, resulting in a prompt box for segmentation. When the detection outer frame det a The intersection-union ratio with the target outer bounding box, and the detection inner bounding box det b When the intersection-union ratio (IU) of the target inner bounding box is greater than a preset IU threshold, the first row of the preliminary box matching matrix is ​​used as the first row of the final box matching matrix, and the second row of the preliminary box matching matrix is ​​used as the second row of the final box matching matrix, thus obtaining the final box matching matrix. If the actual user target region is the target outer bounding box within the target region boundary, the index of the detected outer bounding box is obtained from the first row of the final box matching matrix, and the detected outer bounding box is det... a As a cue box for each image in the incremental test set; if the actual user target area is the inner bounding box of the target area, the index of the detected outer bounding box is obtained from the second row of the final box matching matrix, and the detected inner bounding box is det b As a prompt box for each image in the incremental test set; When the detection outer frame det a The intersection-union ratio (CUB) with the target outer bounding box is greater than a preset CUB threshold, and the detected inner bounding box det b If the intersection-union ratio (IU) of the inner bounding box with the target bounding box is not greater than a preset IU threshold, the first row of the preliminary box matching matrix is ​​used as the final box matching matrix. If the actual user target region is the outer bounding box of the target region, the index of the detected outer bounding box is obtained from the final box matching matrix, and the detected outer bounding box is set to `det`. a As the prompt box for each image in the incremental test set; if the actual user target area is the inner box of the target area bounding box, then select the one with the largest area from all the user intent area detection boxes as the prompt box for each image in the incremental test set. When the detection outer frame det a The intersection-union ratio (CUB) with the target outer bounding box is not greater than a preset CUB threshold, and the detected inner bounding box det b When the intersection-union ratio (IU) of the detected inner bounding box with the target inner bounding box is greater than a preset IU threshold, the second row of the preliminary box matching matrix is ​​used as the final box matching matrix. If the actual user target region is the target inner bounding box within the target region boundary, the index of the detected inner bounding box is obtained from the final box matching matrix, and the detected inner bounding box is det... b As the prompt box for each image in the incremental test set; if the actual user target area is the target outer box within the target area boundary box, then select the one with the largest area from all the user intent area detection boxes as the prompt box for each image in the incremental test set. When the detection outer frame det a The intersection-union ratio with the target outline and the detection inner frame det b If the intersection-to-union ratios with the target inner frame are all not greater than the preset intersection-to-union ratio threshold, S44 does not need to be executed.

7. The user-customized intent understanding method for planar scanned images as described in claim 5, characterized in that, The fixed size is set to 400×300.

8. The user-customized intent understanding method for planar scanned images as described in claim 6, characterized in that, The crossover ratio threshold is set to 0.

3.

9. A user-customized intent understanding system for planar scanned images, characterized in that, include: The dataset acquisition module is used to acquire the base domain dataset and the target domain dataset. The base domain dataset includes several planar scan images and several pixel-level masks corresponding to the planar scan images. The base domain dataset is divided into a base domain test set and a base domain training set. The target domain dataset is divided into N groups of incremental subsets. Each planar scan image in the target domain dataset corresponds to a label file. Each label file contains the labeling status, the height, width, and rotation angle of the image corresponding to the label file. The incremental test set construction module is used to divide the planar scan images into unlabeled, partially labeled, and fully labeled categories in each incremental subset. For each fully labeled planar scan image in each incremental subset, one planar scan image is selected as a template image. A template set is constructed using N template images, where each template image must have a rotation angle of 0 in its corresponding annotation file. In each incremental subset, the unlabeled, partially labeled, and template images are removed, and the remaining planar scan images in each incremental subset, along with the base domain test set, constitute the incremental test set. The basic pixel-level mask acquisition module is used to train a base domain segmentation model on the base domain training set, and use the trained base domain segmentation model to infer the incremental test set to obtain the basic pixel-level mask for each image in the incremental test set. The target pixel-level mask acquisition module is used to acquire a trained target detection model and a trained segmentation model, and to construct a target domain inference model using the target detection model, the segmentation model, a template matching module and a box matching module; the target domain inference model is used to infer the target pixel-level mask of each image in the incremental test set. The final pixel-level mask acquisition module is used to input the basic pixel-level mask and the target pixel-level mask together into the self-lifting module to obtain the final pixel-level mask for each image in the incremental test set; The specific process of the self-elevation module is as follows: First, the basic pixel-level mask and the target pixel-level mask are compared one by one and the same pixels are selected. They are then fused in a bitwise AND manner to obtain the preliminary mask for each image in the incremental test set. Next, obtain the hierarchical contours in the preliminary mask of each image in the incremental test set. Select the contour with the largest area in the hierarchical contours as the target contour of each image in the incremental test set. Treat all pixels contained inside the target contour as the foreground part and fill the inside of the target contour to obtain the final pixel-level mask of each image in the incremental test set.