Method and computer program product for retrieving visually narrative book pages

Abstract

A picture book page retrieval method includes: a character facial feature extraction step, obtaining the character facial feature vector of each character facial patch in each page image of the picture book; a page-appearing character recognition step, obtaining the page-appearing character tag corresponding to each page image; a page semantic acquisition step, obtaining the page semantic vector of each page image; a database establishment step, storing the page-appearing character tags and page semantic vectors in a page retrieval database corresponding to page index information; and a composite retrieval step, performing a composite retrieval based on the page retrieval database for user queries, including a page semantic retrieval and a page-appearing character retrieval, and outputting the corresponding page retrieval results.

Claims

1. A picture book page retrieval method, applied to a picture book page retrieval system, to output page retrieval results for a picture book based on a user query. The picture book page retrieval method includes: a character facial feature extraction step, in which a character facial detection model detects character facial patches of each page image of the picture book, and a character facial feature extraction model converts each character facial patch of each page image into a corresponding character facial feature vector; a page-appearing character recognition step, in which the character facial feature vectors of each page are compared with known character facial feature vectors stored in a preset character feature database to identify the characters appearing in each page image and obtain page-appearing character tags corresponding to each page image; and a page semantic acquisition step, in which a multimodal embedding model converts each page image of the picture book into a corresponding page semantic vector. The system includes a database establishment step, which stores page index information corresponding to the character tags and semantic vectors of each page and the images on each page in a page retrieval database of the picture book page retrieval system; and a composite retrieval step, which performs a composite retrieval including a page semantic retrieval and a page character retrieval for the user query input into the picture book page retrieval system. The page semantic retrieval converts the user query into a page query vector using the multimodal embedding model, and compares the similarity of the page query vector with the page semantic vectors stored in the page retrieval database. The page character retrieval compares the matching of the character identification words contained in the user query with the page character tags stored in the page retrieval database, and outputs the page index information corresponding to the page semantic vectors with high similarity and / or matching page character tags as the page retrieval results.

2. The picture book page retrieval method as described in claim 1, wherein in the step of character recognition on the page, the known character facial feature vector is obtained by detecting the known character facial patch using the character facial detection model for the known character image of the known character in the picture book, and then converting it accordingly using the character facial feature extraction model.

3. The picture book page retrieval method as described in request item 1, wherein in the step of obtaining the semantics of the page, the multimodal embedding model is a Jina-Embedding-v4 model.

4. A picture book page retrieval method, applied to a picture book page retrieval system, for outputting page retrieval results for a picture book based on a user query. The picture book page retrieval method includes: a character facial feature extraction step, using a character facial detection model to detect character facial patches in each page image of the picture book, and using a character facial feature extraction model to convert each character facial patch in each page image into a corresponding character facial feature vector; a character grouping step, using a clustering algorithm to group all the character facial feature vectors according to feature density to obtain one or more character groups; a database establishment step, storing the page index information corresponding to each page image for each character group in a page retrieval database of the picture book page retrieval system; and an image retrieval step, outputting the corresponding page index information as the page retrieval result based on the user query input into the picture book page retrieval system, wherein... The user query is generated by selecting a representative image of a character from a character gallery provided by the picture book page retrieval system. The picture book page retrieval system outputs the page index information of the corresponding character group as the page retrieval result based on the selected representative image of the character. Alternatively, the user query is generated by uploading an image of a character to be queried to the picture book page retrieval system. The picture book page retrieval system detects the facial patch of the character to be queried using the character facial detection model, and then converts it into a facial feature vector of the character to be queried using the character facial feature extraction model. By comparing the facial feature vector of the character to be queried with the facial feature vectors of the character group stored in the page retrieval database, the page index information of the corresponding character group is output as the page retrieval result.

5. The picture book page retrieval method as described in request item 1 or 4, wherein the picture book is a comic book.

6. The picture book page retrieval method as described in request item 1 or 4, wherein in the character facial feature extraction step, the character facial detection model is a YOLO (You Only Look Once) object detection model.

7. The picture book page retrieval method as described in request item 1 or 4, wherein in the character facial feature extraction step, the character facial feature extraction model is a Contrastive Language–Image Pre-training (CLIP) model.

8. The picture book page retrieval method as described in claim 1 or 4, wherein the page index information includes the book identification code, page number, series identification code, volume number, and image file path address corresponding to the page image.

9. The picture book page retrieval method as described in claim 4, wherein in the role grouping step, the clustering algorithm is a density-based spatial clustering of applications with noise (DBSCAN) algorithm.

10. A computer program product for retrieving picture book pages, which is loaded onto a computer and executes the picture book page retrieval method as described in any one of claims 1 to 9.

Images