Photo album management system and photo album management method based on multiple modes

Abstract

The invention relates to a photo album management system and a photo album management method based on multiple modes. The photo album management system comprises a multi-mode CLIP model, a text feature extraction model, an image feature extraction model and an image feature extraction model, wherein the multi-mode CLIP model comprises a text feature extraction model TextModel and an image feature extraction model ImageModel; the vector database is used for storing feature vectors of the photos and related information of the feature vectors; the automatic classification module is used for automatically classifying the imported photos according to a plurality of preset category labels; the picture searching module is used for searching similar pictures according to the pictures provided by the user; and the text picture searching module is used for searching similar pictures according to the text description input by the user.

Description

Technical Field

[0001] The present invention relates to image processing and information retrieval technology, and in particular to an album management system and method for digital photos / pictures, and in particular to an album management system and an album management method for classifying and searching digital photos using a multimodal algorithm. Background Art

[0002] Multimodality generally refers to the use or analysis of multiple different types of data or information sources simultaneously in the fields of artificial intelligence, human-computer interaction, data analysis, etc. These data sources may include but are not limited to text, images, sound, video, etc. Multimodal systems can understand, process and integrate information from different modalities to provide more comprehensive and accurate understanding and response.

[0003] A photo is a static image that captures and records real-world scenes through photography technology. Digital photos are the most common form of photos in daily life. People take digital photos through mobile devices to capture scene information, convey emotional information, and save personal memories. Therefore, a large number of digital photos are generated. Furthermore, how to effectively manage a large number of digital photos is a problem that needs to be solved. Common usage scenarios include large quantities of digital photos being transferred from mobile devices to personal computers, so a system needs to be deployed on the computer side to manage digital photos.

[0004] At present, digital (photo / image) album management is generally stored by date of the photo / image, or the photos are manually classified and manually moved to the corresponding folder for classification each time. When the number of photos is too large, it is not conducive to searching for photos when they are needed later, and it takes too much energy to search in the album manually.

[0005] With the development of digital photography technology, the number of photos taken and stored by users in daily life has increased dramatically. How to effectively manage and retrieve these photos has become an urgent problem to be solved. Traditional album management methods usually rely on users to manually classify and search, which is not only time-consuming and labor-intensive, but also easily leads to confusion in photo management.

[0006] The existing technology has deficiencies in automatic classification and efficient search, so a new album management system and management method are needed to improve the user experience.

[0007] The information included in this background section of the present specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be construed as subject matter that will limit the scope of the present invention. Summary of the invention

[0008] The present invention is proposed in view of the above and other more concepts.

[0009] The present invention proposes a multimodal-based album management system and method for more efficient and convenient management of digital photos / pictures.

[0010] The purpose of the present invention is to provide a multimodal album management system and an album management method, which solves the problem of photo classification and search through automatic classification, image search and text search functions. Specifically, one or more embodiments of the multimodal album management system according to the present invention may have the following functions.

[0011] Automatic photo classification function: The system presets multiple tags (such as "landscape" and "animals"). When users import photos, the photo feature vector is extracted through a multimodal algorithm, and the similarity is calculated with the feature vector of the preset tag. The photos are automatically labeled and stored in the vector database.

[0012] Image search function: The user can provide a picture, and the system will extract the feature vector of the picture, calculate the similarity with the feature vectors of all photos stored in the database, and return the paths of the 10 photos with the highest similarity.

[0013] Image search by text: The user enters a text description, the system extracts the feature vector of the text, calculates the similarity with the feature vector of the photos stored in the database, and returns the paths of the 10 photos with the highest similarity.

[0014] The present invention mainly solves the following problems:

[0015] In order to solve the problem of photo classification, the present invention designs an automatic photo classification function in the system. The system will preset some tags, such as "landscape" and "animal". When a photo is imported into the system, the system will extract the feature vector of the photo based on the multimodal algorithm, and calculate the similarity with the feature vector of the pre-extracted tag (text). The tag with the highest similarity will be stored in the vector database together with the feature vector of the photo. The picture / photo is labeled and classified invisibly, and the user does not need to drag it to the corresponding folder.

[0016] In order to solve the problem of photo search, the present invention designs a search-by-image function in the system. If the user needs to find out whether there are similar pictures in the album, this function can be used. The user provides a picture and starts the system's search-by-image function. The picture will be extracted with a multimodal algorithm. The feature vector of the picture will be calculated by the calculation function of the vector database to calculate the similarity with the feature vectors of all pre-extracted stored photos, and the system paths of the 10 most similar photos will be returned and displayed.

[0017] Still solving the problem of photo search, the present invention designs a text search function in the system. Thanks to the multimodal algorithm model, when a user enters a text description in the album system, the multimodal algorithm in the system will extract the feature vector of the text. The feature vector is calculated by the calculation function of the vector database to calculate the similarity with the feature vectors of all pre-extracted stored photos, and the system paths of the 10 photos with the highest similarity are returned and displayed.

[0018] According to one aspect of the present invention, a multimodal album management system is provided, which includes: a multimodal CLIP model, which includes a text feature extraction model TextModel and an image feature extraction model ImageModel; a vector database for storing feature vectors of photos and related information thereof; an automatic classification module for automatically classifying imported photos according to a plurality of preset category labels; a picture search module for finding similar photos according to pictures provided by a user; and a text search module for finding similar photos according to text descriptions input by a user.

[0019] According to one embodiment, the automatic classification module of the album management system is configured to:

[0020] -Preset multiple category labels, and use the text feature extraction model TextModel to extract feature vectors of the labels;

[0021] -When a photo is imported, the image feature extraction model ImageModel is called to extract the feature vector of the photo; and

[0022] - By calculating the similarity between the feature vector of the photo and the feature vector of the preset category label, selecting the label with the highest similarity, and storing the label with the highest similarity together with the feature vector of the photo in the vector database.

[0023] According to one embodiment, the text feature extraction model TextModel extracts the feature vector of the tag as a text feature Feature text =TextModel(text), where Feature text It is a 512-dimensional feature vector, specifically expressed as: Feature text =[a1,a2,a 3, …,a 512 ], where Feature text is the text feature vector; TextModel(text) is a simple expression for text feature extraction, that is, the text is input into the text feature extraction model for feature extraction; a1, a2, ..., a512 are the individual values ​​in the vector.

[0024] According to one embodiment, the image feature extraction model ImageModel extracts the feature vector of the photo as the picture feature Feature image = ImageModel(image), where Feature image It is a 512-dimensional feature vector, specifically expressed as: Feature image =[b1,b2,b3,…,b 512 ], where Feature image is the image feature vector; ImageModel(image) is a simple expression for image feature extraction, that is, the image is input into the image feature extraction model for feature extraction; b1, b2, ..., b512 are the individual values ​​in the vector.

[0025] According to one embodiment, the image search module is configured to:

[0026] -After the user provides a picture, the image feature extraction model ImageModel is called to extract the feature vector of the picture;

[0027] - calculating the similarity between the feature vector of the one picture and the feature vectors of all the photos stored in the vector database;

[0028] - Sort the similarities from large to small, and return the system paths of the 10 photos corresponding to at least the top 2 similarities, for example the top 10, in that order (if there are less than 10 photos in the system, return the system paths of all photos in that order from large to small similarities), and display them to the user.

[0029] According to one embodiment, the similarity is calculated using the following formula:

[0030]

[0031] Among them, Similarity is the cosine similarity between the image feature vector and the text feature vector; Feature image is the image feature vector; Feature text is the text feature vector.

[0032] According to one embodiment, the text-based image search module is configured to:

[0033] -After the user inputs a text description, the text feature extraction model TextModel is called to extract the feature vector of the text;

[0034] - calculating the similarity between the feature vector of the text and the feature vectors of all photos stored in the vector database;

[0035] - Sort the similarities from large to small, and return the system paths of the 10 photos corresponding to at least the top 2 similarities, for example the top 10, in that order (if there are less than 10 photos in the system, return the system paths of all photos in that order from large to small similarities), and display them to the user.

[0036] According to one embodiment, the vector database adopts the Milvus vector database.

[0037] According to another aspect of the present invention, a photo album management method is also disclosed, comprising the following steps: deploying a multimodal CLIP model and presetting multiple category labels, wherein the multimodal CLIP model includes a text feature extraction model TextModel and an image feature extraction model ImageModel; importing photos and extracting feature vectors of the photos; automatically classifying the photos according to the preset multiple category labels; a user searches for similar photos through a picture search module based on the provided pictures; and a user searches for similar photos through a text search module description based on the input text.

[0038] According to an embodiment, the method is performed using the multimodal-based album management system as described above.

[0039] The multimodal album management system and method provided by the present invention can effectively solve the deficiencies of traditional photo classification and search, and improve the efficiency and experience of user photo management. Through the automatic classification function, users do not need to manually organize photos; through the image search and text search functions, users can quickly find the required photos, which greatly improves the convenience of photo management.

[0040] The present invention realizes an efficient album management system and an album management method through innovative multimodal algorithms and vector database technology, has broad application prospects, and can provide users with a more intelligent photo management experience.

[0041] More embodiments of the present invention can also achieve other advantageous technical effects that are not listed one by one. These other technical effects may be partially described below and are predictable and understandable to those skilled in the art after reading the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above features and advantages of these embodiments and other features and advantages and the manner in which they are achieved will become more apparent and embodiments of the present invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0043] Figure 1 The following is a schematic operation flow example of an album management system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In the following description of the drawings and specific embodiments, the details of one or more embodiments of the present invention will be described. Other features, objects and advantages of the present invention can be clearly seen from these descriptions, drawings and claims.

[0045] It should be understood that the illustrated and described embodiments are not limited in application to the details of the construction and arrangement of the components set forth in the following description or illustrated in the accompanying drawings. The illustrated embodiments may be other embodiments and may be implemented or executed in various ways. Each example is provided by explaining the disclosed embodiments rather than limiting them. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments of the present invention without departing from the scope or essence of the present invention. For example, a feature illustrated or described as part of one embodiment may be used together with another embodiment to still produce another embodiment. Therefore, the present invention discloses such modifications and variations that fall within the scope of the appended claims and their equivalent elements.

[0046] Likewise, it is to be understood that the phrases and terms used in this application are for descriptive purposes and should not be considered as limiting. The use of "include", "comprise" or "have" and variations thereof herein is intended to openly include the items listed thereafter and their equivalents and additional items.

[0047] In addition, the terms "picture" and "photograph" used in this application can be used interchangeably.

[0048] The present invention will be described in more detail below with reference to specific embodiments of the present invention.

[0049] Figure 1 The following is a schematic operation flow example of an album management system according to an embodiment of the present invention.

[0050] A multimodal album management system according to an embodiment of the present invention may include:

[0051] -Multimodal CLIP model, which may include a text feature extraction model TextModel and an image feature extraction model ImageModel;

[0052] -Vector database, which can be used to store feature vectors of photos and related information;

[0053] - Automatic classification module, which can be used to automatically classify imported photos according to multiple preset category labels;

[0054] - Image search module, which can be used to find similar photos based on the image provided by the user; and

[0055] -Text search module, which can be used to find similar photos based on the text description entered by the user.

[0056] The multimodal album management system of the present invention can realize the following functions / operations.

[0057] System deployment: Deploy the multimodal model CLIP model, which is divided into the text feature extraction model TextModel and the image feature extraction model ImageModel.

[0058] Label preset: The system presets category labels (such as "landscape", "animals", etc.), and uses TextModel to extract the 512-dimensional feature vectors of these labels and store them in the Milvus vector database. Users can also customize labels.

[0059] Photo import: supports single and batch import of photos. Each imported photo extracts feature vectors through ImageModel, calculates cosine similarity with label feature vectors, automatically tags and stores relevant information.

[0060] Search images by category: The user enters a category in the search box, and the system filters and displays the results based on the stored tags.

[0061] Image search: The user provides a picture, the system extracts its feature vector, calculates the similarity with the feature vectors of photos in the database, and returns the 10 photos with the highest similarity.

[0062] Search images by text: The user enters a text description, the system extracts the text feature vector and calculates the similarity with the photo feature vector, and returns the 10 photos with the highest similarity.

[0063] like Figure 1 As shown, an exemplary operation flow of a multimodal album management system according to an embodiment of the present invention may include the following steps.

[0064] Step 1: The album management system deploys the multimodal model CLIP model. The model can be mainly divided into two parts, the text feature extraction model TextModel and the image feature extraction model ImageModel.

[0065] Step 2: The album management system presets some category labels, such as "landscape", "animal", "person", etc. The text feature extraction model TextModel deployed in step 1 will extract features from these label texts. The extracted text features are Feature text =TextModel(text),Feature text is a 512-dimensional feature vector, the specific form is:

[0066] Feature text =[a1,a2,a 3, …,a 512 ],

[0067] Among them, Feature text is the text feature vector; TextModel(text) is a simple expression for text feature extraction, that is, the text is input into the text feature extraction model for feature extraction; a1, a2, ..., a512 are the individual values ​​in the vector.

[0068] The above feature vectors and the corresponding label text are stored in a vector database, and the vector database can use the Milvus vector database. In addition, the user can also set a text description as a category label; like the above preset labels, the extracted feature vectors and the corresponding label text are stored in the vector database.

[0069] Step 3: Import photos. Import the images / photos into the album for management. The system supports single and batch import. For each imported photo, the album management system will call the image feature extraction model ImageModel deployed in step 1 to extract the image features, that is,

[0070] Feature image =ImageModel(image),Feature image is a 512-dimensional feature vector, the specific form is:

[0071] Feature image =[b1,b2,b 3, …,b 512 ],

[0072] Among them, Feature image is the image feature vector; ImageModel(image) is a simple expression for image feature extraction, that is, the image is input into the image feature extraction model for feature extraction; b1, b2, ..., b512 are the individual values ​​in the feature vector.

[0073] Featureimage The similarity comparison algorithm of the vector database and all the extracted label feature vectors in step 2 are used to calculate the similarity. The cosine similarity can be used. The formula is as follows:

[0074]

[0075] Among them, Similarity is the cosine similarity between the image feature vector and the text feature vector; Featureimage is the image feature vector of the imported photo; Featuretext is the text feature vector of the preset label.

[0076] The vector database will give the label text information with the highest similarity. The extracted image features, image related information (such as photo name, storage time, etc.) and calculated label text information will be stored in the vector database as the data of the photo in the system.

[0077] Step 4: Use the category search. Enter the category in the search box, and the database in the album management system will filter the data according to the tags in the stored image data, return all the results of the tag, and display them.

[0078] Step 5: Use image search. The user prepares an image image1 in advance and clicks the image search function. The album management system will call the image feature extraction model ImageModel deployed in step 1 to extract the image features of the image. The extracted image features are also 512-dimensional feature vectors, named Feature1, and the formula is as follows:

[0079] Feature1=ImageModel(image1),Feature1=[i1,i2,i 3, …,i 512 ]

[0080] Feature1 represents the image feature vector of image1. ImageModel(image1) is a simple expression for extracting the image features of image1. i1,i2,i 3, …,i 512 Represents each value in the feature vector. Feature1 calculates the similarity of the corresponding feature vectors of all the photos extracted in step 3 through the similarity comparison algorithm of the vector database. The cosine similarity can be used, and the calculation formula is the same as that in step 3. The vector database sorts the results from large to small by similarity, takes the first 10 data, and finally extracts the pre-stored picture name to locate the picture, and finally displays it.

[0081] Step 6: Use text search. The user enters the text text2 in the search box and clicks search. The album management system will call the text feature extraction model TextModel deployed in step 1 to extract the text features of the text. The extracted text features are also 512-dimensional feature vectors, named Feature2, and the formula is as follows:

[0082] Feature2=TextModel(text2),Feature2=[t1,t2,t 3, …,t 512 ]

[0083] Feature2 represents the text feature vector of text2. TextModel(text2) is a simple expression for extracting text features of text2. t1, t2, t3, …, t 512 Represents each value in the feature vector.

[0084] Feature2 uses the similarity comparison algorithm of the vector database to calculate the similarity with the corresponding feature vectors of all the images / photos extracted in step 3. The cosine similarity can be used, and the calculation formula is the same as step 3. The vector database sorts the results from large to small by similarity, takes the first 10 data, and finally extracts the pre-stored picture name to locate the picture, and finally displays it.

[0085] The foregoing description of several embodiments of the present invention is presented for illustrative purposes. The foregoing description is not intended to be exhaustive, nor is it intended to limit the present invention to the precise steps and / or forms disclosed, and it is apparent that many modifications and variations may be made in light of the above teachings. The scope of the present invention and all equivalents are intended to be defined by the appended claims.

Claims

1. A photo album management system based on multimodality, characterized in that: The photo album management system comprises: The multimodal CLIP model includes a text feature extraction model TextModel and an image feature extraction model ImageModel; Vector database, used to store feature vectors of photos and related information; Automatic classification module, used to automatically classify imported photos according to multiple preset category labels; Image search module, used to find similar photos based on the image provided by the user; and The text-based image search module is used to find similar photos based on the text description entered by the user.

2. The photo album management system according to claim 1, characterized in that: The automatic classification module of the album management system is configured to: -Preset multiple category labels, and use the text feature extraction model TextModel to extract feature vectors of the labels; -When a photo is imported, the image feature extraction model ImageModel is called to extract the feature vector of the photo; and - By calculating the similarity between the feature vector of the photo and the feature vector of the preset category label, selecting the label with the highest similarity, and storing the label with the highest similarity together with the feature vector of the photo in the vector database.

3. The photo album management system according to claim 2, characterized in that: The text feature extraction model TextModel extracts the feature vector of the label as a text feature vector Feature text =TextModel(text), where Feature text It is a 512-dimensional feature vector, specifically expressed as: Feature text =[a1,a2,a3,…,a 512 ], TextModel(text) is a simple expression for text feature extraction, a1, a2, ..., a 512 are the values ​​in the text feature vector.

4. The photo album management system according to claim 2, characterized in that: The image feature extraction model ImageModel extracts the feature vector of the photo as the picture feature vector Feature image = ImageModel(image), where Feature image It is a 512-dimensional feature vector, specifically expressed as: Feature image =[b1,b2,b3,…,b 512 ], ImageModel(image) is a simple expression for image feature extraction, b1, b2, ..., b 512 are the values ​​in the image feature vector.

5. The photo album management system according to any one of claims 1 to 4, characterized in that: The image search module is configured to: -After the user provides a picture, the image feature extraction model ImageModel is called to extract the feature vector of the picture; - calculating the similarity between the feature vector of the one picture and the feature vectors of all the photos stored in the vector database; - Sort the similarities from large to small, return the system paths of the corresponding photos with at least the top 2 similarities in the order of similarity, and display them to the user.

6. The photo album management system according to claim 5, characterized in that: The similarity is calculated using the following formula: Among them, Similarity is the cosine similarity between the image feature vector and the text feature vector, Feature image is the image feature vector, Feature text is the text feature vector.

7. The photo album management system according to any one of claims 1 to 4, characterized in that: The text-based image search module is configured to: -After the user inputs a text description, the text feature extraction model TextModel is called to extract the feature vector of the text; - calculating the similarity between the feature vector of the text and the feature vectors of all photos stored in the vector database; - Sort the similarities from large to small, return the system paths of the corresponding photos with at least the top 2 similarities in the order of similarity, and display them to the user.

8. The photo album management system according to any one of claims 1 to 4, characterized in that: The vector database adopts the Milvus vector database.

9. A photo album management method, characterized in that: The method comprises the following steps: Deploy a multimodal CLIP model and preset multiple category labels, wherein the multimodal CLIP model includes a text feature extraction model TextModel and an image feature extraction model ImageModel; Importing a photo and extracting a feature vector of the photo; Automatically classifying the photos according to the preset multiple category tags; Users search for similar photos based on the provided pictures through the image search module; and Users can search for similar photos based on the text they input through the text search module.

10. The method according to claim 9, characterized in that The method is performed using the multimodal-based album management system described in any one of claims 1-8.

Images