A method and system for multimodal feature recommendation

Abstract

The application discloses a kind of method and system for multimodal feature recommendation, comprising: extracting three kinds of features of target artwork, including: context data, size data, image data;Attention network is constructed by self-attention mechanism;Attention network uses vector to carry out weight calculation;The similarity of three kinds of features is sorted in ascending order in artwork recommendation candidate set, then the serial number obtained is weighted average by weight matrix, by modeling target artwork and candidate artwork set, potential features are mined, the correlation between the similarity of three kinds of features is found, and then the weight distribution between the three similarity serial numbers is determined, in accordance with the final similarity of artwork from large to small, form artwork recommendation set.The application has the advantages that: solve the problem of cold start, effectively fuse multimodal features, applied to artwork multimodal feature recommendation, and the recommendation effect is better.

Description

Technical Field

[0001] This invention relates to the field of intelligent information recommendation technology, and in particular to a method and system for multimodal feature recommendation. Background Technology

[0002] In recent years, with the rapid development of the internet, modern society has become an information- and digital society, with data flooding the world and information overload becoming the norm. However, faced with massive amounts of data, users' utilization rate of information has actually decreased, resulting in the problem of information overload. Recommendation systems are one of the key technologies for effectively solving this problem. In fact, with the rapid development of the internet, the Internet of Things (IoT), and cloud computing technologies, personalized recommendation systems have become standard features of internet products. The internet information that users encounter in e-commerce, video, news, and music is now closely related to recommendation systems. As early as 1992, research began on a system for email filtering, which used collaborative filtering to filter out emails that users disliked. In 1994, the Resnick online news system marked the beginning of recommendation systems as an independent research field. This system made two main contributions: first, the first proposal of a recommendation algorithm based on collaborative filtering (CF); and second, the establishment of a mathematical model for recommendation tasks. In 1998, item-based collaborative filtering algorithms emerged. These algorithms could quickly update recommendations based on new items browsed by users and provide simple interpretations of the results. Between 2000 and 2010, companies like Netflix, Google, and Facebook launched online websites based on recommendation systems, promoting the application and development of these systems and providing personalized recommendation services. In 2012, deep learning technology was gradually applied to the field of recommendation systems. In 2016, Google's recommendation team proposed the Wide & Deep model, which combines the advantages of both wide and deep models for recommendation ranking. The wide model handles the memory task, while the deep model handles the inductive generalization task. This was a successful attempt to apply deep learning to recommendation systems. In 2020, a two-dimensional classification scheme was provided to reasonably categorize recommendation systems. Several typical deep learning recommendation models were also introduced, and their advantages and disadvantages were analyzed. From email filtering to personalized product recommendations, recommendation systems are ubiquitous in our daily lives. Whether it's news, videos, shopping, or travel, recommendation systems help us deal with information overload and interpret user interests in some way.

[0003] Existing technology 1

[0004] Content-based recommendation (CB) algorithms calculate the similarity between item features and then recommend items based on user preferences. For example, when recommending artwork... Figure 1 As shown, the recommendation process can be described as follows: First, select the features of the artworks involved in the similarity calculation, such as the artwork name and type. Through similarity calculation, it is found that artwork 2 and artwork 3 are similar. Then, in the user's historical interest preferences, it is found that user 3 likes artwork 3. Since artwork 2 and artwork 3 have a high similarity, it is determined that user 3 may also be interested in artwork 2. Therefore, artwork 2 is recommended to user 3.

[0005] Disadvantages of existing technology 1

[0006] ① Because it only makes recommendations based on the user's historical preferences, it may ignore changes in the user's interests and fail to uncover their potential hobbies; ② It only makes recommendations based on a single content feature and does not take into account content features of more modalities, which will make the recommendation results inaccurate.

[0007] Existing technology 2

[0008] Collaborative filtering (CF) recommendation algorithms are based on the core idea of ​​"grouping similar things together." This model treats all users as a whole, assigning different weights to each user based on their position in the rating matrix and their historical behavior, thereby achieving personalized recommendations. It can be divided into two categories: ① User-based collaborative recommendation (UCF), which achieves recommendations by calculating the similarity between users, such as... Figure 2 As shown. It is known that users 1 and 2 are similar to user 3, and both users 1 and 2 like artwork 1. Therefore, according to UCF (User-Centered Collaborative Recommendation), artwork 1 will be recommended to user 3; ② Item-based collaborative recommendation (ICF) achieves recommendations by statistically analyzing the similarity of items in user preferences, such as... Figure 3 As shown. It is known that both User 1 and User 2 like Artwork 1 and Artwork 2. Based on this, it is determined that Artwork 1 and Artwork 2 are similar. Therefore, if User 3 likes Artwork 2, according to ICF, Artwork 1 will be recommended to User 3.

[0009] Disadvantages of existing technology 2

[0010] Recommendation algorithms based on collaborative filtering have the following drawbacks: ① Cold start problem: unable to recommend new items or new users; ② Poor scalability of the algorithm.

[0011] Existing technology three

[0012] Recommendation algorithms based on association rules calculate similarity by relating items. These relationships are represented in two ways: frequent itemsets and association rules. As the name suggests, frequent itemsets refer to the relationships between multiple elements. A set of elements that appear frequently is called a frequent itemset, such as {wine, diapers, soy milk} in Table 1. Association rules, on the other hand, refer to the relationships between two elements, such as diapers and wine in Table 1, indicating that someone who buys diapers is likely to buy wine as well.

[0013] Table 1 Association Analysis Table

[0014]

[0015] Disadvantages of existing technology three

[0016] The disadvantages of recommendation algorithms based on association rules are: ① there is a cold start problem for new users and new items; ② for popular products, the recommendations may be "biased", resulting in inaccurate recommendation results. Summary of the Invention

[0017] This invention addresses the shortcomings of existing technologies by providing a method and system for multimodal feature recommendation. It solves the cold start problem for new items and, by incorporating user behavior features for recommendation, also solves the cold start problem for new users.

[0018] This invention is based on an improvement to the content recommendation algorithm. The key improvement is the fusion recommendation of multimodal content features. The algorithm is designed with an attention-weighted ranking fusion mechanism to effectively fuse multimodal features.

[0019] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0020] A method for multimodal feature recommendation includes the following steps:

[0021] S1: Extract three features of the target artwork, including: contextual data, size data, and image data;

[0022] S2: Constructing an attention network through a self-attention mechanism;

[0023] S3: The attention network uses three vectors, Q, K, and V, to calculate the weights;

[0024] Q represents the three features of the target artwork, K represents the three features of the artworks in the recommended candidate set other than the target artwork in the database, and V represents the similarity index value of the three features of the artworks in the recommended candidate set.

[0025] S4: Sort the similarity of the three features in the art recommendation candidate set from smallest to largest, and then take a weighted average of the obtained numbers through a weight matrix. By modeling the target artwork and the candidate artwork set, we can mine potential features, find the correlation between the similarity of the three features, and then determine the weight distribution between the three similarity numbers.

[0026] S5 uses a weighted sorting algorithm, and its steps are as follows:

[0027] S51: Read the similarity of three features of an artwork;

[0028] S52: Construct an artwork feature similarity matrix;

[0029] S53: For the three similarity features of artworks, generate serial numbers vertically in ascending order;

[0030] S55: Construct an artwork similarity index matrix;

[0031] S55: Use an attention network to learn the weights of each feature of an artwork laterally;

[0032] S56: Calculate a weighted average of the similarity scores of the three features of the artwork;

[0033] S57: Normalize the obtained result values ​​to represent the final similarity.

[0034] S6: Sort the artworks from highest to lowest based on their final similarity to form a set of recommended artworks.

[0035] Furthermore, the computation process of the attention network in S3 is as follows:

[0036] S31: Input the similarity of the three features of the target artwork;

[0037] S32: Initialize the Q, K, and V matrices using linear transformations;

[0038] S33: Calculate the correlation between Q of the target artwork and K in all candidate sets;

[0039] S34: Combine the obtained correlation with V to perform weight calculation and transformation;

[0040] S35: Perform Softmax normalization on the weights;

[0041] S36: Output the attention weight matrix for the artwork.

[0042] Furthermore, the similarity of the artworks described in S4 is calculated as follows:

[0043]

[0044]

[0045] Where n represents the number of candidate artworks for recommendation, m represents the number of features involved in the artwork recommendation, and ω ij This represents the weight coefficient of the j-th feature of the i-th artwork. R represents the similarity index of the j-th feature of the i-th artwork. i sim represents the index of the i-th artwork after feature similarity weighted fusion. i This represents the similarity between the i-th artwork and the target artwork.

[0046] Furthermore, the similarity of size data is obtained through cosine distance, the similarity of context data is obtained through the Doc2vecDBOW model, and the similarity of image data is obtained through the TCRN model.

[0047] The present invention also discloses a system for multimodal feature recommendation, which consists of a database, a server, and a client. This system is used to implement the above-mentioned method.

[0048] The database includes: user attribute table, user behavior table, administrator attribute table, and artwork attribute table;

[0049] The user attribute table is mainly used to store attribute information related to the user;

[0050] The user behavior table is mainly used to collect interaction data between users and the system;

[0051] The administrator attribute table is used to store basic information related to the administrator;

[0052] The artwork attribute table is used to store the attribute information of artworks. The table mainly includes fields such as artwork ID, artwork name, artist name, artwork category, artwork subcategory, artwork description, artwork size, and artwork URL. The artwork URL field can be used to download the corresponding image data of the artwork.

[0053] The server-side functions include: registration and login functions, art uploading, modification, deletion, search, collection and recommendation, and an art backend management system designed as a visual operation page for the database.

[0054] The client supports user registration and login, art browsing, art search, art collection, and art recommendation functions.

[0055] Furthermore, the interactive pages of the database in the system are designed using ElementUI technology, and the database is a relational database of MySQL; the server is developed using VS Code software, mainly using the Vue and Flask frameworks for design; the client is iOS system software, developed using Xcode, and the client interface design is implemented using the SwiftUI framework.

[0056] Compared with the prior art, the advantages of the present invention are as follows:

[0057] 1. The proposed algorithm does not have the problem of cold start for new items. It integrates user behavior features for recommendation and also solves the problem of cold start for new users.

[0058] 2. Multimodal content features were fused for recommendation, and an attention-weighted ranking fusion mechanism was designed to effectively fuse multimodal features.

[0059] 3. When applied to multimodal feature recommendation of artworks, the recommendation effect is better. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the structure of a content-based recommendation algorithm in the existing technology.

[0061] Figure 2 This is a graph of an existing technology two recommendation algorithm based on user collaboration;

[0062] Figure 3 This is a graph of the existing technology 3, which is a recommendation algorithm based on item collaboration.

[0063] Figure 4 This is a flowchart of a method and system for multimodal feature recommendation according to an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of the self-attention network architecture according to an embodiment of the present invention;

[0065] Figure 6 This is a matrix diagram of the weighted sorting algorithm according to an embodiment of the present invention;

[0066] Figure 7 This is an ER model diagram of the recommendation system according to an embodiment of the present invention;

[0067] Figure 8 This is a timing diagram of the recommendation system App in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0069] When using multimodal features for art recommendation, different art features have different weights when fused for final recommendation. To address this issue, this embodiment designs a weighted ranking algorithm fusion mechanism with attention. The attention mechanism is used to model the preference weights of different features (including visual features, contextual features, and size features) to achieve reasonable weight allocation. The algorithm model is named VstarNet, where V represents visual features, S represents size features, T represents contextual features, A represents attention, R represents recommendation, and Net represents the neural network model. The implementation process of the VstarNet model is as follows: Figure 4 As shown.

[0070] Since the recommendation algorithm of this invention is based entirely on the recommendation of artwork content, and the three similarities of artworks extracted and calculated in the early stage belong to sequential data and have sequential characteristics, the self-attention mechanism is used to construct the attention network in the research. The core of the self-attention mechanism is to calculate its own weight by analyzing the potential feature connections within the learning data. This is also the biggest difference between the self-attention mechanism and the ordinary attention mechanism. Figure 5 The attention network architecture used in this invention is shown.

[0071] like Figure 5 As shown, the attention network uses three vectors, Q, K, and V, for weight calculation, where Q represents the query, K is the key, and V is the value, and KV represents a key-value pair. In this embodiment, Q represents the three features of the target artwork, K represents the three features of the artworks in the recommendation candidate set, and V represents the similarity index value corresponding to the three features of the artworks in the recommendation candidate set. The calculation process of this network is as follows:

[0072]

[0073] The core idea of ​​the ranking method is to represent the final result value by averaging the ranking index of the probability values ​​(rather than the probability values ​​themselves). In this embodiment, the attention-weighted ranking algorithm sorts the similarity of the three features in the art recommendation candidate set from smallest to largest (starting from 0), and then performs a weighted average of the resulting indexes. The weight allocation is implemented through an attention network. By modeling the target artwork and the candidate artwork set, latent features are mined, the correlation between the three feature similarities is found, and thus the weight allocation among the three similarity indices is determined. For example, the final similarity between artwork M and artwork N is sim MN RMN sim1 represents the final similarity index of artworks M and N, and sim1 represents the visual similarity between artworks M and N. sim1 represents the visual similarity index of artworks M and N, and sim2 represents the contextual similarity of artworks M and N. sim3 represents the context similarity index of artworks M and N, and sim3 represents the size similarity of artworks M and N. Let represent the size similarity index of artworks M and N. The final similarity score between artworks M and N is calculated as shown in Equation 1 below:

[0074]

[0075] Where ω1, ω2, and ω3 represent the attention weights for the artwork's visual appeal, context, and size, respectively, R min R max These represent the minimum and maximum similarity indices after weighted averaging, respectively. The above formula explains the principle of the weighted ranking algorithm from the micro perspective of two artworks. The macro perspective of the entire artwork recommendation algorithm is represented by the following formula 2:

[0076]

[0077] Where n represents the number of candidate artworks for recommendation, m represents the number of features involved in the artwork recommendation, and ω ij This represents the weight coefficient of the j-th feature of the i-th artwork. R represents the similarity index of the j-th feature of the i-th artwork. i sim represents the index of the i-th artwork after feature similarity weighted fusion. i Let represent the similarity between the i-th artwork and the target artwork. The execution flow of the weighted sorting algorithm in this embodiment is as follows:

[0078]

[0079]

[0080] The artworks are sorted from highest to lowest based on their final similarity, forming a set of recommended artworks. Figure 6 The matrix representation of the above formula provides a more intuitive understanding of the algorithm's computational principle. In summary, the weighted sorting algorithm proposed in this embodiment can be extended to multiple features, not just three, and the weight coefficients can be obtained through other methods, not necessarily using an attention network. This provides ideas and a theoretical foundation for subsequent related research.

[0081] Software component:

[0082] The basic development environment for the software system is macOS 13, and the primary development language is Python. The system architecture is a client / server (C / S) structure, employing a front-end / back-end separation technology. To achieve the best user experience, the entire software system consists of three parts: a database, a server, and a client. The database's interactive pages are designed using ElementUI technology, and the database itself is a MySQL relational database. The server is developed using VS Code, primarily employing the Vue and Flask frameworks. The client is an iOS system software, developed using Xcode, and its interface design is implemented using the SwiftUI framework. Specific configurations are shown in Table 2.

[0083] Table 2 System Development Environment Configuration

[0084]

[0085] Database design

[0086] (1) ER model diagram

[0087] An ER diagram, also known as an entity-relationship diagram, is a conceptual model used to reflect the real world, describing the attributes and relationships between entities. From the requirements analysis, we can see that the art recommendation system has three entities: artworks, users, and administrators. Users can browse and collect artworks, and administrators can manage users and artworks. The relationship between users and artworks is m:n, the relationship between administrators and users is 1:m, and the relationship between administrators and artworks is 1:n. The ER model diagram of the art content recommendation system is as follows: Figure 7 As shown, entities are represented by yellow boxes, relationships between entities are represented by green diamonds, and attributes of each entity are represented by gray ellipses.

[0088] (2) Data table design

[0089] The ER diagram shows that the attributes of the three entities correspond to three data tables: the user attribute table, the administrator attribute table, and the artwork attribute table. Furthermore, the relationships between the entities can be analyzed using a user behavior table to collect interaction data between users and artworks.

[0090] 1) User attribute table

[0091] The user attribute table is mainly used to store attribute information related to users. The specific table structure is shown in Table 3.

[0092] Table 3 User Attribute Table

[0093]

[0094]

[0095] 2) User Behavior Table

[0096] The user behavior table is mainly used to collect interaction data between users and the system. The specific table structure is shown in Table 4.

[0097] Table 4 User Behavior Table

[0098]

[0099] 3) Administrator Attribute Table

[0100] Corresponding to the user attribute table, the administrator attribute table is used to store basic information related to the administrator. The specific table structure is shown in Table 5.

[0101] Table 5 Administrator Attribute Table

[0102]

[0103] 4) Artwork Attribute Table

[0104] As the name suggests, the artwork attribute table is used to store the attribute information of artworks. According to the description in Chapter 3 of this embodiment, the table mainly includes fields such as artwork ID, artwork name, artist name, artwork category, artwork subcategory, artwork description, artwork size, and artwork URL. The artwork URL field can be used to download the corresponding image data of the artwork. The specific table structure is shown in Table 6.

[0105] Table 6. Artwork Attributes Table

[0106]

[0107] The above four data tables were built in the MySQL database, and an ETL process was created to store historical data during the recommendation process, laying a good data foundation for the system's recommendation effect.

[0108] Server-side implementation

[0109] The server-side, or backend management system of the entire software, is responsible for completing the various functional modules of the system. The server-side of the art recommendation system mainly implements the registration and login of administrators and users, the management of artworks, and the recommendation of artworks. This paper mainly introduces the technical framework used in the design of the recommendation system server-side and the implementation of the corresponding functions.

[0110] (1) Technical Framework

[0111] Python was used as the server-side development language, VS Code was used as the development tool, the Flask framework was used to complete the functions of uploading, deleting, storing, recommending and user login of artworks, and the ElementUI framework based on Vue 2.0 was used to design and build the server-side backend visual management page.

[0112] Flask, a lightweight web development framework based on Python, emerged in 2010. It is essentially a kernel, with almost all additional functionality requiring third-party extensions. Due to its innovative design philosophy of "cross-platform" technology, it enables the rapid construction of highly scalable and flexible systems. While not the most famous web framework, Flask is arguably one of the most resilient. It does not pre-install a database; users can choose MySQL or NoSQL databases.

[0113] ElementUI is a desktop UI framework based on Vue.js 2.0. It was developed and launched by the front-end team of Ele.me, a Chinese company. Its visual design is more in line with the aesthetic experience of Chinese people, and it is one of the most popular and widely used UI frameworks in the field of front-end design in my country.

[0114] (2) Server-side function implementation

[0115] The main functions implemented include registration and login, artwork uploading, modification, deletion, searching, and recommendation. An artwork backend management system was designed, essentially a visual database operation page. The `Art` module is defined to implement artwork management functions, where `Art_add` represents artwork uploading, `Art_update` represents artwork modification, `Art_delete` represents artwork deletion, `Art_list` represents artwork searching, `Art_collect` represents artwork collection, and `Art_Rcommand` represents artwork recommendation.

[0116] ① Registration and login function

[0117] Define a Login module to handle registration and login functionality. The Adm class manages administrator registration and login, and the User class manages user registration and login. The specific implementation steps are as follows, using xxx to represent the Adm or User class:

[0118] Step 1: Administrator & User Registration. Call the Login.xxx.setName() method to set the name, Login.xxx.setAccount() to set the login account, Login.xxx.setPassword() to set the login password, Login.xxx.setSex() to set the gender of the user & administrator, and Login.xxx.setAge() to set the age, etc.

[0119] Step 2: After successful registration, design the login page using ElementUI and SwiftUI respectively;

[0120] Step 3: Enter your username and password on the login page to log in. Call the `Login.xxx.verify_password()` method to verify the username and password. If the entered username is correct and matches the password successfully, continue to the next step.

[0121] Step 4: The server calls the Login.xxx.post() method to submit a login request. If the HTTP code is 400, it means that the account does not exist and you need to register first; if the HTTP code is 401, it means that the password is wrong; if the HTTP code is 201, it means that the login is successful and you are taken to the system page.

[0122] ② Artwork upload function

[0123] It supports uploading information such as artwork name, artist, category, subcategory, artwork description, and artwork image. The artwork image must be uploaded in Base64 format. By clicking the upload button on the operation page and filling in the relevant artwork information, the `request.get_data()` method is used to retrieve the page interaction information. The system verifies whether the collected artwork name and artwork image are empty. If either is empty, the system will display the message "Name and image cannot be empty". Otherwise, the system uses the `Art.query.filter().add()` method to verify whether the artwork already exists in the artwork attribute table. If the HTTP return code is 401, the system will report "Add failed, the artwork already exists". If the HTTP return code is 201, the system will indicate "Add successful" and add all the artwork information to the artwork attribute table.

[0124] ③ Artwork editing function

[0125] It supports modifying the information of uploaded artworks. By clicking the modify button on the operation page, you can modify the basic information of the artwork, call the request.get_data() method to obtain the page interaction information, and use the Art.query.filter().update() method to submit the modification request. If the HTTP return code is 200, it means "modification successful" and the artwork information will be updated in the artwork attribute table.

[0126] ④ Artwork deletion function

[0127] Clicking the delete button next to the artwork triggers the request.get_data() method to retrieve page interaction information. The Art.query.filter().delete() method is then used to submit a deletion request. If the HTTP return code is 204, a "deletion successful" message is displayed, and the artwork is removed from the data table.

[0128] ⑤ Artwork search function

[0129] There is a search bar in the upper left corner of the operation page. You can search by entering the name of the artwork and other information. The request.get_data() method is called to obtain the page interaction information. The Art.query.filter().all() method is used to submit the search request. The data table is matched. If the HTTP return code is 200, it means "operation successful" and the search results are displayed on the page.

[0130] ⑥ Art collection function

[0131] Art collection is one of the client's features. When you enter the artwork details page and click the "Collect" button, the `request.get_data()` method is called to get the page interaction information. The `collection_add()` method is used to add the artwork to your collection, the `collection_delete()` method is used to remove it from your collection, and the `collection_list()` method is used to display the list of your artwork collection.

[0132] ⑦ Art Recommendation Function

[0133] This recommendation system is based on artwork content. It calculates the fusion similarity of the target artwork with artworks in the database in terms of multimodal features, sorts them in descending order of similarity, and selects the top-10 artworks to form a recommendation list. The specific algorithm implementation and the determination of the N value are described in detail in Chapter 4. It mainly involves processes such as user information acquisition, user browsing information acquisition, and user collection information acquisition. The specific implementation steps are as follows:

[0134] Step 1: Obtain the information of the artworks currently viewed by the user through the user's operation behavior on the client, and obtain ArtFavoriteList (the user's list of artwork collections) from the user behavior table according to the user ID;

[0135] Step 2: Use the Art_Rcommand.getart() method to obtain all artwork information and extract the corresponding image visual, context, and size information to form a multimodal feature artwork recommendation candidate set;

[0136] Step 3: By analyzing user behavior such as browsing and collection information, different attention weights are assigned to the three modalities of the artwork;

[0137] Step 4: Combine the VstarNet recommendation algorithm from Chapter 4 to obtain the recommended list for the user under the current artwork.

[0138] Client-side implementation

[0139] The art recommendation system client is an iOS app that supports user registration and login, art browsing, art search, art collection, and art recommendation. This section mainly introduces the technical framework required for the client implementation and the implementation of the above functions.

[0140] (1) Technical Framework

[0141] The client application uses Xcode as its development tool and Swift as its programming language for iOS. Swift is a new programming language introduced by Apple at WWDC 2016, and it offers the following advantages:

[0142] ① High efficiency. Swift overcomes the long-standing dilemma of other languages ​​being unable to achieve both. C boasts top-notch execution performance, but its development speed is far slower than scripting languages ​​like JavaScript and Python. Python offers very fast development speed, but its execution performance cannot compare to C and Java. Swift strikes a balance between development speed and execution performance. For example, a commonly used search algorithm implemented in Swift is 2.6 times faster than in Objective-C and 8.4 times faster than in Python 2.7.

[0143] ② High security. Swift prioritizes security from its inception, and many of its design features stem from security considerations. Furthermore, Xcode can dynamically compile Swift code, revealing bugs in real time.

[0144] ③ Concise and easy to understand. Swift's syntax is very clear and relatively easy to read and write. This is because Swift has removed many legacy conventions, such as semicolons, line breaks, and parentheses. Swift code is closer to simple English, making coding more natural and allowing developers to spend less time finding problematic code.

[0145] (2) Client Function Implementation

[0146] The client-side main functions include user registration and login, artwork details, collection, and recommendation features. After users log in to the app using their account and password, the front-end display and call flow of the recommendation system app are represented using a UML sequence diagram, such as... Figure 8 As shown.

[0147] The specific implementation steps are as follows:

[0148] Step 1: Users log in with their account and password, enter the recommendation system homepage, and browse artworks by category;

[0149] Step 2: Users can click on artworks that interest them to enter the artwork's details page, where they can add the artwork to their favorites or recommend it.

[0150] Step 3: When the user clicks the "Favorite" button on the details page, the server-side Art_collent class will be called to perform the artwork favorite operation, and then the artwork will appear on the favorites page;

[0151] Step 4: When a user clicks the "You May Also Like" button on the details page, the server-side Art_Rcommand class will be called to calculate the similarity of artworks, sort them, and display the recommendation results.

[0152] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A method for multi-modal feature recommendation, characterized in that, The method comprises the following steps: S1: extracting three features of the target artwork, including context data, size data, and image data; S2: constructing an attention network through a self-attention mechanism; S3: the attention network uses Q, K, and V vectors to calculate weights; Q represents the three features of the target artwork, K represents the three features of the artworks in the recommendation candidate set in the database except the target artwork, and V represents the sequence number value of the similarity of the three features of the artworks in the recommendation candidate set; The calculation process of the attention network is as follows: S31: inputting the similarity of the three features of the target artwork; S32: using linear transformation to initialize the Q, K, and V matrices; S33: calculating the correlation of Q of the target artwork and K of all candidate sets; S34: performing weight calculation and conversion on the obtained correlation combined with V; S35: performing Softmax normalization operation on the weights; S36: outputting the artwork attention weight matrix; S4: sorting the similarity of the three features in the artwork recommendation candidate set from small to large, then performing weighted average on the obtained sequence numbers through the weight matrix, modeling the target artwork and the candidate artwork set, mining potential features, finding the correlation between the similarities of the three features, and then determining the weight distribution between the sequence numbers of the three similarities; S5 uses a weighted ranking algorithm, and the steps are as follows: S51: reading the similarity of the three features of the artwork; S52: constructing an artwork feature similarity matrix; S53: generating sequence numbers from small to large in the vertical direction according to the similarity of the three features of the artwork; S55: constructing an artwork similarity sequence number matrix; S55: using the attention network to learn the feature weights of the artwork in the horizontal direction; S56: performing weighted average on the sequence numbers of the three features of the artwork; S57: performing normalization on the obtained result value to represent the final similarity; S6: sorting the final similarity of the artwork from large to small to form an artwork recommendation set.

2. The method for multi-modal feature recommendation of claim 1, wherein: The similarity of the size data is obtained through the cosine distance, the similarity of the context data is obtained through the Doc2vecDBOW model, and the similarity of the image data is obtained through the TCRN model. ， ， wherein n represents the number of the art recommendation candidate set, m represents the number of the features participating in the art recommendation, represents the weight coefficient of the mth feature of the nth art, represents the similarity order number of the mth feature of the nth art, represents the order number after the feature similarity weighting fusion of the nth art, represents the similarity of the nth art to the target art.​​​​​​ 3. The method for multi-modal feature recommendation of claim 1, wherein: The system is composed of a database, a server, and a client, and is used for implementing the method for multi-modal feature recommendation according to any one of claims 1 to 3; 4. A system for multi-modal feature recommendation, characterized in that: The database comprises a user attribute table, a user behavior table, an administrator attribute table, and an artwork attribute table; The user attribute table is used to store attribute information related to the user; The user behavior table is used to collect interaction data between the user and the system; The administrator attribute table is used to store basic information related to the administrator; The artwork attribute table is used to store attribute information of the artwork, and the table comprises the fields of work ID, work name, author name, work category, work subcategory, work description, work size, and work URL, wherein the work URL field is used to download the corresponding image data of the work; The system is composed of a database, a server, and a client, and is used for implementing the method for multi-modal feature recommendation according to any one of claims 1 to 3; The database comprises a user attribute table, a user behavior table, an administrator attribute table, and an artwork attribute table; The user attribute table is used to store attribute information related to the user; The user behavior table is used to collect interaction data between the user and the system; The administrator attribute table is used to store basic information related to the administrator; The artwork attribute table is used to store attribute information of the artwork, and the table comprises the fields of work ID, work name, author name, work category, work subcategory, work description, work size, and work URL, wherein the work URL field is used to download the corresponding image data of the work; The server functions include registration and login functions, artwork uploading, modification, deletion, search, collection and recommendation, and the design artwork background management system is a visual operation page of the database; The client supports user registration and login, artwork browsing, artwork search, artwork collection, and artwork recommendation functions.

5. The system for multi-modal feature recommendation of claim 4, wherein: The interactive page of the database in the system is designed using ElementUI technology, and the database uses MySQL relational database; the server uses vscode software development, uses VUE and flask framework method for design; the client is IOS system software, the development tool is Xcode, and SwiftUI framework is used to realize the interface design of the client.

Images