Multi-modal interaction and content generation method and system for intelligent photo album

Abstract

The invention relates to the technical field of man-machine interaction and intelligent content management, in particular to a multi-mode interaction and content generation method and system for an intelligent photo album. Comprising the steps of obtaining user interaction data and determining initial hotspot content; an attention mechanism model is adopted to calculate the association strength of the content subject type and the picture style feature, and a preference vector is obtained; extracting a preference vector change trend, adjusting an attenuation element weight, and generating an updated display priority sequence; determining a photo album content display sequence through the display priority sequence; acquiring real-time behavior feedback data, and optimizing the preference vector to obtain a refined interest model; making a recommendation strategy according to the interest model, and generating a photo album layout template; and generating a final output sequence according to the layout template, fusing the main body type and the picture style, and outputting a personalized photo album version. According to the invention, the personalized and intelligent level of photo album display is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction and intelligent content management, and in particular to a multi-modal interaction and content generation method and system of an intelligent photo album. BACKGROUND

[0002] In the field of digital content recommendation, personalized services have become the core of improving user experience and enhancing platform competitiveness. With the diversification and dynamic changes of user needs, how to accurately match user interests and provide tailor-made experiences has become a key direction for industry development. Personalized recommendation not only affects user satisfaction, but also directly determines the user stickiness and long-term development of the platform.

[0003] However, existing recommendation methods have obvious shortcomings in dealing with changes in user interests. Although many systems can make content recommendations based on historical data, they often fail to adapt to user interest fluctuations at different times and scenarios in a timely manner. Existing technologies are mainly limited to analyzing user historical behavior, and lack of capturing and understanding real-time behavior. This limitation leads to a deviation between recommended content and user current needs, affecting user acceptance and interaction effect.

[0004] The deeper challenge lies in how to dynamically adjust the priority of content recommendation to better adapt to users' changing interest preferences. User behavior when browsing content, such as click frequency, dwell time, etc., actually reflects their immediate feedback on content. If these feedback cannot be captured and applied to the recommendation algorithm in a timely manner, there will be a disconnection between content display and user interests. For example, a user may show strong interest in landscape pictures at a certain time period and frequently click on related content, but the system still pushes irrelevant content, even after the user's interest shifts to other topics. This lag will lead to a decline in user experience and greatly reduce the effectiveness of the platform's content recommendation. SUMMARY

[0005] The present application provides a multi-modal interaction and content generation method and system of an intelligent photo album to improve the personalization and intelligence level of photo album display.

[0006] In a first aspect, the present application provides a multi-modal interaction and content generation method of an intelligent photo album, comprising: S1, obtaining interaction data of a user when browsing a photo album, and determining initial hot content according to the interaction data; S2, using an attention mechanism model to calculate the correlation strength between the content subject type and the content picture style features of the initial hot content, and obtaining a preference vector based on the correlation strength; S3, extracting a change trend from the preference vector, adjusting a decay element weight according to the change trend, and generating an updated display priority sequence; S4, determining a display order of the album content through the display priority sequence; S5, obtaining real-time behavior feedback data, and optimizing the preference vector according to the real-time behavior feedback data to obtain a refined interest model; S6, formulating a recommendation strategy according to the interest model, and generating an album layout template; S7, generating a final output sequence according to the layout template, and outputting a personalized album version by fusing the subject type and the picture style.

[0007] In a second aspect, the present application provides a multi-modal interaction and content generation system of an intelligent album, and the system comprises: A data collection and analysis module is configured to obtain interaction data of a user when browsing an album, and determine initial hot content according to the interaction data; A preference feature extraction module is configured to calculate an association strength between a content subject type and a content picture style feature of the initial hot content by using an attention mechanism model, and obtain a preference vector based on the association strength; A priority dynamic adjustment module is configured to extract a change trend from the preference vector, adjust a decay element weight according to the change trend, and generate an updated display priority sequence; A content sorting and display module is configured to determine a display order of the album content through the display priority sequence; A model real-time optimization module is configured to obtain real-time behavior feedback data, and optimize the preference vector according to the real-time behavior feedback data to obtain a refined interest model; A layout strategy generation module is configured to formulate a recommendation strategy according to the interest model, and generate an album layout template; An album fusion output module is configured to generate a final output sequence according to the layout template, and output a personalized album version by fusing the subject type and the picture style.

[0008] The technical scheme provided in the application improves the personalization degree and user experience of album content display through the multi-modal interaction and content generation method of the intelligent album. The method captures the interaction data of the user in the browsing process in real time, analyzes the correlation strength between the content and the user interest based on the attention mechanism model, and accurately matches the user interest. By dynamically adjusting the display priority and recommendation strategy, the method can respond to changes in user interest in a timely manner, avoiding the recommendation lag problem existing in traditional methods, and ensuring that the content display is always synchronized with the user demand. In addition, based on the real-time behavior feedback and interest model optimization of the user, personalized album layout templates are generated to provide users with unique and personalized visual experience. This adaptive optimization capability improves the recommendation accuracy and enhances the interaction effect between users and the platform, reduces user churn caused by inaccurate recommendations, thereby improving the user stickiness and long-term value of the platform, and further promoting the sustainable development and competitiveness of the platform.

[0009] The third aspect of the application provides a computer device, the memory stores machine readable instructions executable by the processor, when the computer device runs, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the multi-modal interaction and content generation method of the intelligent album. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical scheme of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings based on these drawings without creative labor.

[0011] Figure 1 The flowchart of the multi-modal interaction and content generation method of the intelligent album of the application; Figure 2 The flowchart of the dynamic optimization and personalized display of the intelligent album content of the application; Figure 3 The interest score comparison analysis before and after the gradient descent optimization of the application; Figure 4 The structural schematic diagram of the multi-modal interaction and content generation system of the intelligent album of the application; Figure 5 The structural schematic block diagram of the multi-modal interaction and content generation device of the intelligent album of the application. DETAILED DESCRIPTION

[0012] The embodiments of the present application provide a multi-modal interaction and content generation method and system of an intelligent photo album. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" or "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to such processes, methods, products or devices.

[0013] For ease of understanding, the specific processes of the embodiments of the present application are described below. Please refer to Figure 1 One embodiment of the multi-modal interaction and content generation method of the intelligent photo album in the embodiments of the present application includes the following steps. Step S1, obtaining interaction data of a user when browsing a photo album, and determining initial hot content according to the interaction data.

[0014] In a specific embodiment, the process of performing step S1 can specifically include the following steps: By recording the interaction data of the user when browsing the photo album, analyzing the click frequency and the stay time in the interaction data, and determining the attention degree of the user to different content; According to the attention degree, filtering out content higher than a preset attention degree threshold as initial hot content; Analyzing the initial hot content, determining the content subject type and the content picture style feature of the initial hot content; Statistically analyzing the content subject type and the content picture style feature to generate an initial content preference distribution; Calculating a matching score between the initial hot content and the initial content preference distribution; If the matching score is lower than a preset matching threshold, reanalyzing the interaction data to update the initial hot content.

[0015] Specifically, to record user interaction data while browsing photo albums, the system analyzes the click frequency and dwell time for each photo to determine the user's level of interest in different content. Click frequency and dwell time are two key indicators used by the system to filter high-interest content as initial hot topics. For example, when the click frequency exceeds a preset interest threshold or the dwell time exceeds a set standard, the system considers the content to have high interest and thus designates it as initial hot topics. The system uses image recognition algorithms to extract the subject type and stylistic features of each hot topic. Subject type can be identified using a Convolutional Neural Network (CNN). Convolutional layers are used to extract edge and texture features, while pooling layers are used to reduce dimensionality, ultimately classifying images into types such as "people" and "landscapes." Stylistic features can be extracted by calculating indicators such as color saturation and contrast. Based on the identified subject type and stylistic features, the system performs statistical analysis to generate an initial content preference distribution. For example, the system can calculate the frequency of occurrence of different subject types and stylistic features and use a Gaussian Mixture Model (GMM) to fit these features. Gaussian mixture models optimize parameters using the expectation-maximization algorithm, estimating the mean and variance of each feature to generate a preference distribution. For each feature of the preference, it can be expressed by the following formula:

[0016] in, Given eigenvalues The probability density under the following conditions It is the first The weights of a Gaussian distribution It has a mean Covariance Matrix Gaussian distribution, It is an eigenvalue. It refers to the quantity of the mixed components.

[0017] After generating an initial preference distribution, the system calculates the matching degree between each piece of content and that distribution, typically using cosine similarity or Euclidean distance to measure the similarity between features. For example, if the subject type and style features of an image match the subject type and style features preferred by the user, the image will have a higher matching score; conversely, it will have a lower score. If the matching score is lower than a preset matching threshold (e.g., 0.8), the initial hot content is considered inaccurate and needs to be updated. The matching degree can be calculated using the following formula:

[0018] in, and These are the feature vectors of the two images. Represents the dot product. and is the norm of the vector. If the matching degree is lower than the set matching threshold (e.g. 0.5), the system will re-collect the user's interaction data and adjust the attention threshold of the filtered content. Preferably, a dynamic threshold can be set, which is dynamically adjusted according to the user's historical behavior. When the matching score is lower than the preset matching threshold, new hot content is re-filtered until the accuracy reaches the preset standard. In this way, the system can continuously optimize the selection of hot content to ensure that the final displayed content is highly matched with the user's preferences. Each iteration process can improve the accuracy of personalized recommendation, thereby providing a better photo album browsing experience for the user.

[0019] In step S2, the attention mechanism model is used to calculate the correlation strength between the content subject type of the initial hot content and the content picture style feature, and the preference vector is obtained based on the correlation strength.

[0020] In a specific embodiment, the process of step S2 can specifically include the following steps: The attention mechanism model is used to calculate the correlation strength between the content subject type of the initial hot content and the content picture style feature. According to the correlation strength, the content picture style preference vector corresponding to each content subject type is calculated by using a weighted average algorithm as the initial preference vector. The initial preference vector is normalized to obtain a standardized preference representation. The inclination degree of the user to different content subject types is calculated according to the preference representation. The distribution characteristics of the user's interest are further analyzed through the inclination degree. If the distribution characteristics meet the preset concentration condition, the weight of the corresponding element in the initial preference vector is adjusted to obtain the final preference vector.

[0021] Specifically, the attention mechanism model is used to calculate the correlation strength between the content subject type of the initial hot content and the content picture style feature. The attention mechanism can automatically weight different features according to the input content features, so that the model can focus on the part of the content related to the current task. In this embodiment, the subject type and style feature of each photo are extracted by image recognition algorithm, and the attention mechanism model is used to calculate the correlation strength between each subject type and style feature. The correlation strength can be represented by the following formula:

[0022] wherein, is the correlation strength, is the transpose symbol, is the query vector, representing the feature vector of the content subject type, is the key vector, representing the feature vector of the content picture style feature, is the dimension of the key vector, To normalize the results, ensuring that the weight of each feature is between 0 and 1. By calculating the resulting correlation strength, the system will use a weighted average algorithm to calculate the preference vector of each content subject type corresponding to the content picture style. For example, assuming that the subject type of a picture is "person", and the style feature is "warm tone", according to the calculated correlation strength, the system will assign a weighted average value to the pairing of the subject type and the style feature, thereby obtaining a preliminary preference vector, which can be represented as:

[0023] wherein, is the correlation strength of the th content subject type and style feature, is the preference vector of the feature, is the number of features. The preference vector will serve as the initial preference vector, describing the user's overall preference for different subject types and style features. The initial preference vector is normalized to obtain a standardized preference representation. The normalization operation can ensure that the sum of the weights of each element is 1, thereby eliminating the bias between features. According to the standardized preference representation, the system can calculate the inclination degree of the user to different content subject types. The inclination degree can be obtained by calculating the similarity between each subject type and the user preference vector. For example, for a subject type "person", its inclination degree can be calculated by cosine similarity.

[0024] Further, through inclination degree analysis, the system can find the concentration degree of the user's interest in certain content subject types. If the user's preference for certain subject types is more concentrated (i.e. the inclination degree of certain subject types is significantly higher than that of other subject types), the system will adjust the weight of the corresponding elements in the initial preference vector to ensure better reflection of the user's current interest. This adjustment can be achieved by adjusting the weight of high inclination degree elements in the vector and reducing the weight of low inclination degree elements, thereby generating the final preference vector. For example, if the user's interest in "person" type is higher than that in "landscape" type, the system will enhance the weight of "person" type and reduce the weight of "landscape" type, thereby updating the initial preference vector.

[0025] Step S3, extract the change trend from the preference vector, adjust the weight of the decay element according to the change trend, and generate an updated display priority sequence.

[0026] In a specific embodiment, the process of performing step S3 can specifically include the following steps: extract the change trend over time from the preference vector using time series analysis method; determine whether the change trend exceeds a preset trend threshold, and if so, identify the decay element and reduce the weight value corresponding to the decay element; recompute the preference vector distribution based on the adjusted weights; calculate a matching score based on the preference vector distribution and the content feature vector of each photo, and generate a display priority sequence according to the score.

[0027] Specifically, the time series analysis method is used to extract the change trend over time from the preference vector. Time series analysis tracks and analyzes user interaction data over different time periods to identify patterns of interest changes. For example, suppose a user frequently browses landscape photos in the past month, the system will track this change and identify fluctuations in user interest. If the user's interest has changed significantly, such as suddenly showing more interest in "person" photos, the system can capture this trend and make appropriate adjustments. The system will determine whether the change trend exceeds a preset trend threshold. The trend threshold is a set standard to determine whether the user's interest has changed significantly. For example, if the set threshold is 20%, when the user's interest in a certain category increases by more than 20%, the system will consider it a change worth attention. If the change exceeds this threshold, the system will further analyze the specific content of the interest change.

[0028] Once the interest change exceeds the threshold, the system will identify the decay element, i.e. the feature whose user interest has significantly weakened. For example, if the user's interest in "person" photos has significantly decreased, the system will reduce the weight of the "person" category, and correspondingly increase the weight of "landscape" photos. These changes reflect the adjustment of user interest, and the identification of decay elements helps the system respond and update the recommended content in a timely manner. Based on the adjustment of the decay element, the system will recompute the distribution of the preference vector. The adjusted preference vector will more accurately reflect the user's current interest. For example, if the user's interest has changed, the system will increase the weight of the new preferred category and reduce the weight of the old preferred category, so as to better match the user's current needs.

[0029] Further, the system calculates a matching score based on the preference vector distribution and the content feature vector of each photo. The content feature vector includes information such as the subject type (e.g. person, landscape) and the color tone (e.g. warm, cool) of the photo. By calculating the matching degree of each photo with the user's preference, the system can determine the relative importance of each photo. Photos with higher matching scores represent a better fit with the user's interest and have higher priority. The system generates a display priority sequence according to the matching scores. Photos with higher scores will be displayed first to ensure that the user sees content that matches their current interest first. For example, the system can first display a landscape photo that best matches the user's current preference, and then display a person photo. This process ensures that the display of the album content always aligns with the user's latest interest, providing a personalized and efficient recommendation experience.

[0030] Step S4, determining the display order of the album content according to the display priority sequence.

[0031] In a specific embodiment, the process of performing step S4 can specifically include the following steps: reordering the album content according to the display priority sequence; analyzing the reordered album content, integrating the content subject type and content picture style features of each photo, and generating a content feature vector; calculating a matching score using the current user's preference vector and the content feature vector; if the matching score is lower than a preset interest threshold, adjusting the position of some elements in the display priority sequence; generating a new content ordering scheme according to the adjusted sequence; determining the display order of the album content in the next round based on the new content ordering scheme.

[0032] Specifically, the album content is re-ordered according to a display priority sequence. The display priority sequence is generated based on the matching scores of each photo with the current user's interest, with higher-scored photos being displayed first. The system places these higher-scored photos at the front and lower-scored photos at the back, ensuring that the content that best matches the user's interest is presented to the user first. The re-ordered album content is analyzed, and the content subject type and content visual style features of each photo are integrated to generate a content feature vector for each photo. The content subject type can include "people", "landscape", "animals", etc., and the visual style features can include color tone (such as warm tone, cool tone), lighting effect, composition method, etc. By extracting these features, the system generates a content feature vector for each photo, which accurately represents the various visual features of the photo. The system calculates the matching degree score between the current user's preference vector and the content feature vector of each photo. The preference vector represents the user's interest distribution, for example, the user's preference weight for people photos can be higher, and the preference weight for landscape photos can be lower. By calculating the similarity between the user's preference and the photo content features, the system generates a matching degree score for each photo. The higher the matching degree score, the more the photo matches the user's current interest, and the more likely it is to be displayed first. If the matching degree score is lower than the preset interest threshold, the system will consider that the photo does not match the user's interest enough and cannot meet the requirements of personalized display. At this time, the system will adjust the position of some elements in the display priority sequence, i.e. move the lower-scored photos to the back of the display, and the higher-scored photos to the front. In this way, the system can ensure that the content of the album display is as consistent as possible with the user's interest.

[0033] Further, according to the adjusted sequence, the system generates a new content ordering scheme. The new ordering scheme will rearrange the display order of each photo in the album to ensure that the content that best matches the current user's interest is displayed first. After each update, the system will adjust the display order of the album according to the new ordering scheme, so that the content of the album always remains consistent with the user's latest interest. Based on the new content ordering scheme, the system determines the display order of the album content in the next round. This process is dynamic, meaning that as the user's interest changes, the order of the album display will be constantly adjusted to ensure that the content displayed each time meets the user's personalized needs to the greatest extent. This ensures that the album display is always intelligent and timely in responding to the user's interest, providing a highly personalized visual experience. Figure 2 The figure shows the process of intelligent album content dynamic optimization and personalized display.

[0034] Step S5, real-time behavior feedback data is obtained, and the preference vector is optimized based on this to obtain a refined interest model.

[0035] In a specific embodiment, the process of performing step S5 can specifically include the following steps: By recording the user's behavior feedback data in real-time album browsing, the feedback data is analyzed to obtain click-stay features; Using the click-stay features as a supervisory signal, a gradient descent algorithm is used to optimize the dynamic change parameters in the preference vector; The preference vector is updated by the optimized dynamic change parameters to obtain a refined interest model; The latest behavior feedback data is input into the interest model to obtain an instant interest score; If the instant interest score is lower than the preset instant interest threshold, the gradient descent algorithm is repeatedly used to optimize the dynamic change parameters in the preference vector until the instant interest score reaches the preset instant interest threshold, and the final refined interest model is obtained.

[0036] Specifically, the system records the user's click count, stay duration and other information according to the user's real-time behavior feedback data in the album, thereby extracting click-stay features that reflect the user's interest intensity in different photos. The system uses these click-stay features as a supervisory signal to optimize the dynamic change parameters in the preference vector through a gradient descent algorithm. The gradient descent algorithm calculates the influence of each element of the preference vector on the current error, and then adjusts the weights in the preference vector according to the size of the error. This process reduces the error and optimizes the model through backpropagation of error, until the preference vector can more accurately reflect the user's interest. During the optimization process, the learning rate controls the step size of each update, ensuring stable convergence of the algorithm.

[0037] Through multiple iterations of optimization, the system generates a refined interest model that can better adapt to the user's latest interest changes. Each user's behavior feedback affects the update of the interest model, making the model more in line with the user's instant needs. Then, the system inputs the latest behavior data into the refined interest model to calculate the instant interest score. If the instant interest score is lower than the preset interest threshold, the system will again use the gradient descent algorithm to optimize the dynamic change parameters in the preference vector until the score reaches the preset interest threshold. Through this repeated optimization process, the preference vector can be gradually adjusted to ensure that the system's recommended content always meets the user's latest interests, thereby improving the accuracy and personalization of recommendations.

[0038] Taking the user's browsing of landscape and portrait photos as an example, by recording the user's clicks and dwell time in real time, the system extracts the user's longer dwell time on landscape photos and higher click frequency than on portrait photos. Through these behavior data, the system calculates the click dwell feature value of landscape photos as 0.8 and that of portrait photos as 0.3. The system uses these features as a supervisory signal and uses a gradient descent algorithm to optimize the preference vector. Initially, the user's preference vector is (portrait 0.5, landscape 0.5). After gradient descent optimization, the system finds that the matching degree of landscape photos is higher, and the weight of the landscape category in the preference vector is increased to 0.7, while the weight of the portrait category is reduced to 0.3. After 10 iterations, the user's instant interest score when browsing landscape photos increases from 0.75 to 0.85, and the score of portrait photos decreases from 0.65 to 0.6. The system adjusts the recommendation order according to the user's latest interest data and displays landscape photos in front, significantly improving the user's click rate and dwell time. Through this optimization, the user's satisfaction with the recommended content improves by about 20%, and the overall interaction rate of the album increases by 15%, demonstrating the effectiveness of the model in real-time adjustment of recommended content and the improvement of personalized recommendations. Reference Figure 3 The figure shows the comparison of interest scores before and after gradient descent optimization.

[0039] Step S6, formulating a recommendation strategy according to the interest model and generating an album layout template.

[0040] In a specific embodiment, the process of performing step S6 can specifically include the following steps: Based on the refined interest model, obtain the hot spot recognition result, and formulate a recommendation strategy according to the hot spot recognition result; Extract elements with priority higher than a preset priority threshold from the recommendation strategy as high-priority elements; Generate an album layout template according to the high-priority elements; Calculate the fit degree of the album layout template and the current user's preference vector; If the fit degree is lower than a preset fit threshold, adjust the priority order in the recommendation strategy; Update the album layout template through the adjusted strategy to obtain the final album layout template.

[0041] Specifically, based on the refined interest model, the system identifies the user's current focus theme or content through a hot spot recognition algorithm. Assuming that the user frequently browses landscape photos recently and the user's interest in portrait photos has significantly decreased, the system will identify "landscape" as the user's most focused content category according to this hot spot recognition result. The system formulates a recommendation strategy based on this hot spot recognition result, listing landscape photos as the recommended priority elements and reducing the recommendation priority of portrait photos.

[0042] The system extracts elements with priority higher than a preset priority threshold (e.g. 0.7) from the recommendation strategy as high-priority elements. For example, assuming the priority of landscape photos is 0.8 and the priority of portrait photos is 0.5, the system will display landscape photos as high-priority elements. Based on these high-priority elements, the system generates a photo album layout template. The design of this layout template takes into account the current interest preferences of the user, ensuring that the most interesting content is displayed first. For example, the system may arrange landscape photos on the first few pages of the photo album and arrange portrait photos at the back. Then, the system calculates the fit degree of the photo album layout template with the current user's preference vector, i.e. the consistency between the order of recommended content and the user's actual interests. Assuming the user's preference vector is (landscape 0.7, portrait 0.3) and the content of the layout template is (landscape 0.8, portrait 0.2), the fit degree is 0.9, which matches the user's interests.

[0043] If the fit degree is lower than a preset fit threshold (e.g. 0.8), the system considers that the current layout template does not match the user's interests well and will adjust the priority order in the recommendation strategy. For example, if the priority of portrait photos is mistakenly set to be higher, the system will adjust the strategy to lower the priority of portrait photos and increase the weight of landscape photos. Through this adjustment, the system updates the photo album layout template and generates the final layout scheme.

[0044] Step S7, generate a final output sequence according to the layout template, and fuse the subject type and the picture style to output a personalized photo album version.

[0045] In a specific embodiment, the process of performing step S7 can specifically include the following steps: According to the photo album layout template, determine the content composition of the final output sequence; Input the content composition, the content subject type corresponding to the content composition, and the content picture style feature into the sequence generation model to generate a personalized photo album version; Calculate the negative cross-entropy loss of the personalized photo album version and the current user's preference vector as a consistency score; If the consistency score does not reach a preset consistency threshold, repeatedly adjust the parameters of the sequence generation model until the consistency score reaches the preset consistency threshold; Update the sequence generation model according to the adjusted parameters; Generate the final personalized photo album version based on the updated sequence generation model.

[0046] Specifically, the system determines the content composition of the final output sequence based on the album layout template. The album layout template has been optimized according to the user's interest preferences and contains a priority ranking of various types of content. For example, after the interest in landscape photos increases, the system will prioritize landscape photos and place them in front of the album, and prioritize portrait photos and place them in the back. After determining the content composition of the album, the system inputs these contents along with their corresponding subject types (such as people, landscapes, etc.) and picture style features (such as color tone, light and shadow, etc.) into the sequence generation model. The sequence generation model is responsible for generating personalized album versions based on the input features, ensuring that the order, type, and style of photos in the album are maximally matched with the user's preferences.

[0047] Further, the system calculates the negative cross-entropy loss of the personalized album version and the current user's preference vector as the consistency score. The negative cross-entropy loss is used to measure the difference between the generated album version and the user's preferences, and the smaller the loss, the higher the degree of fit between the album and the user's interests. Assuming that the current user's preference vector is (landscape 0.7, portrait 0.3), if the generated album version prioritizes landscape photos and reduces the display of portrait photos, the consistency score will be higher. If the consistency score is lower than the preset consistency threshold (such as 0.8), the system will consider that the current generated album version does not meet the user's interests and needs to be adjusted.

[0048] When the consistency score does not reach the threshold, the system will repeatedly adjust the parameters of the sequence generation model, such as optimizing the weights in the model or adjusting the algorithm for sorting album content, until the consistency score reaches the preset standard. After each adjustment, the system updates the parameters of the sequence generation model to better reflect the user's actual interests and preferences. Based on the adjusted model parameters, the system generates the final personalized album version, ensuring that the displayed content not only meets the user's interest preferences, but also provides the best user experience in terms of visual style, photo order, etc. Through this continuous optimization process, the system can achieve highly personalized album recommendations, maximizing user satisfaction.

[0049] The above describes the method of multi-modal interaction and content generation of the intelligent album in the embodiments of the present application. The following describes the multi-modal interaction and content generation system of the intelligent album in the embodiments of the present application. Please refer to Figure 4 An embodiment of the multi-modal interaction and content generation system of the intelligent album in the embodiments of the present application includes: A data collection and analysis module for obtaining interaction data of a user when browsing an album, and determining initial hot content based on the interaction data; A preference feature extraction module for calculating the correlation strength between the content subject type and the content picture style feature of the initial hot content using an attention mechanism model, and obtaining a preference vector based on the correlation strength; The priority dynamic adjustment module is configured to extract a change trend from the preference vector, adjust the decay element weight according to the change trend, and generate an updated display priority sequence; The content ranking display module is configured to determine a display order of the album content according to the display priority sequence; The model real-time optimization module is configured to obtain real-time behavior feedback data, and optimize the preference vector according to the real-time behavior feedback data to obtain a refined interest model; The layout strategy generation module is configured to formulate a recommendation strategy according to the interest model, and generate an album layout template; The album fusion output module is configured to generate a final output sequence according to the layout template, and fuse the subject type and the picture style to output a personalized album version.

[0050] Through the cooperation of the above-mentioned components, the system can dynamically adjust the display order of the album content according to the behavior data of the user, and ensure that the content displayed each time can meet the interest needs of the user to the greatest extent. The data collection and analysis module obtains the interactive data of the user when browsing the album in real time, identifies the content that the user is most interested in, and forms the preliminary hot content. Then, the preference feature extraction module calculates the correlation strength between the content subject type and the picture style feature based on these hot contents by using the attention mechanism model, so as to obtain the accurate preference vector. The priority dynamic adjustment module dynamically adjusts the display priority of each content according to the change trend of the user interest, so as to ensure that the content display order conforms to the current interest of the user. The content ranking display module determines the display order of the album according to the updated priority sequence, and the model real-time optimization module continuously optimizes the preference vector according to the real-time behavior feedback of the user, so as to ensure that the interest model is kept up-to-date and refined. Finally, the layout strategy generation module formulates a personalized recommendation strategy in combination with the interest model, generates an album layout template, and the album fusion output module generates a final personalized album version according to the layout template, so as to ensure that the content seen by the user not only conforms to the interest of the user, but also provides the best visual experience and interactive feeling.

[0051] The above Figure 4 The multi-modal interaction and content generation system of the intelligent album in the embodiment of the application is described in detail from the perspective of the modularized functional entities, and the multi-modal interaction and content generation device of the intelligent album in the embodiment of the application is described in detail from the perspective of hardware processing.

[0052] Referring to Figure 5 In the embodiment of the application, a multi-modal interaction and content generation device of an intelligent album is also provided, which can be a server, and the internal structure thereof can be as shown in Figure 5The multi-modal interaction and content generation device of the smart photo album comprises a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the multi-modal interaction and content generation device of the smart photo album comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the multi-modal interaction and content generation device of the smart photo album is used to store the corresponding data in the embodiment. The network interface of the multi-modal interaction and content generation device of the smart photo album is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to realize the above method.

[0053] Those skilled in the art can understand that, Figure 5 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the multi-modal interaction and content generation device of a smart photo album to which the scheme of the present application is applied.

[0054] The above-described embodiments are only used to illustrate the technical scheme of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical scheme recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical scheme deviate from the spirit and scope of the technical scheme of the embodiments of the present application.

Claims

1. A method for multi-modal interaction and content generation of an intelligent photo album, characterized in that, The method comprises: S1, obtaining the interaction data of the user when browsing the album, and determining the initial hot content according to the interaction data; S2, calculating the correlation strength between the content subject type and the content picture style feature of the initial hot content by using an attention mechanism model, and obtaining a preference vector based on the correlation strength; S3, extracting a change trend from the preference vector, adjusting the weight of the decay element according to the change trend, and generating an updated display priority sequence; S4, determining the display order of the album content through the display priority sequence; S5, obtaining real-time behavior feedback data, and optimizing the preference vector based on the same to obtain a refined interest model; S6, formulating a recommendation strategy according to the interest model, and generating an album layout template; S7, generating a final output sequence according to the layout template, and fusing the subject type and the picture style to output a personalized album version.

2. The method of multi-modal interaction and content generation for smart photo album as claimed in claim 1, wherein, The S1 comprises: By recording the interaction data of the user when browsing the album, the click frequency and the stay time in the interaction data are analyzed to determine the attention degree of the user to different contents; The contents with an attention degree higher than a preset attention degree threshold are selected as the initial hot content; The initial hot content is analyzed to determine the content subject type and the content picture style feature of the initial hot content; The content subject type and the content picture style feature are statistically analyzed to generate an initial content preference distribution; The matching score between the initial hot content and the initial content preference distribution is calculated; If the matching score is lower than a preset matching threshold, the interaction data is reanalyzed to update the initial hot content.

3. The method of multi-modal interaction and content generation for smart photo album as claimed in claim 1, wherein, The S2 comprises: The correlation strength between the content subject type and the content picture style feature of the initial hot content is calculated by using an attention mechanism model; According to the correlation strength, a content picture style preference vector corresponding to each content subject type is calculated by using a weighted average algorithm as an initial preference vector; The initial preference vector is normalized to obtain a standardized preference representation; The inclination degree of the user to different content subject types is calculated according to the preference representation; The distribution characteristics of the user's interest are further analyzed through the inclination degree; If the distribution characteristics meet a preset concentration condition, the element weight corresponding to the initial preference vector is adjusted to obtain a final preference vector.

4. The method of multi-modal interaction and content generation for smart photo album of claim 3, wherein, The S3 comprises: The change trend over time is extracted from the preference vector by using a time series analysis method; It is judged whether the change trend exceeds a preset trend threshold, if yes, a decay element is identified, and the weight value corresponding to the decay element is reduced; The preference vector distribution is recalculated according to the adjusted weight; The matching score is calculated based on the preference vector distribution and the content feature vector of each picture, and a display priority sequence is generated according to the score. 5.The method of claim 1, wherein, The S4 comprises: The album content is reordered according to the display priority sequence; The content subject type and the content picture style feature of each picture are integrated to generate a content feature vector by analyzing the reordered album content; The matching degree score is calculated by using the preference vector of the current user and the content feature vector. If the matching score is lower than a preset interest threshold, positions of some elements in the display priority sequence are adjusted; A new content sorting scheme is generated according to the adjusted sequence; A display order of the next round of album content is determined based on the new content sorting scheme.

6. The method of multi-modal interaction and content generation for smart photo album of claim 3, wherein, The S5 includes: By recording the behavior feedback data of the user in the album browsing in real time, the feedback data is analyzed to obtain a click-stay feature; The click-stay feature is used as a supervision signal, and a gradient descent algorithm is used to optimize the dynamic change parameters in the preference vector; The preference vector is updated by the optimized dynamic change parameters to obtain a refined interest model; The latest behavior feedback data is input into the interest model to obtain an instant interest score; If the instant interest score is lower than a preset instant interest threshold, the gradient descent algorithm is repeatedly used to optimize the dynamic change parameters in the preference vector until the instant interest score reaches the preset instant interest threshold, and a final refined interest model is obtained.

7. The method of multi-modal interaction and content generation for smart photo album as claimed in claim 6, wherein, The S6 includes: Based on the refined interest model, a hot spot recognition result is obtained, and a recommendation strategy is formulated according to the hot spot recognition result; Elements with a priority higher than a preset priority threshold are extracted from the recommendation strategy as high-priority elements; An album layout template is generated according to the high-priority elements; The fit degree of the album layout template and the preference vector of the current user is calculated; If the fit degree is lower than a preset fit threshold, the priority order in the recommendation strategy is adjusted; The album layout template is updated by the adjusted strategy to obtain a final album layout template.

8. The method of multi-modal interaction and content generation for smart photo album of claim 1, wherein, The S7 includes: The content composition of the final output sequence is determined according to the album layout template; The content composition, the content subject type corresponding to the content composition, and the content picture style feature are input into a sequence generation model to generate a personalized album version; The negative cross-entropy loss of the personalized album version and the preference vector of the current user is calculated as a consistency score; If the consistency score does not reach a preset consistency threshold, the parameters of the sequence generation model are repeatedly adjusted until the consistency score reaches the preset consistency threshold; The sequence generation model is updated according to the adjusted parameters; A final personalized album version is generated based on the updated sequence generation model.

9. A multi-modal interaction and content generation system of a smart album, configured to implement the multi-modal interaction and content generation method of the smart album according to any one of claims 1-8. The multi-modal interaction and content generation system of the intelligent album includes: A data collection and analysis module is configured to obtain interaction data of a user when browsing an album, and determine initial hot spot content based on the interaction data; A preference feature extraction module is configured to calculate the association strength between the content subject type and the content picture style feature of the initial hot spot content by using an attention mechanism model, and obtain a preference vector based on the association strength; A priority dynamic adjustment module is configured to extract a change trend from the preference vector, adjust the weight of a decay element according to the change trend, and generate an updated display priority sequence; A content sorting and display module is configured to determine the display order of album content by using the display priority sequence; A model real-time optimization module is configured to obtain real-time behavior feedback data, and optimize the preference vector based on the real-time behavior feedback data to obtain a refined interest model. A layout strategy generation module is configured to formulate a recommended strategy according to the interest model, and generate an album layout template; An album fusion output module is configured to generate a final output sequence according to the layout template, and fuse subject types and picture styles, and output a personalized album version.

10. A device for multi-modal interaction and content generation of a smart album, characterized in that, The multi-modal interaction and content generation device for an intelligent album includes a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory, so that the multi-modal interaction and content generation device for an intelligent album performs the multi-modal interaction and content generation method for an intelligent album in any one of claims 1-8.

Images