Content recommendation method and apparatus, electronic device, and computer readable medium

Abstract

This application discloses a content recommendation method, apparatus, electronic device, and computer-readable medium, relating to the field of mobile terminal technology. The method includes: acquiring the display content corresponding to the target interface currently displayed on the screen of the electronic device; determining the subject information corresponding to the user's query subject based on the display content; determining the recommended content corresponding to the subject information; and displaying the recommended content through the target interface. Therefore, the recommended content presented to the user is related to the subject information within the currently displayed interface, providing more relevant recommended content to the user's actual needs, thereby improving the relevance and accuracy of the recommendations.

Description

Technical Field

[0001] This application relates to the field of mobile terminal technology, and more specifically, to a content recommendation method, apparatus, electronic device, and computer-readable medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI assistants are being used more and more widely across various industries, playing a particularly important role in providing personalized services. Personalized recommendation systems, as one of the core technologies of AI assistants, utilize user behavior, preferences, and needs data to provide customized recommendations and services. In recent years, with the emergence and development of large-scale models (such as GPT series, BERT, T5, etc.), query recommendation systems based on these models have become a significant innovation in the field of AI assistants, providing users with more intelligent and accurate recommendation services. However, current personalized recommendation systems may not recommend content that is currently of most interest to the user, leading to inappropriate content recommendations. Summary of the Invention

[0003] This application proposes a content recommendation method, apparatus, electronic device, and computer-readable medium to improve the above-mentioned deficiencies.

[0004] In a first aspect, this application provides a content recommendation method applied to an electronic device. The method includes: obtaining the display content corresponding to the target interface currently displayed on the screen of the electronic device; determining the subject information corresponding to the user's subject to be queried based on the display content; determining the recommended content corresponding to the subject information; and displaying the recommended content through the target interface.

[0005] Secondly, this application also provides a content recommendation device applied to an electronic device, the device comprising: an acquisition unit, a determination unit, a recommendation unit, and a display unit. The acquisition unit is used to acquire the display content corresponding to the target interface currently displayed on the screen of the electronic device; the determination unit is used to determine the subject information corresponding to the user's subject to be queried based on the display content; the recommendation unit is used to determine the recommended content corresponding to the subject information; and the display unit is used to display the recommended content through the target interface.

[0006] Thirdly, this application also provides an electronic device, comprising: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the methods described above.

[0007] Fourthly, this application also provides a computer-readable medium storing processor-executable program code that, when executed by the processor, causes the processor to perform the above-described method.

[0008] The content recommendation method, apparatus, electronic device, and computer-readable medium provided in this application, in scenarios where recommended content needs to be pushed to a user, obtain the display content corresponding to the target interface currently displayed on the screen of the electronic device, and determine the subject information corresponding to the user's query subject based on the display content. This query subject can be considered as an object that the user may currently or may be interested in querying or searching for. Then, recommended content corresponding to the subject information is determined; the recommended content is displayed through the target interface. Therefore, the recommended content presented to the user is related to the subject information within the currently displayed interface, providing recommended content more relevant to the user's actual needs, thereby improving the relevance and accuracy of the recommendations.

[0009] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart of a content recommendation method provided in an embodiment of this application is shown;

[0012] Figure 2 A schematic diagram of a target interface provided in an embodiment of this application is shown;

[0013] Figure 3 A flowchart of a content recommendation method provided in another embodiment of this application is shown;

[0014] Figure 4 A flowchart of a content recommendation method according to another embodiment of this application is shown;

[0015] Figure 5 A framework diagram of a content recommendation method provided in an embodiment of this application is shown;

[0016] Figure 6A schematic diagram of a target interface provided in another embodiment of this application is shown;

[0017] Figure 7 A block diagram of a content recommendation device according to an embodiment of this application is shown;

[0018] Figure 8 A structural block diagram of the electronic device provided in an embodiment of this application is shown;

[0019] Figure 9 An embodiment of this application shows a storage unit for storing or carrying program code that implements the content recommendation method according to an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] With the rapid development of artificial intelligence (AI) technology, AI assistants are being used more and more widely across various industries, playing a particularly important role in providing personalized services. Personalized recommendation systems, as one of the core technologies of AI assistants, utilize user behavior, preferences, and demand data to provide users with customized recommended content and services. In recent years, with the emergence and development of large-scale models (such as the GPT series, BERT, T5, etc.), query recommendation systems based on large-scale models have become a significant innovation in the field of AI assistants, providing users with more intelligent and accurate recommendation services.

[0023] In the AI ​​assistant industry, personalized recommendation systems typically employ a three-stage recommendation process, a highly efficient and widely used architecture. This three-stage recommendation system consists of three phases: Recall, Ranking, and Re-ranking. 1. Recall Phase: The goal of this phase is to quickly filter out a candidate set that the user might be interested in from the recommendation query content library, reducing the computational burden of the subsequent ranking phase. Recall strategies can be based on various methods, such as content preference, collaborative filtering, popular item recall, and strategy recall. To increase diversity, a multi-path recall structure is often used, combining the results of multiple recall methods. 2. Ranking Phase: After obtaining a smaller candidate set in the recall phase, the ranking phase aims to refine the rating and ranking of these candidate items based on the user's historical behavior and preferences. This phase typically uses machine learning models, such as logistic regression, gradient boosting machines (GBDT), or deep learning models, to predict the user's click probability or rating for each candidate item. 3. Re-ranking Phase: After ranking, the re-ranking phase further optimizes the ranking results to improve the accuracy and diversity of the recommendations. This stage may take into account more business rules and user feedback, such as novelty, diversity, user feedback and long-term interests, in order to adjust the final recommendation list.

[0024] In addition, the entire recommendation system architecture includes a data layer and a model layer. The data layer is responsible for collecting and processing feature data of users, items, and scenarios, while the model layer is responsible for training and deploying recommendation models. Furthermore, to improve recommendation performance, end-to-end interventions are implemented, including cold start strategies and exploration and exploitation (E&E) strategies.

[0025] In summary, modern recommendation systems, through close collaboration across these three stages, can provide users with personalized, real-time, and accurate recommendation results.

[0026] With the rapid development of artificial intelligence technology, AI assistants have become an indispensable part of smartphones. The application of AI assistants in smartphones has greatly enhanced the user experience and promoted the intelligence, personalization, and automation of mobile phone functions. However, the inventors have discovered the following shortcomings in current mobile phone intelligent assistant recommendation systems:

[0027] 1. Lack of screen information perception: Existing recommendation systems rely heavily on users' explicit / implicit behaviors from feature collection to sample generation, lacking the ability to perceive user mobile phone screen information. The recommendation query is not related to the user's screen, resulting in low accuracy and relevance of personalized recommendations. Users have poor satisfaction with personalized recommendations in application scenarios. Here, query refers to the request, question or instruction entered by the user, which is the basic form of interaction between the user and the intelligent assistant.

[0028] 2. Limitations of Fixed Queries: Existing recommender systems rely on fixed-configuration queries and query sets, which are typically generated based on predefined rules and user behavior patterns. This approach limits the flexibility and adaptability of recommender systems, making it difficult to cope with the diversity and complexity of user needs.

[0029] 3. Limitations of Query Sources: Recommended queries are mainly written and reviewed manually. This method is inefficient, costly, and difficult to adapt to rapidly changing user needs and market trends.

[0030] Therefore, to overcome the above-mentioned shortcomings, embodiments of this application provide a content recommendation method applied to electronic devices, such as smartphones and tablets. Figure 1 As shown, the method may include: S101 to S104.

[0031] S101: Obtain the display content corresponding to the target interface currently displayed on the screen of the electronic device.

[0032] It is understood that the displayed content refers to the content contained within the target interface. The target interface can be the interface that the user is currently browsing on the screen of the electronic device. The displayed content can be images, videos, text, etc. within the interface. Specifically, the text can be text entered by the user or text converted from voice input and displayed on the interface. The images and videos can be the content that the user is currently browsing.

[0033] One implementation method is to determine the currently displayed interface of the target application as the target interface after the user runs the target application on their screen and enters query content within the target application. For example, after the user opens the target interface, an operation to capture a screenshot of the target interface can be triggered, and the screenshot image can be identified to obtain the displayed content. Alternatively, the screenshot of the target interface can be captured after detecting that the user has performed a specified operation within the target interface. This specified operation may include the user triggering a specified control, which is used by the user to enter query content within the target interface. For example, the user can operate the target control to enter text or voice; therefore, the specified operation can be triggered by the specified control.

[0034] Alternatively, the specified operation can also be that the number of queries entered by the user on the target interface is greater than a specified number. The query can be a novelty search request entered by the user on the target interface. For example, if the user enters "What kind of flower is this?", this query corresponds to one query request. The specified number is used to determine whether the user has a high demand for queries on the current interface. For example, if the number of queries entered by the user on the target interface is greater than the specified number, it can be determined that the user has a demand for content on the interface. However, if the number of queries entered by the user on the target interface is less than or equal to the specified number, it can be determined that the user does not have a demand for queries on the interface. For example, the user may quickly jump to another interface.

[0035] S102: Determine the subject information corresponding to the subject to be queried by the user based on the displayed content.

[0036] It is understandable that, since the displayed content can include images, videos, text content, etc. within the target interface, the user's current focus within the target interface can be determined by analyzing the displayed content. For example, if a user is browsing a book within the target interface, and the target interface displays the book's cover image, then by identifying the cover image, the name of the book being browsed can be determined, and that book name can be used as the subject to be queried.

[0037] It should be noted that the subject information corresponding to the subject to be queried may include the description of the subject to be queried. The description may include a specific description of the subject to be queried. The user indicates that they need to know the specific identity and attribute information of the subject to be queried. For example, if the mobile phone is the subject to be queried, the relevant attributes may include battery life, camera performance, screen, performance, etc. The specific identity refers to the name, model or type of the subject to be queried.

[0038] S103: Determine the recommended content corresponding to the main information.

[0039] Understandably, once the subject of the query is determined, relevant information can be extracted from the corresponding databases or knowledge sources. This information can come from local databases, the internet, or broader knowledge bases. Specifically, structured or unstructured data sources (such as Wikipedia, news APIs, proprietary databases, etc.) are used to retrieve relevant information about the subject. For example, if the subject is "OPPO mobile phone," the system might extract information such as the price, features, and sales volume of OPPO mobile phones from the knowledge base as recommended content.

[0040] One implementation method involves searching for content matching the target subject within current trending information and using it as recommended content. This can be done by statistically analyzing content with a pageview count exceeding a threshold within a preset time period, identifying it as trending information, and then extracting the target subject from this trending information (i.e., specific product names, locations, names, etc.). The method then searches for trending subjects that match the target subject and uses the corresponding trending information as recommended content. Alternatively, recommended content can be determined based on the trending information corresponding to the matched target subjects. Specifically, the trending information corresponding to the matched target subjects can be used as candidate information, and a preset number of trending information items can be selected as recommended content. For example, a preset number of trending information items can be randomly selected as recommended content; alternatively, the matching degree between each trending information item and the target subject information can be sorted from high to low, and the top preset number of trending information items can be selected as recommended content.

[0041] As another implementation method, recommended content can be determined using a large language model. Specifically, after determining the subject information, a large language model can be used to generate recommended content. This recommended content can be personalized suggestions, information supplements, or action recommendations based on the user's query intent. Large language models (such as GPT) provide relevant recommendations based on the subject information.

[0042] For example, the subject information includes not only the identity information of the subject to be queried, but also the query intent corresponding to the subject to be queried. The query intent is used to determine which specific attribute of the subject the user may want to query. For example, if the subject to be queried is a mobile phone, the query intent could be the price or camera performance of the subject.

[0043] As one implementation method, the query intent can be determined by recognizing the displayed content. Specifically, the displayed content may include query content already entered by the user. Based on this query content, the user's query intent can be determined. For example, if the displayed content includes an image of a mobile phone, image analysis can determine that the subject to be queried is a mobile phone. If the query request entered by the user on the target interface involves the price of a mobile phone, then the user's query intent can be determined to be related to price. Of course, even if the user has not entered any query content on the target interface, analyzing the image can also determine the query intent. For example, by recognizing at least one image on the target interface, the auxiliary information corresponding to the subject in the image can be determined. This auxiliary information may include the subject's actions, expressions, clothing, makeup, and items being used. Based on this auxiliary information, the query intent can be determined. For example, if the auxiliary information is the subject's actions, the type of actions can be determined, and the query intent can be determined based on the type. For example, if the action type is martial arts, the query intent may be to learn about the martial arts related content of the subject to be queried. Or, if the subject is a person, the auxiliary information may include clothing, and the clothing can be used as the query intent, that is, the query intent is to understand the person's clothing style or clothing recommendations.

[0044] As another implementation, the query intent can be determined based on the user's preference information. Specifically, the user's preference information can determine the content that the user is interested in regarding the subject to be queried. This preference information can be determined by the user's historical query content regarding the subject to be queried.

[0045] Therefore, based on the obtained subject information and query intent, recommended content is generated.

[0046] It's understandable that the recommended content is a query suggestion sent to the user; each recommended content can be understood as a query that the user can trigger to receive a response. In other words, the recommended content is a query suggestion.

[0047] S104: Display the recommended content through the target interface.

[0048] In one implementation, the recommended content can be displayed directly on the target interface, for example, through pop-ups, floating prompts, floating buttons, notification bars, etc.

[0049] like Figure 2 As shown, the target interface displays a book cover image, and the book's name is represented by ABCD. Therefore, the identified main information can be the book's name, i.e., ABCD. The corresponding recommended content for this book is as follows: Figure 2As shown, the recommendations can include "What famous quotes does this book contain?", "How to judge the quality of a book?", and "Where is the author of this book from?". This recommendation content 201 can be displayed within the target interface. It can be seen that each recommendation is displayed sequentially above the input control 202. Therefore, the user can click on the recommendation to receive the corresponding response. Thus, the recommendation content displayed in this embodiment can include query suggestions, and each recommendation can serve as a query suggestion pushed to the user.

[0050] Of course, users can also enter new query content through the input control 202. This input control supports input of text, images, videos, and also voice input.

[0051] Therefore, in this embodiment of the application, the recommended content presented to the user is related to the main information in the interface currently displayed on the screen, and can provide recommended content that is more relevant to the user's actual needs, thereby improving the relevance and accuracy of the recommendations.

[0052] Please see Figure 3 This application provides a content recommendation method, which is applied to the above-mentioned electronic device. Specifically, the method includes: S301 to S305.

[0053] S301: Obtain the display content corresponding to the target interface currently displayed on the screen of the electronic device.

[0054] In one implementation, the displayed content is a screenshot of the target interface. For example, when the target interface is displayed on the screen of the electronic device, the electronic device automatically performs a screenshot operation to obtain a screenshot image corresponding to the target interface. The timing of the electronic device performing the screenshot operation can be referred to the foregoing content and will not be repeated here.

[0055] S302: Identify the screenshot image of the target interface and determine the main object.

[0056] In one implementation, the screenshot image includes a video or picture viewed by the user. The image corresponding to the video or picture in the screenshot image is named the target image. By identifying the target image, the main object in the image can be determined.

[0057] Specifically, an image may typically contain multiple target objects. For example, a landscape photo may include mountains, houses, grasslands, trees, flowers, animals, clouds, etc., while a portrait photo may contain people, objects used by people, and objects around people (e.g., clamps, vehicles, etc.). By identifying the target image, the main object in the target image can be determined.

[0058] For example, the subject in an image typically refers to the most prominent, significant, or important part or object that occupies a primary position in the image. The subject is usually the focal point or core of the image; it is the main object or content that draws the viewer's attention. In other words, the subject usually refers to the most important target or object in the image. For instance, given a street scene image, the "cars," "people," or "buildings" might be considered the subject. Object detection and segmentation algorithms can identify the main subject in an image.

[0059] Specifically, object detection is a technique for identifying the location (i.e., bounding box) and category (e.g., people, vehicles, animals, etc.) of different objects (including main objects) in an image. For example, the YOLO (You Only Look Once) model can be used to extract the main object from an image. Alternatively, an image segmentation model can be used to identify the subject information in a screenshot. Specifically, an image segmentation model divides an image into multiple regions, typically each representing an object or a part of the image. Image segmentation can be used to more accurately identify the contours and boundaries of main objects. For example, this image segmentation model can be semantic segmentation or instance segmentation. Furthermore, other models can be used to analyze and obtain the main object, such as deep learning models and visual attention mechanism models.

[0060] S303: Determine the subject information corresponding to the subject to be queried by the user based on the subject object.

[0061] After identifying the main object, the system can determine the main object that the user is currently viewing on the screen, which can serve as the user's current focus. Then, based on the main object, the system determines the subject information corresponding to the subject the user is querying.

[0062] As one implementation method, the subject object can be used as the subject to be queried by the user. Then, the identity information of the subject object is used as the subject information of the subject to be queried. The identity information can be the name, type, brand, model and other information that can characterize the identity of the subject object.

[0063] As another implementation method, a portion of the subject objects can be identified from the subject objects as the subject to be queried, thereby obtaining the subject information of the subject to be queried.

[0064] For example, one could collect entity information from the historical queries entered by each user within a specified time period as a reference subject, count the number of times each entity information was queried, and use the entity information with a query count greater than a specified number as a reference entity. Among the aforementioned subject objects, one could determine the object that matches the reference entity as the subject to be queried.

[0065] It should be noted that the historical query content can include text or image content. For the text content, entity information can be the identification information of the query object itself (such as product name, location, person name, etc.). Typically, the query content includes descriptive information corresponding to the entity information, which refers to the characteristics or attributes of the query object (such as the product's color, price, function, etc.); time information: referring to the time or time period related to the query (such as the time of the event, the scheduled date, etc.); quantity information: such as specific numerical information like quantity, size, capacity, etc.; and relationship information: describing the relationships or dependencies between the query objects (such as the relationship between a person and a company). Generally, Natural Language Processing (NLP) technology can be used to identify the core entities, actions, attributes, etc., in the sentence, thereby extracting entity information and its corresponding descriptive information.

[0066] Therefore, entity information in a query can be viewed as the query object corresponding to that query. An entity refers to a specific object in the text that has a particular meaning. Typically, this specific object can be uniquely identified or recognized within a certain context. In Natural Language Processing (NLP) and information extraction, entities are often called named entities, including but not limited to the following types: person entities, location entities, and object entities. Object entities can refer to product entities; for example, an object entity can be a specific commodity, brand, or technological product. For instance, if the query is "Is this phone expensive?", then in this query, the phone is the query object, and the price is the descriptive content, i.e., attribute information. Therefore, the electronic device needs to first understand the query object, specifically the phone's identity information, such as its model and brand. This identity information can usually be obtained through the query results corresponding to that phone.

[0067] Therefore, by using the above method, reference entities within a specified time period can be statistically obtained. These reference entities are then matched with the main objects obtained by recognizing screenshots, and the matched main objects are used as the subject to be queried. The statistically obtained reference entities can be based on the target user's historical query content within the specified time period, or they can be historical query content entered by multiple users. The target user refers to the user using the target interface of the electronic device; for example, it could be the user currently logged into the application corresponding to that target interface.

[0068] In other embodiments, when there are multiple subject objects determined by recognizing screenshots of the target interface, the determination can also be based on the query content entered by the user in the target interface. Specifically, the implementation method for determining the subject information corresponding to the user's subject to be queried based on the subject object can be: determining the target query content entered by the user in the target interface during the current time period; and determining the subject information corresponding to the user's subject to be queried based on the subject object and the target query content.

[0069] It should be noted that the current time period can be the time period between the moment the target interface is opened and the current moment. In other words, the target query content refers to the query content entered by the user on the target interface after the target interface is opened. The moment the target interface is opened can be the moment when the target application corresponding to the target interface is switched from the closed state to the foreground running state for the first time, and the target interface within the application is running in the foreground. It can be understood as the moment when the target application is launched and the target interface is opened.

[0070] Alternatively, the current time period can also refer to the time period of the day corresponding to the current moment. This is achieved by pre-dividing each day's 24 hours into multiple time periods, with the time period corresponding to the current moment being considered the current time period. Furthermore, the current time period can also be the current date or the current season; there are no restrictions on this.

[0071] The implementation method for determining the subject information corresponding to the user's subject to be queried based on the subject object and the target query content can be as follows: determine the entity information in the target query content, determine historical entities based on the entity information of the target query content, take the subject object that matches the historical entity as the subject to be queried, and obtain the subject information corresponding to the subject to be queried.

[0072] In one implementation, the subject information to be queried can be the identity information of the subject, such as the name of an item or the identity information of a person. The identity information of a person can include physical characteristics such as gender, age, and height. In another implementation, in addition to the identity information corresponding to the subject to be queried, the subject information can also include descriptive information corresponding to the subject to be queried. The description of the descriptive information can be referred to the foregoing content and will not be repeated here. This descriptive information can indicate the direction or intent of the query for the subject to be queried. For example, if the query content is "Is this mobile phone expensive?", then the query object, i.e., the subject to be queried, is the mobile phone, and the descriptive information is "price." Therefore, the subject information corresponding to the subject to be queried includes the model and price of the mobile phone.

[0073] Understandably, this descriptive information can also be determined through the aforementioned target query content. Specifically, the descriptive information for each target query content is identified, and target descriptive information is determined based on the descriptive information of each target query content, serving as the descriptive information corresponding to the subject to be queried. For example, the number of each descriptive information is counted, and descriptive information with a number greater than a specified threshold is used as target descriptive information.

[0074] For example, suppose a user is browsing a video on a target interface of a target application. The video on the target interface shows two people. The user enters a query through a query assistant on the target interface, for example, asking how to compose a group photo. Analyzing the query, the determined description information is "group photo". Assuming that "group photo" can be used as the target description information and thus as the main information of the query object, when determining the recommended content, query suggestions related to group photos may be used as the recommended content.

[0075] S304: Determine the recommended content corresponding to the main information.

[0076] S305: Display the recommended content through the target interface.

[0077] It's important to note that since the main information is identified based on a screenshot of the screen, the recommended content is highly relevant to what the user is currently viewing. Therefore, the recommended content presented to the user is related to the main information displayed on the screen, providing recommendations that are more relevant to the user's actual needs, thus improving the relevance and accuracy of the recommendations. Furthermore, the recommended content can be displayed within the target interface, allowing users to quickly obtain query suggestions related to that content while focusing on it. Users can then access corresponding responses through these suggestions, facilitating a quick understanding of the information related to the screen content and reducing the complexity of manually entering queries.

[0078] Please see Figure 4 This application provides a content recommendation method, which is applied to the above-mentioned electronic device. Specifically, the method includes: S401 to S405.

[0079] S401: Obtain the display content corresponding to the target interface currently displayed on the screen of the electronic device.

[0080] S402: Determine the subject information corresponding to the subject to be queried by the user based on the displayed content.

[0081] As one implementation method, the method for determining the subject information can refer to the foregoing embodiments, and will not be repeated here. In the embodiments of this application, it is assumed that the subject information is the subject to be queried determined based on the recognition operation of the screenshot image of the target interface. The subject information is the identity information corresponding to the subject to be queried, and may also include descriptive information in addition to the identity information.

[0082] S403: Determine reference information, the reference information including at least one of the user's corresponding preference information and scenario context information, wherein the scenario context information includes the user's historical query content.

[0083] As one implementation method, the scenario context information can be the aforementioned target query content, i.e., the user's historical query content, which corresponds to the current time period. The implementation method for the current time period can refer to the aforementioned content and will not be repeated here.

[0084] User preference information can be determined based on user operation data of the target application corresponding to the target interface. Alternatively, it can be determined by combining operation data from other applications. It should be noted that this operation data is collected with the user's authorization. Of course, user preference information can also be data actively entered by the user. This user preference information can represent at least one of the query subject the user is interested in and the corresponding descriptive information of interest for that query subject. For example, if a user is interested in the camera performance of a mobile phone, then the subject of interest is "mobile phone," and the corresponding descriptive information of interest is "camera." Based on the user's preference information, search suggestions related to the mobile phone's camera can be pushed to the user.

[0085] S404: Determine the recommended content based on the subject information and the reference information.

[0086] As mentioned earlier, contextual information reflects the user's historical queries within the current time frame on the target interface, representing the user's current query preferences. User preference information, on the other hand, is a summary of the user's interests through statistical data analysis. This interest can include the subject of interest, or the subject of interest and its corresponding descriptive information. Therefore, both contextual information and user preference information reflect the user's degree of preference for different query suggestions. Consequently, the aforementioned reference information can be used to determine recommended content.

[0087] As one implementation method, assuming the reference information includes user preference information, determining the recommended content based on the subject information and the reference information using a large language model can be achieved by inputting the subject information and user preference information into the large language model to obtain the recommended content. The subject information includes the identity information of the subject object obtained beforehand through recognition of a screenshot of the screen.

[0088] Specifically, by capturing and processing screen images, image recognition technologies (such as deep convolutional neural networks (CNN), YOLO, etc.) are used to extract information about the main objects in the images. The main objects can be people, products, text, plants, landscapes, or other objects.

[0089] Then, the identity information of the subject is extracted from the recognized image. For example, if the subject is a person, identity information such as the person's name, position, and related events may be obtained. This identity information can be obtained by identifying the image of the subject in the screenshot and then determining the subject's corresponding identity information via the internet based on that image. For example, the obtained identity information of the subject may include: Subject type: 'person', Name: 'Zhang San', Position: 'Product Manager', Related field: 'technology'.

[0090] User preference information can be determined through users' historical behavioral data. For example, users may frequently browse technology products, news articles, or content from certain brands.

[0091] Then, the subject information and user preference information are input into the large language model. After receiving this information, the large language model will combine the subject identity and user preferences to generate recommended queries. The model will understand the user's needs based on the input information and generate query content that may be attractive to the user.

[0092] Specifically, based on the input information, the large language model generates relevant queries according to the identity information of the subject. For example, if the subject is "Zhang San, a product manager in the technology field," the model might generate queries such as "Zhang San's latest experience in technology product management" or "the workflow of a product manager in the technology field." In other words, the large model determines multiple candidate contents based on the subject information, and then selects the final content from these candidate contents based on the user's preference information. Specifically, the determined user preference information is input into the personalized ranking model along with the subject information. The personalized ranking model scores the candidate contents based on the user preference information, determines the score corresponding to each candidate content, sorts the candidate contents according to the score from high to low, and selects the top N candidate contents as recommended content, where N is a positive integer greater than or equal to 1. This score characterizes the relevance between the candidate content and the user preference information; specifically, the score is positively correlated with this relevance.

[0093] For example, the large model generates five screen-related queries (potential content) for each entity (i.e., subject information), corresponding to each entity. For instance, if an image contains five entities, it will generate 25 screen-related queries. These 25 queries, generated by the large model based on screen entities, are then combined with the user's personalized profile information (i.e., user preference information) and input into the personalized ranking model of the recommendation system. This results in the output of three screen queries most likely to be clicked by the user. In other words, these 25 queries, along with the user profile, are input into the personalized ranking model. The personalized ranking model can use machine learning-based ranking algorithms, such as Gradient Boosting Tree (GBDT) and neural network ranking models, to score and rank each query. The model predicts which queries are most likely to be clicked by the user based on the degree of match between the query and user preferences, historical click behavior, and other features, thus generating recommended content.

[0094] For example, assuming the target interface displays an image of a person introducing a mobile phone, the screenshot of the interface can identify at least two subjects: the mobile phone and the person. The subject information corresponding to the mobile phone and the person can be obtained. The subject information corresponding to the mobile phone includes the mobile phone model, detailed specifications, reviews, price, purchase recommendations, release date, user reviews, etc. The subject information corresponding to the person can include the person's name, background information, past speeches, relevant technical articles, social media information, etc.

[0095] Using large language models (such as GPT-4), the system can generate a set of alternative content based on relevant information for each subject to be queried. For example, if the phone model is Find X8, the corresponding alternative content would be "differences between Find X8 and Find X8 Pro", "what is the most powerful feature of Find X8", "is Find X8 worth buying", "best accessories for Find X8", and "how is the processor performance of Find X8". If the person's name is Zhang San and his identity is a scientist, the corresponding alternative content could include "Zhang San's latest technical speech", "Zhang San's views on artificial intelligence", "Zhang San's product management experience", "Zhang San's performance at the 2024 technology conference", and "Zhang San's personal social media accounts".

[0096] Next, the user's preference information and the aforementioned 25 queries are input into the recommendation system. The personalized ranking model will sort these queries based on the user's preferences and output the three queries most likely to attract the user's clicks.

[0097] Specifically, user profiles include information across multiple dimensions, such as the user's areas of interest, historical behavior, preferences, and search history. For example, areas of interest include technology, product reviews, artificial intelligence, and electronic products; search history shows that the user has previously searched for content related to smartphones and product managers, and enjoys tech news and product review articles; click behavior shows that the user frequently clicks on content related to new product launches, tech industry reports, and smart device reviews.

[0098] Then, these 25 queries, along with the user profile, are input into a personalized ranking model. This model uses machine learning-based ranking algorithms, such as Gradient Boosting Tree (GBDT) and neural network ranking models, to score and rank each query. The model predicts which queries are most likely to be clicked based on how well they match the user's preferences, thus assigning a score to each candidate. A higher score indicates a higher predicted probability of being clicked. For example, suppose the model outputs three candidate options: "What is the strongest feature of Find X8?", "Is Find X8 worth buying?", and "Zhang San's views on artificial intelligence." It can be seen that the recommendation system selects N candidate options from multiple subjects as recommended content. Alternatively, it can determine N candidate options corresponding to each subject to be queried as recommended content.

[0099] As another implementation, assuming the reference information includes scene context information, similar to the above method, after identifying the subject to be queried in the target interface of the screen, the subject information corresponding to the subject to be queried is determined. Based on the large language model, the candidate content corresponding to each subject information is determined. Then, the candidate content and scene context information are input into the personalized ranking model. The personalized ranking model scores the candidate content through the scene context information, determines the score corresponding to each candidate content, sorts each candidate content according to the score from high to low, and selects the top N candidate contents as recommended content, where N is a positive integer greater than or equal to 1.

[0100] It's understandable that contextual information refers to the user's actions and inputs within the current interface, particularly their query behavior and needs within the current time period. This information is crucial for personalized ranking because it helps the system understand the user's immediate needs, thereby generating more accurate recommendations. The personalized ranking model assigns a score to each candidate, reflecting the degree to which that candidate matches the contextual information. In other words, the score characterizes the relevance of the candidate to the user's current query, specifically its relevance to the topic and keywords used in the query, with the score positively correlated with the degree of relevance.

[0101] Taking Find X8 and Zhang San as examples again, unlike the above implementation method, this implementation method can push content based on scene context information instead of user interest preferences. That is, it determines the content that the user is currently highly interested in by using scene context information.

[0102] Contextual information refers to the user's actions and inputs within the current interface, especially their query behavior and needs within the current time period. This information is crucial for personalized ranking because it helps the system understand the user's immediate needs, thereby generating more accurate recommendations. For example, if a user enters "Find X8 performance review" in the search box on the target interface, it indicates that the user is looking for information related to the iPhone 15's performance. Additionally, if the user is browsing smartphone pages and recently viewed content related to "Find X8 Pro," the system might assume the user is more interested in performance comparisons of the Find X8. The final recommended content might then include: "Differences between Find X8 and Find X8 Pro," "What are the strongest features of the Find X8," and "Is the Find X8 worth buying?"

[0103] It can be seen that determining recommended content based on user preference information and determining recommended content based on scene context information are different. The former has a broader scope and can determine recommended content based on the user's overall interests, while scene context information can focus more on the user's query needs in the current time period and has greater real-time performance.

[0104] Therefore, considering that when recommending content to users, the recommended content can be determined by combining user preference information and scene context information, as another implementation method, assuming that the reference information includes scene context information and user preference information, similar to the above method, after identifying the subject to be queried in the target interface of the screen, the subject information corresponding to the subject to be queried is determined, and the candidate content corresponding to each subject information is determined based on the large language model. Then, the candidate content, scene context information and user preference information are input into the personalized ranking model. The personalized ranking model scores the candidate content through scene context information and user preference information, determines the score corresponding to each candidate content, sorts each candidate content according to the score from high to low, and selects the top N candidate contents as recommended content, where N is a positive integer greater than or equal to 1.

[0105] Specifically, based on contextual information (such as user query behavior, current page content, and operation history), the relevance of each candidate item to the contextual information is evaluated to determine a first score for each candidate item. Then, each candidate item is scored according to user preference information to obtain a second score. Finally, the first and second scores are weighted and summed to obtain the score for each candidate item. The weight corresponding to the contextual information is the first weight, and the weight corresponding to the user preference information is the second weight. These first and second weights can be set based on actual scenario needs. If it is desired that the recommended content is more focused on the user's current query needs, the first weight can be set greater than the second weight; if it is desired to recommend richer content to the user, the second weight can be set greater than the first weight.

[0106] S405: Display the recommended content through the target interface.

[0107] like Figure 5 As shown, leveraging the deep learning and natural language processing capabilities of large language models, the system can analyze user screen information, scene context information, and behavioral preference information in real time, and generate relevant queries. These queries not only contain the main information of the current user's mobile phone screen, but also predict other content that the user may be interested in based on the user's context information and behavioral preference information.

[0108] Specifically, the screenshot of the target interface is input into the Xiaobu Assistant APP. The Xiaobu Assistant client takes a screenshot of the user's current interface and calls the screen recognition SDK. The screenshot image is input to the on-device image subject label recognition model. The model outputs the subject labels that may exist in the image, outputting them in descending order of confidence. Based on the confidence, the top k labels are extracted as the subject labels of the user interface, that is, the k subjects to be queried are determined.

[0109] The system retrieves the user's historical query content within the target interface as contextual information. This screen information and contextual information are then input into the personalized recommendation model. The personalized recommendation model then retrieves the user's corresponding preference information from the data center. Finally, it inputs the screen subject information, contextual information, and preference information into the large model service, which includes a large language model and a personalized ranking model. The large language model generates queries in real time based on the current screen subject tags, which are added to the recommendation system as a recall. These queries are then combined with the user's personalized profile information (i.e., the user's preference information) and contextual information and input into the personalized ranking model. The ranking model scores the user's interest in these queries based on click probability and returns the three most interesting queries to the user.

[0110] To improve the accuracy and relevance of recommendations, a KTO (Knowledge, Tolerance, and Optimization) human preference alignment algorithm is employed. This algorithm adjusts the real-time generation strategy of the large language model based on real-time user click feedback. In this way, the system can continuously learn and improve, providing more personalized and relevant recommendations. Specifically, the system continuously monitors user behavior, such as clicks, searches, and dwell time. This behavioral data reflects users' current interests and preferences for certain content. For example, if a user clicks on a recommended query suggestion, it indicates that they are interested in that content. Based on this immediate feedback, the KTO algorithm adjusts the generation strategy of the large language model. This means that when the system detects that certain content types or features are more attractive to users than other content, the recommendation strategy is quickly adjusted. For example, if a user is more interested in "OPPO's Find X8 performance review," the system will prioritize similar content in the next recommendation.

[0111] In addition, through the real-time error detection system and the offline avoidance system, queries that may lead to incorrect answers can be detected and corrected in a timely manner, reducing the occurrence of bad experiences.

[0112] Specifically, real-time error-aware systems are used to monitor and identify potential problems in user input or model output. These problems can arise from various factors, such as language model misunderstandings, ambiguity, missing context, or the inherent vagueness of the input itself. For example, the system analyzes user queries in real time to identify potential errors or inconsistencies. For instance, some queries may be poorly worded, grammatically incorrect, ambiguous, or involve complex background knowledge. As the model generates an answer, the real-time error-aware system reviews the model's output to ensure the answer is reasonable, accurate, and consistent with the user's intent. Once a potential error is identified, the system makes immediate adjustments or provides feedback through built-in mechanisms. For example, the system may automatically modify the generated answer, flag the output, or suggest more information to the user.

[0113] Offline avoidance systems refer to systems that proactively analyze and correct potential erroneous answers in a non-real-time environment to avoid problems in real-world applications. This typically involves offline analysis of large amounts of historical data and model output to identify patterns or situations that may lead to errors and implement avoidance measures. Specifically, offline avoidance systems usually collect data from historical user interactions, including user feedback, model output errors, and common query problems. By analyzing this data, the system can identify certain common error patterns or potential causes of poor user experiences. Based on the analysis results, the system establishes a set of avoidance rules or strategies for the model to prevent common errors. For example, certain types of queries may trigger system errors, and offline avoidance systems will prevent this from happening by adjusting the model's generation strategy or adding filtering mechanisms. Offline avoidance systems may also involve regularly updating the model's knowledge base or fine-tuning the model to avoid known errors and vulnerabilities.

[0114] These two systems typically work together to improve system robustness and user experience. The real-time error detection system is responsible for detecting and correcting errors in real time during user interaction, reducing the occurrence of incorrect answers. The offline avoidance system, on the other hand, identifies common problems that the system may encounter in advance through the analysis of historical data, and avoids these problems by adjusting the model or setting rules, ensuring that users receive correct answers and effective assistance at every stage of the user experience.

[0115] Furthermore, after determining the recommended content corresponding to the subject information, the method further includes obtaining user operation data based on the recommended content; and updating the large language model based on the operation data.

[0116] Specifically, user action data is acquired. In a recommendation system, when recommended content is presented to a user, all of the user's various behaviors are recorded and converted into action data. This action data typically includes: click behavior, browsing time, and bounce behavior. For a recommendation query, click behavior refers to clicking the query suggestion; browsing time can be the time the user spends on the query suggestion and its corresponding response; and bounce behavior refers to the user closing the query suggestion or swiping it out of the display area of ​​the screen.

[0117] After acquiring user action data, the recommendation system needs to process and analyze this data to extract useful features. For example, specific analyses may include identifying user interest trends and preferences by analyzing user clicks and browsing behavior. For instance, if a user frequently clicks on health-related content, the recommendation system might consider the user to have a high level of health concern; and analyzing user preferences for certain types of content, such as text, video, or product categories.

[0118] Then, based on user action data, the language model can be fine-tuned to better understand user interests and preferences. For example, by collecting user behavior data and feedback, transforming this action data into a format suitable for model training, and using this behavior data to fine-tune the original large language model, customized training makes the model more personalized in recommendation scenarios.

[0119] Therefore, the system can adjust its recommendation strategy based on user feedback and behavioral data to improve user satisfaction and loyalty.

[0120] like Figure 6 As shown, Figure 6 (a) Figure 6 (b) and Figure 6 (c) demonstrates how recommended content is displayed when different types of content are displayed within the target interface.

[0121] like Figure 6 As shown in (a), the main information determined through screen recognition (i.e., identifying screenshots) includes glasses and group photography. Then, the recommended content determined based on this main information, scene context information, and user preference information includes: "What role does the glasses in the picture suit?", "What is the style of the glasses in the picture?", and "How to compose a group photo?". Figure 6 As shown in (b), if the main information identified through screen recognition includes flowers and grassland, then the recommended content can include "What kind of flower is this?", "What are some tips for taking landscape photos?", and "When does this flower bloom?". Figure 6 As shown in (c), if the main information identified through screen recognition includes books, then the identified recommended content includes: "What are some famous quotes in this book?", "How to distinguish between good and bad books?", and "Where is the author of this book from?"

[0122] Therefore, in this embodiment of the application, by combining the text generation capabilities of a large language model and the image analysis capabilities of computer vision, a query related to the user's current screen information is generated in real time, and the scene context information and user preference information are comprehensively considered to provide richer and more accurate recommended content.

[0123] This system leverages the deep learning and natural language processing capabilities of large language models, combined with user mobile phone screen information, to generate relevant queries. It incorporates screen information perception capabilities, employing computer vision technology to perceive the main elements of the mobile phone screen, thereby improving the intelligent assistant's understanding of user needs. It can generate queries relevant to the user's current mobile phone screen in real time, enhancing the intelligent assistant's response speed and recommendation quality. It utilizes the KTO (Knowledge, Technology, and Objective) human preference alignment algorithm to evaluate and improve the relevance between generated queries and user needs. Furthermore, it continuously improves recommendation strategies based on user feedback and behavioral data to enhance user satisfaction and loyalty.

[0124] Please see Figure 7 The diagram shows a structural block diagram of a content recommendation device 700 provided in an embodiment of this application. The device may include: an acquisition unit 701, a determination unit 702, a recommendation unit 703, and a display unit 704.

[0125] The acquisition unit 701 is used to acquire the display content corresponding to the target interface currently displayed on the screen of the electronic device.

[0126] The determining unit 702 is used to determine the subject information corresponding to the subject to be queried by the user based on the displayed content.

[0127] Furthermore, the displayed content is a screenshot of the target interface, and the determining unit 702 is also used to identify the screenshot of the target interface, determine the subject object, and determine the subject information corresponding to the user's subject to be queried based on the subject object.

[0128] Furthermore, the determining unit 702 is also used to determine the target query content entered by the user in the target interface during the current time period; and to determine the subject information corresponding to the subject to be queried by the user based on the subject object and the target query content.

[0129] The recommendation unit 703 is used to determine the recommended content corresponding to the subject information.

[0130] Furthermore, the recommendation unit 703 is also used to determine reference information, which includes at least one of the user's corresponding preference information and scene context information, wherein the scene context information includes the user's historical query content; and to determine the recommended content based on the subject information and the reference information.

[0131] Furthermore, the recommendation unit 703 is also used to acquire user operation data based on the recommended content; and to update the large language model based on the operation data.

[0132] Display unit 704 is used to display the recommended content through the target interface.

[0133] Furthermore, the display unit 704 is also used to obtain the response content corresponding to the query suggestion if a user trigger operation on the query suggestion is detected, and to display the response content in the target interface.

[0134] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the described devices and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0135] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0136] Please refer to Figure 8 This document illustrates a structural block diagram of an electronic device according to an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0137] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0138] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0139] Please refer to Figure 9 This diagram illustrates a structural block diagram of a computer-readable medium provided in an embodiment of this application. The computer-readable medium 900 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0140] The computer-readable medium 900 may be an electronic storage device such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable medium 900 includes a non-volatile computer-readable storage medium. The computer-readable medium 900 has storage space for program code 910 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 910 may, for example, be compressed in a suitable form.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A content recommendation method characterized by, Applied to electronic devices, the method includes: Obtain the display content corresponding to the target interface currently displayed on the screen of the electronic device; Based on the displayed content, determine the subject information corresponding to the subject to be queried by the user; Determine the recommended content corresponding to the main information; The recommended content is displayed on the target interface.

2. The method according to claim 1, characterized in that, The displayed content is a screenshot of the target interface, and the step of determining the subject information corresponding to the user's subject to be queried based on the displayed content includes: Identify the screenshot image of the target interface to determine the main object; Based on the subject object, determine the subject information corresponding to the subject to be queried by the user.

3. The method according to claim 2, characterized in that, The step of determining the subject information corresponding to the user's subject to be queried based on the subject object includes: Determine the target query content entered by the user on the target interface within the current time period; Based on the subject object and the target query content, determine the subject information corresponding to the subject to be queried by the user.

4. The method according to claim 1, characterized in that, Determining the recommended content corresponding to the subject information includes: Determine reference information, which includes at least one of the user's preference information and scenario context information, wherein the scenario context information includes the user's historical query content; The recommended content is determined based on the subject information and the reference information.

5. The method according to claim 4, characterized in that, After determining the recommended content corresponding to the subject information, the method further includes: Obtain user action data based on the recommended content; The large language model is updated based on the operational data, wherein the large language model is used to determine the recommended content based on the subject information and the reference information.

6. The method according to claim 1, characterized in that, The recommended content is a query suggestion.

7. The method according to claim 5, characterized in that, After displaying the recommended content through the target interface, the process further includes: If a user triggers an action in response to a query suggestion, the corresponding response content is retrieved and displayed on the target interface.

8. A content recommendation device, characterized in that, Applied to electronic devices, the device includes: The acquisition unit is used to acquire the display content corresponding to the target interface currently displayed on the screen of the electronic device; A determining unit is used to determine the subject information corresponding to the user's subject to be queried based on the displayed content; The recommendation unit is used to determine the recommended content corresponding to the subject information; A display unit is used to display the recommended content through the target interface.

9. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable medium stores processor-executable program code, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1-7.

Images