Service pushing method and device and electronic equipment

Abstract

The invention discloses a service pushing method and device and electronic equipment, and belongs to the technical field of data processing. Obtaining page information of the electronic equipment, context information corresponding to the page information and personal preference information of a user, wherein the page information is information contained in a first page which is being displayed by the electronic equipment; inputting the page information, the context information and the personal preference information into a multi-modal large model, predicting the intention of the user through the multi-modal large model, and outputting recommendation information matched with the intention; and carrying out service pushing based on the recommendation information.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a service push method, device and electronic device. Background Art

[0002] With the development of Internet technology, electronic devices can provide users with more and more services through applications. For example, electronic devices can provide users with ticket purchasing services, multimedia playback services, reservation services, etc., to solve all aspects of problems encountered in users' lives and bring convenience to users' lives.

[0003] In the related art, in order to push services that better meet the needs of users, users are usually required to provide information related to their needs one by one. For example, when a user wants to search for information about performances that he or she is interested in recently to purchase tickets, the user often needs to provide the electronic device with relevant information such as the type of performance he or she wants to watch, the ticket price budget range, location preference, preferred time, etc. This requires the user to interact with the electronic device multiple times, which is cumbersome. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a service push method, device and electronic device, which can determine the user's intention based on the page browsed by the user when the user browses the page, so as to automatically push services that meet the user's real intention, so that the user can directly access the required service based on the push, without the user providing too much information, thereby reducing the user's interactive operations.

[0005] In a first aspect, an embodiment of the present application provides a service push method, including:

[0006] Inputting the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predicting the user's intention through the multimodal large model, and outputting the recommended information matching the intention;

[0007] Based on the recommended information, service push is carried out.

[0008] In a second aspect, an embodiment of the present application provides a service push device, characterized in that it includes:

[0009] A processing module, used for inputting the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predicting the user's intention through the multimodal large model, and outputting recommendation information matching the intention;

[0010] The push module is used to push services based on recommended information.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0014] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0015] In an embodiment of the present application, the page information of the first page displayed on the display screen and the context information of the page information are input into the multimodal large model, the user's intention is predicted by the multimodal large model, and the recommended information matching the intention is output; based on the recommended information, the service is pushed. According to the embodiment of the present application, when the user browses a page, the user's intention can be directly determined based on the page browsed by the user, so that the service that meets the real intention is automatically pushed to the user, without the user providing information, reducing the user's interactive operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a service push method provided by some embodiments of the present application;

[0017] Figure 2 It is a page schematic diagram provided by some embodiments of the present application;

[0018] Figure 3 It is a page schematic diagram provided by some embodiments of the present application;

[0019] Figure 4 It is a page schematic diagram provided by some embodiments of the present application;

[0020] Figure 5 It is a page schematic diagram provided by some embodiments of the present application;

[0021] Figure 6 It is a flowchart of a service push method provided by some embodiments of the present application;

[0022] Figure 7It is a page schematic diagram provided by some embodiments of the present application;

[0023] Figure 8 It is a page schematic diagram provided by some embodiments of the present application;

[0024] Fig. 9 It is a flowchart of a service push method provided by some embodiments of the present application;

[0025] Fig.10 is a schematic diagram of a dialog box provided by some embodiments of the present application;

[0026] Fig.11 is a schematic diagram of a dialog box provided by some embodiments of the present application;

[0027] Fig.12 is a schematic diagram of a service push device provided by some embodiments of the present application;

[0028] Fig.13 is a schematic block diagram of an electronic device provided by some embodiments of the present application;

[0029] Fig.14 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] Page access entry signs refer to signs or indicators used to guide users to a specific page. Page access entry signs can be expressed in many different forms, such as URLs, sharing cards, text links, graphic buttons, graphic codes, etc.

[0033] Card: It is an information carrier with a closed outline, which provides various information of various applications in a condensed form, intuitively and quickly, and is used for the display and interaction of various program information. In the Android system, a card is a window interface that can be stacked and presented. The shape of a card is usually a rounded rectangle, similar to the shape of a credit card, but it can also be other shapes. Cards make it more convenient and intuitive for users of electronic devices to obtain information and perform operations, and therefore are widely used. In the embodiments of the present application, the size, position and shape of the card are not limited. A share card is a card that can be used to share specific content. A share card can contain information in various forms such as text, pictures, videos, etc., and provide a jump link. Users can jump to a specified page by clicking on the share card.

[0034] A URL usually consists of a protocol, domain name, path, parameters, etc. The protocol includes but is not limited to "http" or "https". Users can accurately access the target page by clicking on the URL.

[0035] Text links usually appear as blue underlined text, such as the common text links such as "Home", "Product Introduction", "Contact Us" in the navigation bar of a web page. When a text link is clicked, it leads to a specific page.

[0036] Graphic buttons can be buttons with specific icons and text, such as buttons with text such as "Buy Now" or "View Details", buttons with simple graphic logos such as a small house, etc. Users can enter the corresponding page by clicking these buttons.

[0037] Graphic codes are usually QR codes, bar codes, etc. Users can quickly enter the corresponding web page or application page by scanning the graphic code with a scanning device.

[0038] A control refers to a visual element in a graphical user interface (GUI) that is used to implement a specific function or operation. A control can have many different forms, such as buttons, text boxes, drop-down lists, check boxes, sliders, labels, etc.

[0039] It should be noted that the acquisition, storage, use and processing of data in the embodiments of the present application are in compliance with the relevant provisions of national laws and regulations.

[0040] The following, in conjunction with the accompanying drawings, describes in detail the service push method, device, equipment, storage medium and program product provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0041] Combine the following Figure 1-Figure 4The service push method provided in the embodiment of the present application is described in detail. It should be noted that the execution subject of the service push method provided in the embodiment of the present application can be an electronic device, and the electronic device includes but is not limited to a smart phone, a laptop computer, a tablet computer, a car computer, a smart TV, etc. In the embodiment of the present application, the service push method provided in the embodiment of the present application is described by taking the execution of the service push method by an electronic device as an example.

[0042] The service push method provided in the embodiment of the present application can be applied to the scenario of backward service push. Among them, one specific application scenario is that when a user browses an application that can order concert tickets, the ticket purchase information of the concert tickets that the user wants to buy is automatically pushed to the user. Another specific application scenario is that when a user browses an application that can provide music playback services, music that meets the user's preferences is automatically played for the user.

[0043] See also Figure 1 , is a flow chart of a service push method provided in an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps 110 to 120, which are described below respectively.

[0044] Step 110. Input the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predict the user's intention through the multimodal large model, and output recommended information matching the intention.

[0045] Here, the display screen refers to a screen that is built into or connected to an electronic device. For example, if the electronic device is a mobile phone, the display screen may be the screen of the mobile phone; if the electronic device is a computer, the display screen may be the screen of the computer itself or an extended screen connected to the computer. The display screen is used to display information such as text and images generated by the electronic device to the user through display technology. In addition, the display screen can also interact with the user through technologies such as touch screens.

[0046] The first page is the page being displayed on the display screen, that is, the page being browsed by the user. The first page can be a page in any application in the electronic device. Applications in electronic devices include, but are not limited to, ticket purchasing applications, music applications, information applications, office applications, shopping applications, learning applications, navigation applications, social applications, and the like. When a user needs a certain type of service, he or she will open an application in the electronic device that can provide such services to obtain the corresponding services. Therefore, the pages browsed by the user in the electronic device are usually related to the services required by the user. Therefore, the page information in the pages browsed by the user can usually reflect the user's service needs. Based on this, in this embodiment, the page being displayed on the display screen is taken as the first page, so as to obtain the user's true intention based on the page information of the first page, thereby determining the user's service needs.

[0047] For example, see Figure 2 When the user wants to order concert tickets, he opens the application capable of ordering concert tickets in the electronic device and browses page 200 thereof. At this time, the electronic device determines page 200 as the first page.

[0048] For example, see Figure 3 When the user wants to listen to music, he opens the application that can provide music playback service in the electronic device and browses page 300 therein. At this time, the electronic device determines page 300 as the first page.

[0049] The page information of the first page may be all the information of the first page displayed on the display screen, or may be part of the information of the first page displayed on the display screen. The page information of the first page includes, but is not limited to, various types of information such as text, pictures, videos, audios, links, tables, forms, various controls, etc. in the first page.

[0050] For example, see Figure 2 , the first page is page 200, and page 200 includes four concert promotion pictures of singer A, singer B, singer C, and singer D, which are concert promotion picture 211 of A's world tour concert, concert promotion picture 212 of B's ​​national tour concert, concert promotion picture 213 of C's first personal concert, and concert promotion picture 214 of D's national tour concert. The electronic device can obtain information contained in at least one of the concert promotion pictures 211-214 in page 200 as page information of the first page.

[0051] For example, see Figure 3 , the first page is page 300, and page 300 includes four playlist cover pictures of singer A, singer B, singer C, and singer D, which are playlist cover picture 311 corresponding to singer A's music collection, playlist cover picture 312 corresponding to singer B's music collection, playlist cover picture 313 corresponding to singer C's music collection, and playlist cover picture 314 corresponding to singer D's music collection. The electronic device can obtain information contained in at least one of the four playlist cover pictures 311-314 displayed on page 300 as page information of the first page.

[0052] The context information corresponding to the page information of the first page refers to the surrounding information associated with the page information. The context information corresponding to the page information includes, but is not limited to, information in other pages associated with specific objects, events, texts or situations in the page information, information input by the user, etc. As mentioned above, the page information is the page information in the application corresponding to the service currently required by the user. The page information can reflect the service currently required by the user, and the context information corresponds to the page information, so the context information can also reflect some of the user's current service needs. Therefore, the context information based on this page information can more accurately understand the user's real service needs.

[0053] For example, the first page is Figure 2 In the page 200 shown, the context information of the page information may include but is not limited to the concert-related information and singer-related information previously browsed or input by the user.

[0054] For example, the first page is Figure 3 In the page 300 shown, the context information of the page information may include but is not limited to music-related information and singer-related information previously browsed or input by the user.

[0055] In some embodiments of the present application, before the above step 110, the electronic device may first obtain the page information of the first page displayed on the display screen and the context information of the page information, and then input the obtained page information and context information into the multimodal large model through the above step 110 to predict the recommended information.

[0056] In some embodiments of the present application, the electronic device can obtain the page information of the first page through screenshot technology and image analysis technology. When the electronic device displays the first page on the display screen, it can take a screenshot through a preset screenshot tool to obtain a screenshot image, input the screenshot image into a model or a large model with image understanding ability, and perform image understanding on the screenshot image through the model or the large model to obtain the information contained in the screenshot image, and determine the information contained in the screenshot image as the page information of the first page. Among them, the model or large model with image understanding ability includes but is not limited to open source or self-developed models or large models with image understanding ability. For example, models or large models with image understanding ability include but are not limited to YOLO (You Only Look Once), RCNN (Region-based Convolutional Neural Network) series, LayoutLM, Visual GLM-6B, GPT-4o, Gemini, etc.

[0057] In some embodiments of the present application, after obtaining the page information of the first page, the electronic device may obtain the context information of the page information through a web crawler, a preset interface function, or the like.

[0058] After obtaining the page information of the first page and the context information of the page information, the page information and the context information are input into the multimodal large model through the above step 110, so as to predict the user's intention through the multimodal large model, and output the recommended information matching the intention. Among them, after receiving the input page information and context information, the multimodal large model can integrate the information of different modes in the page information and context information, so as to more accurately understand and analyze the page information and context information, and then accurately determine the user's intention. After obtaining the user's intention, the multimodal large model can search the Internet to find information matching the user's intention, and determine the information matching the user's intention as the recommended information. The multimodal large model used in this embodiment includes but is not limited to open source or self-developed artificial intelligence large models such as GPT-4o, Gemini, Kosmos-1, and Palm-E.

[0059] The multimodal big model is a pre-trained artificial intelligence big model that can process and fuse multiple different modal data. Multiple different modal data include but are not limited to different types of data forms such as text, image, audio, video, etc. The multimodal big model has multimodal understanding capabilities. Therefore, the user's true intention can be accurately determined through the multimodal big model. Online search refers to the act of retrieving required information through a computer network. Online search through the multimodal big model can find recommended information that matches the user's intention in the vast ocean of information, improve the richness and comprehensiveness of the recommended information, and obtain real-time updated information to improve the timeliness of the recommended information.

[0060] For example, the first page is as follows Figure 2 In the page 200 shown, the page information of the first page includes the information contained in the concert promotion picture 211, the concert promotion picture 212, the concert promotion picture 213 and the concert promotion picture 214, and the context information of the page information includes information related to singer A. Based on this, after the page information and the context information are input into the multimodal big model, the multimodal big model determines through understanding and analysis that the user currently wants to buy A's concert ticket. Based on this, the multimodal big model performs an online search to obtain concert information related to A's concert, and then outputs the searched concert information as recommended information. Among them, the concert information includes but is not limited to concert description information, ticket information, etc. The concert description information includes but is not limited to concert time, performance venue, ticket price, precautions, etc.

[0061] For example, the first page is as follows Figure 3 In the page 300 shown, the page information of the first page includes the information contained in the song list cover picture 311, the song list cover picture 312, the song list cover picture 313 and the song list cover picture 314, and the context information of the page information includes information related to singer A. Based on this, after the page information and the context information are input into the multimodal large model, the multimodal large model determines through understanding and analysis that the user currently wants to listen to A's music. Based on this, the multimodal large model performs an online search to obtain music information related to A, and then outputs the searched music information as recommendation information. Among them, the music information includes but is not limited to music collections, albums, etc.

[0062] It should be noted that the recommendation information finally determined may include the page information of the first page, or may not include the page information of the first page, but include other information in addition to the page information. The above examples are merely exemplary and do not constitute a limitation of the embodiments of the present application.

[0063] Step 120: Push service based on the recommendation information.

[0064] In some embodiments of the present application, after determining the recommended information that meets the user's intention, the electronic device determines one or more services associated with the recommended information in the electronic device, and pushes one or more services associated with the recommended information to the user. The services associated with the recommended information may include services in the application to which the first page belongs, or services in other applications other than the application to which the first page belongs.

[0065] For example, the first page is as follows Figure 2 In the page 200 shown in FIG. 1 , the recommended information determined in step 110 is concert information related to A's concert. When the electronic device determines that tickets for A's concert have been sold, the electronic device can push services based on the recommended information by, for example, Figure 4 The floating window 410 shown pushes services to the user. Figure 4As described, the floating window 410 includes concert description information and ticket purchase order, wherein the concert description information includes "I have found out that there is a world tour concert of A in Changsha on April 15, and I have ordered a ticket for the inner field at a price of 1,580 for you", and the ticket purchase order is an order corresponding to the inner field ticket that has been ordered, and the ticket purchase order includes the name of the concert "A's World Tour Concert", the concert venue, the concert time, the ticket price and other information. Among them, the ticket purchase order can be a paid order or an unpaid order, which can be determined according to the user's payment settings. For example, if the user turns on the automatic payment function and the electronic device can automatically complete the order payment, then the ticket purchase order is a paid order. If the user does not turn on the automatic payment function and the electronic device cannot automatically complete the order payment, then the ticket purchase order is an unpaid order. The user can jump to the payment page by clicking on the ticket purchase order in page 400 to complete the payment. If the electronic device determines that the tickets for A's concert have not yet been sold, the electronic device can push the concert description information and the reservation order to the user through the floating window 410 when pushing the service based on the recommended information, wherein the concert description information includes "I have found that there is A's world tour concert in Changsha on April 15, and I have reserved tickets for you at the price of 1,580 yuan for the inner field tickets", and the reservation order is an order for reserving the inner field tickets, and the reservation order includes the concert name "A's world tour concert", the concert location, the concert time, the ticket price and other information. In this way, after the concert tickets are on sale, the electronic device can automatically grab tickets based on the reservation order.

[0066] For example, the first page is as follows Figure 3 In the page 300 shown in FIG. 1 , the recommended information determined in step 110 includes music information related to A, and the music information includes A's music collection. Based on this, when the electronic device pushes a service based on the recommended information, it can do so as follows: Figure 5 The music playing component 510 shown plays the music in the music collection.

[0067] The service push method provided in the embodiment of the present application inputs the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predicts the user's intention through the multimodal large model, and outputs the recommended information matching the intention; based on the recommended information, the service is pushed. According to the embodiment of the present application, when the user browses a page, the user's intention can be directly determined based on the page browsed by the user, so as to automatically push services that meet the real intention to the user, without the user providing information, thereby reducing the user's interactive operations.

[0068] In some embodiments, the electronic device can directly determine all the information in the first page as the page information of the first page. However, considering that not all the information displayed in the first page may be related to the service required by the user, if all the information displayed in the first page is directly determined as the page information and input into the multimodal large model, it may cause the multimodal large model to determine the user intention inaccurately, thereby affecting the accuracy of the final service push. In view of this, in order to improve the accuracy of service push, when not all the information in the first page is related to the service required by the user, the user can select information related to the service required by the user from the first page, and then the electronic device determines the information selected by the user as the page information of the first page to be input into the multimodal large model. Based on this, see Figure 6 Before the above step 110, the electronic device can obtain the page information of the first page through the following steps 610-620.

[0069] Step 610: Receive selection input for the first page.

[0070] Here, the selection input is an operation for selecting an area of ​​interest in the first page. The area of ​​interest is a page area in the first page where information related to the service required by the user is located. In the case where not all the information in the first page is related to the service required by the user, the user can select the area of ​​interest in the first page through the selection input, so that the electronic device obtains the page information to be input into the multimodal macro model from the area of ​​interest.

[0071] In some embodiments of the present application, the selection input may be a user's adjustment input to a selection box, wherein the selection box is used to circle a page area in a page displayed on a display screen, and the user can adjust the page area circled by the selection box by adjusting the size or position of the selection box. Based on this, the user can call up the selection box when selecting an area of ​​interest in the first page, and the called selection box can be suspended and displayed in the first page. The user can adjust the position or size of the selection box by dragging the corners or edges of the selection box, so that the selection box selects the area of ​​interest to the user in the first page. Among them, the calling method of the selection box includes but is not limited to calling up by voice command, calling up from a shortcut interface, or calling up by shortcut keys. For example, the user can call up the selection box by inputting a voice command of "calling up the selection box" to the electronic device. The user can also call up the selection box by opening the shortcut interface of the electronic device and clicking and inputting the selection box logo in the shortcut interface. The user can also call up the selection box by clicking the shortcut key used to call up the selection box on the electronic device.

[0072] In some embodiments of the present application, when the display screen is a touch screen, the selection input may include at least one of the following: fingertip touch input and knuckle touch input. Among them, fingertip touch input refers to input achieved by operating the surface of the touch screen using the user's finger. Knuckle touch input refers to input achieved by operating the surface of the touch screen using the user's knuckle. When browsing the first page, the user can draw a selection box around the page area where the information of interest is located in the first page through fingertip touch input or knuckle touch input, thereby encircling the area of ​​interest through the selection box.

[0073] For example, when a user wants to buy concert tickets, he opens an application in an electronic device that can buy concert tickets and browses the application. Figure 7 In the page 200 shown, the application opened by the user can be used to purchase other tickets besides concert tickets, such as Figure 7 As shown, page 200 includes, in addition to concert promotion pictures 211, 212, 213, and 214, two drama promotion pictures, namely, drama promotion picture 221 corresponding to drama E and drama promotion picture 222 corresponding to drama F. Figure 7 As shown, the user can make selection inputs on page 200, and circle the page area where the concert-related information is located in page 200 through the selection box 710. The circled page area includes concert promotion picture 211, concert promotion picture 212, concert promotion picture 213 and concert promotion picture 214.

[0074] For example, when a user wants to listen to music, he opens an application in an electronic device that can play music and browses the application. Figure 8 In the page 300 shown in FIG. 1 , the application opened by the user can play audio books in addition to music, such as Figure 8 As shown, page 300 includes, in addition to playlist cover images 311, playlist cover images 312, playlist cover images 313, and playlist cover images 314, two audiobook cover images, namely, audiobook cover image 321 corresponding to audiobook G and audiobook cover image 322 corresponding to audiobook H. Figure 8 As shown, the user can make selection inputs on page 300 by circling the page area where music-related information is located in page 300 through selection box 810. The circled page area includes playlist cover image 311, playlist cover image 312, playlist cover image 313 and playlist cover image 314.

[0075] Step 620. In response to the selection input, the information in the page area selected by the selection input is determined as the page information input into the multimodal macro model.

[0076] In some embodiments of the present application, the electronic device determines the page area selected by the selection input in response to the user's selection input of the first page, obtains the information in the page area, and determines the information in the page area as the page information to be input into the multimodal macro model. Because the page area selected by the user through the selection input is the area of ​​interest, the information in the page area is information related to the service required by the user, so that the page information related to the service required by the user can be obtained.

[0077] For example, see Figure 7 The electronic device determines the information contained in the concert promotion pictures 211-214 circled in the selection box 710 as the page information to be input into the multimodal macro model.

[0078] For example, see Figure 8 The electronic device determines the information contained in the playlist cover images 811-814 circled in the selection box 810 as the page information to be input into the multimodal large model.

[0079] In some embodiments of the present application, when the electronic device determines the page information to be input into the multimodal large model based on the page area selected by the selection input, it can use the local screenshot technology to only take a screenshot of the page area selected by the selection input, and then use the model or large model with image understanding ability to perform image understanding on the image obtained by the local screenshot, thereby obtaining the page information to be input into the multimodal large model. Among them, the model or large model with image understanding ability includes but is not limited to open source or self-developed models or large models with image understanding ability. For example, models or large models with image understanding ability include but are not limited to YOLO (You Only Look Once), RCNN (Region-based Convolutional Neural Network) series, LayoutLM, VisualGLM-6B, GPT-4o, Gemini, etc.

[0080] Through the above technical scheme, the page information to be input into the multimodal large model can be screened according to the actual needs of the user, and the information in the first page that is not related to the service required by the user can be avoided from being input into the multimodal large model. In this way, the recommended information output by the multimodal large model can be more in line with the actual needs of the user, thereby improving the accuracy of service push.

[0081] In some embodiments, when the selection input includes a knuckle touch input, the knuckle touch input includes tapping the screen with the knuckle and circling an area along a reference direction from the tapping position. The reference direction includes but is not limited to a clockwise direction and a counterclockwise direction. Based on this, when the user needs to circle the information on the first page, the user can use the knuckle to touch the screen and keep the knuckle from leaving the screen, and then drag the knuckle along the reference direction to draw a closed shape around the information to be selected.

[0082] Based on the above, see Fig. 9 , the above step 620 can be implemented through the following steps 910-930.

[0083] Step 910: Obtain knuckle operation parameters of knuckle touch input.

[0084] Among them, the knuckle operation parameters include at least one of the following: touch area, touch pressure, and circle selection trajectory.

[0085] In some embodiments of the present application, the electronic device can obtain the touch pressure of the knuckle touch input through the pressure sensing function on the display screen.

[0086] In some embodiments of the present application, the electronic device can detect the distribution and shape characteristics of touch points in the knuckle touch input through the touch sensing layer on the display screen, and then obtain the touch area of ​​the knuckle touch input.

[0087] In some embodiments of the present application, the electronic device can obtain the circle selection trajectory of the knuckle touch input through a trajectory tracking function.

[0088] Step 920: Determine the knuckle selection area based on the knuckle operation parameters.

[0089] In some embodiments of the present application, the electronic device can identify whether it is a knuckle touch input based on the touch area or touch pressure of the knuckle touch input. Since the pressure exerted by the knuckles on the display screen is different from the pressure of ordinary finger touch, the electronic device can detect this specific pressure change through the built-in pressure sensor to identify the knuckle touch input. The contact area between the knuckles and the display screen is also unique. The contact area of ​​the knuckles is relatively small and the shape is more regular, which is significantly different from the large area contact of the fingertips. Therefore, the electronic device can determine whether it is a knuckle touch input by analyzing the distribution and shape changes of the touch points.

[0090] Furthermore, after the electronic device recognizes the start of the knuckle touch input based on the touch area or touch pressure of the knuckle touch input, it uses the trajectory tracking function to monitor the position changes of the touch points in real time as the knuckles move on the display screen, and connects these position points to form a trajectory. By continuously tracking the movement trajectory of the knuckles, the boundary of the circled area is accurately determined, thereby determining the knuckle circled area. For example, when a user draws a circle on the display screen with the knuckles, the electronic device can determine the radius and center position of the circular circled area based on the movement path of the knuckles. When a user draws a rectangular frame on the display screen with the knuckles, the electronic device can determine the positions of the corner points and sides in the rectangular circled area based on the movement path of the knuckles.

[0091] In some embodiments of the present application, in the process of determining the knuckle circle selection area, the electronic device can also dynamically adjust the trajectory of the knuckle touch input according to the moving speed and moving direction of the knuckle. If the knuckle moves faster, the electronic device can appropriately expand the range of the circle selection area. The electronic device can also make real-time corrections to the trajectory according to the moving direction of the knuckle to ensure that the boundary of the circle selection area is consistent with the user's intention. For example, when the user changes direction during the circle selection process, the system will promptly adjust the shape of the circle selection area to adapt to the new direction change. In this way, the knuckle circle selection area can completely contain the content that the user wants to select.

[0092] In some embodiments of the present application, in order to allow the user to clearly see the area they have circled, the electronic device can also display the boundaries and range of the circle in real time on the display screen. For example, the electronic device can present the circled area in a highlighted display or a semi-transparent overlay, so that the user can intuitively understand whether their selection is accurate. As the knuckles move, the boundaries of the circled area will be updated in real time, allowing the user to adjust the circled range at any time to ensure that the correct content is selected.

[0093] When the user has completed the selection, he can confirm the selected area by clicking a confirmation button or lifting his finger joint. The electronic device will finally determine the selected area after receiving the confirmation signal. Before confirmation, the user can adjust the selected area at any time, or cancel the selection operation and make a new selection. This design can improve the accuracy and flexibility of user operations.

[0094] Step 930: Determine the knuckle circled area as the page information for inputting the multimodal large model.

[0095] After determining that the user has completed the selection through the knuckle touch input, the electronic device determines the knuckle selected area as the page information of the input multimodal large model.

[0096] Knuckle touch technology determines the user's operation intention by identifying the contact area, force, vibration frequency and other characteristics of the knuckles and the screen, which is significantly different from the touch methods of other parts such as the fingertips. This unique recognition method enables electronic devices to accurately distinguish between normal finger touch operations and knuckle touch inputs, reducing the possibility of false touches. In addition, when using electronic devices in daily life, users' hands may inadvertently touch the screen, but because the hardness and touch method of the knuckles are different from other parts of the fingers, the system will not misjudge these ordinary touches as knuckle touch inputs, thereby reducing unnecessary function triggering and ensuring the accuracy and stability of user operations.

[0097] In some embodiments, the context information of the page information includes at least one of the following: page information in the second page, and page information associated with the page information in the history page. The second page is the page indicated by the page access entry identifier on the first page; the history page is the page displayed by the display screen in the page display time period before the first page is displayed.

[0098] Here, page access entry identifiers include but are not limited to URLs, graphic codes, sharing cards, and other identifiers that can achieve page jumps. Among them, the URL is the most direct page access entry identifier, which clearly points to a specific resource location on the network; graphic codes such as QR codes can also be used as page access entries. Graphic codes include access information in an encoded manner, and can jump directly to the corresponding resource page after scanning the code; sharing cards present page access entries, such as URLs, in a more friendly and attractive way, and may integrate multiple entries or provide some additional guidance information. For example, in an online learning platform, the course URL is a direct way to access course resources, and making course information into QR codes or sharing cards allows users to more easily quickly enter the course learning page from different scenarios such as offline promotional activities and mobile device sharing.

[0099] In this embodiment, the second page is the page corresponding to the page access entry identifier contained in the page information of the first page. The second page corresponding to the page access entry in the first page is an important part of the context information. The page access entry can guide the user to other pages of the same application or different applications, providing the user with more references and background knowledge so that the user can understand the content of the first page more deeply. For example, the first page is Figure 2 The page 200 shown includes a URL Http / / xxxx, and the page pointed to by the URL includes relevant introductions to the four concerts corresponding to the concert promotion pictures 211, 212, 213 and 214. For another example, the first page is as follows Figure 3The page 300 shown includes a QR code 330, and the page pointed to by the QR code 330 includes relevant introductions to four playlists corresponding to playlist cover images 311, 312, 313, and 314. Based on this, determining the second page corresponding to the page access entry identifier included in the page information of the first page as the context information of the page information can facilitate a deeper understanding of the page information.

[0100] In this embodiment, the historical page is a page displayed by the display screen in a page display time period before the display screen displays the first page. The page display time period can be set according to actual conditions. For example, the historical page can be a page displayed within two hours before the display screen displays the first page. Usually, there is a certain correlation between the content browsed by the user in a time period. Therefore, the historical pages browsed by the user before browsing the first page may contain information associated with the page information in the first page. Based on this, in this embodiment, the electronic device uses the page information associated with the page information of the first page in the historical pages displayed by the display screen in the page display time period before the display screen displays the first page as context information, so as to obtain more comprehensive information and facilitate a more accurate understanding of the user's intention.

[0101] In some embodiments of the present application, in order to obtain page information in historical pages that is associated with page information of a first page, the electronic device can, after obtaining the page information of the first page, perform keyword extraction on the page information, and search for synonyms and associated words corresponding to the extracted keywords from a third-party vocabulary to expand the keywords and obtain a final keyword set, and then filter the information in the obtained historical pages based on the keyword set, select information that matches any keyword in the keyword set, and use the information that matches any keyword as context information corresponding to the page information.

[0102] For example, the first page is as follows Figure 2 The page 200 shown in FIG. 1 shows the page information of the first page. Figure 7 The information related to the concert circled in the selection box 710 shown, the electronic device extracts keywords from the page information of the first page, and obtains the keywords "concert", "A", "B", "C" and "D". Keywords such as "performance" and multiple singer names are obtained through keyword expansion, and "concert", "A", "B", "C" and "D" and multiple singer names are combined into a keyword set. Based on the keyword set, the electronic device filters the information in the historical pages, and obtains information about singer "A" that matches the keyword set, and uses "A" as context information of the page information.

[0103] For example, the first page is as follows Figure 3The page 300 shown in FIG. 1 is as follows: Figure 8 The information related to the playlist is circled in the selection box 810 shown. The electronic device extracts keywords of the page information of the first page by performing keyword extraction, and obtains keywords such as "playlist", "song", "collection", "A", "B", "C" and "D". Keywords such as "music", "album" and multiple singer names are obtained by keyword expansion. "Playlist", "song", "collection", "A", "B", "C", "D", "music", "album" and multiple singer names form a keyword set. The electronic device filters the information in the historical pages based on the keyword set, and obtains the singer "A" that matches the keyword set, and then uses "A" as the context information of the page information. Through keyword matching, page information associated with the page information of the first page can be quickly filtered out from the historical pages.

[0104] In some embodiments, a software agent capable of communicating with a user is installed in the electronic device. The software agent can understand the instructions input by the user and reply, perform tasks, etc. based on the instructions input by the user. Among them, the instructions input by the user include but are not limited to voice instructions, text instructions, etc. Taking voice instructions as an example, the software agent can perceive the user's voice instructions through voice recognition, understand and make decisions using natural language processing technology, and then take action through voice synthesis or displaying the results on the screen. When the user has a service demand, the user can wake up the software agent of the electronic device and then communicate with the software agent through a dialog box. Based on this, in order to push the service more accurately, before the above step 110, a dialogue message can be received first. The dialogue message refers to the message in which the user asks questions or queries the content to the software agent when the user is communicating with the software agent.

[0105] For example, see Fig.10 When users want to buy concert tickets, they open Figure 2 The page 200 shown in the figure can be used to input the inquiry command "Are there any performances in the near future?" into the dialog box 1010 for communicating with the software agent by voice input or text input.

[0106] For example, see Fig.11 When the user wants to listen to music, he can open Figure 3 The page 300 shown in the figure can be used to input the command "recommend music" into the dialog box 1110 used for communicating with the software agent by voice input or text input.

[0107] Accordingly, in the above step 110, the page information of the first page, the context information of the page information and the dialogue message are input into the multimodal big model, the user's intention is predicted by the multimodal big model, and the recommendation information matching the intention is output.

[0108] Through the above technical solution, services can be pushed to users based on data in two dimensions: the pages browsed by users and the instructions input by users, thereby improving the accuracy of service push.

[0109] In some embodiments, if before the above step 110, the user has a dialogue with the software agent through a dialog box, then in the above step 120, the electronic device can display the recommended services to the user through the dialog box. Fig.10 As shown, the dialog box 1010 shows the user the concert description information "I have found that A's world tour concert will be held in Changsha on April 15, and I have reserved tickets for you at a price of 1580 yuan" and the reservation order. Fig.11 As shown, the music playing component 510 for playing the music collection of A is displayed to the user through the dialog box 1110. In this way, when the service is pushed to the user, the user can still continue to browse the page.

[0110] In some embodiments, when a conversation message is received, the context information of the page information may further include historical conversation messages associated with the conversation message.

[0111] In some embodiments of the present application, the electronic device can store the dialogue messages in the dialog box 1010. The dialogue messages stored by the electronic device include dialogue messages input by the user in the dialog box, and dialogue messages replied by the software agent in response to the dialogue messages input by the user. Based on this, after receiving the dialogue message input by the user, the electronic device can filter the stored historical dialogue messages based on the received dialogue message, so as to determine the historical dialogue messages associated with the received dialogue message.

[0112] In some embodiments of the present application, when filtering historical conversation messages, the electronic device can extract keywords from the received conversation messages, and expand the extracted keywords with synonyms or associated words based on a third-party keyword library, thereby obtaining a keyword set corresponding to the received conversation message, and then filter the historical conversation messages stored within a period of time based on the keyword set, thereby filtering out historical conversation messages that match any keyword in the keyword set, and using the filtered historical conversation messages as historical conversation messages associated with the received conversation message. Among them, the certain time period can be pre-set according to actual conditions, for example, the certain time period can be within the last two days, so that the historical conversation messages within the last two days are filtered.

[0113] Exemplarily, the conversation message received by the electronic device is "What performances are there in the near future?", and the electronic device obtains the keyword "performance" by performing keyword extraction on the conversation message, and obtains keywords such as "concert", "drama", "theater", "stage play", "magic", "cross talk" and multiple artist names through keyword expansion. "Performance", "concert", "drama", "theater", "stage play", "magic", "cross talk" and multiple artist names are combined to obtain a keyword set, and the electronic device traverses the historical conversation messages used in the past two days based on the keyword set, and determines that the historical conversation messages include singer "A", and then uses "A" as the historical conversation message corresponding to the received conversation message.

[0114] Exemplarily, the conversation message received by the electronic device is "recommended music". The electronic device obtains the keyword "music" by performing keyword extraction on the conversation message, and obtains keywords such as "song" and multiple singer names through keyword expansion. "Music", "song" and multiple singer names form a keyword set. Based on the keyword set, the electronic device traverses the historical conversation messages used in the past two days, and determines that the historical conversation messages include singer "A", then uses "A" as the historical conversation message corresponding to the received conversation message.

[0115] The most recent conversation message entered by the user in the dialog box and the historical conversation messages associated with the conversation message can more accurately express the user's current real needs. Therefore, through the above technical solution, the electronic device uses the historical conversation messages associated with the received conversation message as the context information corresponding to the page information, which can enable the multimodal large model to more accurately predict the user's intentions based on the context information, thereby improving the accuracy of the recommendation service.

[0116] In some embodiments, considering that users have different personal context information, their service needs are usually different. Therefore, in order to further improve the accuracy of service push and make the service finally pushed more in line with the real needs of users, the user's personal context information can be obtained before the above step 110. The user's personal context information refers to a series of information related to the user's specific scene and state. In this embodiment, the user's personal context information includes at least one of the following: physical environment information, emotional information, and personal preference information. Among them, physical environment information refers to information related to the physical environment in which the user is located. The user's emotional information is used to indicate the user's emotional state. The user's personal preference information refers to the user's behavior patterns and preferences with certain regularity and repetition.

[0117] In some embodiments of the present application, the physical environment information includes but is not limited to time-related information and space-related information. Among them, the time-related information is used to characterize the time period in which the user is located. The space-related information is used to characterize the place where the user is located. Exemplarily, the time period in the time-related information can be divided into morning, noon and evening. The place in the space-related information can be divided into home, office, leisure place, shopping place, sports place, natural environment, etc.

[0118] In some embodiments of the present application, the electronic device can obtain time-related information through a built-in clock unit. For example, the time period corresponding to the morning is pre-set to be from 7:00 to 11:59 in the morning, the time period corresponding to noon is from 12:00 to 13:59, and the time period corresponding to the evening is from 19:00 to 23:59. If the time of the clock unit is 8 o'clock, the current time-related information is determined to be morning, if the time of the clock unit is 14 o'clock, the current time-related information is determined to be noon, and if the time of the clock unit is 20 o'clock, the current time-related information is determined to be evening. It should be noted that in addition to this method, the electronic device can also use other methods to obtain time-related information, for example, through image analysis and other methods, which are not specifically limited in this embodiment.

[0119] In some embodiments of the present application, the electronic device can obtain spatial related information through the real-time visual function of the multimodal large model. The real-time visual function mainly refers to the ability of the multimodal large model to quickly process, analyze and understand the visual information (including images, videos, etc.) obtained in real time, and to make corresponding responses or provide relevant results based on this information. When the electronic device needs to obtain spatial related information, it can turn on the image acquisition module such as the camera, collect the environmental image around the user in real time, and input the environmental image into the multimodal large model, so as to analyze the environmental image through the real-time visual function of the multimodal large model to obtain the physical environment information of the user's environment. It should be noted that in addition to this method, the electronic device can also use other methods to obtain spatial related information, such as through satellite positioning, etc., which is not specifically limited in this embodiment.

[0120] In some embodiments of the present application, the user's emotional state can be divided into multiple types such as happiness, sadness, anger, anxiety, etc.

[0121] In some embodiments of the present application, the electronic device can obtain the user's emotional information through the emotion perception capability of the multimodal large model. The emotion perception capability of the multimodal large model refers to the ability to understand, recognize and analyze the emotional tendencies, emotional states and emotion-related nuances contained in text, voice, images, videos and other information, and to respond accordingly or provide relevant results based on these emotional information. When obtaining the user's emotional information, the electronic device can obtain the text data input by the user or collect the user's audio data in real time through a microphone, input the text data or audio data into the multimodal large model, and analyze the text data or audio data through the emotion perception capability of the multimodal large model, thereby obtaining the user's emotional information.

[0122] In other embodiments of the present application, the electronic device can obtain the user's emotional information based on the user's tone. Different tones can reflect different emotional states of the speaker. For example, a cheerful tone is usually accompanied by positive emotions, such as excitement, joy, happiness, etc. When a person says in a brisk tone: "Wow, it's such a good day today!" You can clearly feel his happy mood. On the contrary, a low tone may imply negative emotions, such as frustration, sadness, anger or anger. For example, the tone of "Alas, I didn't do well in this exam again" conveys the speaker's lost emotions. Based on this, the electronic device can obtain the text data input by the user or collect the user's audio data in real time through a microphone, analyze the text data or audio data, determine the modal particles contained therein, and obtain the user's emotional information based on the correlation between the modal particles and the emotional state. Among them, modal particles include but are not limited to "yeah", "wow", "yeah", "ne", "la", "ah", "ah", "hum", "bah", etc. The relationship between modal particles and emotional states includes, but is not limited to, "Yeah", "Wow", "Yeah", "Ne", and "La", and the corresponding emotional states are happy, "Ah" and "Ah" are disappointed, and "Hum" and "Pooh" are angry. For example, the audio data of the user acquired by the electronic device is "Wow! This view is so beautiful!", which contains the modal particle "Wow", and "Wow" corresponds to the emotional state of happy. Therefore, based on this audio data, it can be determined that the user's emotional information is happy. For another example, the text data input by the user acquired by the electronic device is "Alas, I didn't do well in this exam again", which contains the modal particle "Alas", and "Alas" corresponds to the emotional state of disappointed. Therefore, based on this text data, it can be determined that the user's emotional information is disappointed.

[0123] In some embodiments of the present application, the user's personal preference information includes but is not limited to daily routine preferences, consumption preferences, entertainment preferences, time arrangement habits, etc. Among them, consumption preferences include but are not limited to product category preferences, brand preferences, price range preferences, etc. Entertainment preferences include but are not limited to movie subject preferences, music type preferences, game type preferences, etc. For example, a user's daily routine preference is to get up at 6 o'clock in the morning and go to bed at 9:30 in the evening. The consumption preferences include liking to buy electronic products, liking popular brands, and liking a price range of 50 yuan to 1,500 yuan. Entertainment preferences include liking horror movies, pop music, and action games. Time arrangement preferences include liking to go out and spend on weekends.

[0124] In some embodiments of the present application, the electronic device can obtain the pre-stored personal preference information of the user from the designated location. The electronic device can provide the user with a page for setting the personal preference information in advance, allowing the user to actively select their preferences and habits. The electronic device stores the personal preference information set by the user in the designated location according to the user's selection. In this way, when the user's personal preference information needs to be obtained, the electronic device can directly and quickly obtain the pre-stored personal preference information of the user from the designated location.

[0125] In some other embodiments of the present application, the electronic device can obtain the user's personal preference information based on the user's historical behavior data. The historical behavior data includes but is not limited to browsing behavior data, interactive behavior data, purchase behavior data, etc. The browsing behavior data includes but is not limited to page browsing records, dwell time, browsing paths, etc. The interactive behavior data includes but is not limited to click behavior, search behavior, evaluation and comment, sharing and forwarding, etc. The purchase behavior data includes but is not limited to purchase records, shopping cart behavior, payment behavior, etc. When it is necessary to obtain the user's personal preference information, the electronic device obtains the user's historical behavior data, analyzes the historical behavior data through specific algorithms or models such as collaborative filtering algorithms, content filtering algorithms, deep learning algorithms, clustering algorithms, etc., understands the user's behavior patterns and habits, and thus obtains the user's personal preference information. Among them, the collaborative filtering algorithm includes user-based collaborative filtering and project-based collaborative filtering. For example, the electronic device can analyze the historical behavior data of the current user and the historical behavior data of other users, calculate the similarity between users using methods such as cosine similarity, find a group of similar users with a high similarity to the current user among other users, and determine the current user's personal preference information based on the preferences of this group of similar users.

[0126] In some other embodiments of the present application, the electronic device can obtain the user's personal preference information based on the content actively recorded by the user. Among them, the content actively recorded by the user includes but is not limited to the content actively input and recorded by the user in various applications such as photo albums, notes, calendars, etc. Among them, the content actively recorded by the user in the photo album includes but is not limited to the images stored in the photo album, the content actively recorded by the user in the notes includes but is not limited to the recorded study notes, work notes, life notes, etc., and the content actively recorded by the user in the calendar includes but is not limited to the recorded work arrangements, personal activities, important events, etc. When it is necessary to obtain the user's personal preference information, the electronic device obtains the content actively recorded by the user in the electronic device, and obtains the user's personal preference information by analyzing and learning the content actively recorded by the user. For example, the electronic device obtains the images stored by the user in the photo album. If the user's photo album has a large number of natural scenery photos, such as mountains, forests, oceans, etc., it means that the user may have a strong interest in nature and outdoor activities. This may mean that the user likes activities such as hiking, camping, photography, etc., and may have a demand for outdoor equipment, travel destination recommendations, etc. If the photo album is mainly photos of people, especially photos with family and friends, it means that the user values ​​interpersonal relationships and may like social activities, gatherings, etc. Users may have preferences for gifts, event planning, parent-child activities, etc. If there are many food photos in the album, it means that the user may be a food lover and is interested in restaurant recommendations, cooking books, food shows, etc. If there are many pet photos in the album, it may indicate that the user likes animals and has a demand for pet supplies, pet care services, etc. Electronic devices can also obtain the learning notes, work notes, and life notes recorded by users in the note application. According to the subject area to which the learning notes recorded by the user belong, the user's academic interests or career development direction can be judged. For example, if there are many computer science-related notes, it may indicate that the user has a preference for programming, software development, and technology news. If it is a literary note, the user may be interested in books, literary activities, writing tools, etc. The learning methods recorded in the notes can also reflect the user's preferences. For example, users who like to take notes with mind maps may pay more attention to logical thinking and visual learning, and have a demand for mind mapping software, learning skills books, etc. From work notes, you can understand the user's occupation type and then infer the user's work-related preferences. For example, a salesperson's notes may involve customer information, sales skills, etc., and the user may be interested in sales training, customer relationship management software, etc. Designers' notes may contain creative inspirations, design cases, etc., and they may have requirements for design software and design materials. Based on the project themes in work notes, we can determine the fields involved in the projects that users are involved in, such as environmental protection projects, scientific and technological research and development projects, etc., and infer the user's attention and interest in the field. For example, users who participate in environmental protection projects may have preferences for sustainable development products and environmental protection activities. Users may record their hobbies in life notes, such as painting, music, and handicrafts.This can directly reflect the user's personal preferences and recommend relevant courses, tools, activities, etc.

[0127] Accordingly, in the case where the user's personal context information is obtained, in the above step 110, the personal context information, the page information of the first page, and the context information of the page information can be input into the multimodal big model together, so that the multimodal big model can predict the user's intention based on the page information, context information, and personal context information, and output the recommended information that matches the intention. If a conversation message input by the user is received before the above step 110, the page information of the first page, the context information of the page information, the conversation message, and the personal context information can be input into the multimodal big model together in the above step 110, so that the multimodal big model can predict the user's intention based on the page information, context information, the conversation message, and the personal context information, and output the recommended information that matches the intention.

[0128] For example, when a user wants to buy concert tickets, the page information obtained by the electronic device includes: Figure 2 The information and website address http: / / xxxx in the concert promotion picture 211, concert promotion picture 212, concert promotion picture 213 and concert promotion picture 214 shown, the context information obtained includes "A", the user's personal situation information obtained includes that the user's permanent residence and current location are both Shenzhen, the user prefers to buy front-stage tickets for concerts, and they are all weekend shows, the user has a record of purchasing concert tickets in the surrounding cities of Changsha and Guangzhou, etc., the above page information, context information and personal situation information are all input into the multimodal big model, and the multimodal model determines based on the above information that the user's intention is to buy front-stage tickets for A's concert in Shenzhen, Changsha or Guangzhou, and the shows are weekend shows. Based on this, the multimodal big model searches and finds that A will start a world tour in April, and the tour addresses include Guangzhou and Changsha, where the performance time in Guangzhou is weekdays and the performance time in Changsha is weekends. Therefore, the multimodal big model finally obtains the recommended information that matches the user's intention, which is ticket information related to the front-stage tickets for A's concert in Changsha in April.

[0129] For example, when a user wants to listen to music, the page information obtained by the electronic device includes: Figure 3The information and QR code 330 contained in the playlist cover image 311, playlist cover image 312, playlist cover image 313 and playlist cover image 314 shown, the context information obtained includes "A", the user's personal situation information obtained includes that it is currently morning, the user is in the office, the user's emotional state is happy, the user likes pop music, etc. The above page information, context information and personal situation information are all input into the multimodal big model. Based on the above information, the multimodal determines that the user's intention is to listen to cheerful pop songs sung by A. Based on this, the multimodal big model obtains 4 cheerful pop songs sung by A through search queries, so the multimodal big model uses the song information of these 4 cheerful pop songs as recommendation information matching the user's intention.

[0130] Through the above technical solution, by adding the user's personal context information to the input data of the multimodal big model, the multimodal big model can comprehensively understand and analyze the user's intentions from multiple perspectives and multiple aspects, determine more accurate user intentions, and thus output recommendation information that is more in line with the user's real needs, thereby improving the readiness of service push.

[0131] The service push method provided in the embodiment of the present application can be executed by a service push device. In the embodiment of the present application, the service push device provided in the embodiment of the present application is described by taking the service push method executed by the service push device as an example.

[0132] See also Fig.12 , which is a schematic diagram of a service push device provided in some embodiments of the present application, such as Fig.12 As shown, the device 1200 includes the following modules:

[0133] The processing module 1210 is used to input the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predict the user's intention through the multimodal large model, and output the recommended information matching the intention;

[0134] The push module 1220 is used to push services based on the recommendation information.

[0135] The service push device provided in the embodiment of the present application inputs the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predicts the user's intention through the multimodal large model, and outputs the recommended information matching the intention; based on the recommended information, the service is pushed. According to the embodiment of the present application, when the user browses a page, the user's intention can be directly determined based on the page browsed by the user, so as to automatically push services that meet the real intention to the user, without the user providing information, thereby reducing the user's interactive operations.

[0136] In some embodiments, the apparatus 1200 further includes:

[0137] A receiving module, configured to receive a selection input of the first page before inputting the page information of the first page displayed on the display screen and the context information of the page information into the multimodal macro model;

[0138] The processing module 1210 is further configured to, in response to a selection input, determine the information in the page area selected by the selection input as page information input into the multimodal macro model.

[0139] In some embodiments, the selection input includes at least one of the following: fingertip touch input, knuckle touch input.

[0140] In some embodiments, when the selection input includes a knuckle touch input, the knuckle touch input includes an input of tapping the screen with a knuckle and circling an area along a reference direction from the tapped position.

[0141] In some embodiments, the processing module 1210 is specifically configured to:

[0142] Acquire knuckle operation parameters of knuckle touch input, wherein the knuckle operation parameters include at least one of the following: touch area, touch pressure, and circle selection trajectory;

[0143] Based on the knuckle operation parameters, determine the knuckle circle selection area;

[0144] The knuckle circled area is determined as the page information of the input multimodal large model.

[0145] In some embodiments, the context information of the page information includes at least one of the following: page information in the second page, page information in the history page associated with the page information;

[0146] Among them, the second page is the page indicated by the page access entry identifier on the first page; the historical page is the page displayed by the display screen during the page display time period before the first page is displayed.

[0147] In some embodiments, the apparatus 1200 further includes:

[0148] A receiving module, configured to receive a dialog message before inputting the page information of the first page and the context information of the page information into the multimodal macro model;

[0149] The processing module 1210 is also used to input the page information of the first page, the context information of the page information and the dialogue message into the multimodal big model, predict the user's intention through the multimodal big model, and output the recommendation information matching the intention.

[0150] In some embodiments, the context information of the page information includes:

[0151] The historical conversation messages associated with the conversation message.

[0152] In some embodiments, the apparatus 1200 further includes:

[0153] An acquisition module, used to acquire the user's personal context information, the personal context information including at least one of the following: physical environment information, emotional information, and personal preference information;

[0154] The processing module 1210 is also used to input the page information of the first page, the context information of the page information and the personal situation information into the multimodal big model, predict the user's intention through the multimodal big model, and output the recommendation information matching the intention.

[0155] The service push device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0156] The service push device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0157] The service push device provided in the embodiment of the present application can achieve Figures 1 to 11 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0158] Alternatively, if Fig.13As shown, an embodiment of the present application also provides an electronic device 1300, including a processor 1301 and a memory 1302, and the memory 1302 stores a program or instruction that can be executed on the processor 1301. When the program or instruction is executed by the processor 1301, the various steps of the above-mentioned service push method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0159] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0160] Fig.14 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0161] The electronic device 1400 includes but is not limited to: a radio frequency unit 1401, a network module 1402, an audio output unit 1403, an input unit 1404, a sensor 1405, a display unit 1406, a user input unit 1407, an interface unit 1408, a memory 1409, and a processor 1410 and other components.

[0162] Those skilled in the art will appreciate that the electronic device 1400 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1410 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.14 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0163] Processor 1410, configured to input page information of a first page displayed on the display screen and context information of the page information into a multimodal macro model, predict the user's intention through the multimodal macro model, and output recommendation information matching the intention;

[0164] The display unit 1406 is used to push services based on the recommendation information.

[0165] The electronic device provided in the embodiment of the present application inputs the page information of the first page displayed on the display screen and the context information of the page information into the multimodal large model, predicts the user's intention through the multimodal large model, and outputs the recommended information matching the intention; based on the recommended information, the service is pushed. According to the embodiment of the present application, when the user browses a page, the user's intention can be directly determined based on the page browsed by the user, so as to automatically push services that meet the real intention to the user, without the user providing information, thereby reducing the user's interactive operations.

[0166] In some embodiments, the user input unit 1407 is used to receive a selection input of the first page before inputting the page information of the first page and the context information of the page information displayed on the display screen into the multimodal macro model;

[0167] The processor 1410 is further configured to, in response to a selection input, determine information in a page area selected by the selection input as page information input into the multimodal macro model.

[0168] In some embodiments, the selection input includes at least one of the following: fingertip touch input, knuckle touch input.

[0169] In some embodiments, when the selection input includes a knuckle touch input, the knuckle touch input includes an input of tapping the screen with a knuckle and circling an area along a reference direction from the tapped position.

[0170] In some embodiments, the processor 1410 is specifically configured to:

[0171] Acquire knuckle operation parameters of knuckle touch input, wherein the knuckle operation parameters include at least one of the following: touch area, touch pressure, and circle selection trajectory;

[0172] Based on the knuckle operation parameters, determine the knuckle circle selection area;

[0173] The knuckle circled area is determined as the page information of the input multimodal large model.

[0174] In some embodiments, the context information of the page information includes at least one of the following: page information in the second page, page information in the history page associated with the page information;

[0175] Among them, the second page is the page indicated by the page access entry identifier on the first page; the historical page is the page displayed by the display screen during the page display time period before the first page is displayed.

[0176] In some embodiments, the input unit 1404 is used to receive a dialog message before inputting the page information of the first page and the context information of the page information into the multimodal macro model;

[0177] Processor 1410 is further configured to input page information of the first page, context information of the page information, and a dialog message into the multimodal big model, predict the user's intention through the multimodal big model, and output recommendation information matching the intention.

[0178] In some embodiments, the context information of the page information includes:

[0179] The historical conversation messages associated with the conversation message.

[0180] In some embodiments, the processor 1410 is further used to obtain personal context information of the user, where the personal context information includes at least one of the following: physical environment information, emotional information, and personal preference information;

[0181] Processor 1410 is further configured to input page information of the first page, context information of the page information, and personal situation information into the multimodal big model, predict the user's intention through the multimodal big model, and output recommendation information matching the intention.

[0182] It should be understood that in the embodiment of the present application, the input unit 1404 may include a graphics processor (Graphics Processing Unit, GPU) 14041 and a microphone 14042, and the graphics processor 14041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1406 may include a display panel 14061, and the display panel 14061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1407 includes a touch panel 14071 and at least one of other input devices 14072. The touch panel 14071 is also called a touch screen. The touch panel 14071 may include two parts: a touch detection device and a touch controller. Other input devices 14072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0183] The memory 1409 can be used to store software programs and various data. The memory 1409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1409 may include a volatile memory or a non-volatile memory, or the memory 1409 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0184] The processor 1410 may include one or more processing units; optionally, the processor 1410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1410.

[0185] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned service push method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0186] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0187] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned service push method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0188] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0189] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned service push method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0190] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0191] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0192] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A service push method, characterized in that: include: Inputting page information of a first page displayed on the display screen and context information of the page information into a multimodal large model, predicting the user's intention through the multimodal large model, and outputting recommendation information matching the intention; Based on the recommendation information, service push is performed.

2. The method according to claim 1, characterized in that Before inputting the page information of the first page and the context information of the page information into the multimodal macro model, the method further includes: receiving a selection input for the first page; In response to the selection input, information in the page area selected by the selection input is determined as the page information input into the multimodal macro model.

3. The method according to claim 2, characterized in that The selection input includes at least one of the following: fingertip touch input and knuckle touch input.

4. The method according to claim 3, characterized in that In the case where the selection input includes the knuckle touch input, the knuckle touch input includes an input of tapping the screen with a knuckle and circling an area along a reference direction from the tapped position.

5. The method according to claim 4, characterized in that The step of determining the information in the page area selected by the selection input as the page information input into the multimodal macro model includes: Acquire a knuckle operation parameter of the knuckle touch input, wherein the knuckle operation parameter includes at least one of the following: touch area, touch pressure, and circle selection track; Based on the knuckle operation parameters, determining a knuckle circle selection area; The knuckle circled area is determined as the page information input into the multimodal macro model.

6. The method according to claim 1, characterized in that The context information of the page information includes at least one of the following: page information in the second page, page information in the history page associated with the page information; Among them, the second page is the page indicated by the page access entrance identifier on the first page; the historical page is the page displayed on the display screen during the page display time period before the first page is displayed.

7. The method according to any one of claims 1 to 5, characterized in that Before inputting the page information of the first page and the context information of the page information into the multimodal macro model, the method further includes: Receive conversation messages; The step of inputting the page information of the first page and the context information of the page information into the multimodal macro model includes: The page information of the first page, the context information of the page information and the dialog message are input into the multimodal macro model.

8. The method according to claim 7, characterized in that The context information of the page information includes: A historical conversation message associated with the conversation message.

9. The method according to any one of claims 1 to 5, characterized in that: Before inputting the page information of the first page and the context information of the page information into the multimodal macro model, the method further includes: Acquire personal context information of the user, the personal context information including at least one of the following: physical environment information, emotional information, and personal preference information; The step of inputting the page information of the first page and the context information of the page information into the multimodal macro model includes: The page information of the first page, the context information of the page information and the personal situation information are input into the multimodal macro model.

10. A service push device, characterized in that: include: a processing module, configured to input page information of a first page displayed on the display screen and context information of the page information into a multimodal macro model, predict the user's intention through the multimodal macro model, and output recommendation information matching the intention; The push module is used to push services based on the recommendation information.

11. The device according to claim 10, characterized in that The device also includes: A receiving module, configured to receive a selection input of the first page before inputting the page information of the first page displayed on the display screen and the context information of the page information into the multimodal macro model; The processing module is further configured to, in response to the selection input, determine the information in the page area selected by the selection input as the page information input into the multimodal macro model.

12. The device according to claim 11, characterized in that The selection input includes at least one of the following: fingertip touch input and knuckle touch input.

13. The device according to claim 12, characterized in that In the case where the selection input includes the knuckle touch input, the knuckle touch input includes an input of tapping the screen with a knuckle and circling an area along a reference direction from the tapped position.

14. The device according to claim 13, characterized in that The processing module is specifically used for: Acquire a knuckle operation parameter of the knuckle touch input, wherein the knuckle operation parameter includes at least one of the following: touch area, touch pressure, and circle selection track; Based on the knuckle operation parameters, determining a knuckle circle selection area; The knuckle circled area is determined as the page information input into the multimodal macro model.

15. The device according to claim 10, characterized in that The context information of the page information includes at least one of the following: page information in the second page, page information in the history page associated with the page information; Among them, the second page is the page indicated by the page access entrance identifier on the first page; the historical page is the page displayed on the display screen during the page display time period before the first page is displayed.

16. The device according to any one of claims 10 to 15, characterized in that: The device also includes: A receiving module, configured to receive a dialog message before inputting the page information of the first page and the context information of the page information into the multimodal macro model; The processing module is also used to input the page information of the first page, the context information of the page information and the dialogue message into the multimodal big model, predict the user's intention through the multimodal big model, and output recommendation information matching the intention.

17. The device according to claim 16, characterized in that The context information of the page information includes: A historical conversation message associated with the conversation message.

18. The device according to any one of claims 10 to 15, characterized in that: The device also includes: An acquisition module, used to acquire the user's personal context information, wherein the personal context information includes at least one of the following: physical environment information, emotional information, and personal preference information; The processing module is also used to input the page information of the first page, the context information of the page information and the personal situation information into the multimodal big model, predict the user's intention through the multimodal big model, and output recommendation information matching the intention.

19. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the service push method according to any one of claims 1 to 9 are implemented.

Images