Mobile terminal interactive tutorial search system, method, apparatus, and electronic device

Abstract

The application provides a mobile terminal interactive tutorial search system, method, device and electronic equipment. When a mobile terminal determines that a user touches a display screen of the mobile terminal and receives inquiry information input by the user, the mobile terminal sends the inquiry information and context information to a server, and then the server processes the inquiry information and the context information to obtain a tutorial for the inquiry information. Finally, the obtained tutorial for the inquiry information is fed back to the mobile terminal, so that the mobile terminal displays the received tutorial to the user. Through the mobile terminal interactive tutorial search system, method, device and electronic equipment provided by the application, the user only needs to input an inquiry information, and the mobile terminal feeds back a tutorial for solving a use problem to the user, so that the whole process is simple and convenient.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a mobile terminal interactive tutorial search system, method, apparatus, and electronic device. Background Technology

[0002] Currently, some users, limited by their cognitive abilities, encounter numerous operational problems when using mobile devices daily (e.g., not knowing how to use a certain function). When faced with these problems, users often resort to seeking help from others. Therefore, providing users with quick and convenient access to tutorials to resolve operational issues on their mobile devices is a crucial and urgent need. Summary of the Invention

[0003] To address the aforementioned issues, the purpose of this application is to provide a mobile terminal interactive tutorial search system, method, apparatus, and electronic device.

[0004] In a first aspect, embodiments of this application provide a mobile terminal interactive tutorial search system, including: a mobile terminal and a server;

[0005] The mobile terminal is configured to, when determining that a user touches the display screen of the mobile terminal and receives an input query from the user, obtain context information about the user's current situation, wherein the context information includes: the user's current location information, the identifier of the application currently being used by the user, the application interface currently opened by the user, and the UI controls touched by the user within the currently opened application interface; send the query information and the context information to the server; receive a usage tutorial from the server in response to the query information, and display the received usage tutorial to the user;

[0006] The server is configured to receive the query information and context information sent by the mobile terminal; process the query information and context information to obtain a usage tutorial for the query information; and feed back the obtained usage tutorial for the query information to the mobile terminal.

[0007] Secondly, embodiments of this application also provide a mobile terminal interactive tutorial search method, used to perform the functions of a server in the mobile terminal interactive tutorial search system described in the first aspect above, the method comprising:

[0008] Receive the query information and the context information sent by the mobile terminal;

[0009] The query information and the context information are processed to obtain a usage tutorial for the query information;

[0010] The obtained usage tutorial for the query information is fed back to the mobile terminal.

[0011] Thirdly, embodiments of this application also provide a mobile terminal interactive tutorial search device, comprising:

[0012] The receiving module is used to receive query information and contextual information sent by the mobile terminal;

[0013] The processing module is used to process the query information and the context information to obtain a usage tutorial for the query information;

[0014] The feedback module is used to provide the mobile terminal with the usage tutorial obtained in response to the query information.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described in any of the second aspects above.

[0016] Fifthly, embodiments of this application also provide an electronic device, the electronic device including a memory, a processor and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor using the steps of the method described in the second aspect above.

[0017] In the solutions provided by the first to fifth aspects of this application, when the mobile terminal determines that the user has touched the display screen of the mobile terminal and receives the user's input query information, it sends the query information and context information to the server. The server then processes the query information and context information to obtain a usage tutorial for the query information. Finally, the obtained usage tutorial for the query information is fed back to the mobile terminal, allowing the mobile terminal to display the received usage tutorial to the user. Compared with related technologies where users can only solve mobile terminal usage problems by asking other users, this solution eliminates the need to ask other users. The user only needs to send the query information to the mobile terminal, and the mobile terminal can obtain the context information of the user's current situation and send the context information and query information to the server. The server processes the query information and context information to obtain a usage tutorial for the query information and sends it to the mobile terminal to display to the user. Thus, when a user encounters a problem using the mobile terminal, the user only needs to input a query information, and the mobile terminal will provide the user with a usage tutorial to solve the problem. The whole process is simple and convenient, greatly improving the user's experience with the mobile terminal. Moreover, the user describes the problem by combining touch screen and voice, avoiding the defect of chaotic voice description when some users do not know how to describe the interface.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This illustration shows the interaction between a mobile terminal and a server in a mobile terminal interactive tutorial search system provided in Embodiment 1 of this application;

[0021] Figure 2 A flowchart of a mobile terminal interactive tutorial search method provided in Embodiment 2 of this application is shown;

[0022] Figure 3 This paper shows a schematic diagram of the structure of a mobile terminal interactive tutorial search device provided in Embodiment 3 of this application;

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application is shown. Detailed Implementation

[0024] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0026] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0027] Currently, some users (such as middle-aged and elderly users, and users with lower levels of education) encounter many operational problems when using mobile devices due to their limited cognitive abilities (e.g., not knowing how to use a certain function). When encountering operational problems, users can only solve them by asking others for help. Therefore, it is a very important and urgent need to help users quickly and conveniently find tutorials to solve operational problems when they encounter them on mobile devices.

[0028] To address the aforementioned issues, research revealed that users sought specific features for their mobile phones, which provided guidance for subsequent technical solutions.

[0029] 1. Some users prefer voice interaction to pure interface operation, as it is more natural and widely accepted by some users.

[0030] 2. Some users, limited by their own cognitive abilities, produce voice messages that are incomplete, redundant, confusing, or overly specific.

[0031] 3. Some users only think of asking questions when they encounter problems, and do not seek help when they do not encounter problems.

[0032] Therefore, it can be determined that the most natural way for some users to seek solutions to problems is to describe the scene in words when they encounter a problem. However, their descriptive abilities are limited, so it is necessary to propose a system that systematically enhances the speech understanding of some users.

[0033] The mobile terminal interactive tutorial search system, method, apparatus, and electronic device proposed in the following embodiments of this application can achieve the following three functions:

[0034] 1. Some users can directly describe phone function problems using a multimodal interaction method combining voice and touch screen. This solves the problem of some users not knowing how to express the operational problems they encounter when they are unfamiliar with the UI elements. For example, a user can click on the product UI on the interface and say "buy it".

[0035] 2. Automatically supplement missing contextual information in some user queries. We utilize contextual information such as application details, interface content, and physical location to supplement incomplete voice text from some users, resolving issues of incomplete or unclear query descriptions and enabling the algorithm to better understand the query intent of certain users.

[0036] 3. Automatically correct non-standard sentence structures in queries. A Large Language Model (LLM) is used, and two prompts are designed to correct the sentence structures of some users, addressing sentence errors. Finally, a classification model is used for intent understanding training.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] Example 1

[0039] See Figure 1 The diagram shown illustrates the interaction between a mobile terminal and a server in a mobile terminal interactive tutorial search system. This embodiment proposes a mobile terminal interactive tutorial search system, which includes a mobile terminal and a server.

[0040] The mobile terminal is configured to, when determining that a user touches the display screen of the mobile terminal and receives an input query from the user, obtain context information about the user's current situation, wherein the context information includes: the user's current location information, the identifier of the application currently being used by the user, the application interface currently opened by the user, and the UI controls touched by the user within the currently opened application interface; send the query information and the context information to the server; receive a usage tutorial based on the query information from the server, and display the received usage tutorial to the user.

[0041] Among them, the inquiry information contains questions that users need to resolve regarding the use of mobile terminals.

[0042] For example, when a user accidentally opens a floating window on social software A but doesn't know how to close it, they can enter the question "How do I close this?" into the mobile device and touch the floating window.

[0043] The specific process of determining the display screen of a user's touch mobile terminal is as follows: when the user's touch operation is captured using accessibility technology, the display screen of the user's touch mobile terminal is determined. The specific implementation process is existing technology and will not be elaborated here.

[0044] The server is configured to receive the query information and context information sent by the mobile terminal; process the query information and context information to obtain a usage tutorial for the query information; and feed back the obtained usage tutorial for the query information to the mobile terminal.

[0045] The query information received by the mobile terminal may be, but is not limited to, text information or voice information.

[0046] The server can process text information, so when the query information is voice information, the mobile terminal can use the voice-to-text conversion program running on the mobile terminal to convert the query information from voice information to text information. The specific process is existing technology and will not be described in detail here.

[0047] In order to obtain contextual information about the user's current situation, the mobile terminal can perform the following steps (1) to (4):

[0048] (1) The process of obtaining the user's current location information includes: when the user inputs the query information, the API search POI function in the map program running on the mobile terminal is used to obtain the geographical locations (such as parks, supermarkets and transportation hubs) around the user's current location that are related to the user's expected phone function, and the geographical location closest to the user is selected as the user's current location information.

[0049] (2) The process of obtaining the application identifier currently used by the user includes: when the user inputs the query information, the application displayed on the mobile terminal screen is identified by monitoring the “TYPE\_WINDW\_STATE\_\\CHANGED” event in the accessibility technology provided in the mobile terminal's operating system, and the application identifier of the identified application is obtained.

[0050] (3) The process of obtaining the user's currently opened application interface includes: when the user inputs a query, the mobile terminal uses accessibility technology to extract the text of the user's currently opened application interface, and matches the extracted text with the pre-annotated interface text. The interface text that is closest to the extracted text is determined from the pre-annotated interface text through the matching method, and the interface name carried in the interface text that is closest to the extracted text is determined as the name of the user's currently opened application interface, thereby obtaining the user's currently opened application interface; wherein, the pre-annotated interface text includes: the correspondence between interface text and interface name;

[0051] (4) The process of obtaining the UI control touched by the user in the currently opened application interface includes: when the user inputs a query, the mobile terminal uses the floating layer above the interface to obtain the click position of the user's currently opened application interface; based on accessibility technology, it obtains the control identifier and image content of each control among all UI controls on the interface, and obtains the position of the UI control according to the control identifier and image content of each control; calculates the distance between the click position of the user's currently opened application interface and the position of each UI control on the interface, and selects the UI control with the smallest distance from the click position as the UI control clicked by the user in the currently opened application interface.

[0052] In step (3) above, the pre-annotated interface text is pre-cached in the mobile terminal.

[0053] The extracted text refers to the text displayed on the user's currently open application interface.

[0054] In step (4) above, the process of using the fuzzy intent recognition algorithm to process the distance between the user's click position on the interface and the position of each UI control among all UI controls on the interface, and to determine the specific UI control touched by the user in the currently opened application interface, is existing technology and will not be described in detail here.

[0055] Through the process described in steps (1) to (4) above, the mobile terminal can obtain contextual information about the user's current situation and send the user's input query information and the obtained contextual information to the server.

[0056] After receiving the query information and context information sent by the mobile terminal, the server processes the query information and context information to obtain a usage tutorial for the query information, including the following steps (1) to (4):

[0057] (1) Use a large language model to perform text correction on the query information to obtain the corrected query information;

[0058] (2) Use a large language model to perform secondary text correction on the corrected query information to obtain the secondary corrected query information;

[0059] (3) The contextual information is concatenated with the second-corrected query information to obtain a query sentence carrying contextual information;

[0060] (4) Process the question carrying contextual information to obtain a usage tutorial for the question information.

[0061] In step (1) above, the large language model, running on the server, is used to perform text correction on the query information input by the user and to understand the user's intent.

[0062] The process of large language models performing text correction on user-input queries and understanding user intent is existing technology and will not be elaborated here.

[0063] The task of text correction is to transform the user's input query into a more easily understood statement. Therefore, the first part of the prompt is set as: "Change sentence B, which is commonly used by elderly people when using mobile phones, into a more easily understood statement." Since the language model automatically expands or rewrites the meaning of some words, the second part of the prompt includes keyword substitution prevention. If one or more keywords from keyword list A appear in the query, please retain these keywords in their entirety without modification or replacement; then, return directly to the corrected query to obtain the corrected query information.

[0064] In one implementation, the keyword list A = ['logistics', 'delete', 'return', 'copy', 'paste', 'shopping cart', 'forward', 'block', 'favorite', 'live stream', 'watering'].

[0065] In step (2) above, the process of using a large language model to perform secondary text correction on the corrected query information to obtain the secondary corrected query information is existing technology and will not be described in detail here.

[0066] The task of secondary correction is to generalize the contextual information in the corrected query information and replace the words in the corrected query information with generic terms; for example, change specific place names in the corrected query information to 'a place'; change fruits, clothes, and electronic products to 'goods'.

[0067] If one or more keywords from keyword list B appear in the corrected query information, please retain these keywords in the corrected query information without modification or replacement; then return directly to the second-corrected query information to obtain the second-corrected query information.

[0068] In one implementation, the keyword list B = ['logistics', 'delete', 'return', 'copy', 'paste', 'shopping cart', 'forward', 'block', 'favorite', 'live stream', 'watering'].

[0069] In step (3) above, optionally, during the splicing process, the query information after the secondary correction is spliced ​​after the context information to obtain a query sentence carrying context information.

[0070] For example, a question carrying contextual information could be: "Social media app A has a floating interface and UI controls. How do I close this?"

[0071] In step (4) above, specifically, in order to obtain a usage tutorial for the query information, the following steps (1) to (3) can be performed:

[0072] (1) Using the application identifier currently used by the user carried in the context information, query the classification model used to classify the application corresponding to the application identifier;

[0073] (2) Use the queried classification model to classify the question sentences carrying contextual information to obtain the tutorial identifier for the usage tutorial of the question information;

[0074] (3) Use the tutorial identifier to query the usage tutorial for the query information, thereby obtaining the usage tutorial for the query information.

[0075] In step (1) above, the server pre-stores the correspondence between application identifiers and classification models.

[0076] Alternatively, the classification model described above can be the BERT model.

[0077] In step (2) above, the specific process of classifying the query sentences carrying contextual information using the queried classification model to obtain the tutorial identifier for the usage tutorial for the query information is existing technology and will not be described in detail here.

[0078] In one implementation, the contextual question is "Social media app A, the main interface and UI controls are floating windows, how do I close them?" The classification model performs a classification operation on this contextual question, and the result is a tutorial identifier for the user guide on "How to close the floating window of social media app A".

[0079] In step (3) above, the server pre-stores multiple user tutorials, and each user tutorial carries a tutorial identifier.

[0080] Then, by using the tutorial identifiers obtained after classification, the server can search for tutorials on "how to close the floating window of social software A".

[0081] In one implementation, the tutorial is a pre-written tutorial text.

[0082] The mobile terminal interactive tutorial search system proposed in this embodiment improves the machine's understanding of the intent of certain users' voice input by combining three methods: multimodal interaction (i.e., a combination of voice and touch screen), supplementing contextual information, and correcting sentences.

[0083] 1. Supplementing contextual information is to address the issue that when some users describe problems using mobile terminals using voice, the voice content is incomplete, such as the lack of contextual information about the application, interface, or geographical location in the sentence.

[0084] 2. Multimodal interaction is designed to address situations where users describe problems using mobile devices via voice but fail to understand how to describe the visual content on the interface, such as pop-ups, overlays, and other UI controls.

[0085] 3. The purpose of sentence correction is to address the issues that some users encounter when describing problems using mobile terminals via voice, such as redundant information, inaccurate descriptions, and overly specific vocabulary.

[0086] Therefore, the mobile terminal interactive tutorial search system proposed in this embodiment innovatively proposes a multimodal interactive query method for certain users: users describe problems by combining touch screen and voice, solving the problem of chaotic voice information description when some users do not know how to describe the interface. Furthermore, in terms of interaction feasibility, the multimodal interaction method is proven to be efficient and simple. A feasibility experiment showed that when 10 elderly people used the multimodal interaction method to query solutions to solve problems using the mobile terminal, 9 of them found the multimodal interaction method to be more efficient and concise than simply asking questions.

[0087] Furthermore, an innovative method for enhancing speech intent understanding using contextual information was proposed: the role of contextual information in elderly people asking questions about mobile phones was discovered, and a feasibility experiment with 10 elderly people demonstrated that the accuracy of tutorial searches using the mobile terminal interactive tutorial search system proposed in this embodiment is 91%, which is a significant improvement compared to the accuracy of searches without contextual information.

[0088] In summary, this embodiment proposes a mobile terminal interactive tutorial search system. When the mobile terminal determines that a user has touched the display screen and receives a query from the user, it sends the query and contextual information to the server. The server then processes the query and contextual information to obtain a usage tutorial specific to the query. Finally, the system returns the usage tutorial to the mobile terminal, allowing it to display the received tutorial to the user. Compared to related technologies where users can only solve mobile terminal usage problems by asking other users, this system eliminates the need for users to ask others. The user simply sends a query to the mobile terminal, which then obtains the contextual information of the user's situation and sends it along with the query to the server. The server processes the query and contextual information to obtain a usage tutorial specific to the query and displays it to the user. Thus, when a user encounters a problem using the mobile terminal, they only need to input a query, and the mobile terminal will provide a solution. The entire process is simple and convenient, greatly improving the user experience. Furthermore, by combining touchscreen input with voice to describe the problem, the system avoids the problem of unclear voice descriptions when some users don't know how to describe the interface.

[0089] Example 2

[0090] This embodiment proposes a mobile terminal interactive tutorial search method, which is used to perform the server functions in the mobile terminal interactive tutorial search system described in Embodiment 1 above.

[0091] See Figure 2 The flowchart shown illustrates a method for searching interactive tutorials on a mobile terminal. This embodiment proposes a method for searching interactive tutorials on a mobile terminal, including the following specific steps:

[0092] Step 200: Receive the query information and the context information sent by the mobile terminal.

[0093] Step 202: Process the query information and the context information to obtain a usage tutorial for the query information.

[0094] In step 202 above, in order to obtain a usage tutorial for the query information, the following steps (1) to (4) can be performed:

[0095] (1) Use a large language model to perform text correction on the query information to obtain the corrected query information;

[0096] (2) Use a large language model to perform secondary text correction on the corrected query information to obtain the secondary corrected query information;

[0097] (3) The contextual information is concatenated with the second-corrected query information to obtain a query sentence carrying contextual information;

[0098] (4) Process the question carrying contextual information to obtain a usage tutorial for the question information.

[0099] In step (4) above, specifically, in order to process the question carrying contextual information and obtain a usage tutorial for the question information, the following steps (41) to (43) can be performed:

[0100] (41) Using the application identifier currently used by the user carried in the context information, query the classification model used to classify the application corresponding to the application identifier;

[0101] (42) Use the queried classification model to classify the query sentences carrying contextual information to obtain the tutorial identifier for the usage tutorial of the query information;

[0102] (43) Use the tutorial identifier to query the usage tutorial for the query information, thereby obtaining the usage tutorial for the query information.

[0103] Step 204: Feed back the obtained usage tutorial for the query information to the mobile terminal.

[0104] In summary, this embodiment proposes a mobile terminal interactive tutorial search method. When the mobile terminal determines that a user has touched the display screen and receives a query from the user, it sends the query and context information to the server. The server then processes the query and context information to obtain a usage tutorial specific to the query. Finally, the obtained usage tutorial is fed back to the mobile terminal, allowing the mobile terminal to display the received tutorial to the user. Compared to related technologies where users can only solve mobile terminal usage problems by asking other users, this method eliminates the need to ask other users. The user only needs to send a query to the mobile terminal, which can then obtain the context information of the user's current situation and send it along with the query to the server. The server processes the query and context information to obtain a usage tutorial specific to the query and sends it to the mobile terminal for display to the user. Thus, when a user encounters a problem using the mobile terminal, they only need to input a query, and the mobile terminal will provide a usage tutorial to solve the problem. The entire process is simple and convenient, greatly improving the user experience. Furthermore, by combining touchscreen input with voice to describe the problem, the method avoids the problem of unclear voice descriptions when some users do not know how to describe the interface.

[0105] Example 3

[0106] This embodiment proposes a mobile terminal interactive tutorial search device for executing the mobile terminal interactive tutorial search method described in Embodiment 2 above.

[0107] See Figure 3 The diagram shows a structural schematic of a mobile terminal interactive tutorial search device. This embodiment proposes a mobile terminal interactive tutorial search device, comprising:

[0108] The receiving module 300 is used to receive query information and context information sent by the mobile terminal;

[0109] Processing module 302 is used to process the query information and the context information to obtain a usage tutorial for the query information;

[0110] Feedback module 304 is used to feed back the obtained usage tutorial for the query information to the mobile terminal.

[0111] Specifically, the processing module 302 is used for:

[0112] The query information is text-corrected using a large language model to obtain the corrected query information.

[0113] A large language model is used to perform secondary text correction on the corrected query information to obtain the secondary corrected query information.

[0114] The contextual information is concatenated with the second-corrected query information to obtain a query sentence carrying contextual information;

[0115] The query containing contextual information is processed to obtain a usage tutorial for the query information.

[0116] In summary, this embodiment proposes a mobile terminal interactive tutorial search device. When the mobile terminal determines that a user has touched the display screen and receives a query from the user, it sends the query and context information to the server. The server then processes the query and context information to obtain a usage tutorial specific to the query. Finally, the usage tutorial is fed back to the mobile terminal, allowing it to display the received tutorial to the user. Compared to related technologies where users can only solve mobile terminal usage problems by asking other users, this device eliminates the need for users to ask others. The user simply sends a query to the mobile terminal, which then obtains the context information of the user's current situation and sends it along with the query to the server. The server processes the query and context information to obtain a usage tutorial specific to the query and displays it to the user. Thus, when a user encounters a problem using the mobile terminal, they only need to input a query, and the mobile terminal will provide a solution. The entire process is simple and convenient, greatly improving the user experience. Furthermore, by combining touchscreen input with voice to describe the problem, the device avoids the problem of unclear voice descriptions when some users do not know how to describe the interface.

[0117] Example 4

[0118] This embodiment proposes a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the mobile terminal interactive tutorial search method described in Embodiment 2 above. For specific implementation details, please refer to Method Embodiment 2, which will not be repeated here.

[0119] In addition, see Figure 4 The diagram shows the structure of an electronic device. This embodiment also proposes an electronic device, which includes a bus 51, a processor 52, a transceiver 53, a bus interface 54, a memory 55, and a user interface 56. The electronic device includes a memory 55.

[0120] In this embodiment, the electronic device further includes: one or more programs stored in the memory 55 and executable on the processor 52, configured to be executed by the processor to perform the one or more programs for the following steps (1) to (3):

[0121] (1) Receive query information and contextual information sent by the mobile terminal;

[0122] (2) Process the inquiry information and the context information to obtain a usage tutorial for the inquiry information;

[0123] (3) Feedback the obtained usage tutorial for the query information to the mobile terminal.

[0124] Transceiver 53 is used to receive and send data under the control of processor 52.

[0125] The bus architecture (represented by bus 51) can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 52 and memory represented by memory 55. Bus 51 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be further described in this embodiment. Bus interface 54 provides an interface between bus 51 and transceiver 53. Transceiver 53 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. For example, transceiver 53 receives external data from other devices. Transceiver 53 is used to transmit data processed by processor 52 to other devices. Depending on the nature of the computing system, a user interface 56 may also be provided, such as a keypad, display, speaker, microphone, or joystick.

[0126] Processor 52 is responsible for managing bus 51 and general processing, such as running general-purpose operating system 551 as described above. Memory 55 can be used to store data used by processor 52 during operation.

[0127] Optionally, the processor 52 may be, but is not limited to, a central processing unit, a microcontroller, a microprocessor, or a programmable logic device.

[0128] It is understood that the memory 55 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 55 of the systems and methods described in this embodiment is intended to include, but is not limited to, these and any other suitable types of memory.

[0129] In some implementations, memory 55 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 551 and application programs 552.

[0130] The operating system 551 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 552 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of the embodiments of this application can be included in the application program 552.

[0131] In summary, this embodiment proposes a computer-readable storage medium and an electronic device.

[0132] When the mobile terminal determines that a user has touched its display screen and receives a user-input query, it sends the query and contextual information to the server. The server then processes the query and contextual information to generate a user guide specific to the query. Finally, the user guide is fed back to the mobile terminal, allowing it to display the received tutorial to the user. Compared to related technologies where users can only resolve mobile terminal usage issues by asking other users, this method eliminates the need for such inquiries. The user simply sends a query to the mobile terminal, which then obtains the contextual information of the user's situation and sends it along with the query to the server. The server processes the query and contextual information to generate a user guide, which is then displayed to the user. Therefore, when a user encounters a problem using the mobile terminal, they only need to input a query, and the mobile terminal will provide a user guide to resolve the issue. The entire process is simple and convenient, greatly improving the user experience. Furthermore, the combination of touchscreen and voice description avoids the problem of unclear voice descriptions when users are unsure how to describe the interface.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A mobile terminal interactive tutorial search system, characterized in that, include: Mobile terminals and servers; The mobile terminal is configured to, when determining that a user touches the display screen of the mobile terminal and receives user-inputted query information, acquire contextual information about the user's current situation, wherein the contextual information includes: the user's current location information, the identifier of the application currently being used by the user, the application interface currently opened by the user, and the UI controls touched by the user within the currently opened application interface; send the query information and the contextual information to the server; receive a usage tutorial from the server in response to the query information, and display the received usage tutorial to the user; the query information includes a problem that the user needs to solve regarding the use of the mobile terminal; The server is configured to receive the query information and context information sent by the mobile terminal; process the query information and context information to obtain a usage tutorial for the query information; and feed back the obtained usage tutorial for the query information to the mobile terminal; the server is configured to process the query information and context information to obtain a usage tutorial for the query information, including: The query information is text-corrected using a large language model to obtain corrected query information; the task of text correction is to correct the query information into a more easily understood statement. The corrected query information is processed by a large language model to obtain a second-corrected query information; the task of the second-corrected text correction is to replace words in the corrected query information with generic terms. The contextual information is concatenated with the second-corrected query information to obtain a query sentence carrying contextual information; The query containing contextual information is processed to obtain a usage tutorial for the query information.

2. The system according to claim 1, characterized in that, The server is configured to process the query information and the context information to obtain a usage tutorial for the query information, including: Using the application identifier currently used by the user carried in the context information, a classification model is retrieved to classify the application corresponding to the application identifier; The query sentences carrying contextual information are classified using the retrieved classification model to obtain tutorial identifiers for the usage tutorials based on the query information; The tutorial identifier is used to retrieve the usage tutorial for the query information, thereby obtaining the usage tutorial for the query information.

3. A method for searching interactive tutorials on mobile terminals, used to perform the functions of a server in the interactive tutorial search system for mobile terminals as described in any one of claims 1-2, characterized in that, The method includes: The system receives the query information and context information sent by the mobile terminal; the query information includes a problem the user needs to solve regarding the use of the mobile terminal. The query information and the context information are processed to obtain a usage tutorial for the query information; The obtained usage tutorial for the query information is fed back to the mobile terminal; The process of processing the query information and the context information to obtain a usage tutorial for the query information includes: The query information is text-corrected using a large language model to obtain corrected query information; the task of text correction is to correct the query information into a more easily understood statement. The corrected query information is processed by a large language model to obtain a second-corrected query information; the task of the second-corrected text correction is to replace words in the corrected query information with generic terms. The contextual information is concatenated with the second-corrected query information to obtain a query sentence carrying contextual information; The query containing contextual information is processed to obtain a usage tutorial for the query information.

4. The method according to claim 3, characterized in that, The process of processing the question carrying contextual information to obtain a usage tutorial for the question information includes: Using the application identifier currently used by the user carried in the context information, a classification model is retrieved to classify the application corresponding to the application identifier; The query sentences carrying contextual information are classified using the retrieved classification model to obtain tutorial identifiers for the usage tutorials based on the query information; The tutorial identifier is used to retrieve the usage tutorial for the query information, thereby obtaining the usage tutorial for the query information.

5. A mobile terminal interactive tutorial search device, used to execute the mobile terminal interactive tutorial search method according to any one of claims 3-4, characterized in that, include: The receiving module is used to receive query information and contextual information sent by the mobile terminal; The inquiry information includes the user's problems with using the mobile terminal; The processing module is used to process the query information and the context information to obtain a usage tutorial for the query information; The feedback module is used to provide the mobile terminal with the usage tutorial obtained in response to the query information. The processing module is specifically used for: The query information is text-corrected using a large language model to obtain corrected query information; the task of text correction is to correct the query information into a more easily understood statement. The corrected query information is processed by a large language model to obtain a second-corrected query information; the task of the second-corrected text correction is to replace words in the corrected query information with generic terms. The contextual information is concatenated with the second-corrected query information to obtain a query sentence carrying contextual information; The query containing contextual information is processed to obtain a usage tutorial for the query information.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the method described in any one of claims 3-4.

7. An electronic device, characterized in that, The electronic device includes a memory, a processor, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor of the steps of the method according to any one of claims 3-4.

Images