Interaction method and device based on large language model, electronic equipment and medium

By identifying text data in images and using large language models for automated interaction, the problem of high interaction complexity in the prior art is solved, and low-cost and low-complexity application operations are achieved.

CN119067078APending Publication Date: 2024-12-03深圳市星桐科技有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411081337.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the interaction method between graphical user interface and language user interface requires users to perform complex operations, resulting in high interaction costs.

Method used

Using an interactive method based on a large language model, by identifying text data in the image uploaded by the user, using preset identification information as a prompt word, input it into the large language model to obtain matching identification information, and executing preset instructions based on this information to realize Automated application interaction.

Benefits of technology

It reduces the operational complexity and time cost of users, reduces the interaction cost, realizes end-to-end application interaction, and avoids the problem of uncontrollable output of large language models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067078A_ABST
    Figure CN119067078A_ABST
Patent Text Reader

Abstract

The invention provides an interaction method and device based on a large language model, electronic equipment and a medium. According to one example of the invention, the method comprises the following steps: identifying text data in an image uploaded by a user; wherein the text data in the image comprises an application function name of the application program; taking preset identification information as a cue word of the large language model; the identification information is used for guiding the large language model to output matching identification information corresponding to the application function name; inputting the text data and the cue words into a large language model to obtain matching identification information output by the large language model; and executing a preset instruction on the application program based on the matching identification information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interaction method, device, electronic device and medium based on a large language model. Background Art

[0002] With the rapid development of information technology, the interaction methods between users and electronic devices are becoming increasingly diverse. Among them, Graphical User Interface (GUI) and Language User Interface (LUI), as traditional and mainstream interaction methods, occupy an important position in various software and applications.

[0003] The interactive mode of the graphical user interface presents the user interface through graphical elements such as windows, icons, menus, and interactive controls such as buttons and sliders, allowing users to perform intuitive operations through devices such as mice or touch screens. The interactive mode of the language user interface is based on natural language processing (NLP) and speech recognition technology. It allows users to interact with the device through voice or text without physically touching the screen or pressing buttons.

[0004] However, the above-mentioned interaction methods, whether through voice commands or graphical interfaces, may require users to perform a series of complex operations to complete specific tasks, which has a high interaction cost. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present application provides an interaction method, device, electronic device and medium based on a large language model.

[0006] According to a first aspect of any embodiment of the present application, there is provided an interaction method based on a large language model, the method comprising:

[0007] Identify text data in an image uploaded by a user; wherein the text data in the image includes an application function name of an application program;

[0008] Using the preset identification information as a prompt word of the large language model; the identification information is used to guide the large language model to output matching identification information corresponding to the application function name;

[0009] Inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model;

[0010] Based on the matching identification information, a preset instruction is executed on the application.

[0011] According to a second aspect of any embodiment of the present application, there is provided an interaction device based on a large language model, the device comprising:

[0012] A text recognition module, used to recognize text data in an image uploaded by a user; wherein the text data in the image includes an application function name of an application program;

[0013] An identification prompt module, used to use preset identification information as a prompt word of the large language model; the identification information is used to guide the large language model to output matching identification information corresponding to the application function name;

[0014] An identification matching module, used for inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model;

[0015] An instruction execution module is used to execute preset instructions on the application based on the matching identification information.

[0016] According to a third aspect of any embodiment of the present application, there is provided an electronic device, including:

[0017] processor;

[0018] a memory for storing processor-executable instructions;

[0019] The processor implements the method described in any embodiment of the present application by running the executable instructions.

[0020] According to a fourth aspect of any embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the method described in any embodiment of the present application is implemented.

[0021] According to a fifth aspect of any embodiment of the present application, a computer program product is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in any embodiment of the present application is implemented.

[0022] The technical solution provided by this application may have the following beneficial effects:

[0023] According to the above embodiment, by identifying the text data of the application function name of the application in the image uploaded by the user, the preset identification information is used as the prompt word of the large language model, the text data and the prompt word are input into the large language model, and the matching identification information output by the large language model is obtained. Based on the matching identification information, the preset instructions are executed on the application corresponding to the application function name. The user only needs to upload the image to automatically complete the whole process of identifying text data, matching the application and executing instructions, thereby reducing the user's operation complexity and time cost, thereby reducing the user's interaction cost.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0026] Figure 1 is a flowchart of an interaction method based on a large language model according to an exemplary embodiment of the present application;

[0027] Figure 2 is a schematic diagram showing an uploaded image according to an exemplary embodiment of the present application;

[0028] Figure 3 is a flowchart of another interaction method based on a large language model according to an exemplary embodiment of the present application;

[0029] Figure 4 is a schematic diagram of application interaction based on a large language model according to an exemplary embodiment of the present application;

[0030] Figure 5 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present application;

[0031] Figure 6 It is a block diagram of an interaction device based on a large language model according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0033] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] Currently, interaction through a graphical user interface or a language user interface may require users to perform a series of complex operations to complete a specific task, which has a high interaction cost.

[0036] In order to solve the above problems, the present application proposes an interaction method based on a large language model (LLM). To further illustrate the present application, the following embodiments are provided:

[0037] See also Figure 1 , Figure 1 This is a flow chart of an interactive method based on a large language model according to an exemplary embodiment of the present application. The interactive method based on a large language model can be applied to electronic devices such as learning machines, mobile phones, computers, digital broadcast terminals, tablet devices, personal digital assistants, smart home appliances, etc.

[0038] Taking the interactive method applied to a learning machine as an example, the method may include the following steps:

[0039] Step 102: Identify text data in the image uploaded by the user; wherein the text data in the image includes the application function name of the application program.

[0040] In this step, the learning machine can recognize the image uploaded by the user through optical character recognition (OCR), convolutional neural network (CNN) and other technologies to obtain text data in the image.

[0041] The image may be a picture obtained by screenshot, photography, etc., and the text data in the image may include the application function name of the application to be interacted with. The application function name may include at least one of the following: function name, program name.

[0042] The function name is the name of a specific application function in the application, such as: dictation practice, follow-up reading practice, etc. The program name is a name that intuitively reflects the function or characteristics of the application, such as: word reading and writing, thinking training, etc.

[0043] See also Figure 2 , Figure 2 1 is a schematic diagram of an uploaded image according to an exemplary embodiment of the present application. Exemplarily, the learning machine recognizes the image uploaded by the user, wherein the dictation exercise is the application function name in the text data in the image.

[0044] The text data recognized by the learning machine are "You haven't finished your homework yesterday", "Remember to turn on the dictation practice of the learning machine after you go home today", and "Do the exercises that you didn't finish yesterday".

[0045] In one embodiment, the learning machine can receive images uploaded by users and input the images into a pre-trained text recognition model. The text recognition model uses ORC technology to perform text recognition on the image and output the text data in the image. The learning machine obtains the text data output by the text recognition model.

[0046] Among them, the text recognition model is an optical character recognition model that extracts text from an image through image processing and pattern recognition technology and converts it into text data that can be recognized by a learning machine.

[0047] As described above, by receiving images uploaded by users, inputting the images into a text recognition model, and obtaining text data output by the text recognition model, the accuracy and speed of text data recognition can be improved, thereby improving the accuracy and response speed of user interaction with the application.

[0048] Step 104: Use the preset identification information as a prompt word of the large language model; the identification information is used to guide the large language model to output matching identification information corresponding to the application function name.

[0049] In this step, in order to avoid the situation where the content output by the large language model cannot be mapped to the application to be interacted with, the learning machine can adopt a conditional generation method to use the preset identification information as the prompt of the large language model.

[0050] The learning machine can add a preset special token to the left and right sides of each identification information in the prompt word. <pack>, add <\pack> to the right of each identifier to separate the identifiers in the prompt word.

[0051] The preset identification information is the identification information of all applications that support user interaction. The identification information is used to guide the large language model to output matching identification information corresponding to the application function name. The identification information may be information such as the package name and encoding of the application. The identification information at least includes matching identification information corresponding to the application function name.

[0052] All applications supporting user interaction may be all applications supporting user interaction in the learning machine, or may be all applications supporting user interaction in the application market, and this embodiment of the present application does not impose any limitation on this.

[0053] A large language model is a type of deep learning model with a large number of parameters. It learns language patterns, grammar, and semantics by processing large amounts of text, thereby understanding and generating human language. Examples include the Chat Generative Pre-Trained Transformer (ChatGPT), the Bidirectional Encoder Representations from Transformers (BERT), etc.

[0054] Step 106: Input the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model.

[0055] In this step, if the text data in the image includes multiple lines of text, the learning machine can integrate each line of text using symbols such as semicolons to obtain integrated text data.

[0056] The learning machine can input text data and prompt words into the large language model. After the large language model understands the semantics of the input text data, it selects matching identification information corresponding to the application function name from the identification information of all application programs supporting user interaction given by the prompt words, and outputs the matching identification information.

[0057] The learning machine obtains matching identification information corresponding to the application function name output by the large language model.

[0058] The matching identification information is used to identify the application to be interacted with, and may be the package name, code, etc. of the application to be interacted with.

[0059] In one embodiment, before inputting the text data and the prompt words into the large language model, the learning machine may use a preset format to distinguish between handwritten text and printed text in the text data to obtain updated text data.

[0060] For example, the learning machine can use the preset format of the "##" separator to add the preset "##" separator to the left and right sides of handwritten text or the left and right sides of printed text, so that the large language model can distinguish between handwritten text and printed text in the text data.

[0061] The preset format is used to assist the large language model in distinguishing between handwritten text and printed text, and may include the use of diacritical marks, text type prompts, and the like.

[0062] As described above, by using a preset format to distinguish between handwritten text and printed text in text data, updated text data is obtained, ensuring that different processing strategies can be applied to text data of different text types when the large language model recognizes text data. Handwritten text may require the allocation of more computing resources or the use of more complex recognition algorithms, while printed text can adopt a more efficient processing method, thereby more accurately and efficiently understanding and analyzing the matching identification information corresponding to the text data.

[0063] Step 108: Based on the matching identification information, execute preset instructions on the application.

[0064] In this step, the learning machine can find the application corresponding to the matching identification information based on the matching identification information, and execute preset instructions on the application.

[0065] Taking the matching identification information as the package name as an example, the learning machine can access the package name of the application by executing the code, and automatically complete the interactive operation of the application based on the preset instructions.

[0066] When the application function name is a function name, the learning machine can execute preset instructions on the application function corresponding to the function name in the application program; when the application function name is a program name, the learning machine can execute preset instructions on the application program.

[0067] The preset instructions are used to perform interactive operations on the application, such as launching the application, clicking on an application function in the application, etc.

[0068] If the text data in the image includes a single application function name, the large language model outputs single matching identification information corresponding to the application function name. The learning machine can execute a preset instruction on the application corresponding to the single application function name based on the matching identification information.

[0069] If the text data in the image includes multiple application function names, the large language model outputs a matching identification information list corresponding to the multiple application function names, and the matching identification information list includes multiple matching identification information corresponding to the multiple application function names.

[0070] The learning machine can parse and traverse the multiple matching identification information in the matching identification information list, and execute preset instructions on a specified application or all of the multiple applications corresponding to the multiple matching identification information.

[0071] In one embodiment, after obtaining the matching identification information output by the large language model, the learning machine can determine whether the application corresponding to the matching identification information has been installed in the learning machine. Through a set distribution mechanism, the user is guided to the application to be interacted with.

[0072] The learning machine can guide the user to the application program when the application program exists in the learning machine, and automatically execute preset instructions on the application program.

[0073] The learning machine can also jump to the download page corresponding to the application when the application does not exist in the learning machine, guide the user to the download page corresponding to the application, and prompt the user whether to perform the download operation.

[0074] As described above, by executing preset instructions on the application when the application exists in the electronic device, invalid operations can be avoided when the application is not installed, thereby saving system resources and user time; when the application does not exist in the electronic device, it jumps to the download page corresponding to the application, without the user having to manually search for the application, which can improve interaction efficiency and user experience.

[0075] The interaction method based on the large language model of this embodiment recognizes the text data of the application function name of the application in the image uploaded by the user, uses the preset identification information as the prompt word of the large language model, inputs the text data and the prompt word into the large language model, obtains the matching identification information output by the large language model, and executes the preset instruction on the application corresponding to the application function name based on the matching identification information. The user only needs to upload the image to automatically complete the whole process of recognizing the text data, matching the application and executing the instruction, thereby reducing the user's operation complexity and time cost, thereby reducing the user's interaction cost.

[0076] Moreover, by adding identification information as a condition to the prompt word, the problem of uncontrollable output of the large language model can be solved, and the situation where the electronic device cannot recognize the erroneous output information can be avoided; since the electronic device can directly recognize the matching identification information output by the large language model, there is no need for tedious post-processing of the application function name, thereby truly realizing end-to-end application interaction.

[0077] In the aforementioned embodiment, it is introduced that by inputting the text data in the image and the identification information of all applications supporting user interaction into the large language model, based on the matching identification information corresponding to the application function name output by the large language model, the preset instructions are automatically executed on the application. In the following embodiment, the fine-tuning training process of the large language model will be introduced in detail, and can be applied to any of the above embodiments.

[0078] In one embodiment, a training image related to the application function name is prepared in advance, and the training text data in the training image includes a plurality of application function names. Through manual annotation, the plurality of application function names in the text content are extracted according to the text content in the training image.

[0079] Preprocess the training matching identifiers corresponding to the training images, replace the extracted multiple application function names with corresponding identifier information, use the identifier information corresponding to the multiple application function names as the training matching identifiers, and use the training matching identifiers as labels for the large language model.

[0080] The learning machine performs text recognition on the training image to obtain training text data in the training image, where the training text data is text data in the training image.

[0081] In order to prevent the large language model from outputting non-existent identification information during the training process, the learning machine inputs the training identification information into the large language model. The training identification information is the identification information of all applications that support user interaction in the application market, and is used to guide the large language model to output the training matching identification corresponding to the training image, and the training identification information at least includes the training matching identification.

[0082] The learning machine inputs the input text consisting of training text data and training identification information and the training matching identification into the large language model in pairs, and uses fine-tuning techniques such as low-rank adaptation (LoRA), full parameter fine-tuning, and adapter model to fine-tune the large language model.

[0083] Before fine-tuning the large language model, the learning machine can also pre-process the training text data and training identification information. Use a preset format to distinguish between handwritten text and printed text in the training text data. For example, the printed text in the training text data is input normally, and "##" distinguishing symbols are added on both sides of the handwritten text in the training text data to obtain the pre-processed training text data.

[0084] Add special words to the left and right of each identification information in the training identification information, and add <pack>, add <\pack> to the right of each identification information to obtain the preprocessed training identification information.

[0085] For example, the application function names extracted from the training image are "dictation" and "follow-up reading", the identification information of the application corresponding to dictation is P1, and the identification information of the application corresponding to follow-up reading is P2. The pre-processed training matching identification is "[P1, P2]", which is a list format that can be directly converted and read by a large language model.

[0086] The input text composed of the preprocessed training text data and training identification information is "Remember to open the dictation app today## and then learn to follow## <pack>P1<\pack> <pack>P2<\pack>”.

[0087] Among them, "Remember to open the dictation app today" is the printed text in the training text data, and "Relearn to read along" is the handwritten text in the training text data. <pack>P1<\pack> <pack>P2<\pack>” is the training identification information.

[0088] As described above, by inputting the training text data and training identification information in the training image into the large language model, using the training matching identification corresponding to the training image as a label, and fine-tuning the large language model, the accuracy of identifying matching identification information and executing preset instructions can be improved.

[0089] In one embodiment, the learning machine can perform text recognition on invalid images that are not related to the application function name to obtain text data corresponding to the invalid image. If the invalid image does not include text, the text data corresponding to the invalid image is empty.

[0090] The text data and training identification information corresponding to the invalid image are input into the large language model, the empty result is used as the label of the large language model, and the large language model is fine-tuned and trained using fine-tuning technology.

[0091] Among them, invalid images are images whose corresponding text data do not contain application function names, such as: animal photos, selfies of people, landscape photos, etc.

[0092] As described above, by inputting the text data and training identification information corresponding to the invalid image that do not contain the application function name into the large language model, and using the empty result as a label, the large language model is fine-tuned to avoid the large language model outputting information that is irrelevant to the matching identification information of the application, resulting in abnormal execution of preset instructions.

[0093] To further introduce the interaction process based on the large language model, Figure 3 A flow chart of another interaction method based on a large language model is shown. Continuing to take the electronic device as a learning machine as an example, the interaction method may include the following steps:

[0094] Step 302: input the training text data and training identification information in the training image into the large language model, use the training matching identification corresponding to the training image as a label, and fine-tune the large language model.

[0095] In this step, the learning machine can use OCR technology to identify the training text data in the training image. The training text data and training identification information are input into the large language model, and the training matching identification corresponding to the training image is used as a label to fine-tune the large language model.

[0096] Step 304: Input the text data and training identification information corresponding to the invalid image into the large language model, use the empty result as a label, and fine-tune the large language model.

[0097] In this step, the learning machine can use OCR technology to identify text data in invalid images. The text data and training identification information corresponding to the invalid images are input into the large language model, and the empty results are used as labels to fine-tune the large language model.

[0098] Step 306: Input the image uploaded by the user into the text recognition model to obtain text data output by the text recognition model.

[0099] In this step, the learning machine can receive images uploaded by users, input the images into the text recognition model, and obtain text data output by the text recognition model.

[0100] See also Figure 4 , Figure 4 A schematic diagram of application interaction based on a large language model is shown. After the user uploads the image, the learning machine can input the image into the text recognition model 41 to obtain the text data output by the text recognition model 41.

[0101] Step 308: The learning machine uses a preset format to distinguish between handwritten text and printed text in the text data to obtain updated text data.

[0102] Step 310: Use the preset identification information as a prompt word of the large language model.

[0103] In this step, the learning machine uses the preset identification information as a prompt word of the large language model to guide the large language model to output matching identification information corresponding to the application function name.

[0104] Step 312: Input the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model.

[0105] In this step, the learning machine inputs the prompt words corresponding to the text data and the identification information into the large language model, and the large language model understands the semantics of the text data and selects the matching identification information corresponding to the application function name from the identification information given by the prompt words, and obtains the matching identification information output by the large language model.

[0106] Please continue reading Figure 4 , the learning machine determines whether the text data in the image is empty. If the text data is empty, the interactive operation is determined to be an invalid operation; if the text data is not empty, the text data and the prompt word are input into the large language model 42 to obtain the matching identification information output by the large language model 42.

[0107] Step 314: Determine whether there is an application program in the learning machine.

[0108] In this step, the learning machine can determine whether there is an application in the learning machine.

[0109] Please continue reading Figure 4 The learning machine can determine whether the matching identification information output by the large language model 42 is valid identification information. If the matching identification information is valid identification information, it is determined whether there is an application in the learning machine; if the matching identification information is not valid identification information, the interactive operation is determined to be an invalid operation.

[0110] If the application program exists in the learning machine, continue to execute step 316;

[0111] If the application does not exist in the learning machine, step 318 is executed.

[0112] Step 316: Execute the preset instruction on the application program.

[0113] In this step, if there is an application in the learning machine, the learning machine guides the user to the application corresponding to the matching identification information and executes preset instructions on the application.

[0114] Please continue reading Figure 4 ,When there is an application in the learning machine, the learning machine automatically executes preset instructions on the application.

[0115] Step 318: Jump to the download page corresponding to the application.

[0116] In this step, if the application does not exist in the learning machine, the learning machine jumps to the download page corresponding to the application and prompts the user whether to perform the download operation.

[0117] Please continue reading Figure 4 , if the application does not exist in the learning machine, the user is directed to the download page of the application.

[0118] Figure 5 1 is a schematic diagram of a structure of an electronic device according to an exemplary embodiment of the present application. The electronic device may be, for example, a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, etc. Figure 5 At the hardware level, the electronic device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other services. The processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it, forming an interactive device based on a large language model at the logical level. Of course, in addition to the software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0119] Figure 6 is a block diagram of an interactive device based on a large language model according to an exemplary embodiment of the present application. Figure 6 The device may include: a text recognition module 602, an identification prompt module 604, an identification matching module 606 and an instruction execution module 608, wherein:

[0120] The text recognition module 602 is used to recognize text data in the image uploaded by the user; wherein the text data in the image includes the application function name of the application program;

[0121] The identification prompt module 604 is used to use the preset identification information as a prompt word of the large language model; the identification information is used to guide the large language model to output the matching identification information corresponding to the application function name;

[0122] The identification matching module 606 is used to input the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model;

[0123] The instruction execution module 608 is used to execute preset instructions on the application based on the matching identification information.

[0124] In one example, the instruction execution module 608, when used to execute preset instructions on the application based on the matching identification information, includes at least one of the following: if the application exists in the electronic device, executing the preset instructions on the application; if the application does not exist in the electronic device, jumping to the download page corresponding to the application.

[0125] In one example, the text recognition module 602, when used to recognize text data in an image uploaded by a user, includes: receiving the image uploaded by the user; inputting the image into a text recognition model, and obtaining the text data output by the text recognition model.

[0126] In one example, the identification matching module 606, before being used to input the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model, further includes: inputting the training text data and training identification information in the training image into the large language model, using the training matching identification corresponding to the training image as a label, and fine-tuning the large language model; wherein the training text data in the training image includes multiple application function names; the training matching identification is identification information corresponding to the multiple application function names; and the training identification information is used to guide the large language model to output the training matching identification.

[0127] In one example, the identification matching module 606, before being used to input the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model, further includes: inputting the text data and training identification information corresponding to the invalid image into the large language model, using the empty result as a label, and fine-tuning the large language model; wherein the text data corresponding to the invalid image does not contain the application function name.

[0128] In one example, the text data includes handwritten text and printed text; the identification matching module 606, before being used to input the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model, further includes: using a preset format to distinguish the handwritten text from the printed text to obtain the updated text data.

[0129] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0130] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0131] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an interaction device based on a large language model to implement any method as described in the above embodiments.

[0132] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present application does not limit this.

[0133] In an exemplary embodiment, a computer program product including a computer program / instruction is also provided. The computer program / instruction can be executed by a processor of an interaction device based on a large language model to implement any of the methods described in the above embodiments.

[0134] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0135] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0136] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.< / pack> < / pack> < / pack> < / pack> < / pack> < / pack>

Claims

1. An interactive method based on a large language model, It is characterized in that The method comprises: Identify text data in an image uploaded by a user; wherein the text data in the image includes an application function name of an application program; Using the preset identification information as a prompt word of the large language model; the identification information is used to guide the large language model to output matching identification information corresponding to the application function name; Inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model; Based on the matching identification information, a preset instruction is executed on the application.

2. The method according to claim 1, It is characterized in that The executing a preset instruction on the application based on the matching identification information includes at least one of the following: When the application program exists in the electronic device, executing a preset instruction on the application program; If the application does not exist in the electronic device, jump to the download page corresponding to the application.

3. The method according to claim 1, It is characterized in that The identifying text data in the image uploaded by the user comprises: receiving the image uploaded by the user; The image is input into a text recognition model to obtain the text data output by the text recognition model.

4. The method according to claim 1, It is characterized in that Before inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model, the method further includes: Inputting the training text data and training identification information in the training image into the large language model, using the training matching identification corresponding to the training image as a label, and fine-tuning the large language model; Among them, the training text data in the training image includes multiple application function names; the training matching identifier is identification information corresponding to the multiple application function names; and the training identification information is used to guide the large language model to output the training matching identifier.

5. The method according to claim 1, It is characterized in that Before inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model, the method further includes: Inputting text data and training identification information corresponding to the invalid image into the large language model, using the empty result as a label, and fine-tuning the large language model; The text data corresponding to the invalid image does not include the application function name.

6. The method according to claim 1, It is characterized in that The text data includes handwritten text and printed text; Before inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model, the method further includes: The handwritten text and the printed text are distinguished using a preset format to obtain updated text data.

7. An interactive device based on a large language model, It is characterized in that The device comprises: A text recognition module, used to recognize text data in an image uploaded by a user; wherein the text data in the image includes an application function name of an application program; An identification prompt module, used to use preset identification information as a prompt word of the large language model; the identification information is used to guide the large language model to output matching identification information corresponding to the application function name; An identification matching module, used for inputting the text data and the prompt word into the large language model to obtain the matching identification information output by the large language model; An instruction execution module is used to execute preset instructions on the application based on the matching identification information.

8. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, It is characterized in that When the instruction is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product having a computer program / instructions stored thereon, It is characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Voice instruction response method and device based on large model fine tuning, equipment and medium

    CN119943046A