Method for acquiring schedule information, electronic device, and readable storage medium

Abstract

The application discloses a method for obtaining schedule information, an electronic device and a readable storage medium, and belongs to the technical field of terminals. The method comprises the following steps: in response to a schedule extraction operation, obtaining target text information, wherein the target text information comprises to-be-extracted schedule information; determining a system prompt corresponding to a business scenario to which the target text information belongs from a plurality of system prompts, obtaining a target system prompt, the system prompt is a task prompt for describing a to-be-reasoned event, and each system prompt in the plurality of system prompts corresponds to a business scenario; obtaining the KV value of the target system prompt from the KV values of the plurality of system prompts in the cache; and taking the target text information and the KV value of the target system prompt as the input of a pre-trained target natural language model, reasoning through the target natural language model, and outputting the schedule information corresponding to the target text information. The application can improve the speed of creating schedule information.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a method for acquiring schedule information, an electronic device, and a readable storage medium. Background Technology

[0002] With the rapid development of terminal technology, electronic devices can typically install various types of third-party social applications, such as instant messaging and ticketing apps. Some messages or service notifications in these third-party social applications often involve schedule information, such as a chat window in an instant messaging application displaying the schedule of a meeting. In some scenarios, users often need to create schedule information related to third-party social applications using their electronic devices. Therefore, how to enable electronic devices to quickly create schedule information has become a hot research topic in this field. Summary of the Invention

[0003] This application provides a method, electronic device, and readable storage medium for obtaining schedule information, which can solve the problem of how to quickly create schedule information in related technologies. The technical solution is as follows:

[0004] Firstly, a method for obtaining schedule information is provided, applied in an electronic device, the method comprising:

[0005] In response to the schedule retrieval operation, the target text information is obtained, which includes the schedule information to be extracted. From multiple system prompts, the system prompt corresponding to the target text information is determined, and the target system prompt is obtained. The system prompt describes the task of the event to be inferred, and each of the multiple system prompts corresponds to a business scenario. The key-value (KV) value of the target system prompt is obtained from the cached KV values ​​of multiple system prompts. The target text information and the target system prompt's KV value are used as input to a pre-trained target natural language model. The target natural language model performs inference and outputs the schedule information corresponding to the target text information.

[0006] In this way, by pre-calculating and caching the key-value (KV) values ​​corresponding to different system prompts, the target system prompts to be used can be determined according to the business scenario during the application process. Then, the KV values ​​corresponding to the target system prompts can be obtained from the cached KV values ​​as part of the model input. This eliminates the need for the target natural language model to calculate the target system prompts, reducing the computational load of the target natural language model and thus accelerating the extraction of schedule information, thereby improving the creation speed of schedule information.

[0007] As an example of this application, determining the system prompt corresponding to target text information from multiple system prompts includes: determining the business scenario corresponding to the target text information based on one or more of the following: keywords related to the target text information, business entry information, application information, and intent description information. Different keywords indicate different business scenarios, business entry information indicates the entry point used to request text information inference, application information indicates the category of the application from which the target text information originates, and intent description information indicates the intent of the target text information. The system prompt corresponding to the determined business scenario is then obtained from the multiple system prompts. Thus, by determining the business scenario of the target text information and obtaining the corresponding target system prompt, the target natural language model can accurately infer based on the target system prompt, improving the accuracy of schedule information extraction.

[0008] As an example of this application, the target text information originates from an image. Accordingly, based on one or more of the keywords, business entry information, application information, and intent description information related to the target text information, the business scenario corresponding to the target text information is determined. This includes: if the first image corresponding to the target text information is obtained through a calendar application request for schedule extraction, and if the first image is obtained by taking a screenshot of a foreground application using an electronic device, then the business scenario corresponding to the target text information is determined based on the application information of the foreground application. If the first image is not obtained by taking a screenshot using an electronic device, then the intent description information corresponding to the first image is determined using a pre-trained intent classification model, and the business scenario corresponding to the target text information is determined based on the intent description information corresponding to the first image. Thus, for the first image, the corresponding business scenario can be determined based on its source or its intent, thereby determining the business scenario corresponding to the target text information.

[0009] As an example of this application, before obtaining the KV value of the target system prompt from the KV values ​​of multiple cached system prompts, the method further includes: for each system prompt among the multiple system prompts, inputting each system prompt into a target natural language model for processing to obtain the KV value of each system prompt. The KV value of each system prompt among the multiple system prompts is then cached. This allows the required KV value to be directly obtained from the cached KV values ​​during application. Furthermore, since the model calculating the KV value corresponding to each system prompt is the target natural language used during application, meaning the KV value is error-free, the target natural language model can accurately infer during application.

[0010] As an example of this application, in response to the schedule extraction operation on the first image, a first recognition result and a second recognition result are determined through a target recognition model. The first recognition result includes the text recognition result of the first image, and the second recognition result includes the edge recognition result and image category of the first image. The edge recognition result includes the image element attribute information or color block attribute information of the first image. Based on the image category of the first image, interference information unrelated to the schedule in the text recognition result and edge recognition result of the first image is filtered out. Based on the filtered first recognition result and filtered second recognition result, the target text information is obtained. In this way, users do not need to manually create schedule information in the calendar application, improving the efficiency of schedule information creation.

[0011] As an example of this application, the image categories include instant messaging chat screenshots, instant messaging notification card screenshots, order screenshots, and other image categories. An instant messaging chat screenshot refers to an image obtained by taking a screenshot of the chat interface in an instant messaging application; an instant messaging notification card screenshot refers to an image obtained by taking a screenshot of the service notification card in an instant messaging application; an order screenshot refers to an image obtained by taking a screenshot of the order interface in the application; and other image categories include images other than instant messaging chat screenshots, instant messaging notification card screenshots, and order screenshots. Specifically, when the first image is an instant messaging chat screenshot, the edge recognition result includes text and image element attribute information; when the first image is one of an instant messaging notification card screenshot, an order screenshot, or an image of another category, the edge recognition result includes color block attribute information.

[0012] Therefore, by categorizing images into the aforementioned categories, targeted image processing methods can be adopted based on image category during the subsequent extraction of schedule information, thereby improving the effectiveness and accuracy of schedule information extraction.

[0013] As an example of this application, the text recognition result includes text line coordinates. Accordingly, based on the image category of the first image, the specific implementation of filtering out schedule-irrelevant interference information from the text recognition result and edge recognition result of the first image may include: If the first image is an instant messaging chat screenshot, filtering out the recognition data corresponding to skewed text lines in the text recognition result of the first image based on the text line coordinates of each text line in the text recognition result of the first image; filtering out the recognition data corresponding to text lines whose line height is less than a target line height in the text recognition result and edge recognition result of the first image based on the text line coordinates of each text line in the text recognition result of the first image, where the target line height is the average line height of all text lines in the first image; and filtering out the recognition data corresponding to the chat timestamp in the first image from the text recognition result and edge recognition result of the first image. Thus, by analyzing the recognition data corresponding to skewed lines, small characters, and chat timestamps in the first image, schedule-irrelevant interference information can be effectively removed, i.e., information that may interfere with schedule information extraction, thereby improving the accuracy of subsequent schedule extraction.

[0014] As an example of this application, the image and text element attribute information in the edge recognition result includes the coordinates and category of the chat element, and the text recognition result also includes the text line recognition content. Accordingly, the specific implementation of filtering the recognition data corresponding to the chat timestamp in the first image from the text recognition result and the edge recognition result of the first image may include: determining chat elements whose category is chat timestamp from the edge recognition result, and obtaining at least one first candidate chat element. Based on the coordinates of each first candidate chat element among the at least one first candidate chat element and the text line coordinates in the text recognition result of the first image, matching the text line recognition content corresponding to each first candidate chat element from the text recognition result of the first image. If it is determined that the target first candidate chat element is a chat timestamp based on the text line recognition content corresponding to the target first candidate chat element, it is determined whether to filter the recognition data corresponding to the target first candidate chat element based on the coordinates of the target first candidate chat element, where the target first candidate chat element is any one of the at least one first candidate chat element. If it is determined that the recognition data corresponding to the target first candidate chat element should be filtered, the recognition data corresponding to the target first candidate chat element is filtered from both the text recognition result and the edge recognition result of the first image.

[0015] In this way, the chat elements that are classified as chat timestamps are first identified based on the edge recognition results. Then, the corresponding text lines are matched from the text recognition results. Based on the matched text lines, the target natural language model determines whether it is a chat timestamp. Finally, the position of the chat element is used to determine whether it is a chat timestamp. This can improve the accuracy of chat timestamp recognition, thereby improving the accuracy of filtering and, in turn, improving the accuracy of calendar information creation.

[0016] As an example of this application, the specific implementation of determining whether to filter out the recognition data corresponding to the target first candidate chat element based on its coordinates may include: if the target first candidate chat element is located in the middle of the first image based on its coordinates, and the area corresponding to the target first candidate chat element includes a single text line, then the recognition data corresponding to the target first candidate chat element is determined to be filtered out. Alternatively, if the target first candidate chat element is located on the right side of the first image based on its coordinates, and the area corresponding to the target first candidate chat element includes a single text line, then if the recognition content of the text line corresponding to the target first candidate chat element only includes a time point and not a date, then the recognition data corresponding to the target first candidate chat element is determined to be filtered out. Thus, determining whether the first candidate chat element is a chat timestamp based on the positional features and text line features can improve the accuracy of the determination.

[0017] As an example of this application, the text recognition result includes text line coordinates and text line recognition content. Based on the image category of the first image, the specific implementation of filtering out interference information unrelated to the schedule from the text recognition result and edge recognition result of the first image may include: if the first image is a screenshot of an instant messaging notification card or an order screenshot, matching the text line recognition content of each color block in the text recognition result of the first image with the coordinates of each color block in the edge recognition result and the text line coordinates in the text recognition result of the first image. For any color block, if it is determined from the text line recognition content of any color block that the color block does not contain schedule-related content, then the recognition data corresponding to that color block is filtered out from the edge recognition result and the text recognition result of the first image. Thus, by filtering out color blocks that do not contain schedule information, subsequent processing can focus only on color blocks containing schedule-related content, improving data processing efficiency and the accuracy of schedule information extraction.

[0018] As an example of this application, the text recognition result includes text line coordinates and text line recognition content. Based on the filtered first and second recognition results, the specific implementation of obtaining the target text information may include: matching the text line recognition content corresponding to each object from the filtered first image's text recognition results, where the object is a chat element or a color block, based on the coordinates of each object in the filtered edge recognition results and the text line coordinates in the text recognition results of the first image. The scene corresponding to the first image is determined according to the text line recognition content corresponding to each object and the image category of the first image. Based on the scene corresponding to the first image, text concatenation is performed on the text line recognition content corresponding to each object to obtain the target text information. Thus, by determining the scene corresponding to the first image and performing text concatenation based on the scene, the final target text information can be made to conform as closely as possible to the rules of natural language.

[0019] In a second aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method for displaying schedule information as described in the first aspect above.

[0020] Thirdly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method for displaying schedule information as described in the first aspect.

[0021] Fourthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the method for displaying schedule information described in the first aspect.

[0022] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating a model inference process according to an exemplary embodiment;

[0024] Figure 2 This is a schematic diagram illustrating an application scenario according to an exemplary embodiment;

[0025] Figure 3 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0026] Figure 4 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0027] Figure 5 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0028] Figure 6 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0029] Figure 7 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0030] Figure 8 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0031] Figure 9 This is a schematic diagram of a frame for displaying schedule information according to another exemplary embodiment;

[0032] Figure 10 This is a schematic diagram of a frame for displaying schedule information according to another exemplary embodiment;

[0033] Figure 11 This is a schematic diagram illustrating a software system of an electronic device according to an exemplary embodiment;

[0034] Figure 12 This is a flowchart illustrating a method for displaying schedule information according to an exemplary embodiment;

[0035] Figure 13 This is a schematic diagram illustrating the processing of an instant messaging chat screenshot according to an exemplary embodiment;

[0036] Figure 14 This is a schematic diagram illustrating the processing of an instant messaging notification card screenshot according to an exemplary embodiment;

[0037] Figure 15 This is a schematic diagram of the architecture of an electronic device according to an exemplary embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0039] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0040] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] Before introducing the method for obtaining schedule information provided in the embodiments of this application, the terms or nouns involved in the embodiments of this application will be briefly explained.

[0042] One-stop office software offers a variety of office functions, such as document processing, spreadsheets, presentation creation, project management, calendar, and email. Users can complete multiple office tasks within a single application, improving work efficiency. One-stop office software typically employs a unified user interface design, making switching and using different functional modules more convenient and consistent.

[0043] Global Favorites: This refers to the ability for users to save information on their electronic devices by swiping down the screen with three fingers.

[0044] Magic Text: This is a feature that quickly extracts text from images. Users can typically enable or disable this feature by going to "Settings > Smart Assistant > Magic Text".

[0045] Large language model (LLM): A model based on machine learning and natural language processing techniques. It can learn to serve the ability of human language understanding and generation by training on a large amount of sample training data. For example, a trained LLM can be used to reason about text information to extract schedule information.

[0046] Prompts include user prompts and system prompts. User prompts are the input text information, i.e., the question actually entered by the user. System prompts are text or instructions added to the user prompt, describing the task prompt for the event to be reasoned about in the user prompt. Typically, both user and system prompts together constitute the input for the LLM, guiding it to generate more accurate and targeted output. A system prompt can be a question, a description, a formatted input, or even keywords. By designing appropriate prompts, the LLM can understand the intent of the question, thereby generating a more accurate answer. For example, the system prompt includes: "The following may contain several events, each event containing the fields <'Title', 'Start Time', 'End Time', 'Repetition Period', 'Location', 'URL Link', 'Participant'>. Each event's fields are output as a single line in JSON format, without additional response.\n\n;", and the user prompt includes: "Wednesday at 3 PM, I need to go to Xinhua Bookstore to get a membership card and buy some books; I'll need a short time to choose. Sunday I need to go to Shanghai General Motors Repair Shop to work on the main engine and transmission, and be responsible for car washing and interior cleaning and maintenance; this is very important. Saturday I need to clean the house, including the refrigerator, TV, and computer; I might be busy then." Based on this, the system prompt and user prompt are concatenated and input into a pre-trained LLM for inference. An example output is: "[{'Title':'Buy some books while applying for a card','Start Time':'Wednesday 3:00pm','Location':'Xinhua Bookstore'},{'Title':'Cleaning the house','Start Time':'Saturday','Location':'Home'},{'Title':'Car wash and interior cleaning and maintenance','Start Time':'Sunday','Location':'Shanghai General Motors repair and maintenance of mainframes and transmissions'}]". It is evident that the system prompt + user prompt is the actual input to the LLM. The system prompt indicates the task to be performed, such as extracting the content of seven fields including title and start time, and returning it in JSON format. The user prompt is the string text to be extracted, i.e., the input text information. The above example output is the actual extracted result, where '[' is the first generated character.

[0047] Vertical domain: refers to industries that provide specific services to a limited group, including entertainment, medical care, environmental protection, education, sports and other industries.

[0048] Currently, various third-party social applications can be installed on electronic devices, including, but not limited to, instant messaging (IM) applications and ticketing applications. Examples of IM applications include WeChat, QQ, DingTalk, and Lark, while ticketing applications include Ctrip, Qunar, Tongcheng, and Fliggy. Users can chat and book tickets through these third-party social applications, and they can also follow official accounts (or notification accounts) of various industries to learn about the areas and events they want to follow through service notification cards published by the official accounts (or notification accounts). In some scenarios, messages from third-party social applications may involve schedule information, such as one or more messages in an instant messaging application's chat window involving the schedule of a certain event. Users often have a need to record schedule information, and in this case, users generally need to manually create schedule information in a calendar application. However, manually creating schedule information is cumbersome, resulting in low efficiency in schedule creation.

[0049] In some embodiments, to improve the efficiency of creating schedule information, electronic devices automatically extract schedule information using a pre-trained natural language model (such as LLM). See, for example... Figure 1 In implementation, the electronic device acquires text information including schedule information (i.e., the user prompt) and the system prompt. The user prompt and the system prompt are concatenated to obtain the instruction. This instruction is then input into a pre-trained LLM for inference, where the LLM extracts the schedule information. The LLM's inference process is iterative; for example, as shown... Figure 1As shown, after the Instruction is input into the LLM, the LLM performs attention calculations based on the Instruction to determine the key-value (KV) value corresponding to the Instruction. As an example, the LLM can cache the KV value corresponding to the Instruction. Then, the LLM determines the hidden layer output vector based on the KV value corresponding to the Instruction. The hidden layer output vector can be used to predict the next character. The LLM calculates the character probability value based on the hidden layer output vector and outputs a token based on the maximum character probability value. The token is the basic unit of output or processing in the LLM; a token can be a word or a character. Afterward, the electronic device appends the token output from the first time to the end of the Instruction, reconstructs the model input data, and inputs the reconstructed input data into the LLM for the second inference. Because the LLM cached the KV value corresponding to the Instruction during the first inference, during the second inference, the LLM can only calculate the KV value corresponding to the newly added token in the input data. Based on the cached KV value and the KV value corresponding to the newly added token, it determines the hidden layer output vector again and outputs the second token based on the re-determined hidden layer output vector. Similarly, the electronic device appends the token output from the nth time to the end of the input data input into the LLM for the nth time, in order to reconstruct the input data used for the (n+1)th inference. For example, the input data used for the (n+1)th inference is Instruction, N1, N2, N3...N i ...N n , where N i The result of the i-th inference is then used. Next, the LLM performs the (n+1)-th inference based on the reconstructed input data until the LLM outputs a termination symbol, ending the inference process. At this point, all output results (i.e., all inferred tokens) are identified as the schedule information to be extracted. The termination symbol is a predefined symbol used during training; its occurrence signifies the end of inference. As an example rather than a limitation, the termination symbol could be the "EOS" symbol.

[0050] In the above embodiments, the inference latency of LLM includes the sum of the inference time of the first token and the inference time of all subsequent tokens (excluding the first token). The inference time of the first token includes the inference time of the user prompt and the inference time of the system prompt. Since the system prompt is usually quite long, with an average length of 100 tokens, there is a fixed latency overhead when extracting schedule information through LLM, while the computing power and power consumption on the edge are limited. To address this, this application provides a method for obtaining schedule information, which enables electronic devices to automatically create schedule information, improving the efficiency of schedule information creation. Furthermore, this method reduces the amount of data that LLM needs to calculate in real time by pre-calculating the KV value of the system prompt, thereby reducing the inference time of LLM, reducing the latency of obtaining schedule information, and improving the speed of obtaining schedule information. The specific implementation can be found below.

[0051] As an example of this application, referring to Table 1, electronic devices can create schedule information from schedules involved in the following scenarios:

[0052] Table 1

[0053]

[0054] The aforementioned images and text include both pictures and text information. That is, the method provided in this application embodiment can automatically extract schedule information not only from text information but also from images. Specifically, the scope supported by the method provided in this application embodiment is shown in Table 2:

[0055] Table 2

[0056]

[0057] As shown in Table 2, electronic devices can extract schedule information from images. These images can be screenshots or photos taken with a camera. Screenshots can be, but are not limited to, full-screen or partial screenshots, and the windows involved can be full-screen, split-screen, or floating windows. Screenshots typically originate from mobile phones, tablets, and PCs. Photos taken with a camera can be, but are not limited to, printed text, handwritten text, and artistic fonts. Additionally, electronic devices can also extract schedule information from text. Text can be, but is not limited to, Text and Webview formats. The text format can be plain text, formatted text, or text mixed with images. The text length can be single paragraph, multi-paragraph, short text, or long text.

[0058] Furthermore, electronic devices can also extract schedule information from speech; for example, they can receive speech through a voice assistant and then convert it into text information for processing. This application's embodiments focus on illustrating the use of an image as the input.

[0059] To facilitate understanding, the application scenarios provided in the embodiments of this application will be introduced below.

[0060] In one example, a multi-turn conversation in a WeChat chat window includes a schedule intent. When a user wants to create related schedule information, they can trigger a screenshot of the application interface where the chat window is located. For example, a user can double-tap the screen in the screenshot trigger area to trigger the screenshot operation, causing the phone to take a screenshot of the WeChat chat window. See below for further details. Figure 2 In Figure (a), the phone displays a screenshot editing interface U1, which shows the instant messaging chat screenshot obtained after taking the screenshot, i.e., image p1. The screenshot editing interface U1 includes a "share" control; users can click the "share" control when they want to create calendar information in image p1. See also... Figure 2 In Figure (b), in response to a user's triggering action on the "Share" control, the phone displays a sharing floating window 10, which includes a calendar icon 11. The user can click on the calendar icon 11. In response to the user's triggering action on the calendar icon 11, the phone begins processing image p1 to extract and create relevant schedule information. For example, see [link to image p1]. Figure 2 In Figure (c), during this process, the phone can display a message saying "Offline parsing of calendar information," so that the user is aware that calendar information is currently being extracted from image p1. See also... Figure 2 In Figure (d), after the schedule information is successfully created, the mobile phone displays the schedule display interface U2. The schedule display interface U2 includes a schedule display window 12, which displays the created schedule information 13. The schedule information 13 includes the title, start time, end time, start date, location, and other content. In this way, the mobile phone achieves the purpose of automatically creating and displaying schedule information, which can avoid the need for users to manually record it and improve the efficiency of creating schedule information.

[0061] As an example of this application, the schedule display window 12 also includes multiple editing controls, allowing users to edit the schedule information 13 created on the mobile phone based on these controls as needed. See, for example... Figure 3 In Figure (a), when a user wants to modify the title of a schedule, they can click the title editing control 14 in the schedule display window 12. In response to the user's click on the title editing control 14, the phone displays the following... Figure 3The target interface U3 (which can be called the smudge interface) shown in Figure (b) displays information related to the schedule in Figure p1. Thus, the user can modify or fill in the schedule title by smudges on the information displayed on the target interface U3. Correspondingly, the phone inputs the smudged content into the title input box 15 of the target interface U3. See, for example... Figure 3 In Figure (b), when a user wants to change the title of the event to "HarmonyOS Chat," the user can sequentially draw "HarmonyOS," "Open," and "Chat" on the target interface U3. Correspondingly, the phone sequentially enters "HarmonyOS," "Open," and "Chat" in the title input box 15. See also Figure 3 In Figure (c), after the user finishes scribbling, the "input" control (or "√" control) on the target interface U3 can be triggered. In response to the user's triggering of the "input" control (or "√" control), the phone resumes displaying the calendar display window 12. At this time, the user can see in the calendar display window 12 that the calendar title has been changed to the content modified by the user through the scribbling operation; that is, the calendar title is changed from "KaiTan" to "HarmonyOS KaiTan". Thus, by displaying calendar-related and scribbable information on the target interface U3, users can quickly modify the calendar title through scribbling, improving the user experience.

[0062] In addition, see Figure 3 As shown in Figure (a), the schedule display window 12 also includes a time editing control. When a user wants to edit the time information in the schedule information 13, they can also modify it based on the time editing control. For example, a user can click on the displayed time information to edit it. In addition, after the user slides down the schedule display window 12, the schedule display window 12 can also provide other editing controls, such as repeat count, reminder time, important reminder, etc. In this way, the user can edit the schedule information based on other editing controls. This embodiment of the application does not limit this.

[0063] As an example of this application, after a user clicks the "√" control in the schedule display window 12, in response to the triggering operation, the phone displays schedule information 13 in the schedule details area of ​​the calendar application, thus allowing the user to view the schedule information 13 from the schedule details area of ​​the calendar application. As an optional example, after a user clicks the "√" control in the schedule display window 12, the phone can also display the schedule information 13 in the form of a card in locations such as the desktop, the negative one screen, or the notification center, so as to facilitate the user's quick viewing later. This application embodiment does not limit this.

[0064] It should be noted that the above example only illustrates how a user triggers the creation of calendar information on their phone via a sharing entry point (i.e., a sharing control). For another example, see... Figure 3 In Figure (a), the mobile phone provides a Magic Text Control 00 in the screenshot editing interface U1. When the user needs the mobile phone to create schedule information based on the image p1, they can click on the Magic Text Control 00 to trigger the mobile phone to create and display schedule information with one click.

[0065] In another example, a user can also trigger the creation and display of schedule information on their phone through any portal. For example, see [link to example]. Figure 4 In Figure (a), after the phone takes a screenshot of the WeChat chat window, the resulting image p1 is automatically saved to the photo library. Therefore, when the user wants the phone to automatically create the schedule information shown in image p1, they can open the screenshot image interface U4 in the photo library, where image p1 is displayed. See also... Figure 4 In diagram (b), the user can select image p1 on the phone and then drag it to the right side of the screen. When the user drags the image to a certain position, the phone displays an application that can receive and process image p1, such as... Figure 4 As shown in Figure (b), the calendar, WeChat, and QQ are displayed. The user can then drag and drop image p1 onto the calendar application and release it. In response to this release, the phone begins processing image p1 to extract and create relevant schedule information. See also... Figure 4 In Figure (c), after the mobile phone creates the schedule information, the schedule information is displayed in the schedule display window 12.

[0066] In another example, a user can select only schedule-related content in image p1, and then trigger the phone to extract schedule information from the selected area. See, for example... Figure 5 In image (a), the user can select a portion of the content in image p1. For example, the screenshot editing interface can provide a control to trigger the selection operation. Once triggered, the user can select the area on image p1. In response to the user's selection, the phone selects the selected area. As an example, the phone can take a screenshot of the selected area. Figure 5 As shown in Figure (b). Afterwards, the user can trigger the phone to extract schedule information from the selected area through interactive entry points such as the "share" control, creating and displaying schedule information related to the content in that area. Thus, by allowing users to select images p1, the amount of data processed by the phone can be reduced, thereby improving the efficiency of schedule information creation.

[0067] It should be noted that the interaction entry points used by the user to trigger the phone to extract schedule information from image p1 in the above application scenarios are merely exemplary. In some embodiments, the phone can also be triggered to extract schedule information from image p1 through other interaction entry points, such as global favorites, etc. This application embodiment does not limit this.

[0068] The application scenarios described above are merely illustrative. Furthermore, mobile phones can also extract schedule information from various types of images in other scenarios. For example, see... Figure 6 In Figure (a), a chat window contains a forwarded image p2 (which could be a screenshot or a photograph), and image p2 contains calendar information. When a user wants to extract the calendar information from image p2, they can drag image p2 to the right side of the phone screen. See [link to relevant documentation]. Figure 6 In image (b), when the user drags the image to a certain location, in response to the user's dragging action, the phone displays an application that can receive and process the image p2, such as... Figure 6 As shown in Figure (b), the calendar, WeChat, and QQ are displayed. After the user drags and drops image p2 onto the calendar application, the phone automatically processes image p2 to extract and create relevant schedule information in response to the user's release action. See also Figure 6 In Figure (c), after the mobile phone creates the schedule information, the schedule information is displayed in the schedule display window 12.

[0069] See Figure 6 As shown in Figure (c), when multiple calendar information is created on the phone, multiple calendar information can be displayed in the calendar display window 12. When the calendar display window 12 cannot display all calendar information, the user can trigger the phone to display hidden calendar information by swiping the calendar display window 12 left or right.

[0070] In addition, the phone not only supports users dragging and dropping images from chat windows to any portal, but also supports dragging and dropping text. For example, if a chat window contains chat text with calendar information, and the user needs to recreate the calendar information within that chat text, they can use the chat text and then... Figure 6 The method shown allows you to drag and drop chat text into the calendar app. The phone then extracts the schedule information from the chat text, creates and displays the relevant schedule information.

[0071] In another example, the phone can also extract schedule information from screenshots of instant messaging notification cards. See, for example... Figure 7Figure (a) shows a schematic diagram of an instant messaging notification card screenshot (i.e., image p3) according to an exemplary embodiment. Image p3 is a screenshot of a notification card published in WeChat, which includes calendar information. When a user wants the phone to create and display the calendar information in image p3, the user can trigger the phone to extract the calendar information according to the operation process described above. For example, the user can trigger the phone to extract the calendar information from image p3 through a sharing entry. Accordingly, the phone extracts the calendar information based on image p3 and then creates or displays the relevant calendar information, for example, see [link to example]. Figure 7 In Figure (b), the mobile phone creates and displays schedule information 60, which includes the theme, ticket number, departure time and departure date, location, etc.

[0072] In another example, the phone can also extract schedule information from order screenshots, where the screenshots can be screenshots of hotel bookings, train ticket orders, plane ticket orders, etc. See, for example... Figure 8 Figure (a) shows a schematic diagram of an order screenshot (i.e., image p4) according to an exemplary embodiment. Image p4 is a screenshot of the interface where the train ticket order is located, and image p4 includes schedule information. When a user wants the phone to create and display the schedule information in image p4, the user can trigger the phone according to the operation process described above. For example, the user can trigger the phone to extract schedule information from image p4 through the Magic text entry. Accordingly, the phone extracts schedule information based on image p4, and then creates or displays the relevant schedule information, such as the displayed schedule information as... Figure 8 As shown in Figure (b), 70, the schedule information 70 includes the theme, train number, departure and arrival times, departure date, location, etc.

[0073] It should be noted that the above application scenarios are all exemplary and do not limit the application scenarios of the methods provided in the embodiments of this application. In another embodiment, the mobile phone can also extract schedule information from other types of images, including images other than instant messaging chat screenshots, instant messaging notification card screenshots, and order screenshots. Of course, other types of images can be images obtained through screenshots or images taken by a camera.

[0074] As described above, schedule information may originate from various data sources, encompassing diverse business scenarios such as instant messaging (IM), orders, IM meetings, and voice assistants. These different scenarios result in varying data formats. For example, different system prompts can be set for different business scenarios, and each system prompt can include a scenario description. This description details the business context. By injecting targeted instructions into the LLM (Local Management System) through different system prompts, the LLM's ability to extract schedule information can be improved.

[0075] Examples of system prompts used in business processes involving the creation of schedule information include:

[0076] (1) "IM Chat": "The following content is a chat conversation between several people, which may contain several events. Each event contains the fields <'Start Time', 'End Time', 'Title', 'Repetition Period', 'Location', 'URL Link', and 'Participants'>. The fields for each event are output as a single line in JSON format. No additional replies are provided."

[0077] (2) "IM Meeting": "The following is a list of meeting notifications, which may contain several events. Each event includes the fields <'Start Time', 'End Time', 'Title', 'Repetition Period', 'Location', 'URL Link', and 'Participants'>. All fields for each event are output as a single line in JSON format. No additional response is required."

[0078] (3) "IM Notification": "The following content may contain several events. Each event includes the fields <'Start Time', 'End Time', 'Title', 'Repetition Period', 'Location', 'URL Link', and 'Participants'>. All fields for each event are output as a single line in JSON format. No additional response will be provided."

[0079] (4) "Voice Assistant": "The following is the content sent by the user to the voice assistant, which may contain several events. Each event contains the fields <'Start Time', 'End Time', 'Title', 'Repetition Period', 'Location', 'URL Link', and 'Participants'>. Each field of each event is output as a single line in JSON format. No additional response is provided."

[0080] (5) "Other Order Screenshots": "The following is the content of the order page, which may contain several events. Each event contains the fields <'Start Time', 'End Time', 'Title', 'Repetition Period', 'Location', 'URL Link', and 'Participants'>. Each event's fields are output as a single line in JSON format. No further response will be provided."

[0081] The business scenarios corresponding to each system prompt in (1)-(5) above are, in order, IM chat service, IM meeting service, IM notification service, voice assistant service and order screenshot service.

[0082] As an example of this application, please see Figure 9 The electronic device can offline calculate the key-value (KV) values ​​corresponding to each of multiple pre-set system prompts, with each system prompt corresponding to a specific business scenario. For example, the electronic device can input each system prompt into a pre-trained LLM (Limited Learning Model) for attention calculation to determine the KV values ​​corresponding to each system prompt. Then, the electronic device caches the KV values ​​corresponding to each system prompt. For instance, the KV values ​​for each system prompt are cached in matrix form, that is, a matrix of KV values ​​between characters in each system prompt. Thus, during application, the electronic device no longer inputs system prompts into the LLM; that is, the input is not the system prompt and the user prompt, but rather reads the required KV values ​​corresponding to the system prompts from the cached KV values ​​as part of the model input. The input consists of the user prompt and the read KV values. This eliminates the need for the LLM to perform real-time calculations of the system prompts, reducing the amount of real-time calculation required by the LLM during application and thereby improving the inference speed of the LLM.

[0083] Please continue reading Figure 9 Taking the first image as an example, where the schedule information to be extracted is to be processed, the application process mainly includes the following steps 1-7:

[0084] 1. Obtain the first image from which the schedule information to be extracted is located.

[0085] The first image can be obtained by taking a screenshot of the application interface of an electronic device, such as taking a screenshot of the chat interface of the WeChat application; or the first image can be obtained by sending it through other electronic devices.

[0086] 2. Receive the first image dragged by the user through the calendar application.

[0087] For example, see Figure 4 In one embodiment, the user can submit the first image to the calendar application by dragging and dropping.

[0088] 3. Determine the business scenario corresponding to the first image.

[0089] As an example of this application, an electronic device can determine the business scenario corresponding to the first image based on one or more of the business entry information, application information, and intent description information related to the first image.

[0090] Business entry information indicates the entry point used to perform text information reasoning. For example, when a user drags and drops the first image into a calendar application to request schedule retrieval, the calendar application information is the business entry information related to the first image.

[0091] Application information can be used to indicate the category of the application from which the first image originates; this information can be the application's name or category, etc. For example, in one possible scenario, see... Figure 2 The first image is a screenshot of the chat interface of an instant messaging application running in the foreground taken by an electronic device. In this case, the application information related to the first image can be identified as the application information of the instant messaging application.

[0092] The intent description information is used to indicate the intent of the first image. As an example, an electronic device can determine the intent description information corresponding to the first image through a pre-trained intent classification model. The pre-trained intent classification model can be obtained by training a neural network model based on training data, and this application embodiment does not limit this.

[0093] As an example of this application, the specific implementation of determining the business scenario corresponding to the first image may include: if the first image corresponding to the target text information is obtained by requesting a calendar application to extract a schedule, and if the first image is obtained by taking a screenshot of the foreground application using an electronic device, then the business scenario corresponding to the target text information is determined based on the application information of the foreground application. If the first image is not obtained by taking a screenshot using an electronic device, then the intent description information corresponding to the first image is determined by a pre-trained intent classification model, and the business scenario corresponding to the target text information is determined based on the intent description information corresponding to the first image.

[0094] In other words, if the first image is a calendar application request for schedule retrieval, the corresponding business scenario can be determined first based on the application information. If it cannot be determined based on the application information, the intent description information can be determined to identify the corresponding business scenario.

[0095] For example, see Figure 4The first image is a screenshot taken by the electronic device of the chat interface of an instant messaging application running in the foreground. The user drags the first image to the calendar application, triggering the electronic device to retrieve the schedule. In this case, the electronic device can recognize that a screenshot was taken of the instant messaging application and that a schedule retrieval request was made through the calendar application. Therefore, it can determine that the business entry information related to the first image is the calendar application information, and the application information related to the first image is "IM chat type". Consequently, it can be determined that the business scenario corresponding to the first image is "IM chat service".

[0096] For example, the first image might also be obtained by another electronic device through a screenshot and then sent to this electronic device, such as via an instant messaging application. The electronic device then requests a schedule through a calendar application. In this case, the electronic device cannot know which application the other electronic device screenshotted, i.e., it cannot determine the application information corresponding to the first image. At this time, the electronic device can determine the intent description information of the first image through a pre-trained intent classification model. For example, if the intent description information indicates that the intent of the first image is IM chat, then the business entry information related to the first image can be determined to be calendar application information, and the intent description information related to the first image can be determined to be IM chat, thereby determining the corresponding business scenario as "IM chat business". As another example, if the intent description information indicates that the intent of the first image is an order, then the corresponding business scenario can be determined to be "order screenshot business".

[0097] It should be noted that the above example uses an image as the object from which the schedule information to be extracted. In another example, if the object from which the schedule information to be extracted is text information, such as text dragged from a text file, the electronic device can determine the corresponding business scenario based on keywords or business entry information related to the text information.

[0098] Different business scenarios can correspond to different keywords, which can be pre-set according to requirements. For example, keywords include voice assistant, IM chat, and orders. For instance, in one possible scenario, if the object to be extracted for schedule information is text information, and the text information includes the keyword "create meeting schedule", the electronic device can determine that the business scenario corresponding to the text information is "IM meeting service" based on this keyword.

[0099] In one possible scenario, the text information is submitted through the voice assistant. That is, the voice assistant receives the voice, converts the voice into text information, and then requests the electronic device to retrieve the schedule from the text information. At this time, the business entry information related to the text information can be obtained as "voice assistant". Based on this business entry information, the business scenario corresponding to the text information can be determined as "voice assistant business".

[0100] It should be noted that when the object for extracting schedule information is the first image, it is necessary to subsequently determine the target text information corresponding to the first image, and then extract the schedule information based on the target text information using LLM. Therefore, the business scenario corresponding to the first image can also be referred to as the business scenario corresponding to the target text information. In this embodiment, the electronic device can use a certain strategy to select which of the following to determine the business scenario corresponding to the text information: keywords, business entry information, application information, and intent description information. For example, the decision can be based on the source of the text information. For instance, if the text information comes from an image, the business scenario can be determined based on the application information and business entry information related to the image. If the application information cannot be determined, the business scenario can be determined based on the intent description information. If the text information comes from voice, such as when schedule extraction is requested through a voice assistant, the business scenario can be directly determined based on the business entry information. Furthermore, if the text information comes from text, the business scenario can be determined based on the keywords in the text information.

[0101] 4. Determine the corresponding system prompt based on the business scenario in the first image.

[0102] As mentioned earlier, different business scenarios correspond to different system prompts. After determining the business scenario corresponding to the first image, the corresponding system prompt can be determined from multiple system prompts. The system prompt corresponding to the first image is the system prompt corresponding to the target text information.

[0103] 5. Retrieve the key-value pair corresponding to the determined system prompt from the cache.

[0104] As mentioned earlier, electronic devices cache key-value (KV) values ​​corresponding to different system prompts. Once the system prompt corresponding to the target text information is determined, the KV value corresponding to the determined system prompt can be retrieved from the cache.

[0105] As an example of this application, the electronic device can perform the operations described in steps 3-5 through the service distribution engine. Upon being triggered, the service distribution engine can respond quickly to obtain the corresponding key-value (KV) values ​​based on the business scenario. Thus, by distributing the task of determining the KV values ​​to the service distribution engine for execution, the electronic device can perform other operations in parallel, such as step 6 below, thereby improving the data processing efficiency of the electronic device.

[0106] 6. Obtain target text information based on the first image.

[0107] The electronic device acquires the text information of the schedule information to be extracted from the first image, and obtains the target text information.

[0108] In implementation, electronic devices can obtain target text information by performing content recognition and format parsing on the first image, followed by preprocessing. Preprocessing includes formatting, layout, line breaks, and line wrapping. See below for a detailed explanation of the implementation. Figure 12 The illustrated embodiment.

[0109] 7. Use the target text information and the obtained KV values ​​as input to the LLM, and extract the schedule information through the LLM.

[0110] In one example, the electronic device inputs the acquired key-value (KV) values ​​along with the target text information into the LLM (Local Management Module) to extract schedule information. That is, in this embodiment, the input data to the LLM does not include the system prompt; instead, it is replaced with the KV values ​​corresponding to the system prompt.

[0111] Thus, by changing the model input from the original user prompt + system prompt to the key-value pairs of user prompt + system prompt, the computational load of LLM can be reduced. For example, if the total length of the user prompt + system prompt is 200 tokens, with 100 tokens for the user prompt and 100 tokens for the system prompt, then LLM only needs to calculate the key-value pairs of the 100 tokens in the user prompt and the input key-value pairs. This saves the computational load of the 100 tokens in the system prompt and reduces the latency of calculating the length of 100 tokens. For example, if the duration of a single token is K, the latency can be reduced by K*L seconds, where K is the length of the system prompt. This allows LLM to quickly extract schedule information.

[0112] It should be noted that the above steps are merely illustrative, and in practice, other operations may be included, such as extracting and preprocessing the target text information, and displaying schedule information. For example, see [link to relevant documentation]. Figure 10 , Figure 10 This is a schematic diagram illustrating a method for displaying schedule information according to an exemplary embodiment. In implementation, the electronic device can pass a first image to be processed to the calendar application through interactive entry points such as sharing, "Anywhere Door," global favorites, or Magic Text. The first image can be a screenshot of a train ticket or plane ticket order, a screenshot of a hotel, restaurant, or entertainment order, a screenshot of a sports or health order, a screenshot of a WeChat mini-program or web page, a screenshot of a service notification card from an IM notification account, or a screenshot of an IM chat conversation, etc. As examples, not limitations, screenshots of train and plane ticket orders can come from applications such as 12306, China Railway, or Ctrip; screenshots of hotel, restaurant, and entertainment orders can come from applications such as Ctrip, Qunar, Tongcheng, Meituan, Fliggy, Dianping, and Damai; screenshots of sports and health orders can come from applications such as Keep, Lianduoduo, and online registration platforms; screenshots of WeChat mini programs and web applications can include content such as performances, flash sales, and marathons; screenshots of service notification cards from IM notification accounts can include notification cards issued by third parties through IM notification accounts, such as hospitals, scenic spot tickets, educational institutions, and insurance companies; and screenshots of IM chat conversations can include content such as work arrangements, invitations, and educational tasks.

[0113] After receiving the first image, the calendar app requests text recognition and edge detection. Text recognition extracts text information from the image, while edge detection identifies chat elements or color blocks within the image. See also Figure 10 Image p1 is passed to the calendar application through the interactive interface. After the calendar application requests the electronic device to perform text recognition, the text recognition result is as follows: Figure 10 As shown in 90.

[0114] Subsequently, the electronic device can filter the text recognition and edge recognition results based on filtering rules, i.e., perform preprocessing to remove irrelevant interference information from the text recognition and edge recognition results. As an example of the application, the preset filtering rules may include, but are not limited to, at least one of the following: 1. Discarding dense text blocks. 2. Discarding small characters. 3. Discarding skewed lines. 4. Discarding floating text in attached images. 5. Discarding the "back" icon in the upper left corner of the image, and the "+" symbol in the image.

[0115] It is worth noting that during the process of extracting the schedule from the first image, if the text recognition results are not filtered and are directly input into the LLM for schedule information extraction, the success rate of LLM recognition is low, usually only about 30%. This is because there are problems such as interference information, formatting line breaks, and loss of layout information. Therefore, in this embodiment, the text recognition results are filtered before extracting the schedule information through LLM, which can improve the accuracy of subsequent LLM schedule information extraction.

[0116] In addition, after receiving the first image, the calendar application requests to determine the business scenario corresponding to the first image, and determines the corresponding system prompt based on the business scenario, thereby obtaining the corresponding key-value value based on the determined system prompt.

[0117] After filtering, the electronic device concatenates the target text information based on the remaining text recognition and edge recognition results. The acquired key-value (KV) values ​​and the target text information are then input into the LLM (Local Management Module) for schedule information extraction. The LLM outputs schedule information based on the input prompts; for example, the output result is as follows: Figure 10 As shown in 91. The electronic device then performs post-processing on the schedule field of the output schedule information to obtain the schedule information to be displayed. For example, the schedule information to be displayed is as follows: Figure 10 As shown in 92. The electronic device can then display the schedule information.

[0118] In one example, post-processing of the schedule field may include, but is not limited to, at least one of the following: 1. Processing according to reminder time rules. 2. Processing according to tomorrow's start time rules. 3. Processing according to detail rules. 4. Processing based on title merging pre-filling rules.

[0119] The reminder time rules include setting the reminder time for schedule information to be earlier than the preset duration of the time information extracted by the LLM. The preset duration can be set according to needs; for example, if the preset duration is 30 minutes, and the start time of the LLM extraction is 8:30, then the reminder time for the schedule information can be set to 8:30. Additionally, the reminder time rules also include handling repeated reminders. For example, if the schedule information extracted by the LLM involves booking numbers on Monday, Tuesday, and Wednesday, then the electronic device will set reminder times for Monday, Tuesday, and Wednesday for that schedule information, instead of setting only one reminder time.

[0120] The "start time tomorrow" rule means that if the time information extracted by the LLM spans days, months, or years, then the time spanning days, months, or years should be added. For example, if the date extracted by the LLM is December 5th, the start time is 23:00, and the end time is 00:30, then the electronic device can add the end time as December 6th 00:30 in the schedule information.

[0121] Details rules refer to adjusting the layout and font size of created schedule information according to the screen size of electronic devices so that it can be displayed correctly and clearly in the schedule details area of ​​the calendar application.

[0122] The title merging and pre-filling rule includes selecting the longest title from multiple titles corresponding to the same time information as the title of the schedule information. Additionally, the rule also includes using a specified title corresponding to the scene of the image as the title of the schedule information. For example, if the scene of the image is about scheduling an appointment and picking up a number for medical treatment, and the title extracted by LLM is "Getting a Number" or "Choosing a Number," then the schedule title can be standardized to "Registering for Medical Treatment." The specified titles corresponding to different scenes can be pre-set as needed.

[0123] It should be noted that the post-processing of the schedule field described above is merely exemplary. In another example, post-processing of the schedule field may also include, but is not limited to, at least one of the following: time similarity processing, discarding empty results, risk control, date standardization, time standardization, symbol standardization, and cleaning up non-natural language titles. Time similarity processing means that if the time in the output schedule information is earlier than the current system time of the electronic device, the closest time to the time in the schedule information is determined based on the current system time, and this determined time is set as the time in the schedule information. For example, if the time in the schedule information is Tuesday, and the current system time is Wednesday, then it is recorded as Tuesday of the following week in the schedule information. Discarding empty results means discarding returned empty fields. Risk control refers to controlling sensitive words. Date standardization means representing the date in the format of year, month, and day. Time standardization means adding information such as am (morning) or pm (afternoon) to the time point. Symbol standardization means unifying symbols to a uniform style. Cleaning up non-natural language titles means that if the schedule title does not include a verb, a verb can be added to the schedule title, or the schedule title can be set to default based on the scene corresponding to the image.

[0124] The software system of the electronic device involved in the embodiments of this application can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system of the electronic device.

[0125] Figure 11 This is a block diagram of a software system for an electronic device provided in an embodiment of this application. See also... Figure 11 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime, the system layer, and the kernel layer.

[0126] The application layer can include a series of application packages, see [link to relevant documentation] Figure 11 The application package can include calendar and other applications, such as instant messaging, ticketing, camera, gallery, calling, maps, navigation, Bluetooth, music, video, SMS, voice assistant, etc.

[0127] Furthermore, as an example of this application, the application layer also includes a schedule management service and a model management service. The schedule management service can be used to provide services to the calendar, or rather, to be invoked by the calendar. For example, the schedule management service can create schedule information and store schedule information and other data for the calendar. The schedule management service can include a schedule creation service (which may be called: intelligent parsing and processing service) and a schedule database (such as: Calendar Provider schedule database). The schedule management service can create schedule information through the schedule creation service and store the created schedule information through the schedule database. The model management service (which may be called: MagicLive large model service) can be used to provide various models for the schedule management service to invoke when needed.

[0128] In one example, the model management service provides a pre-trained LLM, a target recognition model, and a personal behavior feature model. The LLM can be used to recognize (i.e., infer) prompts to determine schedule information. Exemplarily, the LLM can be a natural language processing (NLP) model, which can operate through a natural language unit (NLU). In this embodiment, the pre-trained LLM is referred to as the target natural language model. In some examples, the target natural language model can not only recognize prompts but also extract keywords from a text, such as time and location keywords. Object recognition models can be used for text recognition and edge detection of images, as well as for determining the image category. In one example, the object recognition model includes a first optical character recognition (OCR) model, a second OCR model, and an edge detection model. The first OCR model can be used for text recognition, the second OCR model can be used to determine the image category, and the edge detection model can be used for edge detection of images. There can be multiple edge detection models, and different edge detection models can be used to detect the edges of images of different categories. Personal behavior feature models can be used to determine user profiles based on users' historical behavior data.

[0129] It should be noted that the embodiments in this application are illustrated using a pre-trained LLM as the target natural language model. In another example, the size of the model and whether it is multimodal are not limited.

[0130] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 11 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0131] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java needs to call, and the other part is the core Android library itself. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0132] The system library can include multiple functional modules, such as: surface manager, media libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engines (such as SGL), etc.

[0133] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0134] Electronic devices can interact with the aforementioned modules to implement the method for displaying schedule information provided in this application embodiment. Next, we will combine... Figure 12 This section provides a detailed explanation of the process for extracting and displaying schedule information from images. See also... Figure 12 This method may include the following implementation steps:

[0135] S1201: The calendar application receives the image L to be processed.

[0136] Image L can be a bitmap format image.

[0137] The calendar application receives image L from the interaction entry point. As mentioned earlier, the interaction entry point can be a sharing entry point, a portal entry point, etc. See, for example... Figure 2 In Figure (a), when a user submits image L to the calendar application via the share control, this interaction point is the share entry point.

[0138] S1202: The calendar application sends a schedule creation instruction to the schedule management service, and the schedule creation instruction carries an image L.

[0139] The schedule creation command is used to instruct the creation and display of related schedule information based on image L.

[0140] S1203: The schedule management service determines the business scenario corresponding to image L.

[0141] S1204: The schedule management service determines the corresponding target system prompt based on the business scenario corresponding to image L.

[0142] S1205: The schedule management service retrieves the KV value corresponding to the target system prompt from the cached KV values.

[0143] The cached key-value values ​​include the key-value values ​​corresponding to each system prompt in multiple system prompts.

[0144] As an example of this application, the schedule management service can trigger the business distribution engine to execute operations S1203 to S1205. For details, please refer to [link to implementation details]. Figure 9 The example shown.

[0145] S1206: The schedule management service sends image L to the first OCR model in the model management service.

[0146] In implementation, the schedule management service calls the first OCR model in the model management service and sends image L to the first OCR model so that text recognition processing can be performed by the first OCR model.

[0147] S1207: The first OCR model determines the first recognition result of image L.

[0148] As an example of this application, the first recognition result includes the text recognition result of image L, which includes text block coordinates, text line coordinates, and text line recognition content.

[0149] As an example, the first recognition result also includes target indication information, which can be used to indicate whether the image L input to the first OCR model is a screenshot or a photograph. Exemplarily, the target indication information can be a first identifier, a second identifier, or a third identifier. The first identifier indicates that the image L input to the first OCR model is a screenshot; the second identifier indicates that the image L input to the first OCR model is a photograph, specifically a photograph of a document; and the third identifier indicates that the image L input to the first OCR model is another type of photograph, such as a photograph of an advertisement, road sign, or magazine. The first, second, and third identifiers can be set as needed, for example, the first identifier could be F1, the second identifier F2, and the third identifier F3.

[0150] That is, after the first OCR model receives image L, it performs recognition on image L and outputs the first recognition result of image L.

[0151] S1208: The first OCR model sends the first recognition result to the schedule management service.

[0152] As an example, the schedule management service can cache the first recognition result after receiving it.

[0153] S1209: The schedule management service sends image L to the second OCR model in the model management service.

[0154] In one example, after the schedule management service receives the schedule creation instruction sent from the calendar application, in addition to sending the image L to the first OCR model for text recognition processing, it can also call the second OCR model in the model management service and send the image L to the second OCR model so that the image category can be determined by the second OCR model. That is, the operation of S1206 and the operation of S1203 can be executed in parallel.

[0155] Since the method provided in this application embodiment can extract schedule information from images of different image categories, and the content layout of images of different image categories is different, electronic devices process images of different image categories differently. Therefore, in implementation, after receiving the image L to be processed, the schedule management service not only performs text recognition through the first OCR model, but also inputs image L into the second OCR model to determine the image category of image L.

[0156] As an example of this application, the image categories include instant messaging chat screenshots, instant messaging notification card screenshots, order screenshots, and other image categories. An instant messaging chat screenshot refers to an image obtained by taking a screenshot of the chat interface in an instant messaging application; an instant messaging notification card screenshot refers to an image obtained by taking a screenshot of the service notification card in an instant messaging application; an order screenshot refers to an image obtained by taking a screenshot of the order interface in the application; other image categories include images other than instant messaging chat screenshots, instant messaging notification card screenshots, and order screenshots, and other image categories can be screenshots or photographs.

[0157] S1210: The second OCR model determines the image category of image L.

[0158] After receiving image L, the second OCR model identifies image L and outputs the image category of image L.

[0159] S1211: The second OCR model sends the image category of image L to the schedule management service.

[0160] S1212: The schedule management service sends image L to the edge detection model corresponding to the image category of image L.

[0161] As an example of this application, the model management service provides multiple edge detection models, which can be pre-trained. Different edge detection models can perform edge recognition on images of different image categories. In one example, the multiple edge detection models include a first edge detection model and a second edge detection model. The first edge detection model can be used to perform edge recognition on instant messaging chat screenshots to determine the coordinates and category of chat elements in the instant messaging chat screenshots. The second edge detection model can be used to perform edge recognition on other images besides instant messaging chat screenshots. For example, the second edge detection model can be used to perform edge recognition on instant messaging notification card screenshots, order screenshots, or other types of images to determine the coordinates and category of color blocks in other images.

[0162] Different edge detection models can be obtained by iteratively training an initial training model based on image training samples of the corresponding image category. The image training samples can be obtained in advance through edge annotation as needed. The initial training model can be set as needed, as an example rather than a limitation. The initial training model can be a recurrent neural network (RNN), etc.

[0163] After receiving the image category of image L, the schedule management service determines the edge detection model corresponding to the image category of image L from multiple edge detection models. For example, when image L is... Figure 3 In Figure (a), p1, which is a screenshot of an instant messaging chat, determines the edge detection model corresponding to the image category of image L from multiple edge detection models as the first edge detection model; when image L is... Figure 8 p3 of image (a) (i.e., a screenshot of the instant messaging notification card) or is Figure 9 In the case of p4 of Figure (a) (i.e., the order screenshot), the edge detection model corresponding to the image category of image L is determined from multiple edge detection models as the second edge detection model. Then, the schedule management service calls the determined edge detection model and sends image L to that model to request edge recognition of image L.

[0164] S1213: The edge detection model performs edge recognition processing on image L and outputs the edge recognition results.

[0165] Edge recognition results include image and text element attribute information or color block attribute information.

[0166] In one example, if image L is classified as an instant messaging chat screenshot, the edge detection model corresponding to image L's image category is the first edge detection model. After image L is input into the first edge detection model for processing, the output edge recognition result includes image and text element attribute information. This image and text element attribute information includes the attribute information of chat elements in the instant messaging chat screenshot. The attribute information of chat elements can include the coordinates and category of the chat elements. For example, chat elements include avatar, title, nickname, chat content, chat timestamp, username, and specified identifiers, including the "+" identifier. See, for example... Figure 13 After inputting image L into the first edge detection model, the first edge detection model can determine that the chat elements in image L include... Figure 13 The items marked by the dashed boxes in the image L each have their own coordinates and category. For example, the coordinates of a chat element are the coordinates of the four corners of the area where the chat element is located, and the category is avatar. Optionally, the attribute information of each chat element can also include the chat element's sequence number. The sequence number of each chat element in image L can be set by default according to the sorting of the chat elements in image L.

[0167] In another example, if image L is not an instant messaging chat screenshot, but rather an instant messaging notification card screenshot, order screenshot, or other type of image, the edge detection model corresponding to image L's image category is a second edge detection model. After image L is input into the second edge detection model, the second edge detection model performs color block segmentation, and the output edge recognition result includes color block attribute information. For example, the color block attribute information includes the coordinates, category, and color block number of the color block. See, for example... Figure 14 When image L is a screenshot of an instant messaging notification card, after inputting image L into the second edge detection model, the second edge detection model can determine that the color blocks in image L include... Figure 14 The dashed boxes in the image indicate multiple items, and each color block has its own coordinates and category. For example, the coordinates of the color blocks include the coordinates of the four corners.

[0168] S1214: The edge detection model sends the edge recognition results to the schedule management service.

[0169] As an example, the schedule management service can cache the edge detection results after receiving them.

[0170] It is worth mentioning that after performing text recognition and edge recognition processing on image L through the two branches mentioned above, a first recognition result and a second recognition result can be obtained. The first recognition result includes text recognition results and target indication information, while the second recognition result includes edge recognition results and image category. Since the text recognition result can represent the text content in image L, and the edge recognition result can represent the layout of image L, the subsequent extraction of schedule information based on the first and second recognition results can improve the accuracy of information extraction. The specific implementation can be seen in the following steps.

[0171] It should be noted that there is no strict order of execution between the operations S1203 to S1205, S1206 to S1207, and S1208 to S1214. In one example, the three branches can be executed in parallel.

[0172] S1215: If the image category of image L is an instant messaging chat screenshot, the schedule management service filters the text recognition results and edge recognition results according to the first filtering rule.

[0173] Different image categories correspond to different filtering rules. In implementation, the schedule management service determines the corresponding filtering rules based on the image category of image L, and then filters the text recognition results and edge recognition results of image L according to the determined filtering rules to remove interference information that is irrelevant to the schedule.

[0174] In one example, the instant messaging chat screenshot corresponds to the first filtering rule. The first filtering rule can include filtering out the recognition data corresponding to skewed lines, small text, and chat timestamps. Skewed lines refer to text lines whose tilt angle is greater than a preset angle, which can be set according to needs, such as 10 degrees. Small text refers to text lines whose line height is less than a target line height, which can be the average line height of all text lines in image L. The chat timestamp is a timestamp used to indicate the chat time. Optionally, the first filtering rule can also include filtering out dense text blocks, floating text on the image, the "back" icon in the upper left corner, and the "+" icon in the lower right corner, etc. See, for an example... Figure 13 , Figure 13 According to an exemplary embodiment, the chat elements in image L that need to be filtered are identified, including chat timestamp 1101, small text 1102, slanted lines 1103, and "+" 1004.

[0175] As an example, in the implementation of filtering out the recognition data corresponding to skewed lines, the tilt angle of each text line can be calculated based on the text line coordinates in the text recognition result, thereby determining which text lines are skewed lines, and then deleting the recognition data corresponding to the skewed lines, such as deleting the text line coordinates and text line recognition content of the skewed lines.

[0176] As an example, in the implementation of filtering out the recognition data corresponding to small characters, the line height of each text line in image L can be determined based on the text line coordinates in the text recognition result, thereby filtering out the recognition data corresponding to text lines whose line height is less than the target line height, such as filtering out the text line coordinates and text line recognition content of text lines whose line height is less than the target line height.

[0177] As an example, in the implementation of filtering out the recognition data corresponding to chat timestamps, chat elements whose category is chat timestamp can be determined based on the edge recognition results, obtaining at least one first candidate chat element. Based on the coordinates of each first candidate chat element and the text line coordinates in the text recognition result of image L, the text line recognition content corresponding to each first candidate chat element is matched from the text recognition result of image L. If the target first candidate chat element is determined to be a chat timestamp based on the text line recognition content, it is determined whether to filter out the recognition data corresponding to the target first candidate chat element based on its coordinates. The target first candidate chat element is any one of the at least one first candidate chat element. If it is determined that the recognition data corresponding to the target first candidate chat element should be filtered out, the recognition data corresponding to the target first candidate chat element is filtered out from both the text recognition result and the edge recognition result of image L.

[0178] As mentioned earlier, the edge recognition results of instant messaging chat screenshots include the category of chat elements. Therefore, based on the category of chat elements in the edge recognition results, chat elements whose category is a chat timestamp can be filtered out, obtaining at least one first candidate chat element. Since the edge recognition results do not include text line recognition content, meaning the text content corresponding to each chat element cannot be known, in some cases, the edge detection model may misclassify chat elements whose category is not a chat timestamp as chat timestamps. Therefore, to minimize the filtering out of chat elements that are not chat timestamps, after determining at least one first candidate chat element, the text line recognition content corresponding to each first candidate chat element can be matched from the text recognition results of image L based on the coordinates of each first candidate chat element and the text line coordinates of image L. For example, for any first candidate chat element, based on the coordinates of this first candidate chat element and the text line coordinates of image L, the text line recognition content of at least one text line located within the region corresponding to this first candidate chat element can be determined from the text recognition results of image L, thus matching the text line recognition content corresponding to this first candidate chat element. Subsequently, the schedule management service can determine whether the matched text line is a timestamp using the target natural language model. For example, the schedule management service can send the matched text line to the target natural language model via `getEntity()` to request the model to identify whether the text line is a timestamp. If the target natural language model determines that the text line is a timestamp, then this first candidate chat element is likely a chat timestamp. Otherwise, if the target natural language model determines that the text line is not a timestamp, then this first candidate chat element is not a chat timestamp.

[0179] Since the position of chat timestamps in instant messaging screenshots is generally fixed, such as being centered in the chat window of WeChat, after identifying a first candidate chat element as a chat timestamp through text line content recognition using a target natural language model, we can further determine whether the first candidate chat element is indeed a chat timestamp based on its coordinates, thereby improving the accuracy of chat timestamp recognition.

[0180] As an example of this application, for any one of the at least one first candidate chat elements, the specific implementation of determining whether to filter out the recognition data corresponding to that first candidate chat element based on its coordinates can include the following two cases:

[0181] The first scenario: If the coordinates of the first candidate chat element determine that the first candidate chat element is located in the middle of image L, and the area corresponding to the first candidate chat element includes a single text line, then the recognition data corresponding to the first candidate chat element will be filtered out.

[0182] In most instant messaging applications, the chat timestamp is typically centered in the chat window and consists of only one line of text. Therefore, if a first candidate chat element is located in the center of image L, and the area corresponding to this first candidate chat element contains only a single line of text, and since the target natural language model has already determined that the text line is a timestamp, it can be determined that this first candidate chat element is indeed a chat timestamp. This means that the recognition data corresponding to this first candidate chat element can be filtered out.

[0183] The second scenario: If the coordinates of the first candidate chat element determine that the first candidate chat element is located on the right side of image L, and the area corresponding to the first candidate chat element includes a single text line, then if the recognized content of the text line corresponding to the first candidate chat element only includes time and does not include date, then the recognized data corresponding to the first candidate chat element will be filtered out.

[0184] Because in some instant messaging applications (such as the chat interface forwarded in WeChat), the chat timestamp may be displayed on the right, and the chat timestamp only includes one line of text recognition content, and only includes the time, not the date. Therefore, if it is determined that this first candidate chat element is located on the right side of image L, and the area corresponding to this first candidate chat element contains only a single line of text, it can be determined whether the text recognition content corresponding to this first candidate chat element includes the date. For example, it can be determined whether the text recognition content includes the date using a target natural language model. If it is determined that the date is not included, then this first candidate chat element can be determined to be a chat timestamp, that is, the recognition data corresponding to this first candidate chat element can be filtered out. Of course, if it is determined that the date is included, then it can be determined not to filter it out.

[0185] In one example, for the second case, it is not necessary to determine whether the text line recognition content corresponding to the first candidate chat element only includes time and not date. As long as it is determined that the first candidate chat element is located on the right side of image L and the area corresponding to the first candidate chat element includes a single text line, the schedule management service can determine to filter out the recognition data corresponding to the first candidate chat element.

[0186] If the above process determines that a first-choice chat element is a chat timestamp, the schedule management service removes the corresponding recognition data from the text recognition and edge recognition results of image L. For example, it removes the text line coordinates and text line recognition content corresponding to this first-choice chat element from the text recognition results of image L, and removes the coordinates and category corresponding to this first-choice chat element from the edge recognition results of image L. Of course, if the above process determines that a first-choice chat element is not a chat timestamp, the schedule management service does not filter out the recognition data corresponding to this first-choice chat element.

[0187] It is worth mentioning that by first determining the chat element category as a chat timestamp based on the edge recognition results, then matching the corresponding text line recognition content from the text recognition results, and using the target natural language model to determine whether it is a chat timestamp based on the matched text line recognition content, and then determining whether it is a chat timestamp based on the position of the chat element, the accuracy of chat timestamp recognition can be improved, thereby improving the accuracy of filtering, and further improving the accuracy of calendar information creation.

[0188] It should be noted that the first filtering rule described above is merely an example. When instant messaging chat screenshots originate from different instant messaging applications, the layout of chat elements is usually different, and the chat elements themselves may also differ. This can lead to variations in interfering information in instant messaging chat screenshots from different applications. For example, this typically includes several possible scenarios as shown in Table 3. Therefore, in another example, the first filtering rule may also include other rules for filtering out information unrelated to the chat content.

[0189] Table 3

[0190]

[0191] To effectively filter out interfering information from instant messaging chat screenshots from different instant messaging applications, a first filtering rule can be set according to the union of possible interfering information shown in Table 3. This ensures that interfering information can be effectively removed regardless of the type of instant messaging chat screenshot being processed. For example, the first filtering rule may also include filtering out text, usernames, and specified identifiers from avatars. In implementation, after filtering out the recognition data corresponding to the chat timestamp in image L from the text recognition results and edge recognition results of image L, the schedule management service can determine chat elements unrelated to the chat content from the remaining chat elements in the edge recognition results based on their categories, obtaining at least one second candidate chat element. Based on the coordinates of each second candidate chat element and the text line coordinates in the text recognition results of image L, the text line recognition content corresponding to each second candidate chat element is matched from the text recognition results of image L. The matched text line recognition content and corresponding text line coordinates are then deleted from the text recognition results of image L.

[0192] For example, in the implementation of filtering text from profile pictures, the schedule management service can determine the chat element whose category is profile picture based on the edge recognition result, and then match the text line recognition content corresponding to the chat element in the current recognition result of image L based on the coordinates of the chat element. If there is a matching text line recognition content, the matching text line recognition content and the text line coordinates corresponding to the matching text line recognition content are deleted from the text line recognition result of image L, thereby deleting the text from the profile picture.

[0193] For example, in the implementation of filtering out usernames, the schedule management service can determine the chat element whose category is username based on the edge recognition result, and then match the text line recognition content corresponding to the chat element in the text recognition result of image L based on the coordinates of the chat element. If there is a matching text line recognition content, the matching text line recognition content and the text line coordinates corresponding to the matching text line recognition content are deleted from the text line recognition result of image L, thereby deleting the username.

[0194] For example, in the implementation of filtering specified identifiers, the schedule management service can determine chat elements whose category is the specified identifier based on the edge recognition results. Then, it matches the text line recognition content corresponding to the chat element from the edge recognition results of image L. If the text line recognition content is the specified identifier, such as "+", then the recognition data corresponding to the chat element can be deleted from the text line recognition results of image L, such as deleting the text line coordinates and text line recognition content corresponding to the chat element. The schedule management service can also delete the recognition data corresponding to the chat element from the edge recognition results, such as deleting the coordinates and category of the chat element.

[0195] It should be noted that the above explanation uses an image L as an example of an instant messaging chat screenshot. In another example, if image L is not an instant messaging chat screenshot, but rather an instant messaging notification card screenshot or an order screenshot, the schedule management service filters out interfering information based on the second filtering rule. In one example, in the implementation of filtering based on the second filtering rule, the schedule management service can match the text line recognition content of each color block in the text recognition result of image L with the coordinates of each color block in the edge recognition result and the coordinates of the text lines in the text recognition result of image L. For any color block, if it is determined from the text line recognition content of any color block that the color block does not contain schedule-related content, such as time information and location, then the recognition data corresponding to that color block is filtered out from the edge recognition result and the text recognition result of image L. In one example, the target natural language model can be used to determine whether the color block includes time information and location. For example, the text line recognition content of each color block can be sent to the target natural language model one by one to request the target natural language model to determine whether it includes time information and location.

[0196] When image L is a screenshot of an instant messaging notification card, Table 3 shows that its potential interference includes small text, skewed lines, and a central timestamp. Therefore, in one example, before filtering based on the second filtering rule, the recognition data corresponding to skewed lines, small text, and a central timestamp in the text line recognition content and edge recognition results of image L can be filtered out first, and then filtered out based on the second filtering rule. The filtering of skewed lines, small text, and a central timestamp can be found in the first filtering rule.

[0197] As an example of this application, before filtering, it can be checked whether the number of color blocks in image L is less than a quantity threshold. If the number of color blocks in image L is less than the quantity threshold, it means that there are not many color blocks in image L. In this case, the electronic device can usually process image L, so filtering can be performed according to the second filtering rule. If the number of color blocks in image L is greater than or equal to the quantity threshold, it means that image L contains a large number of color blocks. In this case, filtering can be omitted, and instead, a prompt message can be displayed to guide the user to take a screenshot of image L again, for example, guiding the user to use the electronic device to capture a portion of image L that includes the schedule information. The quantity threshold can be set according to needs; for example, the quantity threshold can be 10.

[0198] As an example of this application, before filtering, the schedule management service can determine whether there are dense text blocks based on the text block coordinates in the text recognition results of image L. If dense text blocks exist, a prompt message can be displayed in the calendar application to inform the user of the existence of dense text blocks, so that the user can re-crop image L as needed. If no dense text blocks exist, the schedule management service performs the filtering operation.

[0199] Or in another example, during the filtering process, if the schedule management service determines the existence of dense text blocks based on the text block coordinates in the text recognition results of image L, then the text line coordinates and text line recognition content in these text blocks are deleted.

[0200] It's worth noting that without filtering, subsequent target natural language models may be unable to accurately extract schedule information, for example, for... Figure 10 Image p1 in the image might have its time extracted as a chat timestamp "20:37" and its address extracted as "No. 5, Danling Street, Haidian District, Beijing, Ningyuan Road," etc. In this embodiment, after determining the first and second recognition results, filtering out interfering information in image L can improve the accuracy of subsequent schedule information extraction.

[0201] As an example rather than a limitation, if image L is a photograph, since the photograph may be skewed or have a background, no filtering is required, and the text recognition results can be used for text concatenation later.

[0202] In another example, if image L is a photograph, the text recognition results and edge recognition results of image L can also be filtered according to the second filtering rule. This application embodiment does not limit this.

[0203] S1216: The schedule management service matches the text lines of the remaining chat elements from the filtered text recognition results based on the filtered edge recognition results.

[0204] The schedule management service uses the coordinates of each chat element in the filtered edge recognition results and the coordinates of the text lines in the filtered text recognition results to match the text line recognition content corresponding to each chat element in the filtered text recognition results.

[0205] If image L is not an instant messaging chat screenshot, the schedule management service matches the text line recognition content in each color block of the filtered edge recognition result with the coordinates of each color block in the filtered edge recognition result and the coordinates of the text line in the filtered text recognition result.

[0206] S1217: The schedule management service identifies the scene corresponding to image L based on the text lines of the remaining chat elements and the image category of image L.

[0207] The scene corresponding to image L refers to the scene involved in the text content of image L.

[0208] If we determine that image L is an instant messaging chat screenshot based on its image category, and further identify the content of the text lines of each filtered chat element to determine that it is a chat conversation, then we can determine that the scenario corresponding to image L is an IM chat scenario.

[0209] In addition, if the image category of image L is other images, such as screenshots of instant messaging notification cards or order screenshots, the schedule management service can also combine the text line recognition content of each matched color block to determine the scenario corresponding to image L, such as a service notification card scenario or a high-speed rail travel order scenario.

[0210] S1218: The schedule management service, based on the scene corresponding to image L, performs chat dialogue splicing on the text lines of each filtered chat element to obtain the target text information.

[0211] In implementation, the schedule management service processes the filtered text lines of each chat element according to the scenario corresponding to image L, performing line breaks, carriage returns, and concatenation. Since chat content is often quite flexible—for example, a complete sentence might be sent as multiple messages—if the scenario corresponding to image L is determined to be an IM chat scenario, the schedule management service can concatenate these multiple messages into a single sentence. Therefore, concatenating chat dialogues based on the scenario corresponding to image L makes the concatenated text more closely resemble natural language. As an example, and not a limitation, the target text information after chat dialogue concatenation is consistent with the content displayed on the drawing interface.

[0212] In one example, the concatenated chat conversation includes the conversation type, title, nickname, and conversation content. The nickname can be customized. For instance, the chat conversation can be concatenated in the following format:

[0213] Conversation topic: IM chat

[0214] Title: xx Group Chat

[0215] Nickname A: xxx

[0216] Nickname B: xxx

[0217] Nickname A: xxxx ......

[0219] It should be noted that S1217 to S1218 are optional operations. In another example, the schedule management service can also perform chat dialogue splicing based on the image category of image L and the text line recognition content of each filtered chat element.

[0220] In addition, if image L is another type of image, such as a screenshot of an instant messaging notification card or an order screenshot, the schedule management service will concatenate the text lines recognized in each filtered color block according to the scenario corresponding to image L. The implementation can be seen in the concatenation of chat dialogues.

[0221] S1219: The schedule management service takes the target text information and the key-value pairs corresponding to the target system prompt as input data and sends the input data to the target natural language model.

[0222] As mentioned earlier, during the image L acquisition stage, the schedule management service can determine the business scenario corresponding to image L, which means it can determine the business scenario to which the target text information belongs. It has already matched the KV corresponding to the target system prompt of the business scenario from the cached KV values. The electronic device can use the KV corresponding to the prompt and the target text information as input data, call the target natural language model, and send the input data to the target natural language model to request the target natural language model to extract the schedule information.

[0223] It should be noted that this application embodiment uses the deployment of the target natural language model in an electronic device as an example for illustration. In another example, the target natural language model can also be deployed in the cloud, and the cloud can provide an interface for the electronic device to call the target natural language model. In this way, when the target natural language model needs to be used, the schedule management service can call the target natural language model through the provided interface. This application embodiment does not limit this.

[0224] S1220: The target natural language model determines schedule information based on input data.

[0225] In one example, the target natural language model outputs schedule information such as Figure 10 As shown in 91.

[0226] S1221: The target natural language model sends schedule information to the schedule management service.

[0227] For example, the target natural language model can send schedule information to the schedule management service in JSON format.

[0228] S1222: The schedule management service performs post-processing on the schedule fields of the schedule information.

[0229] For post-processing of the schedule field in the schedule information, please refer to the previous text.

[0230] In one example, before creating the schedule information, the schedule management service can also invoke a personal behavior feature model to request a query for the user's historical behavior data, such as historical locations. Accordingly, the personal behavior feature model returns the number of historical behaviors. In this way, the schedule management service can predict where the user might go based on the historical behavior data, and then combine this with the schedule information fed back by the target natural language model to create the final schedule information, such as adding the predicted address information to the schedule.

[0231] S1223: The schedule management service displays the processed schedule information in the calendar application.

[0232] For example, if image L is a screenshot of an instant messaging chat, the electronic device displays the following calendar information: Figure 2 As shown in Figure (d) 13.

[0233] In one example, a confirmation request notification can be displayed before the schedule information is shown. After receiving a confirmation instruction from the user triggered by the confirmation request notification, the schedule management service then displays the schedule information in the calendar application.

[0234] As an example of this application, the electronic device also supports users in editing the displayed schedule information. For example, it supports users in modifying the title of the schedule information, such as see [link to relevant documentation]. Figure 3 The illustrated embodiment. During this process, when a user triggers the display of the target interface on the electronic device, it is necessary to display scribbable text on the target interface. To this end, the schedule management service can also request the target natural language model to perform word combination processing on the concatenated text, for example, merging the words "I" and "we" into "we," so that the text can be displayed on the target interface according to the merged words, thus facilitating user scribbling. In implementation, the schedule management service can call the target natural language model via `getWordSegment()` to request the target natural language model to perform word combination processing. Additionally, the schedule management service can also call the target natural language model via `getWordSegment()` to request the target natural language model to determine the topic of the concatenated text, such as specifying that the target natural language model extract topic entities (e.g., meeting, dinner). In this way, the schedule management service can display this topic on the target interface.

[0235] In this embodiment, in response to a schedule extraction operation, target text information to be extracted is obtained. The system prompt corresponding to the business scenario to which the target text information belongs is determined from multiple system prompts, thus obtaining the target system prompt. The target system prompt describes the task for which the time to be inferred in the target text information. The key-value (KV) value of the target system prompt is obtained from the cached KV values ​​of multiple system prompts. The target text information and the target system prompt's KV value are used as input to a pre-trained target natural language model. The target natural language model performs inference and outputs the schedule information corresponding to the target text information. This eliminates the need for users to manually input schedule information item by item in the calendar application, improving the efficiency of schedule information creation. Furthermore, since the electronic device can read the required KV values ​​from the cached KV values ​​as part of the model input during application, the target natural language model no longer needs to perform real-time calculations on the system prompt, reducing the amount of content that the target natural language model needs to calculate in real-time during application, thereby improving the inference speed of the target natural language model.

[0236] It should be noted that the embodiments of this application are illustrated using the extraction of schedule information as an example. In another example, the method provided by the embodiments of this application can also be applied to other scenarios of extracting events, such as answering questions.

[0237] The electronic devices involved in the embodiments of this application may be mobile phones, action cameras (GoPro), digital cameras, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, in-vehicle devices, ultra-mobile personal computers (UMPCs), netbooks, etc., and the embodiments of this application do not limit them.

[0238] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 15The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0239] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0240] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0241] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0242] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0243] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0244] The charging management module 140 receives charging input from a charger, which can be either a wireless or wired charger. The power management module 141 connects to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0245] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices through wireless communication technology.

[0246] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0247] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is an integer greater than 1.

[0248] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0249] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0250] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created by electronic device 100 during use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0251] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D and application processor.

[0252] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch display." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0253] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0254] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for obtaining schedule information, characterized in that, When applied to electronic devices, the method includes: In response to the schedule extraction operation, target text information is obtained, including the schedule information to be extracted; Based on one or more of the keywords, business entry information, application information, and intent description information related to the target text information, the business scenario corresponding to the target text information is determined. Different keywords are used to indicate different business scenarios, the business entry information is used to indicate the entry point used to request text information reasoning, the application information is used to indicate the category of the application from which the target text information comes, and the intent description information is used to indicate the intent of the target text information. From multiple system prompts, obtain the system prompt corresponding to the determined business scenario to obtain the target system prompt. The system prompt is used to describe the task of the reasoning event. Each of the multiple system prompts corresponds to a business scenario. Obtain the key-value value of the target system prompt from the key-value values ​​of multiple cached system prompts; The target text information and the key-value values ​​of the target system prompt are used as inputs to a pre-trained target natural language model. The target natural language model performs inference and outputs the schedule information corresponding to the target text information.

2. The method as described in claim 1, characterized in that, The target text information originates from an image; determining the business scenario corresponding to the target text information based on one or more of the following: keywords, business entry information, application information, and intent description information related to the target text information, includes: If the first image corresponding to the target text information is extracted through a calendar application request, and if the first image is obtained by taking a screenshot of the foreground application through the electronic device, then the business scenario corresponding to the target text information is determined based on the application information of the foreground application. If the first image is not obtained by taking a screenshot from the electronic device, the intent description information corresponding to the first image is determined by a pre-trained intent classification model, and the business scenario corresponding to the target text information is determined based on the intent description information corresponding to the first image.

3. The method as described in claim 1, characterized in that, Before obtaining the key-value value of the target system prompt from the key-value values ​​of multiple cached system prompts, the method further includes: For each of the multiple system prompts, each system prompt is input into the target natural language model for processing to obtain the KV value of each system prompt; The key-value (KV) values ​​of each system prompt among the multiple system prompts are cached.

4. The method according to any one of claims 1-3, characterized in that, The response to the schedule retrieval operation, obtaining the target text information, includes: In response to the schedule extraction operation on the first image, a first recognition result and a second recognition result are determined by the target recognition model. The first recognition result includes the text recognition result of the first image, and the second recognition result includes the edge recognition result and image category of the first image. The edge recognition result includes the graphic element attribute information or color block attribute information of the first image. The target recognition model can determine the text recognition result, edge recognition result and image category of any image. Based on the image category of the first image, filter out interference information unrelated to the schedule from the text recognition results and edge recognition results of the first image; The target text information is obtained based on the filtered first recognition result and the filtered second recognition result.

5. The method as described in claim 4, characterized in that, The image categories include instant messaging chat screenshots, instant messaging notification card screenshots, order screenshots, and other image categories. Instant messaging chat screenshots refer to images obtained by taking a screenshot of the chat interface in an instant messaging application. Instant messaging notification card screenshots refer to images obtained by taking a screenshot of the service notification card in an instant messaging application. Order screenshots refer to images obtained by taking a screenshot of the order interface in the application. Other image categories include images other than those mentioned above. Wherein, if the first image is a screenshot of an instant messaging chat, the edge recognition result includes the image and text element attribute information; if the first image is one of the instant messaging notification card screenshot, the order screenshot, or other types of images, the edge recognition result includes the color block attribute information.

6. The method as described in claim 5, characterized in that, The text recognition result includes text line coordinates; The step of filtering out irrelevant interference information from the text recognition result and edge recognition result of the first image based on the image category includes: If the first image is determined to be an instant messaging chat screenshot based on the image category of the first image, the recognition data corresponding to the skewed text lines in the text recognition result of the first image is filtered out based on the text line coordinates of each text line in the text recognition result of the first image. Based on the text line coordinates of each text line in the text recognition result of the first image, filter out the recognition data corresponding to the text lines whose line height is less than the target line height in the text recognition result of the first image and the edge recognition result. The target line height is the average line height of all text lines in the first image. The recognition data corresponding to the chat timestamp in the first image is filtered out from the text recognition result and the edge recognition result of the first image.

7. The method as described in claim 6, characterized in that, The image and text element attribute information in the edge recognition result includes the coordinates and category of the chat element, and the text recognition result also includes the text line recognition content; The step of filtering out the recognition data corresponding to the chat timestamp in the first image from the text recognition result and the edge recognition result of the first image includes: From the edge recognition results, determine the chat element whose category is chat timestamp, and obtain at least one first candidate chat element; Based on the coordinates of each first candidate chat element in the at least one first candidate chat element and the coordinates of the text line in the text recognition result of the first image, the text line recognition content corresponding to each first candidate chat element is matched from the text recognition result of the first image. If the target first candidate chat element is determined to be a chat timestamp based on the text line recognition content corresponding to the target first candidate chat element, it is determined whether to filter out the recognition data corresponding to the target first candidate chat element according to the coordinates of the target first candidate chat element. The target first candidate chat element is any one of the at least one first candidate chat element. If it is determined that the recognition data corresponding to the first candidate chat element of the target should be filtered out, the recognition data corresponding to the first candidate chat element of the target should be filtered out from the text recognition result of the first image and the edge recognition result.

8. The method as described in claim 7, characterized in that, The step of determining whether to filter out the recognition data corresponding to the first candidate chat element based on its coordinates includes: If, based on the coordinates of the target first candidate chat element, it is determined that the target first candidate chat element is located in the middle of the first image, and the area corresponding to the target first candidate chat element includes a single line of text, then it is determined that the recognition data corresponding to the target first candidate chat element will be filtered out; or, If the target first candidate chat element is located on the right side of the first image based on its coordinates, and the area corresponding to the target first candidate chat element includes a single text line, then if the text line corresponding to the target first candidate chat element only includes a time point and does not include a date, then the identification data corresponding to the target first candidate chat element is determined to be filtered out.

9. The method as described in claim 5, characterized in that, The text recognition result includes text line coordinates and text line recognition content; The step of filtering out irrelevant interference information from the text recognition result and edge recognition result of the first image based on the image category includes: If the first image is determined to be either a screenshot of the instant messaging notification card or a screenshot of the order based on the category of the first image, the text line recognition content in each color block is matched from the text recognition result of the first image based on the coordinates of each color block in the edge recognition result and the coordinates of the text line in the text recognition result of the first image. For any one of the color blocks, if it is determined from the text line recognition content in the color block that the color block does not contain content related to the schedule, then the recognition data corresponding to the color block is filtered out from the edge recognition result and the text recognition result of the first image.

10. The method according to any one of claims 5-9, characterized in that, The text recognition result includes text line coordinates and text line recognition content; The process of obtaining the target text information based on the filtered first recognition result and the filtered second recognition result includes: Based on the coordinates of each object in the filtered edge recognition results and the coordinates of the text lines in the text recognition results of the first image, the text line recognition content corresponding to each object is matched from the filtered text recognition results of the first image, where the object is a chat element or a color block. Based on the text line recognition content corresponding to each object and the image category of the first image, the business scenario corresponding to the first image is determined; Based on the business scenario corresponding to the first image, the text lines corresponding to each object are concatenated to obtain the target text information.

11. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-10.

Images