Methods for displaying schedule information, electronic devices, and readable storage media

Abstract

This application discloses a method, electronic device, and readable storage medium for displaying schedule information, belonging to the field of terminal technology. It includes: responding to a schedule extraction operation, acquiring target text information, the target text information including schedule information to be extracted; extracting schedule information from the target text information using a pre-trained target natural language model to obtain first schedule information; the target natural language model is capable of extracting schedule information from the text information and merging and representing time information from the text information; splitting the time information in the first schedule information to obtain second schedule information, the second schedule information including the split time information and other schedule element information from the first schedule information excluding time information; and displaying the second schedule information. This application can accurately and quickly create and display schedule information.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a method for displaying schedule information, an electronic device, and a readable storage medium. Background Technology

[0002] With the rapid development of terminal technology, electronic devices can typically install various types of third-party social applications, such as instant messaging and ticketing apps. Some messages or service notifications in these third-party social applications often involve schedule information, such as a chat window in an instant messaging app displaying the schedule of a meeting. In some scenarios, users often need to record and display schedule information from third-party social applications on their electronic devices. Therefore, how to enable electronic devices to display schedule information quickly and accurately has become a hot research topic. Summary of the Invention

[0003] This application provides a method, electronic device, and readable storage medium for displaying schedule information, which can solve the problem of how to quickly and accurately display schedule information in related technologies. The technical solution is as follows:

[0004] In a first aspect, a method for displaying schedule information is provided, the method comprising:

[0005] In response to the schedule extraction operation, target text information is acquired, which includes the schedule information to be extracted. A pre-trained target natural language model is used to extract the schedule information from the target text, obtaining first schedule information. This target natural language model can extract the schedule information from the text and combine the time information within the text. The time information in the first schedule information is then split to obtain second schedule information. The second schedule information includes the split time information and other schedule elements from the first schedule information besides the time information. The second schedule information is then displayed.

[0006] In this way, by expressing the same amount of information with fewer fields, the target natural language model needs to focus on fewer fields when extracting schedule information, thereby reducing the computational difficulty of the target natural language model and improving the accuracy of the output results. Furthermore, because the length of the schedule information to be output is reduced, the extraction latency of schedule information can also be decreased.

[0007] As an example of this application, the first schedule information includes field values ​​corresponding to at least one of a plurality of fields, including a title field, a start time field, an end time field, a location field, and a repetition period field. The field value corresponding to the title field is the title. The field value corresponding to the start time field includes one or more of the following: start date, repetition start date, and start time. The field value corresponding to the end time field includes one or more of the following: end date, repetition end date, and end time. The field value corresponding to the repetition period field is the schedule repetition period.

[0008] Thus, by merging the start date, repeat start date, and start time into the start time field, and merging the end date, repeat end date, and end time into the end time field, that is, by combining time information into two fields, the target natural language model's focus on fields is reduced.

[0009] As an example of this application, in the case where the target text information includes a start time and duration description information, but excludes an end time, the field value corresponding to the end time field also includes duration description information. The end time in the first schedule information is represented by the start time and duration description information in the target text information, and the duration description information is used to describe the duration of the schedule. This facilitates the target natural language model in reasoning about the end time in this scenario, improving the intelligence of time reasoning.

[0010] As an example of this application, the end time field of the first schedule information includes a duration description, which describes the duration of the schedule. In this case, the specific implementation of splitting the time information in the first schedule information to obtain the second schedule information includes: splitting the start date from the start time field of the first schedule information as the field value of the first date field; splitting the end date from the end time field of the first schedule information as the field value of the second date field; both the first and second date fields are newly added fields and are different; and determining the field value of the split end time field based on the start time and duration description in the end time field of the first schedule information.

[0011] In this way, by splitting the time information in the first schedule information and representing it through multiple fields, the schedule information can be displayed from multiple dimensions, thus improving the user experience.

[0012] As an example of this application, when the first schedule information does not include periodic schedule information, the first date field is the start date field and the second date field is the end date field; when the first schedule information includes periodic schedule information, the first date field is the repeat start date field and the second date field is the repeat end date field. In this way, the schedule can be reasonably decomposed based on whether the first schedule information includes periodic schedules, improving the accuracy of the decomposition.

[0013] As an example of this application, the specific implementation of determining the field value corresponding to the split end time field based on the start time and duration description information in the field value corresponding to the end time field of the first schedule information includes: adding the start time in the field value corresponding to the end time field to the duration described in the duration description information to determine the field value corresponding to the split end time field. In this way, the end time is determined through calculation, making the subsequent display of schedule information more intuitive.

[0014] As an example of this application, the target natural language model is obtained by training an initial natural language understanding model based on a sample training set. The sample training set includes multiple sets of sample training data. Each set of sample training data includes text training samples and corresponding schedule information samples. The time information in the schedule information samples has been merged. In this way, the target natural language model does not need to focus on a large number of fields during application, and the output length of the schedule information can be reduced, thereby improving the accuracy of inference and reducing the latency of inference.

[0015] As an example of this application, in response to the schedule extraction operation on the first image, a first recognition result and a second recognition result are determined through a target recognition model. The first recognition result includes the text recognition result of the first image, and the second recognition result includes the edge recognition result and image category of the first image. The edge recognition result includes the image element attribute information or color block attribute information of the first image. Based on the image category of the first image, interference information unrelated to the schedule in the text recognition result and edge recognition result of the first image is filtered out. Based on the filtered first recognition result and filtered second recognition result, the target text information is obtained. In this way, users do not need to manually create schedule information in the calendar application, improving the efficiency of schedule information creation.

[0016] As an example of this application, the image categories include instant messaging chat screenshots, instant messaging notification card screenshots, order screenshots, and other categories of images. Instant messaging chat screenshots refer to images obtained by taking a screenshot of the chat interface in an instant messaging application. Instant messaging notification card screenshots refer to images obtained by taking a screenshot of the service notification card in an instant messaging application. Order screenshots refer to images obtained by taking a screenshot of the order interface in the application. Other categories of images include images other than instant messaging chat screenshots, instant messaging notification card screenshots, and order screenshots.

[0017] Specifically, when the first image is a screenshot of an instant messaging chat, the edge recognition result includes image and text element attribute information; when the first image is a screenshot of an instant messaging notification card, an order screenshot, or one of other types of images, the edge recognition result includes color block attribute information.

[0018] Therefore, by categorizing images into the aforementioned categories, targeted image processing methods can be adopted based on image category during the subsequent extraction of schedule information, thereby improving the effectiveness and accuracy of schedule information extraction.

[0019] As an example of this application, the text recognition result includes text line coordinates. Accordingly, based on the image category of the first image, the specific implementation of filtering out schedule-irrelevant interference information from the text recognition result and edge recognition result of the first image may include: If the first image is an instant messaging chat screenshot, filtering out the recognition data corresponding to skewed text lines in the text recognition result of the first image based on the text line coordinates of each text line in the text recognition result of the first image; filtering out the recognition data corresponding to text lines whose line height is less than a target line height in the text recognition result and edge recognition result of the first image based on the text line coordinates of each text line in the text recognition result of the first image, where the target line height is the average line height of all text lines in the first image; and filtering out the recognition data corresponding to the chat timestamp in the first image from the text recognition result and edge recognition result of the first image. Thus, by analyzing the recognition data corresponding to skewed lines, small characters, and chat timestamps in the first image, schedule-irrelevant interference information can be effectively removed, i.e., information that may interfere with schedule information extraction, thereby improving the accuracy of subsequent schedule extraction.

[0020] As an example of this application, the image and text element attribute information in the edge recognition result includes the coordinates and category of the chat element, and the text recognition result also includes the text line recognition content. Accordingly, the specific implementation of filtering the recognition data corresponding to the chat timestamp in the first image from the text recognition result and the edge recognition result of the first image may include: determining chat elements whose category is chat timestamp from the edge recognition result, and obtaining at least one first candidate chat element. Based on the coordinates of each first candidate chat element among the at least one first candidate chat element and the text line coordinates in the text recognition result of the first image, matching the text line recognition content corresponding to each first candidate chat element from the text recognition result of the first image. If it is determined that the target first candidate chat element is a chat timestamp based on the text line recognition content corresponding to the target first candidate chat element, it is determined whether to filter the recognition data corresponding to the target first candidate chat element based on the coordinates of the target first candidate chat element, where the target first candidate chat element is any one of the at least one first candidate chat element. If it is determined that the recognition data corresponding to the target first candidate chat element should be filtered, the recognition data corresponding to the target first candidate chat element is filtered from both the text recognition result and the edge recognition result of the first image.

[0021] In this way, the chat elements that are classified as chat timestamps are first identified based on the edge recognition results. Then, the corresponding text lines are matched from the text recognition results. Based on the matched text lines, the target natural language model determines whether it is a chat timestamp. Finally, the position of the chat element is used to determine whether it is a chat timestamp. This can improve the accuracy of chat timestamp recognition, thereby improving the accuracy of filtering and, in turn, improving the accuracy of calendar information creation.

[0022] As an example of this application, the specific implementation of determining whether to filter out the recognition data corresponding to the target first candidate chat element based on its coordinates may include: if the target first candidate chat element is located in the middle of the first image based on its coordinates, and the area corresponding to the target first candidate chat element includes a single text line, then the recognition data corresponding to the target first candidate chat element is determined to be filtered out. Alternatively, if the target first candidate chat element is located on the right side of the first image based on its coordinates, and the area corresponding to the target first candidate chat element includes a single text line, then if the recognition content of the text line corresponding to the target first candidate chat element only includes a time point and not a date, then the recognition data corresponding to the target first candidate chat element is determined to be filtered out. Thus, determining whether the first candidate chat element is a chat timestamp based on the positional features and text line features can improve the accuracy of the determination.

[0023] As an example of this application, the text recognition result includes text line coordinates and text line recognition content. Based on the image category of the first image, the specific implementation of filtering out interference information unrelated to the schedule from the text recognition result and edge recognition result of the first image may include: if the first image is a screenshot of an instant messaging notification card or an order screenshot, matching the text line recognition content of each color block in the text recognition result of the first image with the coordinates of each color block in the edge recognition result and the text line coordinates in the text recognition result of the first image. For any color block, if it is determined from the text line recognition content of any color block that the color block does not contain schedule-related content, then the recognition data corresponding to that color block is filtered out from the edge recognition result and the text recognition result of the first image. Thus, by filtering out color blocks that do not contain schedule information, subsequent processing can focus only on color blocks containing schedule-related content, improving data processing efficiency and the accuracy of schedule information extraction.

[0024] As an example of this application, the text recognition result includes text line coordinates and text line recognition content. Based on the filtered first and second recognition results, the specific implementation of obtaining the target text information may include: matching the text line recognition content corresponding to each object from the filtered first image's text recognition results, where the object is a chat element or a color block, based on the coordinates of each object in the filtered edge recognition results and the text line coordinates in the text recognition results of the first image. The scene corresponding to the first image is determined according to the text line recognition content corresponding to each object and the image category of the first image. Based on the scene corresponding to the first image, text concatenation is performed on the text line recognition content corresponding to each object to obtain the target text information. Thus, by determining the scene corresponding to the first image and performing text concatenation based on the scene, the final target text information can be made to conform as closely as possible to the rules of natural language.

[0025] As an example of this application, after displaying the schedule information, in response to an editing operation on the title of the schedule information, a target interface is displayed, which includes concatenated text. In response to a selection operation on the text line displayed on the target interface, the selected text is entered into the title input box. In response to an end-of-edit operation, the title is modified to the content entered in the title input box. Thus, by displaying the target interface, users can quickly modify the title by scribbling, improving the user experience.

[0026] As an example of this application, the specific implementation of extracting schedule information from target text information using a pre-trained target natural language model to obtain the first schedule information may include: constructing a prompt based on the target text information and the scene corresponding to the first image, the prompt including scene description information used to describe the scene corresponding to the first image, the scene corresponding to the first image being determined based on the first recognition result and the second recognition result. The prompt is then input into the target natural language model for processing to obtain the first schedule information. Thus, by adding the scene description information of the first image to the prompt, the target word language model can more accurately infer the schedule information.

[0027] In a second aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method for displaying schedule information as described in the first aspect above.

[0028] Thirdly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method for displaying schedule information as described in the first aspect.

[0029] Fourthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the method for displaying schedule information described in the first aspect.

[0030] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a frame for displaying schedule information according to an exemplary embodiment;

[0032] Figure 2 This is a schematic diagram illustrating a schedule information sample corresponding to a text training sample according to an exemplary embodiment;

[0033] Figures 3A-3D This is a schematic diagram illustrating an application scenario according to an exemplary embodiment;

[0034] Figure 4 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0035] Figure 5 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0036] Figure 6This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0037] Figure 7 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0038] Figure 8 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0039] Figure 9 This is a schematic diagram illustrating an application scenario according to another exemplary embodiment;

[0040] Figure 10 This is a schematic diagram of a frame for displaying schedule information according to another exemplary embodiment;

[0041] Figure 11 This is a schematic diagram of a frame for displaying schedule information according to another exemplary embodiment;

[0042] Figure 12 This is a schematic diagram illustrating a schedule information sample corresponding to a text training sample according to another exemplary embodiment;

[0043] Figure 13 This is a schematic diagram illustrating a schedule information sample corresponding to a text training sample according to another exemplary embodiment;

[0044] Figure 14 This is a schematic diagram illustrating a time information splitting according to an exemplary embodiment;

[0045] Figure 15 This is a schematic diagram illustrating a time information splitting according to another exemplary embodiment;

[0046] Figure 16 This is a schematic diagram illustrating a software system of an electronic device according to an exemplary embodiment;

[0047] Figure 17 This is a schematic diagram illustrating an implementation framework for creating schedule information according to another exemplary embodiment;

[0048] Figure 18 This is a flowchart illustrating a method for displaying schedule information according to an exemplary embodiment;

[0049] Figure 19 This is a schematic diagram illustrating the processing of an instant messaging chat screenshot according to an exemplary embodiment;

[0050] Figure 20 This is a schematic diagram illustrating the processing of an instant messaging notification card screenshot according to an exemplary embodiment;

[0051] Figure 21 This is a schematic diagram of the architecture of an electronic device according to an exemplary embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0053] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0054] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0055] Before introducing the method for displaying schedule information provided in the embodiments of this application, a brief explanation of the terms or nouns involved in the embodiments of this application will be given first.

[0056] One-stop office software offers a variety of office functions, such as document processing, spreadsheets, presentation creation, project management, calendar, and email. Users can complete multiple office tasks within a single software, improving work efficiency. One-stop office software typically employs a unified user interface design, making switching and using various functional modules more convenient and consistent.

[0057] Key-value (KV) format: This refers to presenting information in a tabular format (with or without tables). For example, if the presented information includes two columns, the left column is the topic and the right column is the specific content corresponding to the topic.

[0058] Global Favorites: This refers to the ability for users to save information on their electronic devices by swiping down the screen with three fingers.

[0059] Magic Text: This is a feature that quickly extracts text from images. Users can typically enable or disable this feature by going to "Settings > Smart Assistant > Magic Text".

[0060] Prompt: In this embodiment, a prompt refers to the model input data formed by adding a piece of text or instructions to the input text information when using a machine learning model. This includes user prompts and system prompts. The user prompt is the input text information, and the system prompt is the added text or instructions. This guides the machine learning model to generate more accurate and targeted output. A system prompt can be a question, a description, a formatted input, or even keywords. By designing a well-structured prompt, the machine learning model can understand the intent of the question, thereby generating a more accurate answer. In summary, prompts are a widely used technique in machine learning models. They can help users address data bias, improve the controllability of machine learning models, and enhance problem modeling capabilities.

[0061] Vertical domain: refers to industries that provide specific services to a limited group, including entertainment, medical care, environmental protection, education, sports and other industries.

[0062] Currently, various third-party social applications can be installed on electronic devices, including, but not limited to, instant messaging (IM) applications and ticketing applications. Examples of IM applications include WeChat, QQ, DingTalk, and Lark, while ticketing applications include Ctrip, Qunar, Tongcheng, and Fliggy. Users can chat and book tickets through these third-party social applications, and they can also follow official accounts (or notification accounts) of various industries to learn about the areas and events they want to follow through service notification cards published by the official accounts (or notification accounts). In some scenarios, messages from third-party social applications may involve schedule information, such as one or more messages in an instant messaging application's chat window involving the schedule of a certain event. Users often have a need to record schedule information, and in this case, users generally need to manually create schedule information in a calendar application. However, manually creating schedule information is cumbersome, resulting in low efficiency in schedule creation.

[0063] In some embodiments, to improve the efficiency of creating schedule information, see [link to relevant documentation]. Figure 1 Electronic devices acquire text information including schedule details, perform preprocessing on the text, and then input the processed text into a pre-trained model for inference to extract schedule information. The extracted schedule information is then post-processed, including format conversion, before being displayed. The schedule information that users are interested in often involves multiple fields, such as... Figure 1 As shown, common fields involved in schedule information include, but are not limited to, title, start date, end date, start time, end time, location, repetition period, repetition start date, and repetition end date. Specifically, the title field corresponds to the title; the start date field corresponds to the start date; the end date field corresponds to the end date; the start time field corresponds to the start time; the end time field corresponds to the end time; the location field corresponds to the location; the repetition period field corresponds to the schedule repetition period; the repetition start date field corresponds to the repetition start date; and the repetition end date field corresponds to the repetition end date. Therefore, during the model training phase, it is necessary to construct schedule information samples corresponding to the text training samples according to the field formats of the above fields to obtain sample training data. For example, see... Figure 2 In Figure (a), the content of a certain text training sample is "Meetings will be held every Tuesday from October 1st to October 20th at 8:00 AM for approximately two hours in room 501". In this case, when constructing the corresponding schedule information sample, the title field is set to "Meeting", the start date field is empty, the end date field is empty, the start time field is 8:00 AM, the end time field is 10:00 AM, the location field is 501, the repetition period field is every Tuesday, the repetition start date field is October 1st, and the repetition end date field is October 20th. See also... Figure 2In Figure (b), the content of a certain text training sample is "10.18:00-10.20 10:00 Go to Xiamen for vacation". In this case, when constructing the schedule information sample corresponding to this text training sample, the field value corresponding to the title field is set to vacation, the field value corresponding to the start date field is set to October 1st, the field value corresponding to the end date field is set to October 20th, the field value corresponding to the start time field is set to 8:00, the field value corresponding to the end time field is set to 10:00, the field value corresponding to the location field is set to Xiamen, the field value corresponding to the repetition period field is empty, the field value corresponding to the repetition start date field is empty, and the field value corresponding to the repetition end date field is empty. Following this method, a large number of text training samples are obtained, and a schedule information sample corresponding to each text training sample is constructed. Each text training sample and its corresponding schedule information sample are used as a set of sample training data to obtain multiple sets of sample training data. After training the initial model (such as a natural language understanding model) using these multiple sets of sample training data, a trained model can be obtained. Therefore, when extracting schedules using a trained model, the model needs to focus on fields including title, start date, end date, start time, end time, location, repetition cycle, repetition start date, and repetition end date. This large number of fields increases the computational complexity and may lead to more errors in the output. Furthermore, due to the large number of fields, the model needs to output schedule information according to the format of these fields. Since the extraction process is iterative, outputting one token (e.g., a character or word) at a time, the length of the output schedule information is considerable, further increasing the extraction time. Additionally, some scenarios require time information inference, which is often difficult for smaller models to handle.

[0064] To address this, this application provides a method for displaying schedule information, enabling electronic devices to automatically create schedule information and improving the efficiency of schedule information creation. Furthermore, this application increases information density and combines the Chain-of-Thought (COT) method to express the same amount of information using fewer fields, thereby reducing the time required to extract schedule information, lowering the computational difficulty of the model, and improving the accuracy of the model's output results, ensuring that the field values ​​corresponding to all fields of the schedule information are correct as much as possible.

[0065] As an example of this application, referring to Table 1, electronic devices can create schedule information from schedules involved in the following scenarios:

[0066] Table 1

[0067]

[0068] The aforementioned images and text include both pictures and text information. That is, the method provided in this application embodiment can automatically extract schedule information not only from text information but also from images. Specifically, the scope supported by the method provided in this application embodiment is shown in Table 2:

[0069] Table 2

[0070]

[0071] As shown in Table 2, electronic devices can extract schedule information from images. These images can be screenshots or photos taken with a camera. Screenshots can be, but are not limited to, full-screen or region screenshots, and the windows involved can be full-screen, split-screen, or floating windows. Screenshots typically originate from mobile phones, tablets, and PCs. Photos taken with a camera can be, but are not limited to, printed text, handwritten text, and artistic fonts. Furthermore, electronic devices can also extract schedule information from text. Text formats include, but are not limited to, Text and Webview. The text format can be plain text, formatted text, or text mixed with images. The text length can be single-segment, multi-segment, short, or long. In addition, electronic devices can extract schedule information from speech, for example, by receiving speech through a voice assistant and then converting it into text information for processing. This application embodiment focuses on illustrating the use of images as input.

[0072] To facilitate understanding, the application scenarios provided in the embodiments of this application will be introduced below.

[0073] In one example, a multi-turn conversation in a WeChat chat window includes a schedule intent. When a user wants to create related schedule information, they can trigger a screenshot of the application interface where the chat window is located. For example, a user can double-tap the screen in the screenshot trigger area to trigger the screenshot operation, causing the phone to take a screenshot of the WeChat chat window. See below for further details. Figure 3A The phone displays a screenshot editing interface U1, which shows the instant messaging chat screenshot obtained after taking the screenshot, i.e., image p1. The screenshot editing interface U1 includes a "share" control; users can click the "share" control when they want to create calendar information in image p1. See also... Figure 3B In response to the user's triggering action on the "Share" control, the phone displays a sharing floating window 10, which includes a calendar icon 11. The user can click on the calendar icon 11. In response to the user's triggering action on the calendar icon 11, the phone begins processing the image p1 to extract and create relevant schedule information. For example, see [link to example]. Figure 3CDuring this process, the phone can display a message saying "Offline parsing of calendar information," so that the user is aware that calendar information is currently being extracted from image p1. See also... Figure 3D After the schedule information is successfully created, the mobile phone displays the schedule display interface U2, which includes a schedule display window 12. The schedule display window 12 displays the created schedule information 13, which includes the title, start time, end time, start date, location, and other content. In this way, the mobile phone achieves the purpose of automatically creating and displaying schedule information, which can avoid the need for users to manually record it and improve the efficiency of creating schedule information.

[0074] As an example of this application, the schedule display window 12 also includes multiple editing controls, allowing users to edit the schedule information 13 created on the mobile phone based on these controls as needed. See, for example... Figure 4 In Figure (a), when a user wants to modify the title of a schedule, they can click the title editing control 14 in the schedule display window 12. In response to the user's click on the title editing control 14, the phone displays the following... Figure 4 The target interface U3 (which can be called the smudge interface) shown in Figure (b) displays information related to the schedule in Figure p1. Thus, the user can modify or fill in the schedule title by smudged onto the information displayed on the target interface U3. Correspondingly, the phone inputs the user's smudged content into the title input box 15 of the target interface U3. See, for example... Figure 4 In Figure (b), when a user wants to change the title of the event to "HarmonyOS Chat," the user can sequentially draw "HarmonyOS," "Open," and "Chat" on the target interface U3. Correspondingly, the phone sequentially enters "HarmonyOS," "Open," and "Chat" in the title input box 15. See also Figure 4 In Figure (c), after the user finishes scribbling, the "input" control (or "√" control) on the target interface U3 can be triggered. In response to the user's triggering of the "input" control (or "√" control), the phone resumes displaying the calendar window 12. At this time, the user can see in the calendar window 12 that the calendar title has been changed to the content modified by the user through the scribbling operation; that is, the calendar title is changed from "KaiTan" to "HarmonyOS KaiTan". Thus, by displaying calendar-related and scribbable information on the target interface U3, users can quickly modify the calendar title through scribbling, improving the user experience.

[0075] In addition, see Figure 4As shown in Figure (a), the schedule display window 12 also includes a time editing control. When a user wants to edit the time information in the schedule information 13, they can modify it based on the time editing control. For example, a user can click on the displayed time information to edit it. In addition, after the user slides down the schedule display window 12, the schedule display window 12 can also provide other editing controls, such as repeat count, reminder time, important reminder, etc. In this way, the user can edit the schedule information based on other editing controls. This embodiment of the application does not limit this.

[0076] As an example of this application, after a user clicks the "√" control in the schedule display window 12, in response to the triggering operation, the phone displays schedule information 13 in the schedule details area of ​​the calendar application, thus allowing the user to view the schedule information 13 from the schedule details area of ​​the calendar application. As an optional example, after a user clicks the "√" control in the schedule display window 12, the phone can also display the schedule information 13 in the form of a card in locations such as the desktop, the negative one screen, or the notification center, so as to facilitate the user's quick viewing later. This application embodiment does not limit this.

[0077] It should be noted that the above example only illustrates how a user triggers the creation of calendar information on their phone via a sharing entry point (i.e., a sharing control). For another example, see... Figure 3A The phone provides a Magic Text Control 00 in the screenshot editing interface U1. When the user needs the phone to create schedule information based on the image p1, they can click on the Magic Text Control 00 to trigger the phone to create and display the schedule information with one click.

[0078] In another example, a user can also trigger the creation and display of schedule information on their phone through any portal. For example, see [link to example]. Figure 5 In Figure (a), after the phone takes a screenshot of the WeChat chat window, the resulting image p1 is automatically saved to the photo library. Therefore, when the user wants the phone to automatically create the schedule information shown in image p1, they can open the screenshot image interface U4 in the photo library, where image p1 is displayed. See also... Figure 5 In diagram (b), the user can select image p1 on the phone and then drag it to the right side of the screen. When the user drags the image to a certain position, the phone displays an application that can receive and process image p1, such as... Figure 5 As shown in Figure (b), the calendar, WeChat, and QQ are displayed. The user can then drag and drop image p1 onto the calendar application and release it. In response to this release, the phone begins processing image p1 to extract and create relevant schedule information. See also... Figure 5In Figure (c), after the mobile phone creates the schedule information, the schedule information is displayed in the schedule display window 12.

[0079] In another example, a user can select only schedule-related content in image p1, and then trigger the phone to extract schedule information from the selected area. See, for example... Figure 6 In Figure (a), the user can select a portion of the content in image p1. For example, the screenshot editing interface can provide a control to trigger the selection operation. After triggering the control, the user can select the area in image p1. In response to the user's selection operation, the phone selects the selected area. As an example, the phone can take a screenshot of the selected area, such as... Figure 6 As shown in Figure (b). Afterwards, the user can trigger the phone to extract schedule information from the selected area through interactive entry points such as the "share" control, creating and displaying schedule information related to the content in that area. Thus, by allowing users to select images p1, the amount of data processed by the phone can be reduced, thereby improving the efficiency of schedule information creation.

[0080] It should be noted that the interaction entry points used by the user to trigger the phone to extract schedule information from image p1 in the above application scenarios are merely exemplary. In some embodiments, the phone can also be triggered to extract schedule information from image p1 through other interaction entry points, such as global favorites, etc. This application embodiment does not limit this.

[0081] The application scenarios described above are merely illustrative. Furthermore, mobile phones can also extract schedule information from various types of images in other scenarios. For example, see... Figure 7 In Figure (a), a chat window contains a forwarded image p2 (which could be a screenshot or a photograph), and image p2 contains calendar information. When a user wants to extract the calendar information from image p2, they can drag image p2 to the right side of the phone screen, see [reference needed]. Figure 7 In image (b), when the user drags the image to a certain location, in response to the user's dragging action, the phone displays an application that can receive and process the image p2, such as... Figure 7 As shown in Figure (b), the calendar, WeChat, and QQ are displayed. After the user drags and drops image p2 onto the calendar application, the phone automatically processes image p2 to extract and create relevant schedule information in response to the user's release action. See also Figure 7 In Figure (c), after the mobile phone creates the schedule information, the schedule information is displayed in the schedule display window 12.

[0082] See Figure 7As shown in Figure (c), when multiple schedules are created on the phone, multiple schedules can be displayed in the schedule display window 12. When the schedule display window 12 does not display all the schedule information, the user can trigger the phone to display the hidden schedule information by swiping the schedule display window 12 left or right.

[0083] In addition, the phone not only supports users dragging and dropping images from chat windows to any portal, but also supports dragging and dropping text. For example, if a chat window contains chat text with calendar information, and the user needs to recreate the calendar information within that chat text, they can use the chat text and then... Figure 7 The method shown allows you to drag and drop chat text into the calendar app. The phone then extracts the schedule information from the chat text, creates and displays the relevant schedule information.

[0084] In another example, the phone can also extract calendar information from screenshots of instant messaging notification cards. See, for example... Figure 8 Figure (a) shows a schematic diagram of an instant messaging notification card screenshot (i.e., image p3) according to an exemplary embodiment. Image p3 is a screenshot of a notification card published in WeChat, which includes calendar information. When a user wants the phone to create and display the calendar information in image p3, the user can trigger the phone to extract the calendar information according to the operation process described above. For example, the user can trigger the phone to extract the calendar information from image p3 through a sharing entry. Accordingly, the phone extracts the calendar information based on image p3 and then creates or displays the relevant calendar information, for example, see [link to example]. Figure 8 In Figure (b), the mobile phone creates and displays schedule information 60, which includes the theme, ticket number, departure time and departure date, location, etc.

[0085] In another example, the phone can also extract schedule information from order screenshots, where the screenshots can be screenshots of hotel bookings, train ticket orders, plane ticket orders, etc. See, for example... Figure 9 Figure (a) shows a schematic diagram of an order screenshot (i.e., image p4) according to an exemplary embodiment. Image p4 is a screenshot of the interface where the train ticket order is located, and it includes schedule information. When a user wants the phone to create and display the schedule information in image p4, they can trigger the phone according to the operation process described above. For example, the user can trigger the phone to extract schedule information from image p4 through the Magic text entry. Accordingly, the phone extracts schedule information based on image p4 and then creates or displays the relevant schedule information, such as the displayed schedule information as follows: Figure 9 As shown in Figure (b), 70, the schedule information 70 includes the theme, train number, departure and arrival times, departure date, location, etc.

[0086] It should be noted that the above application scenarios are all exemplary and do not limit the application scenarios of the methods provided in the embodiments of this application. In another embodiment, the mobile phone can also extract schedule information from other types of images, including images other than instant messaging chat screenshots, instant messaging notification card screenshots, and order screenshots. Of course, other types of images can be images obtained through screenshots or images taken by a camera.

[0087] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating an implementation framework for a method of displaying schedule information according to an exemplary embodiment. In implementation, the electronic device acquires text information, which can be from images, text, or voice. In the case of images, the electronic device can obtain text information through image recognition processing; the user operation flow is described above. In the case of text, the electronic device can obtain text information through operations such as copying. In the case of voice, the electronic device can obtain text information through speech recognition processing. Subsequently, the electronic device can extract schedule information using a pre-trained target natural language model, and after post-processing the extracted schedule information, displays the processed schedule information.

[0088] As an example of this application, the target natural language model can extract schedule information from text information and combine the time information in the text information for a unified representation. That is, the time information in the schedule information extracted by the target natural language model is represented as a unified whole, rather than being represented as multiple separate fields. See, for example... Figure 11 The schedule information output by the target natural language model includes the corresponding field values ​​of the title field, start time field, end time field, location field, and repetition period field. In other words, the time information in the text information is merged into two fields (start time field and end time field) for encoding and representation. In comparison, the schedule information output by the target natural language model involves fewer fields.

[0089] As an example of this application, the target natural language model is obtained by pre-training. In one example, the target natural language model is obtained by training an initial natural language understanding model based on a sample training set. The sample training set includes multiple sets of sample training data. Each set of sample training data includes text training samples and schedule information samples corresponding to the text training samples. The time information in the schedule information samples has been merged.

[0090] As an example, the initial natural language model is a large language model (LLM) model, which, for example, can be a natural language processing (NLP) model.

[0091] During training, when setting the schedule information sample corresponding to each text training sample, the time information can be merged to obtain the processed schedule information sample. As an example of this application, the start date field, start time field, and repeat start time field can be merged, as can the end date field, end time field, and repeat end time field. Furthermore, if the text training sample does not include an end time point but includes a start time point and duration description information (which describes the duration of the schedule), then when constructing the schedule information sample, the field value in the end time field can be set to the start time point and duration description information from the text training sample.

[0092] For example, see Figure 12 For a text training sample containing cyclical schedule information, such as "Meetings every Tuesday at 8:00 AM for approximately two hours in room 501 from October 1st to October 20th", the corresponding schedule information samples can be changed from the original... Figure 12 The structure shown in Figure (a) is modified as follows: Figure 12 The structure shown in Figure (b) merges the start date (empty) of the start date field and the repeat start date (October 1st) of the repeat start date field into the field value of the start time field, and merges the end date (empty) of the end date field and the repeat end date (October 20th) of the repeat end date field into the field value of the end time field. The end time in the end time field is represented by "8:00" and "approximately two hours." This means that even when the text training sample includes the start time and duration description but not the end time, the end time is not calculated based on the start time and duration description in the text training sample when setting the corresponding schedule information sample. Instead, the end time is represented by the start time and duration description in the text training sample for model learning. This allows the target natural language model to automatically infer the end time for the same scenario in subsequent model applications, improving the model's intelligence.

[0093] See also Figure 13For a text training sample like "Going on vacation to Xiamen from 8:00 AM on October 1st to 10:00 AM on October 20th", which does not include periodic schedule information, the corresponding schedule information sample can be changed from the original one. Figure 13 The structure shown in Figure (a) is modified as follows: Figure 13 The structure shown in Figure (b) merges the start date (October 1st) of the start date field and the repeat start date (empty) of the repeat start date field into the field value of the start time field, and merges the end date (October 20th) of the end date field and the repeat end date (empty) of the repeat end date field into the field value of the end time field. Similarly, if the text training sample includes a start time and duration description information but not an end time, then when setting its corresponding schedule information sample, the end time is not calculated based on the start time and duration description information in the text training sample, but is directly represented by the start time and duration description information in the text training sample.

[0094] When combining and representing time information, time points and dates can be distinguished by preset identifiers, such as "|". Dates and time points can be arranged according to certain rules, such as the date being placed before the time point. This application does not limit this.

[0095] The number of preset identifiers can be set according to needs. For example, if there is a duration description, two preset identifiers can be included, such as... Figure 12 As shown in Figure (b). Alternatively, it can include only one identifier, in which case the time information before the preset identifier can be the date and time point, and the time information after the preset identifier is the duration description information. In the absence of duration description information, it can be segmented by a single identifier.

[0096] As an example of this application, if the text training sample includes an end time and duration description information, but excludes a start time, then when constructing the schedule information sample corresponding to the text training sample, the start time can be represented by the end time and duration description information in the field value corresponding to the start time field. If the text training sample includes a start date and duration description information, but excludes an end date, then when constructing the schedule information sample corresponding to the text training sample, the end date can be represented by the start date and duration description information in the field value corresponding to the end time field. If the text training sample includes an end date and duration description information, then when constructing the schedule information sample corresponding to the text training sample, the start date can be represented by the end date and duration description information in the field value corresponding to the start time field.

[0097] As an example of this application, if the text training sample includes an end time and a time interval duration, where the time interval duration refers to the time between the end time and the next start time, then when constructing the schedule information sample corresponding to the text training sample, the next start time can be represented by the end time and the time interval duration in the field value corresponding to the start time field. For example, if the text training sample is "This meeting ends at 10:00 and will continue in two hours," then the field value corresponding to the start time field in its corresponding schedule information sample could be: |10:00| two hours later.

[0098] As an example of this application, if the text training sample does not include the start time and end time, but only the duration of the time interval, then when constructing the schedule information sample corresponding to the text training sample, the field value corresponding to the start time field can only include the duration of the time interval. For example, if the text training sample is "starting in two hours", then the field value corresponding to the start time field in its corresponding schedule information sample can be: || two hours later.

[0099] The schedule information sample corresponding to each text training sample is set up in this way. Then, each text training sample and its corresponding schedule information sample are used as a set of training data to obtain multiple sets of training data. After iteratively training the initial natural language model using these multiple sets of training data, the target natural language model can be obtained. Thus, in application, after inputting text information into the target natural language model, the target natural language model will output the schedule information according to the modified field specifications, such as the field format corresponding to the title field, start time field, end time field, location field, and repetition period field.

[0100] It's worth noting that by expressing the same amount of information with fewer fields, the information density is increased. This reduces the number of fields the target natural language model needs to focus on when extracting schedule information, thus lowering the computational difficulty of the target natural language model and improving the accuracy of the output. Furthermore, the reduced length of the schedule information to be output also lowers the extraction latency.

[0101] Because the time information in the schedule information output by the target natural language model is represented in a combined manner during application, that is, using fewer fields, therefore, see Figure 11In the post-processing stage, the electronic device performs structure decoding on the schedule information output by the target natural language model to split the merged time information, thereby representing it using more fields individually. As an example of this application, during the splitting process, the start date is extracted from the start time field value of the schedule information output by the target natural language model and used as the field value for the first date field. The end date is extracted from the end time field value of the schedule information output by the target natural language model and used as the field value for the second date field. Both the first and second date fields are newly added fields and are different. Furthermore, if the end time field value of the schedule information output by the target natural language model includes duration description information, the electronic device can also determine the split end time field value based on the start time and duration description information in the end time field value of the schedule information output by the target natural language model.

[0102] As an example rather than a limitation, time information can be split according to preset identifiers. For example, for the field value corresponding to the start time field, the time information before the first preset identifier is the start date, the time information between the first and second preset identifiers is the start time point, and the time information after the second preset identifier is the duration description information.

[0103] When the schedule information output by the target natural language model does not include periodic schedule information, the first date field is the start date field, and the second date field is the end date field. The value of the start date field is the date the schedule begins, and the value of the end date field is the date the schedule ends. When the schedule information output by the target natural language model includes periodic schedule information, the first date field is the repeat start date field, and the second date field is the repeat end date field. The value of the repeat start date field is the date the repeat begins, and the value of the repeat end date field is the date the repeat ends. As an example and not a limitation, electronic devices can determine whether the schedule information includes periodic schedule information based on whether the value of the repeat period field is empty.

[0104] As an example of this application, based on the start time and duration description information in the end time field value of the schedule information output by the target natural language model, the field value corresponding to the split end time field is determined, including: adding the start time and duration description information in the end time field value to determine the field value corresponding to the split end time field. Specifically, electronic devices can identify time entities, for example, through rule matching. For instance, if the time information meets the condition xx:xx, it is determined to be a time point; otherwise, it is duration description information. This determines the start time and schedule duration, and then adding the start time and schedule duration yields the end time.

[0105] Of course, in another example, the electronic device can also input the field value corresponding to the split end time field into other models for time inference separately. For example, input "8:00 approximately two hours" into other models for time inference to determine the end time point. Other models can be the target natural language model or other models. This application embodiment does not limit this.

[0106] For example, see Figure 14 The structure of the schedule information output by the target natural language model is as follows: Figure 14 As shown in Figure (a), this includes the field values ​​corresponding to the title field (meeting), the start time field (October 1st | 8:00), the end time field (October 20th | 8:00 | approximately two hours), the location field (501), and the repetition period field (every Tuesday). In this case, during the splitting process, the start date (October 1st) from the start time field is used as the field value corresponding to the repetition start date field, and the end date (October 20th) from the end time field is used as the field value corresponding to the repetition end date field. The 8:00 from the end time field is then added to 2 hours to determine the final field value corresponding to the end time field. Thus, the result is as follows: Figure 14 The structure shown in Figure (b) is the split schedule information.

[0107] For example, see Figure 15 The structure of the schedule information output by the target natural language model is as follows: Figure 15As shown in Figure (a), the table includes the field values ​​corresponding to the title field (basketball), the start time field (Monday), the end time field (empty), the location field (empty), and the repetition period field (every Monday). In other words, the schedule information output by the target natural language model includes the field value corresponding to at least one of these fields. In this case, during the splitting process, the start date in the start time field is used as the field value corresponding to the repetition start time field. Thus, the following can be obtained: Figure 15 The structure shown in Figure (b) is the split schedule information.

[0108] As an example of this application, if the start time field of the schedule information output by the target natural language model includes an end time and a time interval duration, the electronic device determines the start time based on the end time and the time interval duration. For instance, if the start time field of the schedule information output by the target natural language model has the value: |10:00| two hours later, then the start time can be determined to be 12:00.

[0109] As an example of this application, if the field value corresponding to the start time field of the schedule information output by the target natural language model only includes the duration of the time interval, the electronic device determines the start time point based on the current system time and the duration of the time interval. For example, if the current system time is 5:00, and the field value corresponding to the start time field of the schedule information output by the target natural language model is: || Two hours later, the start time point can be determined to be 7:00.

[0110] In this way, the schedule information output by the target natural language model can be restored to the field format required for daily use. Then, the schedule information can be displayed based on the split structure.

[0111] It should be noted that the above implementation method, which involves outputting schedule information from a target natural language model and then having an electronic device split the schedule information output by the target natural language model, is merely exemplary. In another example, when training the target natural language model, schedule information samples corresponding to the split field formats can also be added to the sample training data. That is, the sample training data includes not only text training samples and merged schedule information samples, but also split schedule information samples. For example, see... Figure 12 The training data includes not only text training samples and such Figure 12 The schedule information sample shown in Figure (b) also includes, for example, Figure 12The schedule information sample shown in Figure (a) is as follows. Thus, during model application, the target natural language model can not only output the merged schedule information, but also, after inputting the merged schedule information into the target natural language model, the target natural language model can further decompose the merged schedule information, thereby outputting the decomposed schedule information, such as... Figure 14 The schedule information shown in Figure (a) is input into the target natural language model and can output something like... Figure 14 The schedule information is shown in Figure (b). Thus, the COT method was used to extract the schedule information.

[0112] It should be noted that this embodiment of the application uses the extraction of schedule information through a target natural language model as an example. In another example, if the text information follows a certain pattern, schedule information can also be extracted through rule matching. For example, if the text information follows the rule "xx time" to "xx", the information before "go" can be extracted as time information, and the information after "go" can be extracted as location. In this case, the electronic device can also extract schedule information according to the above field rules, and this embodiment of the application does not limit this.

[0113] The software system of the electronic device involved in the embodiments of this application can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software system of the electronic device.

[0114] Figure 16 This is a block diagram of a software system for an electronic device provided in an embodiment of this application. See also... Figure 16 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime, the system layer, and the kernel layer.

[0115] The application layer can include a series of application packages, see [link to relevant documentation] Figure 16 The application package can include calendar and other applications, such as instant messaging, ticketing, camera, gallery, calling, maps, navigation, Bluetooth, music, video, SMS, voice assistant, etc.

[0116] Furthermore, as an example of this application, the application layer also includes a schedule management service and a model management service. The schedule management service can be used to provide services to the calendar, or rather, to be invoked by the calendar. For example, the schedule management service can create schedule information and store schedule information and other data for the calendar. The schedule management service can include a schedule creation service (which may be called: intelligent parsing and processing service) and a schedule database (such as: Calendar Provider schedule database). The schedule management service can create schedule information through the schedule creation service and store the created schedule information through the schedule database. The model management service (which may be called: MagicLive large model service) can be used to provide various models for the schedule management service to invoke when needed.

[0117] In one example, the model management service provides pre-trained large language model (LLM), target recognition model, and personal behavior feature model. The LLM model can be used to identify (i.e., infer) prompts to determine schedule information. Exemplarily, the LLM model can be a natural language processing (NLP) model, which can be run through a natural language unit (NLU). In this embodiment, the pre-trained LLM model is referred to as the target natural language model. In some examples, the target natural language model can not only identify prompts but also extract keywords from a text, such as time and location keywords. Object recognition models can be used for text recognition and edge detection of images. They can also be used to determine the image category. In one example, an object recognition model includes a first optical character recognition (OCR) model, a second OCR model, and an edge detection model. The first OCR model can be used for text recognition, the second OCR model can be used to determine the image category, and the edge detection model can be used for edge detection of images. Multiple edge detection models can be used to detect the edges of images of different categories. Personal behavior feature models can be used to determine user profiles based on users' historical behavior data.

[0118] It should be noted that the embodiments in this application are illustrated using a pre-trained LLM model as the target natural language model. In another example, the size of the model and whether it is multimodal are not limited.

[0119] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 16 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0120] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java needs to call, and the other part is the core Android library itself. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0121] The system library can include multiple functional modules, such as: surface manager, media libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engines (such as SGL), etc.

[0122] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0123] Electronic devices can interact through the aforementioned modules to implement the method for displaying schedule information provided in this application embodiment. For example, taking an image as the object from which schedule information is to be extracted, see [link to relevant documentation]. Figure 17In practice, electronic devices can pass images to the calendar application through interactive entry points such as sharing, "Anywhere Door," global favorites, or Magic Text. These images can be screenshots of train or plane tickets, hotel, restaurant, or entertainment bookings, sports or health-related bookings, screenshots of WeChat mini-programs or web applications, screenshots of service notification cards from IM notification accounts, or screenshots of IM chat conversations, etc. As examples, not limitations, screenshots of train and plane ticket orders can come from applications such as 12306, China Railway, or Ctrip; screenshots of hotel, restaurant, and entertainment orders can come from applications such as Ctrip, Qunar, Tongcheng, Meituan, Fliggy, Dianping, and Damai; screenshots of sports and health orders can come from applications such as Keep, Lianduoduo, and online registration platforms; screenshots of WeChat mini programs and web applications can include content such as performances, flash sales, and marathons; screenshots of service notification cards from IM notification accounts can include notification cards issued by third parties through IM notification accounts, such as hospitals, scenic spot tickets, educational institutions, and insurance companies; and screenshots of IM chat conversations can include content such as work arrangements, invitations, and educational tasks.

[0124] After receiving an image, the calendar application requests a target recognition model to perform text recognition and edge detection. Text recognition extracts text information from the image, while edge detection identifies chat elements or color blocks within the image. (See also...) Figure 17 Image p1 is passed to the target recognition model through the interactive entry point. After text recognition by the target recognition model, the text recognition result is as follows: Figure 17 As shown in 90, in addition, after edge recognition by the target recognition model, the coordinates and categories of each chat element in image p1 can be identified.

[0125] Subsequently, the electronic device can filter the text recognition and edge recognition results based on filtering rules, i.e., perform preprocessing to remove irrelevant interference information from the text recognition and edge recognition results. As an example of the application, the preset filtering rules may include, but are not limited to, at least one of the following: 1. Discarding dense text blocks. 2. Discarding small characters. 3. Discarding skewed lines. 4. Discarding floating text in attached images. 5. Discarding the "back" icon in the upper left corner of the image and the "+" symbol in the image. For example, after filtering the text recognition results based on the filtering rules, the following can be removed: Figure 17 The content shown in 91 can be further filtered based on the edge recognition results using filtering rules to remove the coordinates and categories of chat elements that are irrelevant to the schedule.

[0126] In the process of extracting schedules from images, if the text recognition results are not filtered and are directly input into the target natural language model for schedule information extraction, as shown in Table 3, the recognition success rate of the target natural language model is low, usually only about 30%. This is because there are problems such as interference information, formatting line breaks, and loss of layout information. Therefore, in this embodiment, the text recognition results are filtered before the schedule information is extracted by the target natural language model, which can improve the accuracy of the schedule information extracted by the target natural language model.

[0127] Table 3

[0128]

[0129] After filtering, the electronic device concatenates the target text information based on the remaining text recognition and edge recognition results to construct a corresponding prompt. This prompt is then input into the target natural language model (NLP) for extracting schedule information. The NLP outputs schedule information based on the input prompt, for example... Figure 17 Figure 92 illustrates a portion of the processing by the target natural language model. The electronic device then splits the output schedule information into time information segments, performs schedule field post-processing on the split schedule information, and obtains the schedule information to be displayed. For example, the schedule information to be displayed is as follows: Figure 17 As shown in 93. Afterwards, the electronic device can display the schedule information.

[0130] In one example, post-processing of the schedule field may include, but is not limited to, at least one of the following: 1. Processing according to reminder time rules. 2. Processing according to tomorrow's start time rules. 3. Processing according to detail rules. 4. Processing based on title merging pre-filling rules.

[0131] The reminder timing rules include setting the reminder time for schedule information to be earlier than the preset duration of the time information extracted by the target natural language model. The preset duration can be set according to needs; for example, if the preset duration is 30 minutes, and the start time of the target natural language model extraction is 8:30, then the reminder time for the schedule information can be set to 8:30. Additionally, the reminder timing rules also include handling repeated reminders. For example, if the schedule information extracted by the target natural language model indicates that registration is being conducted on Monday, Tuesday, and Wednesday, then the electronic device will set reminder times for Monday, Tuesday, and Wednesday for that schedule information, instead of setting only one reminder time.

[0132] The "start time tomorrow" rule means that if the time information extracted by the target natural language model spans days, months, or years, then the time spanning days, months, or years should be added. For example, if the date extracted by the target natural language model is December 5th, the start time is 23:00, and the end time is 00:30, then the electronic device can add the end time as December 6th 00:30 to the schedule information.

[0133] Details rules refer to adjusting the layout and font size of created schedule information according to the screen size of electronic devices so that it can be displayed correctly and clearly in the schedule details area of ​​the calendar application.

[0134] The title merging and pre-filling rule includes selecting the longest title from multiple titles corresponding to the same time information as the title of the schedule information. Additionally, the rule also includes using a specified title corresponding to the scene of the image as the title of the schedule information. For example, if the scene of the image is about scheduling an appointment and taking a number for medical treatment, and the title extracted by the target natural language model is "taking a number" or "getting a number," then the schedule title can be standardized to "registering for medical treatment." The specified titles corresponding to different scenes can be pre-set as needed.

[0135] It should be noted that the post-processing of the schedule field described above is merely exemplary. In another example, post-processing of the schedule field may also include, but is not limited to, at least one of the following: time similarity processing, discarding empty results, risk control, and cleaning up non-natural language titles. Time similarity processing means that if the time in the output schedule information is earlier than the current system time of the electronic device, the closest time to the time in the schedule information is determined based on the current system time, and this determined time is set as the time in the schedule information. For example, if the time in the schedule information is Tuesday, and the current system time is Wednesday, then it is recorded as Tuesday of the following week in the schedule information. Discarding empty results means discarding returned empty fields. Risk control refers to managing sensitive words. Cleaning up non-natural language titles means that if the schedule title does not contain a verb, a verb can be added to the schedule title, or the schedule title can be set to default based on the scene corresponding to the image.

[0136] Next, combine Figure 18 This section provides a detailed explanation of the process for extracting and displaying schedule information from images. See also... Figure 18 This method may include the following implementation steps:

[0137] S1801: The calendar application receives the image L to be processed.

[0138] Image L can be a bitmap format image.

[0139] The calendar application receives image L from the interaction entry point. As mentioned earlier, the interaction entry point can be a sharing entry point, a portal entry point, etc. See, for example... Figure 3A When a user submits image L to the calendar application via the share control, this interaction point becomes the share entry point.

[0140] S1802: The calendar application sends a schedule creation instruction to the schedule management service, which includes an image L.

[0141] The schedule creation command is used to instruct the creation and display of related schedule information based on image L.

[0142] S1803: The schedule management service sends image L to the first OCR model in the model management service.

[0143] In implementation, the schedule management service calls the first OCR model in the model management service and sends image L to the first OCR model so that text recognition processing can be performed by the first OCR model.

[0144] S1804: The first OCR model determines the first recognition result of image L.

[0145] As an example of this application, the first recognition result includes the text recognition result of image L, which includes text block coordinates, text line coordinates, and text line recognition content.

[0146] As an example, the first recognition result also includes target indication information, which can be used to indicate whether the image L input to the first OCR model is a screenshot or a photograph. Exemplarily, the target indication information can be a first identifier, a second identifier, or a third identifier. The first identifier indicates that the image L input to the first OCR model is a screenshot; the second identifier indicates that the image L input to the first OCR model is a photograph, specifically a photograph of a document; and the third identifier indicates that the image L input to the first OCR model is another type of photograph, such as a photograph of an advertisement, road sign, or magazine. The first, second, and third identifiers can be set as needed, for example, the first identifier could be F1, the second identifier F2, and the third identifier F3.

[0147] That is, after the first OCR model receives image L, it performs recognition on image L and outputs the first recognition result of image L.

[0148] S1805: The first OCR model sends the first recognition result to the schedule management service.

[0149] As an example, the schedule management service can cache the first recognition result after receiving it.

[0150] S1806: The schedule management service sends image L to the second OCR model in the model management service.

[0151] In one example, after the schedule management service receives the schedule creation instruction from the calendar application, in addition to sending the image L to the first OCR model for text recognition processing, it can also call the second OCR model in the model management service and send the image L to the second OCR model so that the image category can be determined by the second OCR model. That is, the operation of S1806 and the operation of S1803 can be executed in parallel.

[0152] Since the method provided in this application embodiment can extract schedule information from images of different image categories, and the content layout of images of different image categories is different, electronic devices process images of different image categories differently. Therefore, in implementation, after receiving the image L to be processed, the schedule management service not only performs text recognition through the first OCR model, but also inputs image L into the second OCR model to determine the image category of image L.

[0153] As an example of this application, the image categories include instant messaging chat screenshots, instant messaging notification card screenshots, order screenshots, and other image categories. An instant messaging chat screenshot refers to an image obtained by taking a screenshot of the chat interface in an instant messaging application; an instant messaging notification card screenshot refers to an image obtained by taking a screenshot of the service notification card in an instant messaging application; an order screenshot refers to an image obtained by taking a screenshot of the order interface in the application; other image categories include images other than instant messaging chat screenshots, instant messaging notification card screenshots, and order screenshots, and other image categories can be screenshots or photographs.

[0154] S1807: The second OCR model determines the image category of image L.

[0155] After receiving image L, the second OCR model identifies image L and outputs the image category of image L.

[0156] S1808: The second OCR model sends the image category of image L to the schedule management service.

[0157] S1809: The schedule management service sends image L to the edge detection model corresponding to the image category of image L.

[0158] As an example of this application, the model management service provides multiple edge detection models, which can be pre-trained. Different edge detection models can perform edge recognition on images of different image categories. In one example, the multiple edge detection models include a first edge detection model and a second edge detection model. The first edge detection model can be used to perform edge recognition on instant messaging chat screenshots to determine the coordinates and category of chat elements in the instant messaging chat screenshots. The second edge detection model can be used to perform edge recognition on other images besides instant messaging chat screenshots. For example, the second edge detection model can be used to perform edge recognition on instant messaging notification card screenshots, order screenshots, or other types of images to determine the coordinates and category of color blocks in other images.

[0159] Different edge detection models can be obtained by iteratively training an initial training model based on image training samples of the corresponding image category. The image training samples can be obtained in advance through edge annotation as needed. The initial training model can be set as needed, as an example rather than a limitation. The initial training model can be a recurrent neural network (RNN), etc.

[0160] After receiving the image category of image L, the schedule management service determines the edge detection model corresponding to the image category of image L from multiple edge detection models. For example, when image L is... Figure 3A In the case of p1, which is a screenshot of an instant messaging chat, the edge detection model corresponding to the image category of image L is determined from multiple edge detection models as the first edge detection model; when image L is... Figure 8 p3 of image (a) (i.e., a screenshot of the instant messaging notification card) or is Figure 9 In the case of p4 of Figure (a) (i.e., the order screenshot), the edge detection model corresponding to the image category of image L is determined from multiple edge detection models as the second edge detection model. Then, the schedule management service calls the determined edge detection model and sends image L to that edge detection model to request edge recognition of image L.

[0161] S1810: The edge detection model performs edge recognition processing on image L and outputs the edge recognition results.

[0162] Edge recognition results include image and text element attribute information or color block attribute information.

[0163] In one example, if image L is classified as an instant messaging chat screenshot, the edge detection model corresponding to image L's image category is the first edge detection model. After image L is input into the first edge detection model for processing, the output edge recognition result includes image and text element attribute information. This image and text element attribute information includes the attribute information of chat elements in the instant messaging chat screenshot. The attribute information of chat elements can include the coordinates and category of the chat elements. For example, chat elements include avatar, title, nickname, chat content, chat timestamp, username, and specified identifiers, including the "+" identifier. See, for example... Figure 19 After inputting image L into the first edge detection model, the first edge detection model can determine that the chat elements in image L include... Figure 19 The dashed boxes in the image indicate multiple items. Each chat element has its own coordinates and category. For example, the coordinates of a chat element are the coordinates of the four corners of the area where the chat element is located, and the category is avatar. Optionally, the attribute information of each chat element can also include the chat element's sequence number. The sequence number of each chat element in image L can be set by default according to the sorting of chat elements in image L.

[0164] In another example, if image L is not an instant messaging chat screenshot, but rather an instant messaging notification card screenshot, order screenshot, or other type of image, the edge detection model corresponding to image L's image category is a second edge detection model. After image L is input into the second edge detection model, the second edge detection model performs color block segmentation, and the output edge recognition result includes color block attribute information. For example, the color block attribute information includes the coordinates, category, and color block number of the color block. See, for example... Figure 20 When image L is a screenshot of an instant messaging notification card, after inputting image L into the second edge detection model, the second edge detection model can determine that the color blocks in image L include... Figure 20 The dashed boxes in the image indicate multiple items, and each color block has its own coordinates and category. For example, the coordinates of the color blocks include the coordinates of the four corners.

[0165] S1811: The edge detection model sends the edge recognition results to the schedule management service.

[0166] As an example, the schedule management service can cache the edge detection results after receiving them.

[0167] It is worth mentioning that after performing text recognition and edge recognition processing on image L through the two branches mentioned above, a first recognition result and a second recognition result can be obtained. The first recognition result includes text recognition results and target indication information, while the second recognition result includes edge recognition results and image category. Since the text recognition result can represent the text content in image L, and the edge recognition result can represent the layout of image L, the subsequent extraction of schedule information based on the first and second recognition results can improve the accuracy of information extraction. The specific implementation can be seen in the following steps.

[0168] S1812: If the image category of image L is an instant messaging chat screenshot, the schedule management service filters the text recognition results and edge recognition results according to the first filtering rule.

[0169] Different image categories correspond to different filtering rules. In implementation, the schedule management service determines the corresponding filtering rules based on the image category of image L, and then filters the text recognition results and edge recognition results of image L according to the determined filtering rules to remove interference information that is irrelevant to the schedule.

[0170] In one example, the instant messaging chat screenshot corresponds to the first filtering rule. The first filtering rule can include filtering out the recognition data corresponding to skewed lines, small text, and chat timestamps. Skewed lines refer to text lines whose tilt angle is greater than a preset angle, which can be set according to needs, such as 10 degrees. Small text refers to text lines whose line height is less than a target line height, which can be the average line height of all text lines in image L. The chat timestamp is a timestamp used to indicate the chat time. Optionally, the first filtering rule can also include filtering out dense text blocks, floating text on the image, the "back" icon in the upper left corner, and the "+" icon in the lower right corner, etc. See, for an example... Figure 19 , Figure 19 According to an exemplary embodiment, the chat elements in image L that need to be filtered are identified, including chat timestamp 1101, small text 1102, slanted lines 1103, and "+" 1004.

[0171] As an example, in the implementation of filtering out the recognition data corresponding to skewed lines, the tilt angle of each text line can be calculated based on the text line coordinates in the text recognition result, thereby determining which text lines are skewed lines, and then deleting the recognition data corresponding to the skewed lines, such as deleting the text line coordinates and text line recognition content of the skewed lines.

[0172] As an example, in the implementation of filtering out the recognition data corresponding to small characters, the line height of each text line in image L can be determined based on the text line coordinates in the text recognition result, thereby filtering out the recognition data corresponding to text lines whose line height is less than the target line height, such as filtering out the text line coordinates and text line recognition content of text lines whose line height is less than the target line height.

[0173] As an example, in the implementation of filtering out the recognition data corresponding to chat timestamps, chat elements whose category is chat timestamp can be determined based on the edge recognition results, obtaining at least one first candidate chat element. Based on the coordinates of each first candidate chat element and the text line coordinates in the text recognition result of image L, the text line recognition content corresponding to each first candidate chat element is matched from the text recognition result of image L. If the target first candidate chat element is determined to be a chat timestamp based on the text line recognition content, it is determined whether to filter out the recognition data corresponding to the target first candidate chat element based on its coordinates. The target first candidate chat element is any one of the at least one first candidate chat element. If it is determined that the recognition data corresponding to the target first candidate chat element should be filtered out, the recognition data corresponding to the target first candidate chat element is filtered out from both the text recognition result and the edge recognition result of image L.

[0174] As mentioned earlier, the edge recognition results of instant messaging chat screenshots include the category of chat elements. Therefore, based on the category of chat elements in the edge recognition results, chat elements whose category is a chat timestamp can be filtered out, obtaining at least one first candidate chat element. Since the edge recognition results do not include text line recognition content, meaning the text content corresponding to each chat element cannot be known, in some cases, the edge detection model may misclassify chat elements whose category is not a chat timestamp as chat timestamps. Therefore, to minimize the filtering out of chat elements that are not chat timestamps, after determining at least one first candidate chat element, the text line recognition content corresponding to each first candidate chat element can be matched from the text recognition results of image L based on the coordinates of each first candidate chat element and the text line coordinates of image L. For example, for any first candidate chat element, based on the coordinates of this first candidate chat element and the text line coordinates of image L, the text line recognition content of at least one text line located within the region corresponding to this first candidate chat element can be determined from the text recognition results of image L, thus matching the text line recognition content corresponding to this first candidate chat element. Subsequently, the schedule management service can determine whether the matched text line is a timestamp using the target natural language model. For example, the schedule management service can send the matched text line to the target natural language model via `getEntity()` to request the model to identify whether the text line is a timestamp. If the target natural language model determines that the text line is a timestamp, then this first candidate chat element is likely a chat timestamp. Otherwise, if the target natural language model determines that the text line is not a timestamp, then this first candidate chat element is not a chat timestamp.

[0175] Since the position of chat timestamps in instant messaging screenshots is generally fixed, such as being centered in the chat window of WeChat, after identifying a first candidate chat element as a chat timestamp through text line content recognition using a target natural language model, we can further determine whether the first candidate chat element is indeed a chat timestamp based on its coordinates, thereby improving the accuracy of chat timestamp recognition.

[0176] As an example of this application, for any one of the at least one first candidate chat element, the specific implementation of determining whether to filter out the recognition data corresponding to that first candidate chat element based on its coordinates can include the following two cases:

[0177] The first scenario: If the coordinates of the first candidate chat element determine that the first candidate chat element is located in the middle of image L, and the area corresponding to the first candidate chat element includes a single text line, then the recognition data corresponding to the first candidate chat element will be filtered out.

[0178] In most instant messaging applications, the chat timestamp is typically centered in the chat window and consists of only one line of text. Therefore, if a first candidate chat element is located in the center of image L, and the area corresponding to this first candidate chat element contains only a single line of text, and since the target natural language model has already determined that the text line is a timestamp, it can be determined that this first candidate chat element is indeed a chat timestamp. This means that the recognition data corresponding to this first candidate chat element can be filtered out.

[0179] The second scenario: If the coordinates of the first candidate chat element determine that the first candidate chat element is located on the right side of image L, and the area corresponding to the first candidate chat element includes a single text line, then if the recognized content of the text line corresponding to the first candidate chat element only includes time and does not include date, then the recognized data corresponding to the first candidate chat element will be filtered out.

[0180] In some instant messaging applications (such as the chat interface forwarded in WeChat), the chat timestamp may be displayed on the right, and it may only contain one line of text for recognition. Furthermore, the chat timestamp only includes the time and not the date. Therefore, if it is determined that this first candidate chat element is located on the right side of image L, and the area corresponding to this first candidate chat element contains only a single line of text, it can be determined whether the text line recognition content corresponding to this first candidate chat element includes the date. For example, the target natural language model can be used to determine whether the text line recognition content includes the date. If it is determined that the date is not included, then this first candidate chat element can be determined to be a chat timestamp, meaning that the recognition data corresponding to this first candidate chat element can be filtered out. Of course, if it is determined that the date is included, then it can be determined not to filter it out.

[0181] In one example, for the second case, it is not necessary to determine whether the text line recognition content corresponding to the first candidate chat element only includes time and not date. As long as it is determined that the first candidate chat element is located on the right side of image L and the area corresponding to the first candidate chat element includes a single text line, the schedule management service can determine to filter out the recognition data corresponding to the first candidate chat element.

[0182] If the above process determines that a first-choice chat element is a chat timestamp, the schedule management service removes the corresponding recognition data from the text recognition and edge recognition results of image L. For example, it removes the text line coordinates and text line recognition content corresponding to this first-choice chat element from the text recognition results of image L, and removes the coordinates and category corresponding to this first-choice chat element from the edge recognition results of image L. Of course, if the above process determines that a first-choice chat element is not a chat timestamp, the schedule management service does not filter out the recognition data corresponding to this first-choice chat element.

[0183] It is worth mentioning that by first determining the chat element category as a chat timestamp based on the edge recognition results, then matching the corresponding text line recognition content from the text recognition results, and using the target natural language model to determine whether it is a chat timestamp based on the matched text line recognition content, and then determining whether it is a chat timestamp based on the position of the chat element, the accuracy of chat timestamp recognition can be improved, thereby improving the accuracy of filtering, and further improving the accuracy of calendar information creation.

[0184] It should be noted that the first filtering rule described above is merely an example. When instant messaging chat screenshots originate from different instant messaging applications, the layout of chat elements is usually different, and the chat elements themselves may also differ. This can lead to variations in interfering information in instant messaging chat screenshots from different applications. For example, this typically includes several possible scenarios as shown in Table 4. Therefore, in another example, the first filtering rule may also include other rules for filtering out information unrelated to the chat content.

[0185] Table 4

[0186]

[0187] To effectively filter out interfering information from instant messaging chat screenshots from different instant messaging applications, a first filtering rule can be set according to the union of possible interfering information shown in Table 4. This ensures that interfering information can be effectively removed regardless of the type of instant messaging chat screenshot being processed. For example, the first filtering rule may also include filtering out text, usernames, and specified identifiers from avatars. In implementation, after filtering out the recognition data corresponding to the chat timestamp in image L from the text recognition results and edge recognition results of image L, the schedule management service can determine chat elements unrelated to the chat content from the remaining chat elements in the edge recognition results based on their categories, obtaining at least one second candidate chat element. Based on the coordinates of each second candidate chat element and the text line coordinates in the text recognition results of image L, the text line recognition content corresponding to each second candidate chat element is matched from the text recognition results of image L. The matched text line recognition content and corresponding text line coordinates are then deleted from the text recognition results of image L.

[0188] For example, in the implementation of filtering text from profile pictures, the schedule management service can determine the chat element whose category is profile picture based on the edge recognition result, and then match the text line recognition content corresponding to the chat element in the current recognition result of image L based on the coordinates of the chat element. If there is a matching text line recognition content, the matching text line recognition content and the text line coordinates corresponding to the matching text line recognition content are deleted from the text line recognition result of image L, thereby deleting the text from the profile picture.

[0189] For example, in the implementation of filtering out usernames, the schedule management service can determine the chat element whose category is username based on the edge recognition result, and then match the text line recognition content corresponding to the chat element in the text recognition result of image L based on the coordinates of the chat element. If there is a matching text line recognition content, the matching text line recognition content and the text line coordinates corresponding to the matching text line recognition content are deleted from the text line recognition result of image L, thereby deleting the username.

[0190] For example, in the implementation of filtering specified identifiers, the schedule management service can determine chat elements whose category is the specified identifier based on the edge recognition results. Then, it matches the text line recognition content corresponding to the chat element from the edge recognition results of image L. If the text line recognition content is the specified identifier, such as "+", then the recognition data corresponding to the chat element can be deleted from the text line recognition results of image L, such as deleting the text line coordinates and text line recognition content corresponding to the chat element. The schedule management service can also delete the recognition data corresponding to the chat element from the edge recognition results, such as deleting the coordinates and category of the chat element.

[0191] It should be noted that the above explanation uses an image L as an example of an instant messaging chat screenshot. In another example, if image L is not an instant messaging chat screenshot, but rather an instant messaging notification card screenshot or an order screenshot, the schedule management service filters out interfering information based on the second filtering rule. In one example, in the implementation of filtering based on the second filtering rule, the schedule management service can match the text line recognition content of each color block in the text recognition result of image L with the coordinates of each color block in the edge recognition result and the coordinates of the text lines in the text recognition result of image L. For any color block, if it is determined from the text line recognition content of any color block that the color block does not contain schedule-related content, such as time information and location, then the recognition data corresponding to that color block is filtered out from the edge recognition result and the text recognition result of image L. In one example, the target natural language model can be used to determine whether the color block includes time information and location. For example, the text line recognition content of each color block can be sent to the target natural language model one by one to request the target natural language model to determine whether it includes time information and location.

[0192] When image L is a screenshot of an instant messaging notification card, Table 4 shows that potential interference includes small text, skewed lines, and a central timestamp. Therefore, in one example, before filtering based on the second filtering rule, the recognition data corresponding to skewed lines, small text, and a central timestamp in the text line recognition content and edge recognition results of image L can be filtered out first, and then filtered out based on the second filtering rule. The filtering of skewed lines, small text, and a central timestamp can be found in the first filtering rule.

[0193] As an example of this application, before filtering, it can be checked whether the number of color blocks in image L is less than a quantity threshold. If the number of color blocks in image L is less than the quantity threshold, it means that there are not many color blocks in image L. In this case, the electronic device can usually process image L, so filtering can be performed according to the second filtering rule. If the number of color blocks in image L is greater than or equal to the quantity threshold, it means that image L contains a large number of color blocks. In this case, filtering can be omitted, and instead, a prompt message can be displayed to guide the user to take a screenshot of image L again, for example, guiding the user to use the electronic device to capture a portion of image L that includes the schedule information. The quantity threshold can be set according to needs; for example, the quantity threshold can be 10.

[0194] As an example of this application, before filtering, the schedule management service can determine whether there are dense text blocks based on the text block coordinates in the text recognition results of image L. If dense text blocks exist, a prompt message can be displayed in the calendar application to inform the user of the existence of dense text blocks, so that the user can re-crop image L as needed. If no dense text blocks exist, the schedule management service performs the filtering operation.

[0195] Or in another example, during the filtering process, if the schedule management service determines the existence of dense text blocks based on the text block coordinates in the text recognition results of image L, then the text line coordinates and text line recognition content in these text blocks are deleted.

[0196] It's worth noting that without filtering, subsequent target natural language models may be unable to accurately extract schedule information, for example, for... Figure 17 Image p1 in the image might extract the time as a chat timestamp "20:37" and the address as "xx5 xx Road", etc. In this embodiment, after determining the first and second recognition results, filtering out the interfering information in image L can improve the accuracy of subsequent schedule information extraction.

[0197] As an example rather than a limitation, if image L is a photograph, since the photograph may be skewed or have a background, no filtering is required, and the text recognition results can be used for text concatenation later.

[0198] In another example, when image L is a photograph, the text recognition results and edge recognition results of image L can also be filtered according to the second filtering rule. This application embodiment does not limit this.

[0199] S1813: The schedule management service matches the text lines of the remaining chat elements from the filtered text recognition results based on the filtered edge recognition results.

[0200] The schedule management service uses the coordinates of each chat element in the filtered edge recognition results and the coordinates of the text lines in the filtered text recognition results to match the text line recognition content corresponding to each chat element in the filtered text recognition results.

[0201] If image L is not an instant messaging chat screenshot, the schedule management service matches the text line recognition content in each color block of the filtered edge recognition result with the coordinates of each color block in the filtered edge recognition result and the coordinates of the text line in the filtered text recognition result.

[0202] S1814: The schedule management service identifies the scene corresponding to image L based on the text lines of the remaining chat elements and the image category of image L.

[0203] The scene corresponding to image L refers to the scene involved in the text content of image L.

[0204] If we determine that image L is an instant messaging chat screenshot based on its image category, and further identify the content of the text lines of each filtered chat element to determine that it is a chat dialogue, then we can determine that the scenario corresponding to image L is an IM chat scenario.

[0205] In addition, if the image category of image L is other images, such as screenshots of instant messaging notification cards or order screenshots, the schedule management service can also combine the text line recognition content of the matched color blocks to determine the scenario corresponding to image L, such as a service notification card scenario or a high-speed rail travel order scenario.

[0206] S1815: The schedule management service, based on the scene corresponding to image L, performs chat dialogue splicing on the text lines of each filtered chat element to obtain the target text information.

[0207] In implementation, the schedule management service processes the filtered text lines of each chat element according to the scenario corresponding to image L, performing line breaks, carriage returns, and concatenation. Since chat content is often quite flexible—for example, a complete sentence might be sent as multiple messages—if the scenario corresponding to image L is determined to be an IM chat scenario, the schedule management service can concatenate these multiple messages into a single sentence. Therefore, concatenating chat dialogues based on the scenario corresponding to image L makes the concatenated text more closely resemble natural language. As an example, and not a limitation, the target text information after chat dialogue concatenation is consistent with the content displayed on the drawing interface.

[0208] In one example, the concatenated chat conversation includes the conversation type, title, nickname, and conversation content. The nickname can be customized. For instance, the chat conversation can be concatenated in the following format:

[0209] Conversation topic: IM chat

[0210] Title: xx Group Chat

[0211] Nickname A: xxx

[0212] Nickname B: xxx

[0213] Nickname A: xxxx ......

[0214] It should be noted that S1814 to S1816 are optional operations. In another example, the schedule management service can also perform chat dialogue splicing based on the image category of image L and the text line recognition content of each filtered chat element.

[0215] In addition, if image L is another type of image, such as a screenshot of an instant messaging notification card or an order screenshot, the schedule management service will concatenate the text lines recognized in each filtered color block according to the scenario corresponding to image L. The implementation can be seen in the concatenation of chat dialogues.

[0216] S1816: The schedule management service constructs a prompt message based on the scene and target text information corresponding to image L.

[0217] Different image categories correspond to different prompt templates. When constructing prompts, the schedule management service can use the prompt template corresponding to the image category of image L. As an example of this application, the constructed prompt includes scene description information, which is used to indicate the scene corresponding to image L. That is, the schedule management service adds scene description information for image L when constructing the prompt.

[0218] It's worth noting that if no scene description information is added when constructing the prompts, the target natural language processing (NLP) model may extract each time and related verbs as a single schedule entry, leading to multiple schedule entries and inaccurate information extraction. Therefore, to ensure accurate schedule information extraction, the schedule management service adds scene description information to the constructed prompts, enabling the NLP model to extract a single, accurate schedule entry.

[0219] For example, if image L is a screenshot of an instant messaging chat, the prompt message generated by the calendar management service could be:

[0220] <|Human|>The following content may be an IM chat conversation. IM chat conversations contain fields such as <title, time, location, and participants>. Each field present in an event is output as a single line in JSON format, without additional replies. \n\nLet's Talk KaiTan

[0221] HarmonyOS Architecture Evolution and Key Technologies

[0222] HDC Together

[0223] Time: 09:00-16:30, October 23

[0224] Location: XX Group Building

[0225] No. 5, xx Street, xx District

[0226] <|Moss|>

[0227] S1817: The schedule management service sends a prompt to the target natural language model.

[0228] After constructing the prompt, the schedule management service calls the target natural language model and sends the constructed prompt to the target natural language model for recognition to extract schedule information. For example, the schedule management service can request schedule information extraction from the target natural language model using the `extractinformation()` function.

[0229] It should be noted that this application embodiment uses the deployment of the target natural language model in an electronic device as an example for illustration. In another example, the target natural language model can also be deployed in the cloud, and the cloud can provide an interface for the electronic device to call the target natural language model. In this way, when the target natural language model needs to be used, the schedule management service can call the target natural language model through the provided interface. This application embodiment does not limit this.

[0230] S1818: The target natural language model determines schedule information based on prompts.

[0231] In one example, the schedule information output by the target natural language model includes: Title: KaiTan Talk, Start Time: October 23, 09:00, End Time: 16:30, Location: No. 5, xx Street, xx District, xx Group Building, Repetition Period: (None).

[0232] S1819: The target natural language model sends schedule information to the schedule management service.

[0233] For example, the target natural language model can send schedule information to the schedule management service in JSON format.

[0234] S1820: The schedule management service decodes the received schedule information.

[0235] The main purpose of this step is to split the time information in the received schedule information. For details on the implementation, please refer to the previous text.

[0236] S1821: The schedule management service performs post-processing on the schedule fields of the decoded schedule information.

[0237] For post-processing of the schedule field in the schedule information, please refer to the previous text.

[0238] In one example, before creating the schedule information, the schedule management service can also invoke a personal behavior feature model to request a query for the user's historical behavior data, such as historical locations. Accordingly, the personal behavior feature model returns the number of historical behaviors. In this way, the schedule management service can predict where the user might go based on the historical behavior data, and then combine this with the schedule information fed back by the target natural language model to create the final schedule information, such as adding the predicted address information to the schedule.

[0239] S1822: The schedule management service displays the processed schedule information in the calendar application.

[0240] For example, if image L is a screenshot of an instant messaging chat, the electronic device displays calendar information such as... Figure 3D As shown in 13.

[0241] In one example, a confirmation request notification can be displayed before the schedule information is shown. After receiving a confirmation instruction from the user triggered by the confirmation request notification, the schedule management service then displays the schedule information in the calendar application.

[0242] As an example of this application, the electronic device also supports users in editing the displayed schedule information. For example, it supports users in modifying the title of the schedule information, such as see [link to relevant documentation]. Figure 4 The illustrated embodiment. During this process, when a user triggers the display of the target interface on the electronic device, it is necessary to display scribbable text on the target interface. To this end, the schedule management service can also request the target natural language model to perform word combination processing on the concatenated text, for example, merging the words "I" and "we" into "we," so that the text can be displayed on the target interface according to the merged words, thus facilitating user scribbling. In implementation, the schedule management service can call the target natural language model via `getWordSegment()` to request word combination processing. Additionally, the schedule management service can also call the target natural language model via `getWordSegment()` to request the target natural language model to determine the topic of the concatenated text, such as specifying that the target natural language model extract topic entities (e.g., meeting, dinner). In this way, the schedule management service can display this topic on the target interface.

[0243] In this embodiment, in response to a schedule extraction operation, target text information is acquired, including schedule information to be extracted. A pre-trained target natural language model is used to extract the schedule information from the target text information to obtain first schedule information. The target natural language model can extract the schedule information from the text information and merge the time information within the text information. The time information in the first schedule information is then split to obtain second schedule information. The second schedule information includes the split time information and other schedule element information (such as the topic) from the first schedule information, excluding the time information. The second schedule information is then displayed. This eliminates the need for users to manually input schedule information item by item in a calendar application, improving the efficiency of schedule information creation.

[0244] Furthermore, since the target natural language model (NLP) integrates time information, it doesn't need to focus on numerous time-related fields, thus reducing the computational complexity and improving the accuracy of schedule information extraction. Additionally, the shorter length of the schedule information output by the NLP reduces processing time, as each output depends on previous ones, given its iterative reasoning nature. This lower processing time further reduces latency. Moreover, the NLP doesn't need to calculate the result in one step; for example, it only needs to infer the start time and duration. This minimizes the risk of inference errors due to the limited computational capabilities of most language models, improving accuracy. Of course, in one possible implementation, using a model with stronger reasoning capabilities could complete the inference in one step.

[0245] It should also be noted that this embodiment of the application uses text information as the input to the target natural language model for illustration. In another example, the input to the target natural language model can also be an image; that is, an image can be directly input into the target natural language model for processing, so that the target natural language model can extract the schedule information from the image. In this case, the training data used in the training stage of the target natural language model can include image training samples and corresponding schedule information samples.

[0246] The electronic devices involved in the embodiments of this application may be mobile phones, action cameras (GoPro), digital cameras, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, in-vehicle devices, ultra-mobile personal computers (UMPCs), netbooks, etc., and the embodiments of this application do not limit them.

[0247] Figure 21This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 21 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0248] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0249] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0250] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0251] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0252] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0253] The charging management module 140 receives charging input from a charger, which can be either a wireless or wired charger. The power management module 141 connects to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0254] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices through wireless communication technology.

[0255] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0256] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is an integer greater than 1.

[0257] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0258] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0259] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created by electronic device 100 during use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0260] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D and application processor.

[0261] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch display." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0262] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0263] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method of displaying schedule information, characterized by, The method includes: In response to the schedule extraction operation, target text information is obtained, including the schedule information to be extracted; The target text information is processed by a pre-trained target natural language model to extract schedule information, thereby obtaining first schedule information. The target natural language model can extract schedule information from the text information and merge the time information in the text information. The first schedule information includes field values ​​corresponding to at least one of multiple fields. The multiple fields include a title field, a start time field, an end time field, a location field, and a repetition period field. The field value corresponding to the title field is the title. The field value corresponding to the start time field includes one or more of the start date, repetition start date, and start time point. The field value corresponding to the end time field includes one or more of the end date, repetition end date, and end time point. The field value corresponding to the repetition period field is the schedule repetition period. The time information in the first schedule information is split to obtain the second schedule information, which includes the split time information and other schedule elements information in the first schedule information other than the time information. Display the second schedule information.

2. The method of claim 1, wherein, The target text information includes the start time and duration description information. If the end time is not included, the field value corresponding to the end time field also includes the duration description information. The end time in the first schedule information is represented by the start time and duration description information in the target text information. The duration description information is used to describe the duration of the schedule.

3. The method of claim 1 or 2, wherein, The field value corresponding to the end time field of the first schedule information includes duration description information, which is used to describe the duration of the schedule. The step of splitting the time information in the first schedule information to obtain the second schedule information includes: Extract the start date from the field value corresponding to the start time field of the first schedule information and use it as the field value corresponding to the first date field; The end date is extracted from the end time field value of the first schedule information and used as the field value of the second date field. Both the first date field and the second date field are newly added fields and are different. Based on the start time and duration description information in the field value corresponding to the end time field of the first schedule information, determine the field value corresponding to the split end time field.

4. The method of claim 3, wherein, If the first schedule information does not include periodic schedule information, the first date field is the start date field and the second date field is the end date field; if the first schedule information includes periodic schedule information, the first date field is the repeat start date field and the second date field is the repeat end date field.

5. The method of claim 3 or 4, wherein, The step of determining the field value corresponding to the split end time field based on the start time point and the duration description information in the field value corresponding to the end time field of the first schedule information includes: The result of adding the start time point in the field value corresponding to the end time field to the duration described in the duration description information is determined as the field value corresponding to the split end time field.

6. The method of any one of claims 1-5, wherein, The target natural language model is obtained by training an initial natural language understanding model based on a sample training set. The sample training set includes multiple sets of sample training data. Each set of sample training data includes text training samples and schedule information samples corresponding to the text training samples. The time information in the schedule information samples has been merged.

7. The method of any one of claims 1-6, wherein, The response to the schedule retrieval operation, obtaining the target text information, includes: In response to the schedule extraction operation on the first image, a first recognition result and a second recognition result are determined by the target recognition model. The first recognition result includes the text recognition result of the first image, and the second recognition result includes the edge recognition result and image category of the first image. The edge recognition result includes the graphic element attribute information or color block attribute information of the first image. The target recognition model can determine the text recognition result, edge recognition result and image category of any image. Based on the image category of the first image, filter out interference information unrelated to the schedule from the text recognition results and edge recognition results of the first image; The target text information is obtained based on the filtered first recognition result and the filtered second recognition result.

8. The method of claim 7, wherein, The image categories include instant messaging chat screenshots, instant messaging notification card screenshots, order screenshots, and other image categories. Instant messaging chat screenshots refer to images obtained by taking a screenshot of the chat interface in an instant messaging application. Instant messaging notification card screenshots refer to images obtained by taking a screenshot of the service notification card in an instant messaging application. Order screenshots refer to images obtained by taking a screenshot of the order interface in the application. Other image categories include images other than those mentioned above. Wherein, if the first image is a screenshot of an instant messaging chat, the edge recognition result includes the image and text element attribute information; if the first image is one of the instant messaging notification card screenshot, the order screenshot, or other types of images, the edge recognition result includes the color block attribute information.

9. The method of claim 8, wherein, The text recognition result includes text line coordinates; The step of filtering out irrelevant interference information from the text recognition result and edge recognition result of the first image based on the image category includes: If the first image is determined to be an instant messaging chat screenshot based on the image category of the first image, the recognition data corresponding to the skewed text lines in the text recognition result of the first image is filtered out based on the text line coordinates of each text line in the text recognition result of the first image. Based on the text line coordinates of each text line in the text recognition result of the first image, filter out the recognition data corresponding to the text lines whose line height is less than the target line height in the text recognition result of the first image and the edge recognition result. The target line height is the average line height of all text lines in the first image. The recognition data corresponding to the chat timestamp in the first image is filtered out from the text recognition result and the edge recognition result of the first image.

10. The method of claim 9, wherein, The image and text element attribute information in the edge recognition result includes the coordinates and category of the chat element, and the text recognition result also includes the text line recognition content; The step of filtering out the recognition data corresponding to the chat timestamp in the first image from the text recognition result and the edge recognition result of the first image includes: From the edge recognition results, determine the chat element whose category is chat timestamp, and obtain at least one first candidate chat element; Based on the coordinates of each first candidate chat element in the at least one first candidate chat element and the coordinates of the text line in the text recognition result of the first image, the text line recognition content corresponding to each first candidate chat element is matched from the text recognition result of the first image. If the target first candidate chat element is determined to be a chat timestamp based on the text line recognition content corresponding to the target first candidate chat element, it is determined whether to filter out the recognition data corresponding to the target first candidate chat element according to the coordinates of the target first candidate chat element. The target first candidate chat element is any one of the at least one first candidate chat element. If it is determined that the recognition data corresponding to the first candidate chat element of the target should be filtered out, the recognition data corresponding to the first candidate chat element of the target should be filtered out from the text recognition result of the first image and the edge recognition result.

11. The method of claim 10, wherein, The step of determining whether to filter out the recognition data corresponding to the first candidate chat element based on its coordinates includes: If, based on the coordinates of the target first candidate chat element, it is determined that the target first candidate chat element is located in the middle of the first image, and the area corresponding to the target first candidate chat element includes a single line of text, then it is determined that the recognition data corresponding to the target first candidate chat element will be filtered out; or, If the target first candidate chat element is located on the right side of the first image based on its coordinates, and the area corresponding to the target first candidate chat element includes a single text line, then if the text line corresponding to the target first candidate chat element only includes a time point and does not include a date, then the identification data corresponding to the target first candidate chat element is determined to be filtered out.

12. The method of claim 8, wherein, The text recognition result includes text line coordinates and text line recognition content; The step of filtering out irrelevant interference information from the text recognition result and edge recognition result of the first image based on the image category includes: If the first image is determined to be either a screenshot of the instant messaging notification card or a screenshot of the order based on the category of the first image, the text line recognition content in each color block is matched from the text recognition result of the first image based on the coordinates of each color block in the edge recognition result and the coordinates of the text line in the text recognition result of the first image. For any one of the color blocks, if it is determined from the text line recognition content of the color block that the color block does not contain any content related to the schedule, then the recognition data corresponding to the color block is filtered out from the edge recognition result and the text recognition result of the first image.

13. The method of any one of claims 7-12, wherein, The text recognition result includes text line coordinates and text line recognition content; The process of obtaining the target text information based on the filtered first recognition result and the filtered second recognition result includes: Based on the coordinates of each object in the filtered edge recognition results and the coordinates of the text lines in the text recognition results of the first image, the text line recognition content corresponding to each object is matched from the filtered text recognition results of the first image, where the object is a chat element or a color block. Based on the text line recognition content corresponding to each object and the image category of the first image, the scene corresponding to the first image is determined; Based on the scene corresponding to the first image, the text lines corresponding to each object are concatenated to obtain the target text information.

14. The method as described in claim 13, characterized in that, After displaying the second schedule information, the method further includes: In response to an editing operation on the title of the second schedule information, a target interface is displayed, the target interface including the target text information; In response to a selection operation on the text line content displayed in the target interface, the text selected by the selection operation is input into the title input box; In response to the end of the editing operation, the title is modified to the content entered in the title input box.

15. The method according to any one of claims 7-14, characterized in that, The step of extracting the schedule from the target text information using a pre-trained target natural language model to obtain first schedule information includes: Based on the target text information and the scene corresponding to the first image, a prompt is constructed. The prompt includes scene description information, which describes the scene corresponding to the first image. The scene corresponding to the first image is determined based on the first recognition result and the second recognition result. The prompt is input into the target natural language model for processing to obtain the first schedule information.

16. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-15.

Images