Schedule management method and device, equipment and storage medium

Abstract

The invention provides a schedule management method and device, equipment and a storage medium, and the method comprises the steps: responding to a schedule management request triggered by a target terminal, and collecting multi-modal data corresponding to the target terminal; performing schedule analysis on the multi-modal data by using a schedule extraction model, and determining target schedule information in the multi-modal data; and generating a target schedule event according to the schedule time information and the schedule event information in the target schedule information. The target reminding schedule can be intelligently created without manual input of a user, so that the operation burden of the user is reduced, the user is prevented from missing important schedules, the intelligent degree of schedule events is improved, and the schedule reminding experience of the user is optimized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a schedule management method, device, equipment and storage medium. Background Art

[0002] At present, with the rapid development of Internet technology and smart terminals, a variety of schedule management applications have appeared on the market, based on which users can manage their schedules and make appointments and reminders. However, existing schedule management applications have limitations, relying on users to manually input and set reminders to manage schedules, which seriously increases the user's operation burden, and this input method is prone to omission of important schedules due to negligence, and cannot meet the user's schedule management needs. Summary of the invention

[0003] The embodiments of the present application provide a schedule management method, apparatus, device and storage medium, which are intended to solve the technical problems in the prior art that schedule management applications have cumbersome input and low intelligence.

[0004] On the one hand, an embodiment of the present application provides a schedule management method, the schedule management method comprising the following steps:

[0005] In response to a schedule management request triggered by a target terminal, collect multimodal data corresponding to the target terminal;

[0006] Performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0007] A target schedule event is generated according to the schedule time information and schedule event information in the target schedule information.

[0008] In a possible implementation of the present application, the collecting multimodal data corresponding to the target terminal includes:

[0009] In response to a multimodal data collection request for the target terminal, screen capture is performed on the target terminal to obtain display image data of the target terminal;

[0010] Using a preset audio collection module to collect audio from the target terminal to obtain terminal audio data of the target terminal;

[0011] The display image data and the terminal audio data are preprocessed to obtain multimodal data of the target terminal.

[0012] In a possible implementation of the present application, performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data includes:

[0013] Using a schedule extraction model to extract features from the multimodal data to obtain multimodal features;

[0014] Performing semantic recognition on the multimodal features to obtain multimodal semantic information contained in the multimodal features;

[0015] Performing schedule extraction on the multimodal semantic information to obtain target schedule information in the multimodal semantic information.

[0016] In a possible implementation of the present application, the step of extracting features from the multimodal data using a schedule extraction model to obtain multimodal features includes:

[0017] Acquire display image data and terminal audio data in the multimodal data;

[0018] Performing text detection on the displayed image data, locating the image text area of ​​the displayed image data, and performing feature extraction on the image text data in the image text area using the first feature module in the schedule extraction model to obtain image text features;

[0019] Using the second feature module in the schedule extraction model to extract audio vectors from the terminal audio data to obtain terminal audio features;

[0020] The image text features and the terminal audio features are aggregated to obtain multimodal features.

[0021] In a possible implementation of the present application, performing schedule extraction on the multimodal semantic information to obtain target schedule information in the multimodal semantic information includes:

[0022] Classifying the multimodal semantic information to obtain an information type corresponding to each piece of the multimodal semantic information;

[0023] If the information type is the same as the preset schedule information type, extracting schedule event information and schedule time information from the multimodal semantic information corresponding to the information type;

[0024] The schedule event information and the schedule time information are determined as target schedule information of the target terminal.

[0025] In a possible implementation of the present application, generating a target schedule event according to the schedule time information and schedule event information in the target schedule information includes:

[0026] Acquire schedule time information and schedule event information in the target schedule information;

[0027] Generating structured schedule data according to the schedule time information, the schedule event information and a preset schedule event template;

[0028] The structured schedule data is imported into a target schedule database corresponding to the target terminal to generate a target schedule event.

[0029] In a possible implementation of the present application, after generating the target schedule event according to the schedule time information and the schedule event information in the target schedule information, the method further includes:

[0030] Obtaining the schedule reminder time and schedule reminder method corresponding to the target schedule event;

[0031] If the terminal system time of the target terminal is equal to the schedule reminder time, the target terminal is driven to output schedule reminder information corresponding to the target schedule event according to the schedule reminder mode.

[0032] On the other hand, the present application provides a schedule management device, the schedule management device comprising:

[0033] A data collection module is configured to respond to a schedule management request triggered by a target terminal and collect multimodal data corresponding to the target terminal;

[0034] A data analysis module, configured to perform schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0035] The schedule generating module is configured to generate a target schedule event according to the schedule time information and schedule event information in the target schedule information.

[0036] On the other hand, the present application also provides a schedule management device, the schedule management device comprising:

[0037] one or more processors;

[0038] Memory; and

[0039] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps of the schedule management method.

[0040] On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the schedule management method.

[0041] In this application, by responding to the schedule management request triggered by the target terminal, the multimodal data corresponding to the target terminal is collected; the schedule extraction model is used to perform schedule analysis on the multimodal data to determine the target schedule information in the multimodal data; and the target schedule event is generated according to the schedule time information and schedule event information in the target schedule information. The multimodal data of the target terminal is collected when the target terminal is running, and the schedule extraction model is used to perform schedule analysis on the multimodal data, so as to determine the target schedule information contained in the multimodal data, and automatically generate the target schedule event according to the schedule time information and schedule event information in the target schedule information. In this way, the target reminder schedule can be created without manual input by the user, thereby reducing the user's operating burden, thereby preventing the user from missing important schedules, improving the intelligence of schedule events, and optimizing the user's schedule reminder experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A schematic diagram of a scenario of a schedule management method according to an embodiment of the present application;

[0044] Figure 2 A flowchart of an embodiment of a schedule management method in an embodiment of the present application;

[0045] Figure 3 A flowchart of an embodiment of extracting target schedule information in a schedule management method provided in an embodiment of the present application;

[0046] Figure 4 A flowchart of an embodiment of performing a schedule reminder in the schedule management method provided in the embodiments of the present application;

[0047] Figure 5 A schematic diagram of the structure of an embodiment of a schedule management device provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the structure of an embodiment of a schedule management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0051] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0052] At present, with the rapid development of Internet technology and smart terminals, a variety of schedule management applications have appeared on the market, based on which users can manage their schedules and make appointments and reminders. However, existing schedule management applications have limitations, relying on users to manually input and set reminders to manage schedules, which seriously increases the user's operation burden, and this input method is prone to omission of important schedules due to negligence, and cannot meet the user's schedule management needs.

[0053] Based on this, the present application proposes a schedule management method, device, equipment and computer-readable storage medium to solve the technical problems in the prior art that the schedule management application input is cumbersome and the intelligence level is low.

[0054] The schedule management method in the embodiment of the present invention is applied to a schedule management device, which is arranged in the schedule management device. The schedule management device is provided with one or more processors, memories, and one or more applications, wherein the one or more applications are stored in the memories and are configured to be executed by the processor to implement the schedule management method; wherein the schedule management device can be an intelligent terminal, such as a mobile phone, a tablet computer, a network device, and a smart computer; optionally, the schedule management device can also be a server, or a service cluster composed of multiple servers.

[0055] like Figure 1 As shown, Figure 1 The following is a schematic diagram of a scenario of the schedule management method of an embodiment of the present application. In the embodiment of the present invention, the schedule management scenario includes a schedule management device 100 (a schedule management device is integrated in the schedule management device 100) and a target terminal 200. A computer-readable storage medium corresponding to the schedule management method is run in the schedule management device 100 to execute the steps of the schedule management method. The target terminal 200 is an intelligent terminal that is connected to the schedule management device 100 and can generate target schedule events. Optionally, in a specific embodiment, the schedule management device 100 and the target terminal 200 can be the same intelligent terminal. In other embodiments, the schedule management device 100 and the target terminal 200 can be different combinations of intelligent terminals. For example, in a specific embodiment, the schedule management device 100 can be a wearable intelligent device that is connected to the target terminal 200, and the target terminal can be an intelligent terminal such as a mobile phone, a tablet computer, a smart TV, and a smart computer.

[0056] Understandably, Figure 1 The schedule management device in the schedule management method scenario shown, or the devices included in the schedule management device, does not constitute a limitation on the embodiments of the present invention, that is, the number of devices and types of devices of the schedule management device included in the scenario of the schedule management method, or the number of devices and types of devices included in each device, do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected in the embodiments of the present invention.

[0057] The schedule management device 100 in the embodiment of the present invention is mainly used to: respond to a schedule management request triggered by a target terminal and collect multimodal data corresponding to the target terminal;

[0058] Performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0059] A target schedule event is generated according to the schedule time information and schedule event information in the target schedule information.

[0060] The schedule management device 100 in the embodiment of the present invention can be an independent schedule management device, such as a smart terminal such as a mobile phone, a tablet computer, a network device, a server, and a smart computer, or it can be a schedule management network or schedule management cluster composed of multiple schedule management devices.

[0061] The embodiments of the present application provide a schedule management method, apparatus, device, and computer-readable storage medium, which are described in detail below.

[0062] It can be understood by those skilled in the art that Figure 1 The application environment shown in is only one of the application scenarios related to the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer schedule management devices, or schedule management network connection relationships shown, such as Figure 1 Only one schedule management device is shown. It can be understood that the scenario of the schedule management method can also include one or more schedule management devices, which is not specifically limited here. The schedule management device 100 can also include a memory for storing multimodal data and other data.

[0063] It should be noted that Figure 1 The scenario diagram of the schedule management method shown is only an example. The scenario of the schedule management method described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.

[0064] Based on the scenario of the above schedule management method, various embodiments of the schedule management method disclosed in the present invention are proposed.

[0065] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a schedule management method in an embodiment of the present application. The schedule management method includes the following steps 201 to 203:

[0066] 201. Responding to a schedule management request triggered by a target terminal, collecting multimodal data corresponding to the target terminal;

[0067] The schedule management method in this embodiment is applied to a schedule management device, and the type and quantity of the schedule management device are not specifically limited. That is, the schedule management device can be one or more intelligent terminals or servers equipped with a schedule extraction model. In a specific embodiment, the schedule management device is a smart phone.

[0068] Specifically, during operation, the schedule management device responds to a schedule management request triggered by a target terminal, wherein the schedule management request is an operation request for automatically creating a target schedule event based on multimodal data of the user in the process of using the target terminal or the schedule management device, after the user agrees. The schedule management request can be triggered by the user actively, for example, after the user agrees to the data collection protocol of the schedule management application in the schedule management application in the schedule management device or the target terminal, the schedule management request is actively triggered by clicking the automatic schedule management button in the schedule management application. In addition, the schedule management request can also be triggered by the device actively, for example, after the user agrees to the data collection protocol of the schedule management application, the schedule management request is automatically triggered by the schedule management process pre-set in the schedule management device or the target terminal.

[0069] The schedule management device and the target terminal may be independent different intelligent terminals, for example, the schedule management device and the target terminal may be the user's smart phone and intelligent wearable device (such as smart glasses and smart watches, etc.) respectively. In addition, in other embodiments, the schedule management device and the intelligent terminal may be the same intelligent device.

[0070] Specifically, after obtaining the schedule management request triggered by the target terminal, the schedule management device also collects the multimodal data corresponding to the target terminal. Among them, the multimodal data is the terminal multimodal data that represents the target terminal collected by the corresponding module during operation, which may include the user's schedule planning. For example, the multimodal data can be the target terminal display image data or the terminal audio data output by the target terminal speaker. Among them, the display image data is real-time image data representing the content of the target terminal display screen. The terminal audio data is the audio information that represents the real-time output of the target terminal speaker or receiver or the real-time input of the microphone. The schedule management device can extract the schedule according to the multimodal data in the subsequent steps, thereby automatically obtaining the schedule information in the multimodal data.

[0071] Specifically, after responding to the schedule management request and collecting usage data with the user's permission, the schedule management device also triggers a multimodal data collection request for the target terminal, and then performs multimodal collection of the real-time operation data of the target terminal according to the multimodal data collection request, thereby obtaining the multimodal data of the target terminal.

[0072] Specifically, after obtaining the multimodal data collection request of the target terminal, the schedule management device also performs screen capture on the target terminal to obtain the display image data of the target terminal. That is, the device sets a background screen collection process in the target terminal, and uses the background screen collection process to collect images of the display screen in the target, thereby obtaining the real-time picture of the display screen of the target terminal and obtaining the display image data of the target terminal. The display image data may be a real-time picture image obtained by collecting the display screen of the target terminal at a specific collection frequency; in addition, the display image data may also be a real-time picture video obtained by recording the display screen of the target terminal in the background.

[0073] Specifically, after obtaining the multimodal data collection request of the target terminal, the schedule management device also uses the preset audio collection module to collect audio from the target terminal to obtain the terminal audio data of the target terminal. That is, when the user makes a voice call or performs a human-computer dialogue operation with an application such as a voice assistant, the preset audio collection module is used to collect audio from the target terminal to obtain the terminal audio data of the target terminal. In a specific embodiment, the preset audio collection module can be a microphone component corresponding to the target terminal.

[0074] Specifically, after obtaining the display image data and terminal audio data collected by the target terminal, which may carry the user's schedule planning information, the schedule management device also pre-processes the display image data and terminal audio data, thereby denoising and segmenting the display image data and terminal audio data, thereby obtaining multimodal data suitable for data processing by the schedule extraction model.

[0075] 202. Performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0076] Specifically, after acquiring the multimodal data of the target terminal, the schedule management device also uses the schedule extraction model to perform schedule analysis on the multimodal data, thereby extracting the target schedule information contained in the multimodal data. The target schedule information is the schedule plan and arrangement information that represents the user at a certain time node or time period in the future. For example, in a specific embodiment, the target schedule information may be the user's business trip plan information one week later. In another specific embodiment, the target schedule information may be a memo reminder event for the user at a certain time point tomorrow.

[0077] Among them, the schedule extraction model is a large model that performs feature extraction and semantic analysis on multimodal data to identify the target schedule information in the multimodal data to protect you. Among them, the large model refers to an artificial neural network model with a very large number of parameters. In the field of artificial intelligence, a large model generally refers to a model with hundreds of millions to trillions of parameters. The model usually needs to be trained on a large-scale data set and requires a large amount of computing resources to be optimized and adjusted. Large models are generally used to solve complex tasks such as natural language processing, computer vision, and speech recognition. In a specific embodiment, the schedule extraction model can be ChatGPT, BERT, XLNet, Zhipu model, C l aude, Moonshot AI model, ChatGLM model, Qianyi Tongwen model, Mini Max model, Spark model, L l ama model, 360GPT model, Qwen model, Baichuan model, Skylark model, Vi voLM model and Wenxin Yiyan and other scale language models, which are not limited in the embodiments of this application.

[0078] Specifically, after acquiring the multimodal data, the schedule management device inputs the multimodal data into the schedule extraction model, uses the schedule extraction model to perform feature extraction and semantic recognition on the multimodal data, thereby obtaining the multimodal semantic information, and performs schedule extraction on the multimodal semantic information, thereby determining whether the multimodal semantic information includes target schedule information of the schedule information type, and then extracting the target schedule information.

[0079] Specifically, after acquiring the multimodal data, the schedule management device extracts the preprocessed display image data and the preprocessed terminal audio data from the multimodal data, and performs feature extraction and semantic recognition on the display image data and the terminal audio data to obtain multimodal semantic information.

[0080] Specifically, the schedule management device uses the schedule extraction model to perform text detection on the displayed image data, thereby determining the text area containing text information in the displayed image data, and then locating the image text area of ​​the displayed image data, and generating a text detection box mark in the image text area. Then, the first feature extraction module in the schedule extraction model is used to perform feature extraction on the image text data in the image text area corresponding to the text detection box mark to obtain image text features. That is, the schedule management device uses the first feature extraction module to convert the text information in the image text area into a text sequence, wherein the text sequence is a text string. In addition, the first feature extraction model also uses a natural language processing method to extract text features from the text sequence, thereby obtaining image text features.

[0081] Specifically, the first feature extraction module performs text embedding processing on the text sequence, that is, converting the text sequence into a high-dimensional vector representation, thereby obtaining the image text features corresponding to the text sequence. The first feature extraction module can be a model combination including a text detection module, a text recognition module and a text feature extraction module.

[0082] Specifically, the schedule management device also uses the second feature extraction module in the schedule extraction model to extract the audio vector of the terminal audio data, thereby obtaining the terminal audio features corresponding to the terminal audio data. For example, in a specific embodiment, the schedule management model can use the second feature extraction module to extract the Mel spectrum coefficients corresponding to the terminal audio data or the speech feature vector corresponding to the terminal language data, thereby obtaining the terminal audio features.

[0083] Specifically, after acquiring the image text features and the terminal audio features, the schedule management device aggregates the image text features and the terminal audio features to obtain multimodal features, and performs speech recognition on the multimodal features to obtain multimodal semantic information contained in the multimodal features.

[0084] Specifically, after acquiring the multimodal features, the schedule management device uses the schedule extraction model to perform text semantic recognition on the image text features in the multimodal features to obtain image text information. The schedule extraction model is also used to perform text conversion on the terminal audio features in the multimodal features to obtain the terminal audio text corresponding to the terminal audio features, and the terminal audio text is subjected to text semantic recognition to obtain audio text information.

[0085] Specifically, after acquiring the image text information and the audio text information, the schedule management device sets the image text information and the audio text information as multimodal semantic information, and performs schedule extraction on the multimodal semantic information to obtain the target schedule information in the multimodal semantic information.

[0086] Optionally, in a specific embodiment, the schedule management device can identify the information type corresponding to the multimodal semantic information through the schedule extraction model, and identify the target schedule information according to the information type.

[0087] 203. Generate a target schedule event according to the schedule time information and schedule event information in the target schedule information.

[0088] Specifically, after acquiring the target schedule information, the schedule management device further generates a target schedule event according to the schedule time information and schedule event information in the target schedule information.

[0089] Specifically, the schedule management device is pre-set with a target schedule database for creating target schedule events, and a preset schedule event template for creating structured schedule data. After obtaining the target schedule information, the schedule management device also obtains the schedule time information and schedule event information in the target schedule information, and generates structured schedule data according to the schedule time information, schedule event information and the preset schedule event template. That is, the schedule management device associates the schedule event information and schedule time information into the preset schedule event template according to the corresponding rules of the preset schedule event template to generate structured schedule data.

[0090] Specifically, after acquiring the structured schedule data, the schedule management device imports the structured schedule data into the target schedule database corresponding to the schedule management application to generate the target schedule event, thereby automatically creating the target schedule event required by the user without manual input by the user.

[0091] Specifically, when the target schedule event is triggered, the schedule management device also reminds the user according to the reminder policy corresponding to the target schedule event, thereby preventing the user from missing important schedules.

[0092] In this embodiment, the schedule management device collects multimodal data corresponding to the target terminal by responding to the schedule management request triggered by the target terminal; performs schedule analysis on the multimodal data using the schedule extraction model to determine the target schedule information in the multimodal data; and generates a target schedule event based on the schedule time information and schedule event information in the target schedule information. The multimodal data of the target terminal is collected when the target terminal is running, and the schedule extraction model is used to perform schedule analysis on the multimodal data, thereby determining the target schedule information contained in the multimodal data, and automatically generating a target schedule event based on the schedule time information and schedule event information in the target schedule information. In this way, the target reminder schedule can be created without manual input by the user, thereby reducing the user's operating burden, thereby preventing the user from missing important schedules, improving the intelligence of schedule events, and optimizing the user's schedule reminder experience.

[0093] like Figure 3 As shown, Figure 3 This is a flow chart of an embodiment of extracting target schedule information in the schedule management method provided in the embodiment of the present application. Specifically, in this embodiment, the schedule management method further includes steps 301 to 303:

[0094] 301. Classify the multimodal semantic information to obtain an information type corresponding to each multimodal semantic information;

[0095] 302. If the information type is the same as the preset schedule information type, extracting schedule event information and schedule time information from the multimodal semantic information corresponding to the information type;

[0096] 303. Determine the schedule event information and schedule time information as target schedule information of the target terminal.

[0097] Based on the above embodiments, in this embodiment, after acquiring the multimodal data, the schedule management device inputs the multimodal data into the schedule extraction model, uses the schedule extraction model to perform feature extraction and semantic recognition on the multimodal data, thereby obtaining the multimodal semantic information, and performs schedule extraction on the multimodal semantic information, thereby determining whether the multimodal semantic information includes target schedule information of the schedule information type, and then extracting the target schedule information.

[0098] Specifically, after acquiring the multimodal semantic information, the schedule management device also classifies the multimodal semantic information to obtain the information type corresponding to each multimodal semantic information, wherein the information type includes a schedule information type and a non-schedule information type.

[0099] Specifically, after obtaining the multimodal semantic information, the schedule management device obtains the time semantic information contained in the multimodal semantic information, and compares the time semantic information with the current system time. If the event semantic information is later than the current system time, the time semantic information is determined as schedule time information, and the event semantic information associated with the schedule time information in the multimodal semantic information is obtained, the event semantic information is determined as schedule event information, and the information type of the schedule time information and the schedule event information is determined as the schedule information type.

[0100] Specifically, after acquiring the information type corresponding to each multimodal semantic information, the schedule management device compares the information type with the schedule information type, thereby determining the target schedule information in the multimodal semantic information.

[0101] Optionally, if the information type is the same as a preset schedule information type, the schedule event information and schedule time information in the multimodal semantic information corresponding to the information type are extracted, and the schedule event information and schedule time information are determined as the target schedule information of the target terminal.

[0102] Specifically, after acquiring the target schedule information, the schedule management device generates a target schedule event according to the schedule time information and schedule event information in the target schedule information, and performs a schedule reminder according to the target schedule event.

[0103] In this embodiment, the schedule management device classifies the multimodal semantic information to obtain the information type corresponding to each multimodal semantic information; if the information type is the same as the preset schedule information type, the schedule event information and schedule time information in the multimodal semantic information corresponding to the information type are extracted; and the schedule event information and schedule time information are determined as the target schedule information of the target terminal. This realizes automatic identification of the user's target schedule information and creation of the corresponding target schedule event without the user manually inputting the schedule information into the schedule management program, thereby reducing the user's schedule management burden.

[0104] like Figure 4 As shown, Figure 4 This is a flow chart of an embodiment of performing a schedule reminder in the schedule management method provided in the embodiment of the present application. Specifically, in this embodiment, the schedule management method further includes steps 401 to 402:

[0105] 401. Obtaining a schedule reminder time and a schedule reminder method corresponding to the target schedule event;

[0106] 402. If the terminal system time of the target terminal is equal to the schedule reminder time, drive the target terminal to output schedule reminder information corresponding to the target schedule event according to the schedule reminder method.

[0107] Based on the above embodiment, in this embodiment, after the schedule management device creates a target schedule event according to the multimodal data, it also obtains the schedule reminder time and schedule reminder method corresponding to the target schedule event.

[0108] Specifically, the schedule management device pre-sets different schedule reminder methods according to the target event information and / or schedule reminder object corresponding to the target schedule event. For example, the schedule reminder method may include a combination of reminder methods such as voice assistant voice reminder, vibration reminder and system notification. Different target event information and schedule reminder objects correspond to different schedule reminder methods. The higher the importance, and when the schedule reminder object is an elderly person, the reminder perception of the corresponding schedule reminder method is stronger, thereby preventing users from missing important schedules. For example, if the target event information with a higher degree of importance or the schedule reminder object is an elderly person, the reminder method corresponding to the target schedule event is a combination of voice reminder and vibration reminder.

[0109] Specifically, after obtaining the schedule reminder time and the schedule reminder method, the schedule management device compares the schedule reminder time with the terminal system time. If the terminal system time is equal to the schedule reminder time, the target terminal is driven or the target management device is used to output the schedule reminder information corresponding to the target schedule event according to the schedule reminder method. In a specific embodiment, the schedule reminder time can be set to the schedule time information of the target schedule event or slightly earlier than the schedule time information.

[0110] In this embodiment, the schedule management device obtains the schedule reminder time and schedule reminder method corresponding to the target schedule event; if the terminal system time of the target terminal is equal to the schedule reminder time, the target terminal is driven to output the schedule reminder information corresponding to the target schedule event according to the schedule reminder method, so as to realize personalized schedule reminder for the user and prevent the user from missing important schedules.

[0111] In order to better implement the schedule management method in the embodiment of the present application, based on the schedule management method, the embodiment of the present application also provides a schedule management device, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a schedule management device provided in an embodiment of the present application. Specifically, the schedule management device 500 includes:

[0112] The data collection module 501 is configured to respond to a schedule management request triggered by a target terminal and collect multimodal data corresponding to the target terminal;

[0113] The data analysis module 502 is configured to perform schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0114] The schedule generating module 503 is configured to generate a target schedule event according to the schedule time information and schedule event information in the target schedule information.

[0115] In a possible implementation of this embodiment, the schedule management device collects multimodal data corresponding to the target terminal, including:

[0116] In response to a multimodal data collection request for the target terminal, screen capture is performed on the target terminal to obtain display image data of the target terminal;

[0117] Using a preset audio collection module to collect audio from the target terminal to obtain terminal audio data of the target terminal;

[0118] The display image data and the terminal audio data are preprocessed to obtain multimodal data of the target terminal.

[0119] In a possible implementation of this embodiment, the schedule management device uses a schedule extraction model to perform schedule analysis on the multimodal data to determine target schedule information in the multimodal data, including:

[0120] Using a schedule extraction model to extract features from the multimodal data to obtain multimodal features;

[0121] Performing semantic recognition on the multimodal features to obtain multimodal semantic information contained in the multimodal features;

[0122] Performing schedule extraction on the multimodal semantic information to obtain target schedule information in the multimodal semantic information.

[0123] In a possible implementation of this embodiment, the schedule management device uses the schedule extraction model to extract features from the multimodal data to obtain multimodal features, including:

[0124] Acquire display image data and terminal audio data in the multimodal data;

[0125] Performing text detection on the displayed image data, locating the image text area of ​​the displayed image data, and performing feature extraction on the image text data in the image text area using the first feature module in the schedule extraction model to obtain image text features;

[0126] Using the second feature module in the schedule extraction model to extract audio vectors from the terminal audio data to obtain terminal audio features;

[0127] The image text features and the terminal audio features are aggregated to obtain multimodal features.

[0128] In a possible implementation of this embodiment, the schedule management device performs schedule extraction on the multimodal semantic information to obtain target schedule information in the multimodal semantic information, including:

[0129] Classifying the multimodal semantic information to obtain an information type corresponding to each piece of the multimodal semantic information;

[0130] If the information type is the same as the preset schedule information type, extracting schedule event information and schedule time information from the multimodal semantic information corresponding to the information type;

[0131] The schedule event information and the schedule time information are determined as target schedule information of the target terminal.

[0132] In a possible implementation of this embodiment, the schedule management device generates a target schedule event according to the schedule time information and schedule event information in the target schedule information, including:

[0133] Acquire schedule time information and schedule event information in the target schedule information;

[0134] Generating structured schedule data according to the schedule time information, the schedule event information and a preset schedule event template;

[0135] The structured schedule data is imported into a target schedule database corresponding to the target terminal to generate a target schedule event.

[0136] In a possible implementation of this embodiment, after the schedule management device generates the target schedule event according to the schedule time information and the schedule event information in the target schedule information, the method further includes:

[0137] Obtaining the schedule reminder time and schedule reminder method corresponding to the target schedule event;

[0138] If the terminal system time of the target terminal is equal to the schedule reminder time, the target terminal is driven to output schedule reminder information corresponding to the target schedule event according to the schedule reminder mode.

[0139] In this embodiment, the schedule management device collects multimodal data corresponding to the target terminal by responding to the schedule management request triggered by the target terminal; performs schedule analysis on the multimodal data using the schedule extraction model to determine the target schedule information in the multimodal data; and generates a target schedule event according to the schedule time information and schedule event information in the target schedule information. The multimodal data of the target terminal is collected when the target terminal is running, and the schedule extraction model is used to perform schedule analysis on the multimodal data, thereby determining the target schedule information contained in the multimodal data, and automatically generating a target schedule event according to the schedule time information and schedule event information in the target schedule information. In this way, the target reminder schedule can be created without manual input by the user, thereby reducing the user's operating burden, thereby preventing the user from missing important schedules, improving the intelligence of schedule events, and optimizing the user's schedule reminder experience.

[0140] The embodiment of the present invention also provides a schedule management device, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of an embodiment of the schedule management device provided in the embodiments of the present application.

[0141] The schedule management device integrates any schedule management device provided by the embodiments of the present invention, and the schedule management device includes:

[0142] one or more processors;

[0143] Memory; and

[0144] One or more applications, wherein the one or more applications are stored in the memory and are configured so that the processor executes the steps of the schedule management method described in any of the above schedule management method embodiments.

[0145] Specifically, the schedule management device may include one or more processors 601 of processing cores, one or more computer-readable storage media memories 602, a power supply 603, an input unit 604 and other components. Those skilled in the art will appreciate that Figure 6 The structure of the schedule management device shown in the figure does not constitute a limitation on the schedule management device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0146] The processor 601 is the control center of the schedule management device. It uses various interfaces and lines to connect various parts of the entire schedule management device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it executes various functions of the schedule management device and processes data, thereby monitoring the schedule management device as a whole. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0147] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the schedule management device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0148] The schedule management device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 603 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0149] The schedule management device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0150] Although not shown, the schedule management device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 601 in the schedule management device will load the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 will run the application stored in the memory 602 to achieve various functions, as follows:

[0151] In response to a schedule management request triggered by a target terminal, collect multimodal data corresponding to the target terminal;

[0152] Performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0153] A target schedule event is generated according to the schedule time information and schedule event information in the target schedule information.

[0154] To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any schedule management method provided in an embodiment of the present invention. For example, the computer program can be loaded by a processor to execute the following steps:

[0155] In response to a schedule management request triggered by a target terminal, collect multimodal data corresponding to the target terminal;

[0156] Performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data;

[0157] A target schedule event is generated according to the schedule time information and schedule event information in the target schedule information.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0159] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0160] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0161] The above is a detailed introduction to a schedule management method provided in an embodiment of the present application. Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A schedule management method, characterized in that: The schedule management method comprises: In response to a schedule management request triggered by a target terminal, collect multimodal data corresponding to the target terminal; Performing schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data; A target schedule event is generated according to the schedule time information and schedule event information in the target schedule information.

2. The schedule management method according to claim 1, characterized in that: The collecting multimodal data corresponding to the target terminal includes: In response to a multimodal data collection request for the target terminal, screen capture is performed on the target terminal to obtain display image data of the target terminal; Using a preset audio collection module to collect audio from the target terminal to obtain terminal audio data of the target terminal; The display image data and the terminal audio data are preprocessed to obtain multimodal data of the target terminal.

3. The schedule management method according to claim 1, characterized in that: The using the schedule extraction model to perform schedule analysis on the multimodal data to determine target schedule information in the multimodal data includes: Using a schedule extraction model to extract features from the multimodal data to obtain multimodal features; Performing semantic recognition on the multimodal features to obtain multimodal semantic information contained in the multimodal features; Performing schedule extraction on the multimodal semantic information to obtain target schedule information in the multimodal semantic information.

4. The schedule management method according to claim 3, characterized in that: The using of the schedule extraction model to extract features from the multimodal data to obtain multimodal features includes: Acquiring display image data and terminal audio data in the multimodal data; Performing text detection on the displayed image data, locating the image text area of ​​the displayed image data, and performing feature extraction on the image text data in the image text area using the first feature module in the schedule extraction model to obtain image text features; Using the second feature module in the schedule extraction model to extract audio vectors from the terminal audio data to obtain terminal audio features; The image text features and the terminal audio features are aggregated to obtain multimodal features.

5. The schedule management method according to claim 4, characterized in that: The extracting schedule from the multimodal semantic information to obtain target schedule information in the multimodal semantic information includes: Classifying the multimodal semantic information to obtain an information type corresponding to each piece of the multimodal semantic information; If the information type is the same as the preset schedule information type, extracting schedule event information and schedule time information from the multimodal semantic information corresponding to the information type; The schedule event information and the schedule time information are determined as target schedule information of the target terminal.

6. The schedule management method according to any one of claims 1 to 5, characterized in that: The generating a target schedule event according to the schedule time information and schedule event information in the target schedule information includes: Acquire schedule time information and schedule event information in the target schedule information; Generating structured schedule data according to the schedule time information, the schedule event information and a preset schedule event template; The structured schedule data is imported into a target schedule database corresponding to the target terminal to generate a target schedule event.

7. The schedule management method according to claim 6, characterized in that: After generating the target schedule event according to the schedule time information and schedule event information in the target schedule information, the method further includes: Obtaining the schedule reminder time and schedule reminder method corresponding to the target schedule event; If the terminal system time of the target terminal is equal to the schedule reminder time, the target terminal is driven to output schedule reminder information corresponding to the target schedule event according to the schedule reminder mode.

8. A schedule management device, characterized in that: The schedule management device comprises: A data collection module is configured to respond to a schedule management request triggered by a target terminal and collect multimodal data corresponding to the target terminal; A data analysis module, configured to perform schedule analysis on the multimodal data using a schedule extraction model to determine target schedule information in the multimodal data; The schedule generating module is configured to generate a target schedule event according to the schedule time information and schedule event information in the target schedule information.

9. A schedule management device, characterized in that: The schedule management device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to implement the steps of the schedule management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps of the schedule management method according to any one of claims 1 to 7.

Images