METHOD, COMPUTER PROGRAM, STORAGE MEDIUM AND DEVICE FOR BUILDING A KNOWLEDGE BASE FOR USER INTERFACES

Description

1. TECHNICAL FIELD

[0001] The field of the invention is that of assisting terminal users.

[0002] More specifically, the invention relates to a solution for building a knowledge base for a user with at least one terminal having a screen.

[0003] Once built (in the sense that it contains collected data), the knowledge base can, for example, be used by an intelligent assistant configured to offer contextual help to the user based in particular on the content of the knowledge base, and therefore based on the user's usage on their (at least one) terminal.

[0004] The term “terminal” includes, but is not limited to, a personal computer (desktop or laptop), a digital tablet, a personal digital assistant, a smartphone, a workstation, etc., or any other device that a user can use to receive, send or search for text and / or image and / or sound and / or video content.

[0005] By "content" we mean in particular, but not exclusively, an email, a message (instant or otherwise), a document, a search (carried out for example with a web browser), a social network news feed, content published on a social network, etc.

[0006] The invention can be applied in many fields, for example in the field of companies that wish to offer innovative services to their employees or customers and help them on a daily basis in several areas (productivity, well-being and ecology) via professional or personal assistants (in B2B (business to business) or B2C (business to consumer)).

[0007] The invention can also be applied in other fields: education (for pupils and students), personal development (for any user), etc. 2. TECHNOLOGICAL BACKGROUND

[0008] Many intelligent assistants aim to help users on their devices (PCs, smartphones, tablets, etc.) by offering proactive, contextual assistance after analyzing the various activities the user has already performed on the applications installed on the device. The data collected and stored in the knowledge base is used, for example, by the intelligent agent to detect repetitions in activities, which could then trigger a notification to the user to assist them or suggest task automation.

[0009] This is especially true now in the world of digital business, which aims to offer intelligent assistants that continuously monitor employees and assist them in improving their productivity, well-being, or environmental impact through energy and digital—and therefore physical—resource conservation. For example, the intelligent assistant in the Gmail application offers features such as automatic completion and drafting of email replies.

[0010] Current smart assistants are integrated into applications. Consequently, each smart assistant integrates, or cooperates with, its own data collection mechanism to create its own knowledge base.

[0011] One drawback is that each knowledge base is linked to an application running on the terminal, and therefore it is necessary to have as many knowledge bases as there are applications.

[0012] Another drawback is that data collection mechanisms need to be updated frequently because applications evolve rapidly. For example, if an application with its own chat system (internal instant messaging) is updated, the module (collection mechanism) that interfaces with this application to obtain the information exchanged via the chat system (i.e., the user's exchanges with other people via the application's internal instant messaging) must also be updated.

[0013] Another drawback is that, for each application, the data collection mechanism is different and must interface with that application's specific APIs (Application Programming Interfaces) (assuming that the application provides APIs, which is not always the case). The increasing number of applications running on devices makes this task increasingly difficult. Prior art US 2017 / 060368 A1 (KOCHURA NADIYA [US]), March 2, 2017, discloses techniques for recognizing user interface components from images (e.g., screenshots / display buffers), associating these components with rendering coordinates, and using this information to drive interactions / actions at the interface level.

[0014] Prior art US 2011 / 047488 A1 (BUTIN EMMA [IL] ET AL), February 24, 2011, discloses the capture of a graphical user interface as a screenshot, the analysis of this image to identify GUI controls (with the possibility of OCR), and the recording of images of controls for subsequent identification / execution in an automated assistance context. Prior art US 2011 / 075940 A1 (DEAVER F SCOTT [US]), March 31, 2011, discloses a computer usage monitoring system that includes capturing images of all or part of the screen, extracting information from the images (including text via OCR), and storing / archiving this information in relation to the observed activity.

[0015] The present application aims to propose improvements to at least some of the drawbacks of the state of the art. 3. SUMMARY

[0016] According to one aspect, the present application concerns a method, implemented by a computing machine, for constructing a knowledge base.

[0017] In certain embodiments of this application, said process includes, during or following the use of a terminal, at least one update of the knowledge base based on information extracted from useful application areas of a digital image of a capture of at least a part of the rendering of a screen of said terminal.

[0018] In certain embodiments of the present application, a method is proposed, implemented by a computing machine (30), for constructing a knowledge base (DB), characterized in that it includes at least one update (S6) of usage information contained in the knowledge base (DB) based on information extracted from useful application areas of a digital image of a capture of at least a part of the rendering of at least one screen of a terminal, the useful application areas being areas containing at least one data provided to or received by at least one application via said at least screen.

[0019] Thus, the proposed solution is based on a new approach consisting of building a knowledge base of a user by exploiting screen capture and image analysis technologies, on a rendering of at least one screen of at least one terminal that he uses (or has just used).

[0020] One advantage of the proposed solution is that it is simple to implement, at least according to some embodiments, since it only requires, in addition to the (at least one) terminal already available to the user, a computing machine (possibly the one already present in the terminal).

[0021] Another advantage of the proposed solution, in at least some embodiments of this application, is that it allows for a generic knowledge base structure, freeing itself, for example, from the need to access the APIs of each application run by the terminal. In other words, because it is not based on the APIs of the application(s) run by the terminal, the proposed solution can, at least in some embodiments, enable the creation of an independent ("agnostic") knowledge base, independent of that application(s), with respect to how information related to their use is collected. Therefore, the proposed solution may require fewer implementation constraints in at least some embodiments of this application.

[0022] Another advantage of the proposed solution, in at least some embodiments of this application, is that an intelligent assistant configured to use the content of this knowledge base can also possess this characteristic of independence from the application(s) used on the terminal. In other words, the intelligent assistant does not have to be application-specific and can cooperate with the generic knowledge base, which can contain information related to several applications (although in a particular implementation, it may also contain information related to a single application).

[0023] Yet another advantage of the proposed solution, at least according to certain implementations, is that even if the application(s) evolve, or if the user adds an application to their terminal, the proposed solution can continue to work without requiring an update, since it relies solely on screen extractions (partial or total).

[0024] In some embodiments of this application, according to a first implementation, the capture covers the entire screen rendering.

[0025] In this first implementation, the process manages the entire screen, without needing to know the number of application windows displayed. An application window is a window linked to an application being executed by the terminal. This first implementation is particularly applicable when the terminal can only display one application window at a time (as is the case with some smartphones, for example). If the terminal supports multi-window functionality (i.e., can display several application windows simultaneously), the first implementation can also be applied, but the process does not manage each application window separately (for example, when the terminal's operating system does not allow retrieval of certain multi-window events: opening an application window, retrieving the position and size of an application window, etc.).

[0026] In some embodiments of this application, according to a second implementation, the capture covers a portion of the screen rendering corresponding to at least one application window displayed on at least one screen.

[0027] In this second implementation, the process can manage several application windows displayed on the screen separately (for example, each application window displayed on the screen). This can therefore, at least in some embodiments, improve the completeness of the information collected in the knowledge base by obtaining information specific to several application windows (for example, each application window) separately.

[0028] In certain embodiments of the present application, according to the second implementation, upon detection of an opening of the application window on the screen, the method includes storing in the knowledge base information on the position, size and / or display rank of the application window, and, for a new update of the knowledge base, the capture is a function of the information on the position, size and / or display rank.

[0029] This allows for the easy capture of, for example, each application window for each new knowledge base update. This assumes that, from its opening to its closing, the same application window is captured multiple times.

[0030] In certain embodiments of this application, in the second implementation, if the terminal allows the simultaneous display of several application windows on the screen, the knowledge base update is performed in a separate knowledge base entry for each of the application windows.

[0031] In this way, in certain embodiments of this application, the knowledge base can be even more comprehensive, as it contains multiple entries, each dedicated to one of the application windows. Such embodiments of this application can thus help intelligent assistants with access to the knowledge base to perform more sophisticated processing and / or achieve better results, such as providing the user with more targeted contextual help.

[0032] According to the invention, if the terminal allows the simultaneous display of several application windows (F1 to F4) on the screen, at least one update (S6) of the knowledge base is made in a separate knowledge base entry for each of the application windows.

[0033] According to the invention, for a useful application area of ​​an application window, at least one update (S6) of the knowledge base is carried out conditionally taking into account a rate of overlap of said useful application area by one or more other application windows.

[0034] For example, in some embodiments of this application, for a useful application area of ​​an application window, each update (S6) of the knowledge base is performed conditionally taking into account a rate of overlap of said useful application area by one or more other application windows.

[0035] In certain embodiments of this application, the coverage rate of said useful application area by one or more other application windows is a function of: information on the position and / or size of said useful application area; information on a display rank of the application window; and / or information on the position, size and / or a display rank of the other application window(s).

[0036] In some embodiments of this application, at least one update (S6) of the knowledge base, for the useful application area of ​​the application window, is performed only when the recovery rate is less than a first recovery value.

[0037] In certain embodiments of this application, according to the second implementation, for a usable application area of ​​an application window, the process includes calculating an overlap rate of said usable application area by one or more other application windows, based on: information on the position and / or size of said useful application area; information on the display rank of the application window; and / or information on the position, size and / or display rank of the other application window(s), and an update of the knowledge base, for the useful application area of ​​the application window, is performed only when the recovery rate is less than a first recovery value.

[0038] Thus, in the case where a useful application area is covered with a coverage rate greater than or equal to the first coverage value, it is possible, for example, to avoid updating the knowledge base with incomplete and / or incomprehensible information.

[0039] According to the invention, according to the second implementation, upon detection of a resizing and / or movement of an application window, the method includes storing in the knowledge base new information on the position, size and / or display rank of the application window, and, for at least one new update of the knowledge base (for example, each update of the knowledge base), the capture is a function of the new information on the position, size and / or display rank.

[0040] In this way, at least according to certain embodiments, a good capture of the application window can be allowed, regardless of how it is modified on the screen, by resizing and / or moving, from its opening to its closing.

[0041] In some embodiments of this application, the useful application areas are areas that are not related to a presentation of features of at least one application run by the terminal and for which an application window is displayed on the screen.

[0042] In other words, the process ignores the areas of an application window reserved for displaying the application's features (menus, buttons, and / or other mechanisms). The remaining (i.e., non-ignored) areas constitute the useful application areas. These contain useful incoming data, which the user provides to the application, and useful outgoing data, which the application provides to the user. Thus, for example, it's possible to avoid adding information to the knowledge base that isn't related to how the user interacts with the application on their device, but is solely related to displaying the application's features (and therefore doesn't reflect the user's actual device usage).

[0043] In some embodiments of this application, the extraction of information from useful application areas includes an extraction belonging to the group comprising: extraction of information on text appearing in useful application areas, by an optical character recognition technique; and extraction of information on image elements appearing in useful application areas, by a computer vision technique.

[0044] In this way, the knowledge base can be enriched with two types of information: that extracted from text and / or that extracted from image elements. This covers most, or even in some cases all, of the useful data (incoming or outgoing) exchanged between the user and the application(s) running on the terminal.

[0045] In some embodiments of this application, the extraction (S4) of information from useful application areas takes into account a detection confidence score.

[0046] In some embodiments of this application, the extraction of information from useful application areas includes taking into account extracted information when it is associated with a detection confidence score greater than a first confidence value.

[0047] In this way, such consideration can, for example, help to improve, at least according to certain implementation methods, the quality of information collected in the knowledge base.

[0048] In some embodiments of this application, the extraction of information from useful application areas includes, after detection that a useful application area contains a video, a subsequent update of the knowledge base based on information extracted from the video.

[0049] This can help avoid using too many system resources (especially computing resources) during real-time execution of the process, while allowing for delayed enrichment (for example, in the evening or at night, when the terminal is less busy) of the knowledge base with information extracted from a video. In a less efficient but less costly variant, because it does not perform the subsequent update, the process can ignore a useful application area if it detects that it contains a video.

[0050] In some embodiments of this application, a new update of the knowledge base is performed on a triggering event belonging to the group comprising a periodic event and an event indicating the end of data entry via one or more input devices of a user interface.

[0051] Increasing the number of updates can help, in certain embodiments of this application, to increase the completeness of the knowledge base. The frequency of the periodic event can be chosen, in some embodiments, to try to best record useful data exchanges between the user and the application(s), while also trying to make the best use of the computing machine's resources. In a variant, the event is recurring but not periodic. The data entry completion event indicates that the user is inactive on their terminal. This is, for example, the end of keyboard input or the end of movement of a pointing device (mouse, trackball, trackpoint, joystick, touchscreen, etc.).

[0052] In some embodiments of this application, at least one update to the knowledge base includes a limited addition of information not already contained in the knowledge base.

[0053] In this way, in certain embodiments, it is possible to limit the writing operations in the knowledge base.

[0054] According to another aspect, a computer program product is proposed which includes program code instructions which, when executed by a computing machine (computer, processor, etc.), cause the aforementioned process to be carried out in any of its different modes of embodiment.

[0055] According to yet another aspect, a computer-readable and non-transient storage medium is proposed, storing a computer program comprising a set of instructions executable by a computing machine (computer, processor, etc.) to implement the aforementioned process in any of its various embodiments.

[0056] According to yet another aspect, a computing machine is proposed that is configured to carry out the aforementioned process in any of its different modes of embodiment. 4. LIST OF FIGURES

[0057] Other features and advantages of the invention will become apparent from the following description, given by way of illustrative and non-limiting example, and the accompanying drawings, in which: [ Fig. 1 ] presents a simplified flowchart of the process according to the invention; [ Fig. 2] is an example of a terminal screen rendering, illustrating an application of the process of the figure 1 ; And [ Fig. 3 ] presents the structure of a computing machine, according to a particular embodiment, configured to carry out the process of the figure 1 . 5. DETAILED DESCRIPTION

[0058] In all figures in this document, identical elements and steps are designated by the same numerical reference.

[0059] We now present, in relation to the organizational chart of the figure 1 A particular embodiment of the method according to the invention involves constructing a knowledge base (referred to as DB, for "Data Base") associated with a terminal user. Once this knowledge base is sufficiently rich in information, an intelligent assistant can use it to offer innovative services such as, for example, learning support.

[0060] The process is implemented by a computing machine (also called a "system" in the rest of the description), an example of whose structure is presented below, in relation to the figure 3 In a first implementation, the computing machine implementing the process is integrated into, or combined with, the user's terminal (this terminal is, for example, a desktop or laptop computer, a tablet, a personal digital assistant, a smartphone, a workstation, etc.). In a second implementation, the computing machine implementing the process is integrated into, or combined with, another device that cooperates with the user's terminal (this other device is, for example, a home gateway, also called an "Internet box").

[0061] In this particular embodiment, it is assumed that the user's terminal supports multi-window functionality, that is, the simultaneous display of several application windows on the terminal screen. As mentioned earlier, an application window is a window linked to the execution of an application by the terminal. Many applications can be executed by the terminal: file management applications (e.g., "Windows Explorer" or Apple's "Finder"); email applications, also called "email clients" (e.g., "Microsoft Outlook" or Apple's "Mail"); instant messaging applications (e.g., "Whatsapp"); multimedia conferencing applications; web browsers (e.g., "Microsoft Internet Explorer" or "Google Chrome"); word processing applications (e.g., "Word"); spreadsheet applications (e.g., "Excel"); presentation applications (e.g., "Microsoft PowerPoint" or Apple's "Keynote"); etc.

[0062] It is also assumed, in the detailed embodiment, that the terminal's operating system allows the retrieval, and provision to the computing machine implementing this process, of certain events related to multi-windowing, such as: an E1 event indicating the opening of a new application window, and containing information on the position, size and / or display rank of the application window; an E2 event indicating the closing of an application window; an E3 event indicating the resizing and / or moving of an application window, and containing information on the new position and / or new size and / or new display rank of the application window.

[0063] Information about the position and / or size can be used to determine the coordinates of the application window on the screen, that is, to know precisely which pixels on the screen correspond to the area of ​​the application window. Consider, for example, the current classic situation where the application window is rectangular and where the information about its position and / or size is formed by an X,Y coordinate (for example, of a corner of the rectangular window) and a pair (height, width). The present invention is not limited to rectangular application windows, but applies regardless of the shape (round, oval, etc.).

[0064] The display rank, also called "ordering value", indicates, for example, whether the window is displayed in the foreground or background, behind one or more other windows.

[0065] In the illustrated embodiment, in a step S0,The computing machine seeks to detect events related to multi-windowing such as at least some of the aforementioned events E1, E2 and / or E3.

[0066] If an E1 event is detected (indicating the opening of a new application window), the process can proceed to a step S1in which the computing machine creates a new entry, also called a new activity, in the knowledge base DB (as illustrated by arrow referenced 1). An activity can therefore be associated, in the illustrated embodiment, with a particular application window displayed on the terminal screen, and can group all the information extracted from that application window (for example, as detailed later, all texts (read or written by the user) appearing in the application window, as well as the results of semantic analysis of images manipulated within the application window) from its opening until its closing. Furthermore, in step S1, the computing machine can store information about the position, size, and / or display rank of the new application window in the knowledge base DB (for example, in a table for managing open windows, where each row is specific to a distinct application window).

[0067] In the illustrated embodiment, in a step S2 The computing machine can capture, for example in the form of a digital image, a part of the screen rendering corresponding to the application window, thanks to the information (stored in step S1) on the position, size and / or display rank of the application window.

[0068] In a step S3 The computing machine can identify useful application areas in the digital image resulting from the capture performed in step S2. The useful application areas are areas that are not related to a presentation of application features for which the aforementioned application window is displayed on the terminal screen in the illustrated embodiment.

[0069] In a specific implementation of step S3, the computing machine can identify the areas of the application window reserved for menus, buttons, and / or other mechanisms used to present the application's functionalities. This identification of areas related to the application's functionalities can, for example, be done by: by combining known image segmentation techniques into rectangular sub-areas with known identification techniques via OCR (Optical Character Recognition), in order to identify common keywords in these rectangular sub-areas (including: "open", "file", "home", "insert", "create", "layout", "quit", "save", "yes", "cancelled", etc.); or via detection of aligned round or rectangular icons, by mechanisms such as: artificial intelligence (AI), automatic classification, computer vision, etc.

[0070] In some embodiments of this application, once identified, the areas reserved for displaying the application's functionalities can be ignored. For example, only the remaining areas of the application window are tagged "useful application areas" and their position on the overall screen is memorized.

[0071] In a step S4,The computing machine can extract information from the useful application areas identified in step S3. In some embodiments of this application, two types of extraction are performed: extraction of information from text appearing in the useful application areas, using an OCR technique, and extraction of information from image elements appearing in the useful application areas, using a computer vision technique (enabling, for example, text recognition, table recognition, or recognition of specific elements (images representing an animal, a vehicle, etc.)). The information that we seek to extract from the useful application areas can be defined more generally as representative of the incoming useful data that the user provides to the application, and / or the outgoing useful data that the application provides to the user.This can therefore include all data entered by the user via one or more input devices of a user interface (for example a keyboard or a pointing device such as a mouse, trackball, trackpoint, joystick, touch screen, etc.), as well as all data received by the user (including responses that he may obtain from the application itself or from contacts with whom he communicates via the application, or from images that he manipulates or views through the application).

[0072] In certain embodiments of this application, a text element (e.g., a line) or image recognized by a recognition technique (OCR, computer vision, etc.) may be ignored if the confidence score associated with the recognition is below a specific threshold (configuration parameter). In other words, extracted information is only taken into account if it is associated with a detection confidence score higher than a first confidence value.

[0073] In some embodiments of this application, if the computing machine detects that a useful application area contains a video, it may decide to ignore it (taking into account, for example, at least one configuration parameter). In another variant, it may decide to process this useful application area later (by updating the knowledge base later, for example, in the evening, based on information extracted from the video) to avoid using too many system resources during real-time processing of step S4.

[0074] In a particular implementation, the computing machine can identify useful application areas that are covered, wholly or partially, by other application windows. This identification can be performed by directly using the open window management table, which contains their position, size, and / or display rank (see step S1). If a useful application area is detected as being partially or totally covered, it can be ignored by the computing machine in step S4 (for example, no information is extracted from it).For example, in some embodiments, the computing machine can calculate an overlap rate of the useful application area by at least one other application window, based on various information (information on the position and / or size of the useful application area, information on the display rank of the application window, and information on the position, size, and / or display rank of the other application window(s), for example), and the useful application area can, for example, be taken into account taking the overlap rate into account. The useful application area can, for example, only be taken into account if the overlap rate is less than a first overlap value.

[0075] Step S4 can be followed by a test step Tin which the computing machine detects whether the application window being processed is a newly opened window (the case, hereafter called the first case, where the present iteration of steps S2 to S4 is the first iteration since the detection of event E1), or an application window that was open (the case, hereafter called the second case, where the present iteration of steps S2 to S4 is not the first iteration since the detection of event E1).

[0076] In the first case, the computing machine can perform, in certain embodiments, a step S6 in which the information (useful data) extracted in step S4 is added to the user's knowledge base DB (as illustrated by the arrow referenced 2), being attached to the activity (knowledge base entry) associated with the current application window (whose opening was detected by event E1).

[0077] In the second case, the computing machine can perform, in certain embodiments, a step S5 in which it identifies, among the information (useful data) extracted in step S4, those (hereafter referred to as "new extracted information") that are not already contained in the knowledge base DB. Then, the computing machine can perform stage S6 But here, for example, only the newly extracted information is added to the knowledge base DB (if there is no new information, step S6 is not performed), and it is linked to the activity associated with the current application window. Thus, if text, for example, has been added (or modified) in a useful application area, the process can extract and / or store it, tagging it, for example, with the status "added" (ADDED) (or "modified" (MODIFIED)).

[0078] As long as the application window remains open and is neither resized nor moved, the computing machine can iterate steps S2 to S6 (as illustrated by the arrow labeled R) to extract further information, continuously enriching the knowledge base (DB) and reporting on user application usage (i.e., tracking the user's actions and interactions with the application and / or their contacts). A new iteration, resulting in a new update to the knowledge base, is performed on a triggering event generated by a trigger module, such as a periodic timeout event (every N seconds, for example) or an event indicating the end of data entry via the user interface's input hardware (keyboard, pointing device, etc.) (an event retrievable via the terminal's operating system).The data entry completion trigger event is used, for example, if the user is entering data when a periodic timeout trigger event is detected (e.g., after N seconds). In this case, the data entry completion trigger event allows the system to wait for the user's input to finish, or pause, before taking another screenshot.

[0079] If an E2 event is detected, indicating the closure of the application window (for example, because the user has closed the application window or quit the application, depending on the terminal used), the process can, in certain embodiments, proceed to a step S7 in which the computing machine can close the associated activity, by changing the status of this activity (which changes for example to "completed") in the knowledge base DB.

[0080] If an E3 event is detected, indicating a resizing and / or movement of the application window, the process can restart, for example, at step S2 to re-identify useful application areas within that window and extract new information (based on the new position, size, and / or display rank of the application window). The open window management table (which stores the list of open windows with their position, size, and / or display rank) can also be updated accordingly.

[0081] All the operations and steps described above can be performed for each application, and therefore for each application window. Thus, in the knowledge base DB, a list of extracted and time-stamped information can be linked to an activity (associated with an application window). Variant of the process

[0082] In one variant, the computing machine may not handle some of the events E1, E2, and / or E3 and may not perform some of the steps S0, S1, and S7. It may, for example, execute the iterative mechanism of steps S2 to S6 as described above with the figure 1 , with the difference that in step S2 it can capture, in the form of a digital image, the entire screen rendering (and not just a part corresponding to a particular application window).

[0083] In other words, in this embodiment, only one activity is considered, and the process regularly recalculates the application areas useful for extracting information to enrich the knowledge base (DB). With this variant, the knowledge base is less complete than in the embodiment of the figure 1 , but this already allows intelligent assistants to offer contextual help services.

[0084] This variant can be applied, for example, when the terminal can only display one application window at a time (such as a smartphone). It can also be applied when the terminal supports multi-window functionality but the processing machine is unable to retrieve the E1, E2, and / or E3 events associated with multi-window operation (for example, because the terminal's operating system does not support it, or for other reasons). Application example

[0085] We will now present an example of the application of the process of figure 1 , in relation to the figure 2 which illustrates an example of rendering a 20-inch screen of a terminal.

[0086] We assume that three application windows F1, F2 and F3 are already open and that the open window management table contains four pieces of information for each window (namely: window identifier, window position, window size and / or window display rank): for window F1: "Multimedia Conference - State of the Art", "position_XY_F1", "Size_F1", "Schedule 1"; for window F2: "File Explorer - This PC", "position_XY_F2", "Size_F2", "Schedule 1"; and for window F3: "IM - Lync", "position_XY_F3", "Size_F3", "Schedule 2".

[0087] It is also assumed that on his terminal (for example his PC), the user received a notification from an instant messaging application and clicked on it, causing a new F4 application window to open (window linked to the instant messaging application).

[0088] In step S0, the computing machine receives an E1 event indicating the opening of window F4 and containing the following information: window identifier (“IM <nomdelapersonne>"), window position ("position_XY_F4"), window size ("Size_F4") and window display rank ("Scheduling 2").

[0089] In step S1, the computing machine creates a new activity ("A1: IM") <nomdelapersonne>" in the knowledge base and stores in the open window management table the aforementioned information relating to window F4.

[0090] In step S2, the computing machine captures, in the form of a digital image, a part of the rendering of screen 20 corresponding to window F4.

[0091] In step S3, the computing machine identifies three useful application areas, referenced ZU_XY_1_F4, ZU_XY_2_F4 and ZU_XY_3_F4 on the figure 2 .

[0092] In step S4, the computing machine extracts information from the useful application areas identified in step S3, for example text information: Information extracted from zone ZU_XY_1_F4: <text1>; information extracted from zone ZU_XY_2_F4: <text2>; and information extracted from zone ZU_XY_3_F4: <text3>.

[0093] In step S6, the computing machine updates the activity "A1: IM <nomdelapersonne>» in the knowledge base, by adding a set of time-stamped information, such as: Identifier of the timestamped information set: "Event1_Timestamp"; information extracted from the ZU_XY_1_F4 area: <textl>; information extracted from zone ZU_XY_2_F4: <text2>; and information extracted from zone ZU_XY_3_F4: <text3>.

[0094] Assume that after N seconds, the computing machine detects a first triggering event and that nothing has changed on the screen since the previous capture (previous step S2). For window F4, the computing machine performs a new iteration of steps S2, S3, and S4 (which will provide the same results as in the previous iteration) and then executes step S5. In this step S5, the comparison with the previously stored knowledge base elements (a set of time-stamped information bearing the identifier "Event1_Timestamp") indicates that no new information has been extracted. The processing stops there (step S6 is not performed).

[0095] It is assumed that after N more seconds, the computing machine detects a second triggering event, and that the user is currently entering text ( <text4>) on the keyboard in the useful area "ZU_XY_3_F4". The computing machine then waits for a third triggering event indicating the end of input (corresponding to an actual end of input or a sufficiently long pause in input to represent the end of input). Following the third triggering event, the computing machine performs a new iteration of steps S2, S3, and S4 for window F4. The new iteration of step S4 does not provide the same results as the previous iteration, since the computing machine extracts the following information from the useful application area ZU_XY_3_F4: <text3>And <text4>(instead of <text3>only). Then the computing machine executes step S5 and the comparison with the knowledge base elements previously stored for window F4 (a batch of timestamped information bearing the identifier "Event1_Timestamp") indicates that new information <text4>has been extracted. Processing continues with step S6 in which the computing machine updates the activity "A1: IM <nomdelapersonne>» in the knowledge base, by adding a new set of time-stamped information, such as: identifier of the timestamped information set: "Event2_Timestamp"; and new information extracted from the ZU_XY_3_F4 area: ADDED = <text4>.

[0096] Let's continue with the example by considering the processing of window F1, on which the user is following a video conference.

[0097] When the application window F1 (linked to a video conferencing application) is opened, the computing machine receives, in step S0, an event E1 indicating the opening of the window F1 and containing the following information: window identifier ("Multimedia Conference - State of the art"), window position ("position_XY_F1"), window size ("Size_F1") and window display rank ("Scheduling 1").

[0098] In step S1, the computing machine creates a new activity ("A2: Multimedia Conference"). <nomdelapersonne>") in the knowledge base and stores in the open window management table the aforementioned information relating to window F1.

[0099] In step S2, the computing machine captures, in the form of a digital image, a part of the rendering of screen 20 corresponding to window F1.

[0100] In step S3, the computing machine identifies two useful application areas, referenced ZU_XY_1_F1 and ZU_XY_2_F1 on the figure 2 .

[0101] In step S4, the computing machine extracts information from the useful application areas identified in step S3, for example text information: Information extracted from zone ZU_XY_1_F1: <vide>(In this example, it is assumed that there is no extracted text because the OCR detection confidence scores do not allow for validation of the extracted text as it is too small); information extracted from the ZU_XY_2_F1 area:<text1'> .

[0102] In step S6, the computing machine updates the activity "A2: Multimedia Conference" <nomdelapersonne>» in the knowledge base, by adding a set of time-stamped information, such as: Identifier of the timestamped information set: "Event1_Timestamp"; information extracted from the ZU_XY_1_F1 area: <vide>; and information extracted from zone ZU_XY_2_F1:<text1'> .

[0103] Assume that after N seconds, the computing machine detects a triggering event and that a new slide has been displayed in the window (specifically in the ZU_XY_2_F1 area) since the previous capture (previous step S2). The computing machine performs a new iteration of steps S2, S3, and S4 for window F1. The new iteration of step S4 does not provide the same results as the previous iteration, since the computing machine extracts the following information from the useful application area ZU_XY_2_F1:<text2'> (instead of<text1'> ). Then the computing machine executes step S5 and the comparison with the knowledge base elements previously stored for window F1 (a batch of timestamped information bearing the identifier "Event1_Timestamp") indicates that new information<text2'> has been extracted.The processing continues with step S6 in which the computing machine updates the activity "A2: Multimedia Conference. <nomdelapersonne>» in the knowledge base, by adding a new set of time-stamped information, such as: identifier of the timestamped information batch: "Event1_Timestamp"; information extracted from the ZU_XY_2_F1 area: ADDED =<text2'> .

[0104] There figure 3 presents an example of the structure of a computing machine 30 to carry out (execute) the process of the figure 1 .

[0105] This structure comprises a random access memory 32 (for example, RAM), a read-only memory 33 (for example, ROM or a hard drive), and a processing unit 31 (equipped, for example, with at least one processor and controlled by at least one computer program 330 stored in the read-only memory 33). At initialization, the code instructions of the computer program 330 are, for example, loaded into the random access memory 32 before being executed by the processor of the processing unit 31.

[0106] This figure 3 illustrates only one particular way, among several possible ways, of implementing a computing machine to carry out (execute) the process. Indeed, the computing machine can be implemented indifferently in the form of a reprogrammable computing machine (a PC, a DSP processor or a microcontroller) executing a program comprising a sequence of instructions, or in the form of a dedicated computing machine (for example a set of logic gates such as an FPGA or an ASIC, or any other hardware module).

[0107] In the case of an implementation in the form of a reprogrammable computing machine, the corresponding program (i.e., the sequence of instructions) may be stored on a removable (such as, for example, a floppy disk, CD-ROM, or DVD-ROM) or non-removable storage medium, this storage medium being partially or fully readable by a computer or processor. Alternatively, the storage medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a method according to the invention.

[0108] Alternatively, the storage medium can be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, radio link, wireless optical link, or other means. The program according to the invention can, in particular, be uploaded to an Internet-type network.< / nomdelapersonne> < / vide> < / nomdelapersonne> < / vide> < / nomdelapersonne> < / nomdelapersonne> < / textl> < / nomdelapersonne> < / nomdelapersonne> < / nomdelapersonne>

Claims

1. Method, implemented by a computing machine (30), for constructing a knowledge base (DB) of a user, the method comprising at least one update (S6) of information contained in the knowledge base (DB) of said user and relating to a use of a terminal of said user, based on information extracted from useful application areas of a digital image of a capture of at least part of the rendering of a screen of the terminal, the useful application areas being areas containing at least one datum supplied by at least one application via said screen or received by said application via a user interface of said terminal and rendered on said screen, characterized in that, when the terminal allows the simultaneous display of multiple application windows (F1 to F4) on the screen, referred to as multi-windowing, said capture relates at least to part of the rendering of the screen corresponding to one of the application windows and said at least one update (S6) is carried out upon detection of an event related to the multi-windowing, and in that, for a useful application area of an application window, said at least one update (S6) is carried out conditionally, taking into account a degree of overlap by which said useful application area is overlapped by one or more other application windows, said useful application areas excluding areas of the application window that are reserved for presenting functionalities of the application.

2. Computing machine comprising at least one processor configured to carry out at least one update (S6) of use information contained in a knowledge base (DB) of a user and relating to a use of a terminal of said user, based on information extracted from useful application areas of a digital image of a capture of at least part of the rendering of a screen of the terminal, the useful application areas being areas containing at least one datum supplied by at least one application via said screen or received by said application via a user interface of said terminal and rendered on said screen, characterized in that the processor is configured so that, when the terminal allows the simultaneous display of multiple application windows (F1 to F4) on the screen, referred to as multi-windowing, said capture relates at least to part of the rendering of the screen corresponding to one of the application windows and said at least one update (S6) is carried out upon detection of an event related to the multi-windowing, and in that, for a useful application area of an application window, said at least one update (S6) is carried out conditionally, taking into account a degree of overlap by which said useful application area is overlapped by one or more other application windows, said useful application areas excluding areas of the application window that are reserved for presenting functionalities of the application.

3. Method according to Claim 1 or computing machine according to Claim 2, characterized in that said capture relates to the entire rendering of the screen (20).

4. Method according to Claim 1, characterized in that it comprises, or computing machine according to Claim 2, characterized in that said processor is configured to carry out, upon detection (E1) of opening of an application window on the screen, storage (S1) of information about the position, size and / or a display rank of the application window in the knowledge base, and in that, for a new update of the knowledge base, the capture (S2) depends on the information about the position, size and / or display rank.

5. Method according to Claim 4, characterized in that, or computing machine, characterized in that said processor is configured so that, at least one update (S6) of the knowledge base is carried out in a distinct entry of the knowledge base for each of the application windows.

6. Method according to Claim 1, characterized in that, or computing machine according to Claim 2, characterized in that, the degree of overlap by which said useful application area is overlapped by one or more other application windows depends on: - information about the position and / or size of said useful application area; - information about a display rank of the application window; and / or - information about the position, size and / or a display rank of the one or more other application window (s) .

7. Method according to Claim 6, characterized in that, or computing machine according to Claim 6, characterized in that said processor is configured so that, at least one update (S6) of the knowledge base, for the useful application area of the application window, is carried out only when the degree of overlap is less than a first overlap value.

8. Method according to any one of Claims 1 or 3 to 7, characterized in that it comprises, or computing machine according to any one of Claims 2 to 7, characterized in that said processor is configured to carry out, upon detection (E3) of resizing and / or movement of an application window, storage of new information about the position, size and / or display rank of the application window in the knowledge base, and in that, for each new update (S6) of the knowledge base, the capture (S2) depends on the new information about the position, size and / or display rank.

9. Method according to any one of Claims 1 or 3 to 8, characterized in that, or computing machine according to any one of Claims 2 to 8, characterized in that said processor is configured so that, said at least one update (S6) is carried out upon detection (E2) of closure of an application window.

10. Method according to any one of Claims 1 or 3 to 9, characterized in that, or computing machine according to any one of Claims 2 to 9, characterized in that said processor is configured so that, the useful application areas are areas that are not linked to presenting functionalities of said at least one application and for which an application window is displayed on the screen.

11. Method according to any one of Claims 1 or 3 to 10, characterized in that, or computing machine according to any one of Claims 2 to 10, characterized in that said processor is configured so that, the extraction (S4) of information from the useful application areas comprises an extraction belonging to the group comprising: extraction of information about text appearing in the useful application areas, using an optical character recognition technique; and extraction of information about image elements appearing in the useful application areas, using a computer vision technique.

12. Method according to any one of Claims 1 or 3 to 11, characterized in that, or computing machine according to any one of Claims 2 to 11 characterized in that said processor is configured so that, the extraction (S4) of information from the useful application areas takes into account a detection confidence score.

13. Computer program product comprising program code instructions that, when they are executed by a computing machine, cause the method according to one of Claims 1 or 3 to 12 to be performed.

14. Non-transient computer-readable storage medium storing a computer program comprising a set of instructions able to be executed by a computing machine for implementing the method according to one of Claims 1 or 3 to 12.