Methods and systems for providing and utilizing sensor data

Abstract

The present invention relates to mobile devices. In particular embodiments, the present invention provides a method comprising obtaining sensing data matching a set of user contexts associated with a plurality of mobile states and conditions. The method further comprises a mobile sensing service detecting the sensing data, managing the sensing data by a user profile, and a context manager providing the sensing data to generate input recommendations for input modules of applications running on a mobile computing device. Other embodiments exist.

Description

Background Technology

[0001] This invention relates to mobile systems and applications.

[0002] Over the past decade, smartphones and other mobile devices have become ubiquitous. As users carry mobile devices in their daily lives, an increasing number of these devices now incorporate various sensors. Mobile sensors are small devices built into smartphones and other mobile devices, including those in vehicles, enabling them to collect and interpret data from the physical world. However, providing sensor data is only the beginning. For mobile devices, it is also important to fully utilize sensor data through their operating systems and applications. While various conventional technologies have been developed in the past, they have all had shortcomings.

[0003] Therefore, new and improved methods and systems are needed for providing and utilizing mobile sensor data. Summary of the Invention

[0004] This invention relates to mobile systems and applications. In a particular embodiment, the invention provides a method comprising obtaining motion sensing data matched with a set of user contexts associated with multiple motion states and conditions. The method further includes a motion sensing service detecting the sensing data, managing the sensing data through a user profile accessible via a context manager, and the context manager providing the sensing data to generate input recommendations for an input module of an application running on the mobile system. Other embodiments also exist.

[0005] A system of one or more computers can be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, when executed, cause the system to perform the corresponding actions. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause that device to perform the corresponding actions. One general aspect of this disclosure includes a method for generating input method recommendations using sensed information. The method includes defining a set of user contexts by a context manager and storing the user contexts as part of a user profile. The method also includes acquiring sensed data from one or more sensors at a first moment. The method further includes providing a set of predefined templates associated with the user contexts and the sensed data. Furthermore, the method includes launching a first application at a second moment and displaying the first application on a graphical user interface. The method also includes receiving text input through an input module in the first application, which is displayed on an input area of ​​the graphical user interface. The method further includes determining a first context based on the text input and the first application. Furthermore, the method includes generating a first recommendation using the predefined templates and the first context. Furthermore, the method includes displaying the first recommendation on an input area of ​​the graphical user interface. Other embodiments of this aspect include a corresponding computer system, apparatus, and computer programs recorded on one or more computer storage devices, each for performing the actions of the method.

[0006] Implementations may include one or more of the following features. In this method, one or more sensors include accelerometers. In this method, the input area includes a touch keyboard and an input suggestion area, where a first recommendation is displayed. In this method, the method may include sending a user profile to a remote memory. In this method, obtaining sensing data includes obtaining user activity, collecting location information, and collecting accelerometer information. In this method, the method may include matching various data types of sensing data with a user context, where a set of predetermined templates are associated with conditions predefined for the user context and sensing data under those conditions. In this method, the method may include: receiving a recommendation request by activating the input module of a first application without typing text input; accessing a user profile to determine whether a user context in a second time-space user context is valid to allow a context manager to access sensing data; determining a condition among the conditions associated with that user context; and identifying a template among the predetermined templates associated with that condition for generating a first recommendation. In this method, the method may include a context manager predefining multiple keywords corresponding to various data types of sensing data based on the user context. The method further includes: receiving a recommendation request based on text input via an input module of a first application; comparing the text input with multiple keywords based on similarity to determine the keyword with the highest similarity; and generating a first recommendation using sensing data corresponding to the keyword. In this method, obtaining sensing data from one or more sensors includes using sensor data services registered to the operating system. The method also includes generating autocomplete suggestions on an input area of ​​a graphical user interface using the first recommendation. Implementations of the technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0007] Another general aspect of this disclosure includes a mobile computing device. The device includes: a housing; a display disposed on the front of the housing, the display including a touchscreen; a storage device for storing an operating system and multiple applications; memory; multiple sensors; and a processor connected to the storage device and the memory. The processor is configured to: execute instructions associated with the operating system to define a set of user contexts via a context manager; execute instructions associated with a first application and generate a first set of context information associated with the first application; execute instructions associated with a second application; execute instructions associated with the operating system to provide a user input interface on the touchscreen for the second application; execute instructions associated with the operating system to provide a first recommendation based on at least the first set of context information; and execute instructions associated with the operating system to display the first recommendation on the user input interface. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0008] Implementations may include one or more of the following features. In this device, the processor is further configured to generate a second recommendation using multiple sensors, which is displayed on a user input interface. In this device, the multiple sensors include one or more selected from accelerometers, gyroscopes, magnetometers, GPS, light sensors, touchscreens, microphones, and cameras. The device may be a mobile phone, tablet, wearable device, or vehicle. Implementations of the technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0009] Another general aspect of this disclosure includes a method for generating input recommendations using motion sensing information. The method includes a context manager defining a set of user contexts associated with multiple motion states. The method also includes a sensing data service collecting sensing data matching the multiple motion states based on one or more sensors. The method further includes defining multiple templates for the multiple motion states under different conditions. Furthermore, the method includes storing multiple keywords corresponding to the sensing data matching the multiple motion states. The method also includes the context manager setting the accessibility of sensing data for each of the multiple motion states, the accessibility being determined by a user profile. The method also includes the context manager monitoring an application. The method includes launching an application, which includes an input module, to initiate a recommendation request to the context manager. Furthermore, the method includes detecting sensing data matching the current motion state. Additionally, the method includes returning input recommendations to the input module. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each for performing the actions of the method.

[0010] Implementations may include one or more of the following features. The method may include: receiving a recommendation request without prompts from an input module; determining that a current motion state is valid, such that a context manager can access sensing data matching the motion state; determining a condition associated with the motion state; detecting sensing data matching the current motion state; and generating an input recommendation using a template for the motion state under that condition and the sensing data. The method may also include: receiving a recommendation request with prompts in the input text via the input module; analyzing the similarity between the prompts and multiple keywords stored in a context manager; determining the keyword with the highest similarity; detecting sensing data corresponding to the keyword at the current moment; and generating an input recommendation using the sensing data. Implementations of the technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0011] It should be understood that the embodiments of the present invention offer numerous advantages over conventional techniques. Among other things, motion sensing data, such as GPS location, accelerometer or magnetometer readings, and microphone input, can be used to improve the user experience on mobile devices by providing users with more accurate and relevant information. For example, sensors that detect a user's location can provide location-based services, such as directions or recommendations for local businesses. For example, using GPS location data, mobile applications can provide location-based services, such as recommending nearby restaurants or providing step-by-step navigation instructions. Accelerometer readings can be used to detect device orientation and adjust the screen accordingly for a more comfortable user experience. Microphone input can be used for voice commands and dictation, allowing hands-free navigation of the device. Furthermore, motion sensing can enhance security by detecting unusual activity or patterns that may indicate security threats. For example, sensors that detect a user's heartbeat can be used to verify user identity or detect fraudulent activity. In various embodiments, data security is maintained because a context manager (which may be implemented as part of the operating system) can manage access to motion sensing data from different applications. Furthermore, the sharing of motion sensing data according to this disclosure can help improve the efficiency of mobile devices by providing data that can be used to optimize system performance or extend battery life. For example, sensors that detect user movement can be used to adjust the device's power consumption based on the user's activity level. Motion sensors can provide additional functionality by enabling mobile devices to interact with the physical world in new ways. For instance, sensors that detect user gestures can be used to control the device without touching it, or sensors that detect ambient light can be used to adjust the device's display brightness.

[0012] Embodiments of the present invention can be implemented in conjunction with existing systems and processes. For example, the mobile sensing data sharing based on the application of the present invention can be used in various systems, including mobile phones, wearable devices, tablets, and vehicles with various sensors. Furthermore, various technologies according to the present invention can be incorporated into existing systems through software installation and updates. Other advantages also exist.

[0013] These and other advantages are achieved by the present invention within the context of known technologies. However, the nature and advantages of the invention can be further understood by referring to the following portions of this specification and the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a simplified block diagram of a mobile device according to an embodiment of the present invention.

[0015] Figure 2 This is a simplified block diagram of a system architecture for content recommendation using motion sensing information according to an embodiment of the present invention.

[0016] Figure 3This is an operational diagram of an exemplary mobile sensing information sharing system according to an embodiment of the present invention.

[0017] Figure 4 This is another operational diagram of the mobile sensing service according to an embodiment of the present invention.

[0018] Figure 5 The present invention illustrates (A) the operation of a user context manager providing motion-sensing recommendations without prompting and (B) the steps for generating recommendations.

[0019] Figure 6 The present invention illustrates a process flow for generating recommendations using motion sensing data with prompts, according to an embodiment of the present invention. Detailed Implementation

[0020] This invention relates to mobile systems and applications. In a particular embodiment, the invention provides a method comprising obtaining motion sensing data matched with a set of user contexts associated with multiple motion states and conditions. The method further includes a motion sensing service detecting the sensing data, managing the sensing data through a user profile accessible via a context manager, and the context manager providing the sensing data to generate input recommendations for an input module of an application running on the mobile system. Other embodiments also exist.

[0021] Motion sensing refers to the acquisition of data from the environment using sensors on mobile devices (e.g., smartphones, tablets, wearable devices, vehicles), including various user contextual information. Many types of sensors have been developed for motion sensing services, such as accelerometers (ACC), gyroscopes, magnetometers, GPS, light sensors, touchscreens, microphones, and cameras. The acquired sensing data can include different types, such as being correlated with various user activities, location, physical states (e.g., walking, running, cycling, on a bus, on a subway, etc.), or other environmental information surrounding the user operating the mobile device or vehicle. Based on sensing data and other system information (e.g., maps, calendars, messages, emails), motion sensing can also infer user activities, such as daily commutes, travel, meetings, etc. These are also data generated from motion sensing.

[0022] The following description is intended to enable those skilled in the art to make and use the invention and incorporate it into specific applications. Various modifications and multiple uses in different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of embodiments. Therefore, the invention is not intended to be limited to the presented embodiments, but should be accorded the broadest scope consistent with the principles and novel features disclosed herein.

[0023] In the following detailed description, numerous specific details are set forth to provide a more complete understanding of the invention. However, it will be apparent to those skilled in the art that implementation of the invention is not necessarily limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detailed form to avoid obscuring the invention. The device may be a mobile phone, tablet computer, or vehicle.

[0024] Readers are advised to note all papers and documents submitted concurrently with this specification and available to the public, the contents of which are incorporated herein by reference. Unless expressly stated otherwise, all features disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent, or similar purpose. Therefore, unless expressly stated otherwise, each disclosed feature is merely one example among a series of equivalent or similar features.

[0025] Furthermore, any element in the claims that does not expressly refer to "means for" or "step for" for performing a specified function should not be construed as conforming to the "means" or "step" provisions of Title 35, Section 112, Subsection 6 of the United States Code. Specifically, the use of "step of" or "step of" in the claims herein is not intended to invoke Title 35, Section 112, Subsection 6 of the United States Code.

[0026] Please note that if labels such as left, right, front, back, up, down, forward, reverse, clockwise, and counterclockwise are used, these labels are for convenience only and are not intended to suggest any particular fixed direction. Rather, they are used to reflect the relative position and / or orientation of parts of an object.

[0027] Figure 1This is a simplified block diagram of a mobile device according to an embodiment of the present invention. This diagram is merely illustrative and should not unduly limit the scope of the claims. Those skilled in the art will recognize many variations, alternatives, and modifications. As shown, device 100, such as a smartphone or tablet, comprises several main components that work together to enable the device to function properly. The mobile device includes a processor 100. As an example, processor 100 includes a central processing unit (CPU) designed for mobile devices such as smartphones, tablets, or laptops. Processor 101 may also include a graphics processing unit (GPU) and a neural processing unit (NPU). The CPU is responsible for executing most instructions received by the computer, the GPU is responsible for rendering graphics, and the NPU is dedicated to artificial intelligence and machine learning tasks. Some examples of mobile processors that include a GPU and an NPU are the Qualcomm Snapdragon 865 and the Apple A14 Bionic. The mobile device also includes a screen 102. For example, screen 102 includes a display device for displaying information such as text, images, and videos on mobile device 100. The screen also allows users to interact with the device, install various applications, and run applications supported by processor 100. Optionally, screen 102 is a touchscreen, which enables interaction with the mobile device through touch. In various embodiments, mobile device 100 may include multiple displays (e.g., front display and rear display, etc.).

[0028] In addition, mobile device 100 includes an input terminal 103 that provides various input methods, such as text input, voice input, image input, or any digital data. Input terminal 103 can be implemented via screen 102 (e.g., a touchscreen) to display the input methods on the screen. Voice or image recognition can also be used as input. Input terminal 103 is also connected to processor 101, allowing input (whether text, voice, image, video, or any digital data) to be processed, stored, and used by the mobile device. Input terminal 103 can be integrated into the operating system software or firmware stored in the mobile device and configured through various interfaces in the software architecture, including input methods for interacting with multiple applications actively running on the mobile device. Optionally, input terminal 103 can be configured with a user interface to determine the accessibility of specific state information for certain active applications (apps) running on the mobile device. Mobile device 100 also includes memory 106, also known as read-only memory (ROM) or internal memory, for permanently storing data such as applications, photos, and videos. Applications include some built-in system applications and third-party applications. In various implementations, the operating system and applications can be stored in different storage locations, wherein the operating system has full access to various hardware components, such as sensor 104, while the applications can only access sensor 104 through the operating system.

[0029] Mobile device 100 also includes memory 107. Memory, typically called random access memory (RAM), is used to temporarily store data while the mobile device is in use. This data is deleted when the mobile device is powered off. Optionally, the operating system of mobile device 100 may include a context manager component with query / return functionality running in the background to interact with multiple active applications to obtain updated state information of the applications. For example, the context manager may be an implementation within a component of the operating system. The context manager component may be connected to memory 107 to store updated state information of each active application. The context manager component may be connected to processor 101 or analyze the updated state information of active applications independently to generate simplified messages that can be shared and are useful to other applications. Sharing functionality may be implemented through an interface between the context manager and the input method of each application. In particular, an input method is software that helps users input text content within an application. Shared information may include motion sensing data collected by mobile device 100.

[0030] In addition, mobile device 100 includes a communication interface 108 that provides various connectivity options, such as Wi-Fi, Bluetooth, and cellular data, enabling it to connect to the internet and other devices. Mobile device 100 may include sensors 104, which can be of various types, such as gyroscopes, accelerometers, magnetometers, light sensors, cameras, and compasses, for detecting motion, orientation, or other status information of the device and active applications on it. The acquired sensing data may include different types associated with various user activities, location, physical states (such as walking, running, cycling, on a bus, on a subway, etc.), or other environmental information surrounding the user operating the mobile device or vehicle. Based on the sensing data and other system information (e.g., maps, calendars, messages, emails), motion sensing can also infer user activities, such as daily commutes, travel, meetings, etc. Mobile device 100 may also include other components, such as one or more cameras and an audio output 105.

[0031] Mobile devices, such as smartphones, tablets, and wearable devices like smartwatches and smart glasses, are designed to be portable and usable in a variety of locations. These devices typically connect wirelessly to the internet and other devices and are used for a range of purposes, including making calls, sending text messages and emails, browsing the internet, and running various applications. Furthermore, mobile devices may include vehicles themselves, such as various electric vehicles, boats, and aircraft.

[0032] In many real-world scenarios, multiple applications are actively running on mobile devices, and users of these applications may want to share information between them in a simple and convenient way. For example, if a user is taking a taxi to the airport, they might want to get their location and traffic information from a ride-hailing app and easily share it via SMS. Or, if a user is listening to a new song on their smartphone's media player, they might want to get the song information and share it to social media apps without manually typing or copying / pasting. In both cases, a cross-application information sharing solution is needed to facilitate the access and sharing of relevant information.

[0033] In a particular embodiment, based on Figure 1A mobile computing device is provided. The device includes a housing having a display disposed on its front side, the display including a touchscreen. Furthermore, the device includes a memory and RAM for storing an operating system and multiple applications. Additionally, the device includes multiple sensors and a processor connected to the storage device and the RAM. The processor is configured to execute instructions associated with the operating system to define a set of user contexts via a context manager, execute instructions associated with a first application and generate a first set of context information associated with the first application, execute instructions associated with a second application, provide a user input interface for the second application on the touchscreen, provide a first recommendation based on at least the first set of context information, and display the first recommendation on the user input interface.

[0034] Implementations of the mobile computing device may include one or more of the following features. In this device, the processor is further configured to generate a second recommendation using multiple sensors, which is displayed on a user input interface. In this device, the multiple sensors include one or more selected from accelerometers, gyroscopes, magnetometers, GPS, light sensors, touchscreens, microphones, and cameras. The device may be a mobile phone, tablet, wearable device, or vehicle. Implementations of the technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0035] Figure 2 This is a simplified block diagram of a system architecture for content recommendation using motion sensing information according to an embodiment of the present invention. This diagram is merely illustrative and should not unduly limit the scope of the claims. Those skilled in the art will recognize many variations, alternatives, and modifications. As shown, the software system 200 for content recommendation using motion sensing information is provided in a mobile device within a software environment associated with an operating system that interacts with third-party applications and system-built-in applications. For example, the software system 200 can use... Figure 1 The illustrated mobile system 100 implementation includes an operating system and applications stored in non-volatile memory and executed by a processor. A user context manager 210 is a component in the software system 200 for managing content recommendation using mobile sensing information. The user context manager 210 registers with the mobile device's operating system to manage various context information associated with multiple applications actively running on the mobile device in the background. The context manager 210 interacts with applications 240, a mobile sensing service provider 250, and system settings 260 via interfaces (e.g., application programming interfaces or APIs) 211, 213, and 214 to perform content recommendation functionality using mobile sensing information.

[0036] Application 240 represents an application running on a mobile device, which can be a first-party application or a third-party application (e.g., SMS applications, WeChat, WhatsApp, etc.). Application 240 prompts the user to input information via input method 220, a software component that helps the user input text content into the application. Input method 220 can provide predictions before the user types text to reduce the user's workload. In an embodiment of the invention, input method 220 sends a request to context manager 210 via application programming interface 211 using request recommendation component 212 to seek recommendations for its text input based on user context information (including some sense data matching the corresponding mobile state, which is associated with the mobile device rooted for software system 200). Context manager 210 returns one or more recommendations generated based on the context information (including sense data obtained through status reporter 215).

[0037] The mobile sensing service 250 is a component of the software system 200 that continuously collects sensing data from one or more sensors and provides the sensing data to the context manager 210 via application programming interface 213. The mobile sensing service 250 is also registered with the operating system as a built-in system service provider. For example, the mobile sensing service 250 connects (controlled by the operating system) to one or more sensors installed in a mobile device to dynamically collect various types of sensing data. The sensing data can be stored and updated in the mobile device's memory or system memory. Optionally, the mobile sensing service 250 can be enabled by an end user to interact with the sensors at specific times to collect specific types of sensing data via interface 213 based on instructions from the context manager 210. The context manager 210 sends instructions to the mobile sensing service 250 based on information carried in the request recommendation component 212 sent from the input method 220. As an example, notifications via interface 213 can be used: when the mobility state changes, the mobile sensing service 250 notifies the context manager 210. As another example, a query / response can be used via interface 213, where context manager 210 can query mobile sensing service 250 to determine whether a state associated with a certain type of sensing data is valid. Mobile sensing service 250 will then return sensing data of the corresponding type that matches the user's context information to context manager 210.

[0038] refer to Figure 2The context manager 210 interacts with system settings 260 via recommendation control 216 through its registration with the operating system. Recommendation control 216 is a component of system settings 260 and provides end users with a switch to control which service provider content (including the data type of sensing data collected from or to be collected by one or more sensors) can be provided to the context manager 210 and used in the input method 220. End users can enable / disable the accessibility of specific types of mobile sensing data through the user interface in system settings 260 (e.g., via privacy or other settings). In other words, end users have the right to decide the accessibility of sensing data provided by mobile sensing service 250, which will be shared with application 240 via input method 220.

[0039] Figure 3 This is an operational diagram of an exemplary mobile sensing information sharing system according to an embodiment of the present invention. This diagram is merely illustrative and should not unduly limit the scope of the claims. Those skilled in the art will recognize many variations, alternatives, and modifications. Figure 3 An example of operating a mobile sensing information sharing system on a mobile device is shown. The user of the mobile device may be on a subway. In step 301, the user launches an SMS application with the intention of sending a short message in the SMS application's input module (i.e., the input method). The input module includes an input area with a touch keyboard and an input suggestion area. For example, the user has just launched the input module and types the text "I am taking" in the input module in step 302. At this time, the user may be in any of the user contexts associated with multiple mobility states (such as running, walking, cycling, on the subway, etc.). After launch, the input module sends a suggestion request to the user context manager in step 303. The user context is predefined and stored in the user context manager.

[0040] When a user uses the SMS application, it is associated with the input module. If the user switches to a different application, such as the WeChat application, the input module will be assigned or associated with the WeChat application. For example, the "reqRecommend" interface between the input module and the user context manager can be defined as follows: String[]reqRecommend(String hint) The input module will retrieve a recommendation request in the form of a string array from the user context manager. This interface can provide multiple recommendations to the input module from the user context manager. These recommendations are generated by the user context manager, which in turn seeks updated user context information from a motion sensing service that continuously senses updated user activity and corresponding types of sensing data using one or more sensors.

[0041] At the current moment (step 304), the user context manager, registered with the mobile device's operating system, sends a "Check User Activity" query to the mobile sensing service via the query / return interface. The mobile sensing service is also registered with the mobile device's operating system. This query will cause the mobile sensing service to continuously sense user activity and collect corresponding types of sensing data at the current moment (step 305). Assuming the user is on a subway, the mobile sensing service can detect user activity in the "Transport_Subway" state and send the current user context (i.e., state information or sensing data type) back to the user context manager in step 306. The user context manager can use this type of sensing data to generate a recommendation return input module in the form of the text "Subway" in step 307.

[0042] After receiving the recommendation from the user context manager, the input module displays the recommendation in step 308 by showing the text "Metro" in the input suggestion area of ​​the input module. The SMS application performs an operation, and in step 309, the user directly selects the recommended text to make typing easier. Optionally, the input module can use the recommendation to generate autocomplete suggestions in the input area of ​​the graphical user interface. Furthermore, in step 310, the user can further edit the text using the SMS application to form a complete short text message and send the text message.

[0043] In some embodiments, data security for any type of motion sensing data is ensured by the end user through a recommended control component built into the system settings. Figure 4 This is another operational diagram of a mobile sensing service according to an embodiment of the present invention. This diagram is merely illustrative and should not be overly limiting of the scope of the claims. Those skilled in the art will recognize many variations, alternatives, and modifications. Figure 4An example of operating a mobile sensing information sharing system is illustrated, in which a user context manager runs in the background to set security measures for each predefined type of sensing data. As shown in the example, the user context manager is a software component registered to the operating system of the mobile device. The mobile sensing service is another software component registered to the operating system for collecting sensing data using one or more sensors in the mobile device, such as user activity, location, accelerometer information, and other motion state information. The user context manager is used to predefine all data types of mobile sensing data, which are associated with a set of user contexts. The predefined data types may include multiple keywords for each data type and are stored in the context manager. The user context is stored as part of a corresponding user profile managed in the system settings. Recommendation controls are a software component inserted into the system settings that is associated with the end user through the user profile. Optionally, the user profile can be sent to remote storage. The user context manager is also designed to enable the mobile sensing service to detect specific types of sensing data associated with specific user context information at specific times, based on user authorization settings via recommendation controls. Furthermore, the user context manager is designed to provide one or more recommendations for input methods of various applications running on the same mobile device by converting the corresponding type of sensing data into text format.

[0044] like Figure 4 As shown, in step 401, the user context manager starts, possibly when a recommendation request is received from an application running on a mobile device via an input method. To generate a return for the recommendation request, the user context manager may need to check all predefined types of sensing data that can match the user context corresponding to the current mobile state (which the mobile sensing service can identify at any time). In step 411, for each type of predefined mobile sensing data, the user context manager sends an "Enable" query to the recommendation control to check if the state corresponding to that data type is valid (i.e., enabled by the end user). Only any type of sensing data enabled by the end user can be shared by the user context manager with the application that issued the recommendation request. In the example, the recommendation control returns "Yes" to the query, and then in step 413, the user context manager enables the mobile sensing service to perform corresponding mobile sensing detection based on the data type of the enabled mobile sensing data. The mobile sensing service returns the corresponding mobile sensing data to the user context manager. For example, if data types associated with "GO_TO_HOME" or "TRANSPORTATION" are enabled, the mobile sensing service will be enabled to detect these types of data accordingly. If the recommendation control returns a query with "No", the user context manager will send a query to check for another data type among all predefined data types.

[0045] In some embodiments, in step 402, the end user updates the user profile via the recommendation control built into the system settings, changing the settings of any predefined data types associated with one or more user contexts. In step 403, the recommendation control notifies the user context manager of the corresponding ON / OFF setting change. The ON state of a data type of sensed data allows the corresponding sensed data provided by the motion sensing service to be accessed by the user context manager for sharing with other applications. If a request for updated motion sensing data is triggered, in step 404, the user context manager sends a command to change (enable or disable) the corresponding motion sensing detection. In step 405, the motion sensing service returns the updated motion sensing data to the user context manager.

[0046] In some embodiments, the application's input method invokes it to retrieve recommendations in the form of an array of strings. This functionality can return multiple recommendations from the user context manager. In one implementation, the user context manager's recommendation request is invoked by launching the input method without the user typing any text. For example, when hint = null, it means a recommendation is requested without any prompt. In this case, a set of templates pre-generated and stored in the user context manager based on predefined rules under multiple conditions for various mobility states (of the mobile device user) can be used as returned recommendations before the user types anything (e.g., when the input method has just been launched). The user context manager has predefined mobility-sensing data types corresponding to mobility states, which are identified by the mobility-sensing service. For example, Table 1 below shows some exemplary templates for mobility states.

[0047] Table 1: Examples of templates for user context manager rules defined under various conditions for each mobility state identified by the mobility sensing service.

[0048] For each state, the user context manager will have predefined rules, and each rule will have conditions and a corresponding template. When a condition becomes true, the template will be used to generate recommendations. Rules can be implemented in different ways. For example, rules can be hard-coded into the user context manager using a programming language, or implemented using a rule engine (e.g., Apache Jena). When the input method calls `reqRecommend()` to the user context manager without any suggestions, the user context manager will iterate through each state and check (by sending a query to the recommendation control built into the system settings) whether the state is valid. If the state is valid (i.e., enabled by the user), `examStateRules()` will be called in the user context manager to check each rule predefined for that state. The user context manager will determine if the condition is true. Then, the (predefined) template corresponding to the condition of the valid state will be selected to generate recommendations.

[0049] In another implementation, the information request invoked by the input method carries a hint implicit in the text typed by the user. For example, when `hint != null`, it means the input method sends a recommendation request with a hint. The hint can be generated by the input method based on what the user has typed. For example, when the user types "My destination address is", the string "My destination address is" can be sent as a hint. If a hint is present, the user context manager compares the hint with multiple keywords that are predefined for each type of sensing data that the mobile sensing service can provide. The data types of sensing data are associated with various user contexts defined by the user context manager. For example, Table 2 below shows some predefined keywords corresponding to some types of mobile sensing data.

[0050] Table 2: Examples of predefined keywords for each sensing data type based on the corresponding user context.

[0051]

[0052] The user context manager compares the similarity between suggestions and keywords and identifies the keyword with the highest similarity. Then, a motion sensing service is enabled to detect (or retrieve) the corresponding motion sensing data, which is then provided to the user context manager. The user context manager transforms the motion sensing data into one of the possible recommendations returned to the input method. Different comparison algorithms are available for our solution. For example, a natural language understanding (NLU) algorithm is used to compare the similarity between text suggestions and keywords.

[0053] Figure 5The diagram illustrates (A) the operation of a user context manager providing motion-sensing recommendations without prompting, and (B) the steps for generating recommendations, according to an embodiment of the present invention. This diagram is merely illustrative and should not unduly limit the scope of the claims. Many variations, alternatives, and modifications will be recognized by those skilled in the art. Figure 5 As shown in section (A), the operation of the user context manager in providing motion-sensing recommendations without prompting is illustrated. In step 501, the input method initiates a recommendation request without prompting. The user context manager is called without prompting by `reqRecommend()`, initiating process 502 to display each motion state to check whether the state identified by the motion sensing service is valid (i.e., the user provides a truth value to allow the type of sensing data detected by the motion sensing service to be accessible to the user context manager). Process 502 is followed by a return notification 503 from the motion sensing service regarding whether the state is true or false. The user context manager then executes process 504 to check the rules under the state. Each rule has a predefined set of templates generated under various conditions for a specific type of sensing data detected under that state. Process 504 is used to determine if the condition of the rule is true so that the corresponding template can be selected. Finally, the user context manager can return the selected template as a recommendation to the input method.

[0054] As an alternative to illustrating how the user context manager provides motion-sensing recommendations without prompts, the process flow is as follows: Figure 5 Method 500 is illustrated in section (B). The user context manager is a software component that runs in the background of the operating system on the mobile device. Method 500 begins the operation of the user context manager at step 510, checking each mobility state after receiving a no-prompt reqRecommend() from the application's input method. Each mobility state corresponds to a user context defined in the user context manager and is associated with the data type of mobility sensing data detected by the mobility sensing service. Method 500 includes step 520, where the user context manager checks for each state whether the currently checked state has predefined rules. Optionally, method 500 includes matching various data types of sensing data with the user context associated with the mobility state of the mobile device. If step 520 outputs "No", method 500 proceeds directly to step 560 to repeat the operation for another state.

[0055] If step 520 outputs "Yes", method 500 includes step 530, where the user context manager further checks whether the conditions of any rule are true. In this step, each condition of the rule is checked. For example, for a rule with the "GO_TO_WORK" state, there are at least two conditions: a normal on-time condition and a late condition. The normal on-time condition is the default condition, and the late condition is used to notify the user of lateness. If step 530 outputs "No", method 500 includes step 540, obtaining the next rule to be checked again in step 520. If no next rule is available (i.e., each rule has been checked in the previous loop), step 540 stops the process. If step 530 outputs "Yes", method 500 includes step 550, generating a recommendation based on the template under that condition (determined to be true), and adding the recommendation to the input method's return list. Each template is predefined for the corresponding condition under the rule. For example, for a rule with the "GO_TO_WORK" state, if the default condition is true, the corresponding template is predefined as "I'm going to work". In one embodiment, a set of predetermined templates are associated with different conditions predefined for the user context and sensing data under those conditions. Method 500 then includes step 560, repeating the operation to provide motion sensing recommendations for another state.

[0056] Figure 6 The flowchart illustrating the steps of generating recommendations using motion sensing data with prompts according to embodiments of the present invention is shown. This figure is merely illustrative and should not be overly limiting of the scope of the claims. Many variations, alternatives, and modifications will be recognized by those skilled in the art. Figure 6Method 600 is illustrated for generating recommendations using motion-sensing data based on prompts given by text input typed by a user as an application on a mobile device. Method 600 begins at step 610 via a user context manager software component registered to the operating system running on the mobile device. The user context manager runs in the background of the operating system. Step 610 is triggered by a `reqRecommend()` function with a prompt initiated by the application through its input module (i.e., input method). The prompt is carried in the text typed by the user in the input area of ​​the input module. `reqRecommend()` is a recommendation request sent to the user context manager. The user context manager has a predefined set of user contexts and stores these user contexts as part of a user profile. The user contexts are associated with various user activities, locations, or other mobility state information that can be recognized by the motion-sensing service. The user context manager also predefines multiple keywords corresponding to various data types of sensed data based on the user contexts. The motion-sensing service is also a software component registered to the operating system and is used to detect all types of sensed data matching this set of user contexts at any given time using one or more sensors. For each type of motion-sensing data, the user context manager predefines some keywords associated with the user context.

[0057] Upon receiving a recommendation request with a prompt, method 600 includes step 620, checking whether keywords are stored in the user context manager for each mobility state. Each mobility state corresponds to a user context defined in the user context manager and is associated with the data type of mobility sensing data detected by the mobility sensing service. If step 620 outputs "No", method 600 proceeds directly to step 660 to repeat the operation for another mobility state.

[0058] If step 620 outputs "Yes", method 600 includes step 630, analyzing the similarity between the prompt and multiple keywords stored in the context manager. If no highly similar keywords are found in the current group, method 600 includes step 640, obtaining the next set of keywords by repeating step 620. If no next set of keywords is found, i.e., each set of keywords has been checked in previous loops, step 640 stops the process. If the keyword with the highest similarity to the prompt is determined, method 600 includes step 650, invoking the mobile sensing service to detect sensing data of the type corresponding to that keyword. If sensing data is detected or retrieved, and the type of sensing data is valid or authorized by the user to allow the user context manager to access it, the sensing data is provided to the user context manager. The user context manager then generates a recommendation by converting the sensing data into text and adds the text to the return list of the input module. After step 650, the next set of keywords is obtained in step 640 for comparison with the prompt. Finally, when there are no more keywords to compare, method 600 stops.

[0059] While the foregoing is a complete description of specific embodiments, various modifications, alternative constructions, and equivalents may be used. Therefore, the foregoing description and illustrations should not be construed as limiting the scope of the invention as defined by the appended claims.

Claims

1. A method for generating input method recommendations using sensing information, the method comprising: A context manager defines a set of user contexts; The user context is stored as part of the user profile. Acquire sensing data from one or more sensors in the first instant; Provide a set of predefined templates associated with the user context and the sensed data; Launch the first application at the second moment; Display the first application in the graphical user interface; Text input is received through the input module in the first application, and the text input is displayed on the input area of ​​the graphical user interface; Based on the text input and the first application, a first context is determined; Using the predefined template and the first context, a first recommendation is generated; as well as The first recommendation is displayed in the input area of ​​the graphical user interface.

2. The method according to claim 1, wherein, The one or more sensors include accelerometers.

3. The method according to claim 1, wherein, The input area includes a touch keyboard and an input suggestion area, with the first suggestion displayed in the input suggestion area.

4. The method of claim 1 further includes sending the user profile to a remote storage.

5. The method according to claim 1, wherein, Obtaining sensor data includes obtaining user activity.

6. The method according to claim 1, wherein, Obtaining sensor data includes collecting location information.

7. The method according to claim 1, wherein, Obtaining sensing data includes collecting accelerometer information.

8. The method of claim 1, further comprising matching various data types of the sensed data with the user context, wherein the set of predetermined templates is associated with conditions predefined for the user context and the sensed data under the conditions.

9. The method according to claim 8, further comprising: By activating the input module of the first application without typing any text, a recommendation request is received; Access the user profile to determine whether a user context in the user context at the second time point is valid to allow the context manager to access the sensed data; Determine one of the conditions associated with the user context; as well as Identify one of the predetermined templates associated with the stated condition to generate the first recommendation.

10. The method according to claim 1, further comprising the context manager predefining multiple keywords corresponding to various data types of the sensed data based on the user context.

11. The method of claim 10, further comprising: The first application receives recommendation requests based on the text input through its input module. The text input is compared with the multiple keywords based on similarity to determine the keyword with the highest similarity. as well as The first recommendation is generated using the sensing data corresponding to the keywords.

12. The method according to claim 1, wherein, Obtaining sensing data from one or more sensors includes using sensor data services registered with the operating system.

13. The method of claim 1, further comprising generating an autocomplete suggestion on the input area of ​​the graphical user interface using the first recommendation.

14. A mobile computing device, comprising: case; A display, disposed on the front of the housing, includes a touchscreen; Memory, used to store the operating system and multiple applications; Memory; Multiple sensors; as well as A processor, connected to the memory and the RAM, wherein the processor is used for: Execute instructions associated with the operating system to define a set of user contexts via a context manager; Execute instructions associated with the first application and generate a first set of context information associated with the first application; Execute instructions associated with the second application; Execute the instructions associated with the operating system to provide a user input interface on the touchscreen for the second application; Execute the instructions associated with the operating system to provide a first recommendation based on at least the first set of context information; and Execute instructions associated with the operating system to display the first recommendation on the user input interface.

15. The apparatus according to claim 14, wherein, The processor is also configured to generate a second recommendation using the plurality of sensors, the second recommendation being displayed on the user input interface.

16. The apparatus according to claim 14, wherein, The plurality of sensors includes one or more selected from accelerometers, gyroscopes, magnetometers, GPS, light sensors, touchscreens, microphones, and cameras.

17. The device of claim 14, comprising a mobile phone, a tablet computer, a wearable device, or a vehicle.

18. A method for generating input recommendations using motion sensing information, the method comprising: The context manager defines a set of user contexts, which are associated with multiple movement states; The sensing data service collects sensing data based on one or more sensors that matches the multiple movement states; Multiple templates are defined for the various movement states under different conditions; Store multiple keywords corresponding to the sensing data that match the multiple movement states; The context manager sets the accessibility of the sensed data for each of the plurality of movement states, and the accessibility is determined by a user profile. The context manager monitors the application; Launch the application, which includes an input module, to send a recommendation request to the context manager; Detect sensor data that matches the current movement state; as well as Return input recommendations to the input module.

19. The method of claim 18, further comprising: Receive the recommendation request without prompting from the input module; Determine that the current movement state is valid, so that the context manager can access the sensing data that matches the movement state; Determine the conditions associated with the movement state; Detect sensing data that matches the movement state at the current moment; as well as The input recommendation is generated using a template for the movement state under the stated conditions and the sensed data.

20. The method of claim 18, further comprising: The input module receives the recommendation request with prompts from the input text. Analyze the similarity between the prompt and the multiple keywords stored in the context manager; Identify the keywords with the highest similarity; Detect sensing data corresponding to the keywords; as well as The input recommendation is generated using the sensed data.

Images