Information processing device, method, and program

Abstract

Traditionally, it has sometimes been difficult for users to set appropriate search criteria for an application. [Solution] An information processing device that runs an operating system which executes estimation processing using a model that has learned the relationship between data corresponding to user behavior and information to be used for search processing in an application, characterized in that when an application executes the estimation processing using the operation history based on user input performed before the use of the application as input data and the output estimation result is recorded, the device retrieves the estimation result from the operating system, executes search processing using the information contained in the estimation result, and displays the result.

Description

Technical Field

[0001] The present invention relates to a technique for improving the usability of an application by using the results of machine learning or the like.

Background Art

[0002] Conventionally, an application has received a user's input operation and provided information corresponding to the operation. For example, by inputting a search word or image data, search results similar to the input are displayed.

[0003] As a search technique using machine learning technology, there is Patent Document 1. In Patent Document 1, a learning device that has learned the relationship between a transaction target and the category to which the transaction target belongs is used to estimate the category to which an object extracted from a captured image input to an application belongs. Then, using the estimated category and the feature amount of the object, a transaction target is searched from an electronic shopping street.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, input of a captured image is required to obtain search results. On the other hand, a user who performs a search may not be able to input appropriate search conditions to the application. When searching for past information or unknown information, appropriate image data may not be available at hand as search conditions, or a search word may not come to mind. In such cases, it may be complicated for the application itself to consider appropriate search conditions.

[0006] Furthermore, users attempting to perform a search within an application may have spoken or taken actions that provide clues to their search before or after launching the application. However, if such actions occurred before the application was launched, the user would need to re-enter the information corresponding to those actions as search criteria after the application is launched. This process can also be difficult for users. [Means for solving the problem]

[0007] Therefore, the present invention is an information processing device on which an operating system is executed that performs estimation processing using a model that has learned the relationship between data corresponding to a user's words and actions and information to be used for search processing in an application, and the operating system is recorded when it performs estimation processing by taking as input data information included in the operation history based on at least one of user input made to the information processing device and user input made to a device that can communicate with the information processing device via a network, an acquisition means for an application that the user has agreed to allow to use the results of the estimation processing, an acquisition means for an application that performs search processing using the information included in the acquired estimation results, and a display control means for an application that displays the results of the search processing. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to reduce the user's operations and tasks on the application by utilizing estimation results that reflect the user's words and actions on other applications before and after the application is launched. [Brief explanation of the drawing]

[0009] [Figure 1] An example of a system configuration is shown. [Figure 2] This figure shows an example of a mobile device hardware configuration. [Figure 3] This figure shows an example of a software module configuration for a mobile device. [Figure 4] An example of data for input to a trained model. [Figure 5] A flowchart illustrating a series of processes, including those utilizing pre-trained models, implemented by the operating system of a mobile device. [Figure 6] An example of data relating to the output obtained using a pre-trained model. [Figure 7] A flowchart illustrating the process of providing estimation results, implemented by the mobile device's operating system, to an application. [Figure 8] A flowchart illustrating the processes implemented by applications that receive information from the operating system. [Figure 9] This is an example of a screen provided by the application in this embodiment. [Figure 10] A flowchart illustrating the processes implemented by the application in the example. [Figure 11] This is an example of a screen provided by an application in an application example. [Modes for carrying out the invention]

[0010] Figure 1 shows an example of the system configuration in this embodiment.

[0011] 101 is a mobile device. Mobile devices 101 include, for example, smartphones, tablets, laptops, and wearable devices. 102 is a voice assistant terminal. The voice assistant terminal 102 receives voice input from the user and responds with search results for that input. 103 is a peripheral device. Peripheral devices 103 include digital home appliances such as televisions, refrigerators, and microwave ovens, as well as in-car terminals such as car navigation systems.

[0012] The mobile device 101 can communicate with the voice assistant terminal 102 and peripheral devices 103 via a network. In this embodiment, communication via a wireless network such as Bluetooth® is given as an example. Other connection methods may be used to achieve communication between devices.

[0013] The mobile device 101 continuously acquires information corresponding to the user's voice input from the voice assistant terminal 102 via communication and records it along with the time. The mobile device 101 continuously acquires information corresponding to the user's voice input, function operation, location information, etc. from the peripheral device 103 via communication and records it along with the time.

[0014] Figure 2 shows an example of the hardware configuration of an information processing device such as a mobile device 101.

[0015] The CPU 201 executes programs stored in the ROM 203 and programs such as an OS (Operating System) and applications loaded from the storage device 204 to the RAM 202. That is, by executing the program stored in a readable storage medium, the CPU 201 functions as each processing unit that executes the processing of each flowchart described later. The RAM 202 is the main memory of the CPU 201 and functions as a work area and the like. The touch panel 206 is the display unit of the mobile device 101 and is also an input device. On the display unit of the touch panel 206, the results of function execution by the OS and applications are displayed. Also, when a user operation on the touch panel 206 is detected, desired control is executed by the corresponding program. Note that the input device of the mobile device 101 is not limited to the touch panel. For example, voice input by the microphone 207 and image input by the camera 209 are also possible. Also, a position measuring device 210 such as GPS also serves as an input device for position information.

[0016] The network I / F 205 is connected to a local network and communicates with devices connected to the network. The short-range communication I / F 208 is an I / F that performs input / output of short-range communication such as Bluetooth or Near Field Communication (NFC), and communicates with the connected devices. Each component of the mobile device 101 is connected to the internal bus 210 and can communicate with each other.

[0017] The voice assistant terminal 102 and the peripheral device 103 also have a hardware configuration equivalent to that of the mobile device 101. That is, at least a processor, a memory, a storage device, a network I / F, a short-range communication I / F, and a mechanism for receiving input are provided. Also, for the peripheral device 103, hardware according to the application will be additionally provided.

[0018] Figure 3 shows an example of the software module configuration of the mobile device 101. In this figure, the main components of the processing are represented by the execution of one or more programs, such as an OS and applications, for this embodiment.

[0019] Modules 301-305 indicate modules that correspond to services provided by the OS.

[0020] The data collection unit 301 records the details of user operations on the touch panel 206 and voice input from the microphone 207, including the time (year, month, day, and time) and location information. It also collects operation history information related to the mobile device 101 or its user (owner) by making requests to the voice assistant terminal 102 and peripheral devices 103. This information is managed in the storage device 204, as shown in Figure 4(a).

[0021] Furthermore, the present invention includes the OS, applications, and combinations thereof installed on the mobile device 101.

[0022] Figure 4(a) is a table that stores information collected by the collection unit 301 regarding manual and voice input operations performed by the user inside and outside the mobile device 101. For applications running on the OS, operation history is automatically recorded if the user consents to allow the recording of operation history to the OS during installation. For services outside the device, operation history can be obtained from each service only if it is possible to cooperate with the OS and the user consents. This table manages the history, with one record recorded in the format defined within each application and service. The contents include time, function information (identification information for functions such as applications and services), user information, location information, and operation details. The information collected by the collection unit 301 may also include program information and usage information provided by peripheral devices 103 such as televisions and digital home appliances.

[0023] The input control unit 302 vectorizes the information collected by the collection unit 301 and inputs it to the analysis service unit 303. The vectorized data is recorded in the input management table shown in Figure 4(b) as needed. For example, the collected information is converted or filtered to become input data. When the analysis service unit 303 processes input at the word level, it divides the information contained in the history operation content into word units, removes pronouns, etc., and extracts necessary information such as nouns to generate input data. The analysis service unit 303 may also use location information as an additional parameter for analysis and estimation processing, so location information is also recorded. The input control unit 302 can operate in response to information collection by the collection unit 301, but it may also operate asynchronously.

[0024] The analysis service unit 303 is a so-called AI (Artificial Intelligence) function that acquires the input data managed in Figure 4(b) and performs estimation processing using a trained model. In addition to the input data managed in Figure 4(b), the trained model used by the analysis service unit 303 also performs estimation processing using location information and the profile information of the owner of the mobile device 101 as parameters, which are linked to the input being analyzed.

[0025] Figure 4(c) shows the owner's profile information registered on the mobile device 101. This includes place of origin, date of birth, gender, hobbies, address, occupation, etc. Additionally, the user can optionally add their OS-supported religion and affiliated communities.

[0026] Here, the analysis service unit 303 provided by the OS can communicate with a cloud service for retraining the model via a network interface 205. The cloud service is a platform provided to realize the processing during the learning phase for creating the model used by the analysis service unit 303. The trained model used in this embodiment is a model that outputs words that can be used as search terms in applications running on the OS, trained by preparing a large amount of data in a predetermined format from input sources that the OS can support. Input sources that the OS can support include the OS assistant function that performs searches on voice input from a microphone provided by the OS, and voice assistant terminals 102 and peripheral devices 103 provided by a predetermined vendor. User profile information managed by the OS vendor's service is also used for training. In this embodiment, the model is generated by supervised learning using SVM (Support Vector Machine) to output search terms that can derive appropriate search results for the aforementioned inputs. However, various other algorithms can be applied, and for example, a neural network incorporating deep learning may be used.

[0027] The Information Provision Unit 304 provides the results of the estimation process performed by the Analysis Service Unit 303 to the linked application. For applications running on the OS, information is provided only if the user consents to accept recommendations based on the estimation results from the Analysis Service Unit 303 during installation or other similar processes. From this point forward, it is assumed that this consent has already been given for applications with the analysis service linkage function enabled. The Information Provision Unit 304 exposes an API (Application Programming Interface) to provide the estimation processing results to the application when the application starts up or upon request from the application.

[0028] The function control unit 305 receives user input and launches applications and other programs.

[0029] Modules 311-313 and 321-323 represent modules corresponding to services provided by two applications (Application A and Application B) installed to run on the OS.

[0030] Functional units 311 and 321 provide functions specific to each application. For example, a web browser analyzes HTML documents related to a website to generate display images and execute scripts. For social networking applications or messaging applications, they acquire and process messages and user information managed in chronological order. Display control units 312 and 322 display the processing results from functional units 311 and 321. Coordination units 313 and 323 function with the user's consent and acquire information via the information provision unit 304. The acquired information is processed by functional units 311 and 321 and displayed by display control units 312 and 322.

[0031] Figure 5 is a flowchart illustrating a series of processes, including those utilizing a pre-trained model, implemented by the operating system of the mobile device 101. The program for this process is assumed to be provided embedded within the OS to utilize the pre-trained model. However, this process may also be implemented by a program acquired externally via a network or other means and subsequently installed.

[0032] This process will be executed periodically, such as every few minutes, while the OS is running. It may also be executed at other times, such as when new operation history is recorded. Furthermore, it will be performed automatically in the background, asynchronously with any processing by the application receiving the information.

[0033] In S501, the collection unit 301 collects history data corresponding to manual and voice input operations performed by the user inside and outside the mobile device 101, and manages it as shown in Figure 4(a). The data that can be collected includes, as mentioned above, user operations on the touch panel 206, voice input from the microphone 207, and operation details recorded by network-connected devices (such as the voice assistant terminal 102 and peripheral devices 103).

[0034] In S502, the input control unit 302 extracts the unprocessed records managed in Figure 4(a) and vectorizes the data (contents) of those records. As mentioned above, this is a process performed by the input control unit 302 for input to the analysis service unit 303. The collected information may be converted or filtered to become input data. The information, including the vectorized data, is managed as shown in Figure 4(b).

[0035] In S503, the input control unit 302 inputs the vectorized input data to the analysis service unit 303. In this case, in addition to inputting the input data of the latest record included in Figure 4(b), multiple different inputs may be made, such as inputting new input data for multiple records all at once.

[0036] In S504, the analysis service unit 303 performs estimation processing using the input data acquired from the input control unit 302 and the trained model. In S505, the analysis service unit 303 records the results of the estimation processing in a table (Figure 6) in the storage device 204. Figure 6 shows the results (output) of the estimation processing obtained by the analysis service unit 303, the time, and the content of the corresponding input data.

[0037] The data recorded in Figure 6 (especially the results of the estimation process as output) will be provided to Application A and Application B by the Information Provision Unit 304.

[0038] Figure 7 is a flowchart illustrating the process by which the analysis service unit 303, implemented by the OS of the mobile device 101, provides the estimation results to the application. The program for this process is intended to be embedded in the OS to utilize a pre-trained model. However, this process may also be implemented by a program acquired externally via a network or other means and subsequently installed.

[0039] Figure 7(a) shows the processing that occurs when an application is launched by the function control unit 305.

[0040] In S701, the function control unit 305 detects an application startup instruction that follows user operation. In S702, the function control unit 305 determines whether the application for which the startup instruction was detected has the analysis service linkage function enabled. If it is enabled, the process proceeds to S703; otherwise, this process ends.

[0041] In S703, the information provision unit 304 obtains estimation results from the table shown in Figure 6. The records obtained are one or more records within a predetermined period from the current time. In this case, information not yet provided to the application may be managed, and only the relevant records may be obtained. In S704, the information provision unit 304 provides the information of the records obtained in S703 to the application as estimation results. When providing the information, only the content included in the output managed in Figure 6 may be provided.

[0042] In addition, in S704, the function control unit 305 may be designed to pass the information of the record acquired in S703 as a parameter when the application is started.

[0043] Figure 7(b) shows the processing when the information provision unit 304 interacts with an application that is currently running. In the mobile device 101, the information provision unit 304 operates as a resident service and executes this processing.

[0044] In S711, the information provision unit 304 determines whether or not it has received a request for estimation results from the application. If the request has been received, it proceeds to S712; otherwise, it proceeds to S714.

[0045] In S712, the information provision unit 304 obtains estimation results from the table shown in Figure 6, similar to S703. In S713, the information provision unit 304 provides the application with the information from the obtained records as estimation results, similar to S704.

[0046] In S714, the information provision unit 304 determines whether or not it has received feedback from the application. The feedback indicates that the provided estimation results have been used. If feedback has been received, the process proceeds to S715; otherwise, it proceeds to S711.

[0047] In S715, the information provision unit 304 identifies the input data entered into the analysis service unit 303 when the estimated results used by the user included in the feedback are obtained, referring to Figure 6. Furthermore, the information provision unit 304 identifies the identification information of the function that is the source of the identified input data, referring to Figure 4(b). In S716, the information provision unit 304 links the identified function identification information with the received feedback information and records it in the storage device 204.

[0048] This recorded data is provided via the OS to the aforementioned cloud service for retraining the model. The retrained and learned model can be downloaded and installed at times such as when the OS is updated.

[0049] Figure 8 is a flowchart illustrating the processing performed by the application that receives information from the information provision unit 304. Here, we will describe the runtime processing of application A, in which the analysis service linkage function is enabled.

[0050] In S801, the display control unit 312 displays the home screen of application A in accordance with the application startup instruction that follows user operation.

[0051] In S802, the functional unit 311 performs a search using the estimation results provided by the information provision unit 304. Specifically, a search is performed using words such as "basketball" and "super play compilation" included in the estimation results, and users who have made related posts are automatically found. The search by the functional unit 311 may be performed by connecting all the provided estimation results with an AND condition, or by using an OR condition. Alternatively, a search may be performed after selectively extracting a portion of the estimation results according to the search target of application A or the search results using an OR condition. The search target may be an external area connected via a network.

[0052] In S803, the display control unit 312 automatically updates the display content based on the search results from the function unit 311.

[0053] Figure 9 shows an example of the screen of application A provided on the updated mobile device 101. 901 displays objects that application A has acquired via the network and that are currently attracting attention from many other users. 902 displays the results of a search process using estimation results provided by the information provision unit 304, labeled as "AI search." This display 902 can represent the results of the processing desired by the user for application A, derived from estimation results output by the OS obtained from the history of the user's actions from before application A was launched until immediately afterward, and from the action of launching application A.

[0054] In S804, the functional unit 311 determines whether a predetermined time has elapsed since the previous processing in S802. If time has elapsed, it proceeds to S805; otherwise, it proceeds to S806. In S805, the linkage unit 313 requests the information provision unit 304 for the estimation results. If the estimation results are received, the processing from S802 onwards is executed again. If there is no difference in the estimation results, the functional unit 311 may skip the search processing in S802 and the update processing in S803. S804 and S805 allow the latest estimation results from the analysis service unit 303 to be applied to the application.

[0055] In S806, the functional unit 311 determines whether or not there has been a user interaction with the screen provided by application A. If there has been a user interaction, the process proceeds to S807; otherwise, it returns to S804.

[0056] In S807, the functional unit 311 executes processing according to user operations. Specifically, if objects or links included in display 901 or display 902 are specified, display switching and processing for providing information according to those specifications are executed.

[0057] In S808, the functional unit 311 determines whether the user operation resulting from the processing in S807 corresponds to the estimation result received from the information provision unit 304. For example, if an object or link included in the display 902 is specified, it is determined that there was a user operation corresponding to the estimation result. If the user operation corresponds to the estimation result, the process proceeds to S809; otherwise, it returns to S804. In S809, the linkage unit 313 sends feedback to the information provision unit 304, including the estimation result corresponding to the user operation.

[0058] (Application Example 1) Figure 10 shows an example of applying the previously described embodiment (Figure 8) to a different application. In Figure 10, the same reference numerals are used for processes that are the same as in Figure 8, and their explanation is omitted here. The differences and their effects will be explained below.

[0059] In S1001, the functional unit 311 determines whether the process executed in S807 was a predetermined user operation. If it was a predetermined user operation, the process proceeds to S1002; otherwise, the process proceeds to S808.

[0060] Specific examples of designated user operations include scrolling by flicking and operations on display objects that correspond to the search function.

[0061] Specifically, if the running application displays user-submitted messages chronologically on a timeline, we assume the user scrolled using a flick rather than a swipe. In this case, the user who performed the flick is expected to want to view messages or image data posted relatively far in the past relative to the current time. If the user is viewing data from several years or even decades ago, the flick scrolling operation will be detected repeatedly.

[0062] Furthermore, regardless of the application, if the operation is performed on a display object that corresponds to a pre-defined search function, it is possible to understand the user's intention to perform a search using a specific search function.

[0063] This application example utilizes the estimation results provided by the information provision unit 304, taking such user intent into account.

[0064] In S1002, the functional unit 311 executes a search process using the estimation results provided by the information provision unit 304 and automatically updates the display content to reflect the search results.

[0065] Figure 11(a) is an example of a screen provided by an application that displays user-submitted messages chronologically on a timeline, reflecting the processing of S1002.

[0066] On this screen, 1101 indicates an additional object that allows users to jump to messages or time information corresponding to the search results in the data included in the estimation results, among the multiple messages contained at the scroll bar's destination. Users may be able to obtain the desired message by selecting one of the objects 1101 that automatically appear while flicking. Alternatively, they can directly scroll to the message corresponding to the search results by flicking.

[0067] Furthermore, if a message corresponding to a search result in S1002 is included among multiple messages displayed while the screen is scrolling via flick, it is possible to stop scrolling while that message is displayed to prevent the user from skipping it.

[0068] Figure 11(b) is an example of a screen that reflects the processing of S1002 when a user selects a specific search function in a running application.

[0069] In Figure 11(b), in addition to the input field where the user can arbitrarily enter search terms, the results of the "AI search" using the estimation results provided by the information provision unit 304 are displayed, as shown in 1102. Here, an automatic search is performed using word search and time period (from the current time to the time included in the estimation results). The user can perform additional operations such as arbitrarily adding search terms using the input field or deleting search terms from the "AI search".

[0070] In this application, by automatically applying the aforementioned estimation results in addition to the user's predetermined actions on the application, it may be possible to quickly provide the user with the search targets they desire. In this application, the user may be able to eliminate tasks such as visually finding the search targets or thinking up and entering appropriate search terms.

[0071] (Application Example 2) The aforementioned OS also includes reporting and bias adjustment functions.

[0072] The reporting function is designed to show the rationale behind how a trained model derived an output from input data during the estimation process. For example, if a single output is selected, the input data and parameters that led to that output are listed, and the extent to which each of the listed data points influenced the derivation of that output is displayed.

[0073] The bias adjustment function is designed to detect and notify the user when the pre-trained model's estimation process tends to produce an output undesirable by the user for a given input. Furthermore, this function also includes the ability to set a transformation rule in the input control unit 302 to reflect the inverse bias in the given input in order to produce the estimation result desired by the user, according to the user's instructions. Once this setting is made, thereafter, when a given input is received, at least some of the information in the input will be modified according to the transformation rule and used in the pre-trained model.

[0074] (Other examples) The present invention also includes apparatuses, systems, and methods configured by appropriately combining the embodiments described above.

[0075] Here, the present invention is a device or system that is the main body for executing one or more software programs that realize the functions of the embodiments described above. Furthermore, a method for realizing the embodiments described above that are executed on that device or system is also part of the present invention. The program is supplied to the system or device via a network or various storage media, and the program is read into one or more memories by one or more computers (CPU, MPU, etc.) of the system or device and executed. In other words, as part of the present invention, the program itself or various storage media that can be read by the computer storing the program are also included. Furthermore, the present invention can also be realized by a circuit (e.g., ASIC) that realizes the functions of the embodiments described above. [Explanation of Symbols]

[0076] 101 Mobile devices 102 Voice Assistant Terminal 103 Peripherals

Claims

1. An information processing device on which an operating system is executed that performs estimation processing using a model that has learned the relationship between data corresponding to user behavior and information that should be used for search processing in the application, When the estimation result output by executing the estimation process in the aforementioned operating system is recorded, using as input data information included in the operation history based on at least one of the user input made to the information processing device and the user input made to a device that can communicate with the information processing device via a network, An acquisition means for obtaining the estimation results from the operating system by an application whose user consents to the use of the results of the estimation process, A search means that performs a search process using the information contained in the estimation result obtained by the application, A display control means for displaying the results of the search process by the aforementioned application, An information processing device characterized by having the following features.

2. The application further includes a request means for requesting the estimation result from the operating system, The information processing apparatus according to claim 1, wherein the acquisition means acquires the estimation result in response to the request.

3. The information processing apparatus according to claim 2, characterized in that the request means periodically requests the estimation result from the operating system.

4. The information processing apparatus according to any one of claims 1 to 3, characterized in that the acquisition means acquires the estimation result from the operating system when the application is started.

5. The information processing apparatus according to any one of claims 1 to 4, further comprising a transmission means for transmitting feedback including the estimation result to the operating system when the application receives a user operation corresponding to the result of a search process using the information contained in the estimation result.

6. When the search means detects a predetermined operation by the user, it performs a search process on the search target corresponding to the predetermined operation using the information contained in the estimation result. The information processing apparatus according to any one of claims 1 to 5, characterized in that the display control means updates the display content by reflecting the display using the results of the search process in the content to be displayed according to the predetermined operation.

7. The information processing apparatus according to claim 6, characterized in that the predetermined operation is a flicking scroll operation, and the scroll destination is a display using the results of the search process.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that the devices that can communicate via the aforementioned network include at least one of a voice assistant terminal, a digital consumer electronics device, and an in-vehicle terminal.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that the user input includes voice input.

10. A method in an information processing device in which an operating system is executed that performs estimation processing using a model that has learned the relationship between data corresponding to user behavior and information to be used for search processing in an application, When the estimation result output by executing the estimation process in the aforementioned operating system is recorded, using as input data information included in the operation history based on at least one of the user input made to the information processing device and the user input made to a device that can communicate with the information processing device via a network, A process of obtaining the estimation results from the operating system by an application whose user has consented to the use of the results of the estimation process, A search step by which the application performs a search process using the information contained in the acquired estimation result, A display control step for displaying the results of the search process by the aforementioned application, A method characterized by having the following:

11. A program for causing a computer to function as each of the means described in any one of claims 1 to 9.

Images