Program, information processing apparatus, method, and system

Abstract

To solve a problem in which conventionally, a user may have difficulty in setting an appropriate search condition to an application.SOLUTION: An information processing apparatus runs an operating system that executes estimation processing using a model that has learned the relationship between data corresponding to user's actions and words and information to be used for search processing in an application. When an estimation result output by executing the estimation processing using, as input data, an operation history based on the user's input performed before the use of the application has been recorded, the application acquires the estimation result from the operating system, executes the search processing by using information included in the estimation result, and performs display using a result of the processing.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a technique for improving the usability of an application by using the results of machine learning or the like.

Background Art

[0002] Conventionally, an application has received a user's input operation and provided information corresponding to the operation. For example, by inputting a search word or image data, search results similar to the input are displayed.

[0003] There is Patent Document 1 as a search technique using machine learning technology. In Patent Document 1, a learning device that has learned the relationship between a transaction target and the category to which the transaction target belongs is used to estimate the category to which an object extracted from a photographed image input to an application belongs. Then, using the estimated category and the feature amount of the object, a transaction target is searched from an electronic shopping street.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, input of a photographed image is required to obtain search results. On the other hand, a user who performs a search may not be able to input appropriate search conditions to the application. When searching for past information or unknown information, appropriate image data may not be available as search conditions, or a search word may not come to mind. In such cases, it may be complicated for the application itself to consider appropriate search conditions.

[0006] In addition, a user who attempts to perform a search in an application may talk or act in a way that provides hints for the search before and after the application is launched. However, if such user speech or actions occur before the application is launched, after the application is launched, the user needs to re-set, as search conditions for the application, information corresponding to such speech or actions. Such operations may also be difficult for the user.

Means for Solving the Problem

[0007] Therefore, the present invention is, for example, Display messages posted among multiple users in chronological order A program incorporated into an information processing apparatus in which an application is installed, together with an operating system, the information processing apparatus using, as input data, data corresponding to an input based on user speech or actions, and using a model learned to output application to be used in the search process An information recording step of recording output information output by an estimation process executed using, as input data, data corresponding to at least one of an input to the information processing apparatus by the user and an input to a device communicable with the information processing apparatus, and, for the application installed in the information processing apparatus, for the search process of posts related to the output information in the application A providing step of providing the output information recorded in the recording step, and in the providing step, the output information is provided as a parameter at the time of launching the application for the user of the information processing apparatus For the application in which a startup instruction following the operation is detected, the detected characterized in that.

Advantages of the Invention

[0008] According to the present invention, it becomes possible to reduce operations and tasks by the user for an application by using an estimation result reflecting user speech and actions performed on something other than the application before and after the application is launched.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0010] FIG. 1 shows an example of the system configuration in this embodiment.

[0011] 101 is a mobile device. The mobile device 101 includes, for example, smartphones, tablets, notebook personal computers, wearable terminals, etc. 102 is a voice assistant terminal. The voice assistant terminal 102 receives the user's voice input and responds with the search results for the input. 103 is a peripheral device. The peripheral device 103 includes digital home appliances such as TVs, refrigerators, and microwave ovens, and in-vehicle terminals such as car navigation systems.

[0012] The mobile device 101 can communicate with the voice assistant terminal 102 and the peripheral device 103 via a network. In this embodiment, communication via a wireless network such as Bluetooth (registered trademark) is taken as an example. Other connection forms may be used to realize communication between devices.

[0013] The mobile device 101 obtains information corresponding to the user's voice input from the voice assistant terminal 102 at any time via communication and records it together with the time. The mobile device 101 obtains information corresponding to the user's voice input, function operation, position information, etc. from the peripheral device 103 at any time via communication and records it together with the time.

[0014] FIG. 2 is a diagram showing an example of the hardware configuration of an information processing device such as the mobile device 101.

[0015] The CPU 201 executes programs stored in the ROM 203 and programs such as an OS (operating system) and applications loaded from the storage device 204 to the RAM 202. That is, by executing the program stored in the readable storage medium, the CPU 201 functions as each processing unit that executes the processing of each flowchart described later. The RAM 202 is the main memory of the CPU 201 and functions as a work area and the like. The touch panel 206 is the display unit of the mobile device 101 and is also an input device. On the display unit of the touch panel 206, the results of function execution by the OS and applications are displayed. Also, when a user operation on the touch panel 206 is detected, the corresponding program executes the desired control. Note that the input device of the mobile device 101 is not limited to the touch panel. For example, voice input by the microphone 207 and image input by the camera 209 are also possible. Also, a position measuring device 210 such as GPS also serves as an input device for position information.

[0016] The network I / F 205 is connected to the local network and communicates with devices connected to the network. The short-range communication I / F 208 is an I / F that performs input / output of short-range communication such as Bluetooth or Near Field Communication (NFC), and communicates with the connected devices. Each component of the mobile device 101 is connected to the internal bus 210 and can communicate with each other.

[0017] The voice assistant terminal 102 and the peripheral device 103 also have the same hardware configuration as the mobile device 101. That is, at least a processor, a memory, a storage device, a network I / F, a short-range communication I / F, and a mechanism for receiving input are provided. In addition, for the peripheral device 103, hardware according to the application will be further added.

[0018] FIG. 3 is a diagram showing an example of the module configuration of the software of the mobile device 101. In this figure, for the purpose of this embodiment, it represents the main body of the processing realized by executing one or more programs such as an OS and an application.

[0019] 301 to 305 indicate modules corresponding to services provided by the OS.

[0020] When the collection unit 301 detects a user operation on the touch panel 206 or a voice input by the microphone 207, it records the content together with the time (year / month / day / hour) and position information. In addition, it makes a request to the voice assistant terminal 102 and the peripheral device 103, and collects by receiving the history information of operations related to the mobile device 101 or the user (owner) of the mobile device 101. These information are managed by the storage device 204 as shown in FIG. 4(a).

[0021] Note that the present invention includes each of the OS and applications installed in the mobile device 101, and further combinations thereof.

[0022] Figure 4(a) is a table storing information collected by the collection unit 301 regarding manual operations and voice input operations performed by the user inside and outside the mobile device 101. For applications operating on the OS, they are automatically recorded when there is consent from the user to permit recording of the operation history with respect to the OS during installation. For services outside the device, operation history can be obtained from each service only when it is possible to cooperate with the OS and there is consent from the user. In this table, the histories recorded one record at a time in the formats defined within each application and service are collectively managed. The content includes time, function information (identification information of functions such as applications and services), user information, location information, operation details, etc. The information collected by the collection unit 301 may include program information and usage information provided by TVs and digital home appliances as peripheral devices 103.

[0023] The input control unit 302 vectorizes the information collected by the collection unit 301 and inputs it to the analysis service unit 303. The vectorized data is recorded in the input management table shown in Figure 4(b) as needed. For example, the collected information is data-converted or filtered to be used as input data. When the analysis service unit 303 processes the input in terms of words, the information included in the operation details of the history is split into word units, and unnecessary information such as pronouns is removed, and necessary information such as nouns is extracted to generate the input data. Also, since the analysis service unit 303 may use location information, etc. as additional parameters for analysis and estimation processing, the location information is also recorded. The input control unit 302 can operate in response to the information collection by the collection unit 301, but it may also operate asynchronously.

[0024] The analysis service unit 303 is a so-called AI (Artificial Intelligence) function that acquires the input data managed in Fig. 4(b) and executes an estimation process using a learned model. The learned model used by the analysis service unit 303 executes an estimation process using, as parameters, position information associated with the input to be analyzed and the profile information of the owner of the mobile device 101, in addition to the input data managed in Fig. 4(b).

[0025] Fig. 4(c) shows the profile information of the owner registered in the mobile device 101. It includes place of birth, date of birth, gender, hobbies, address, occupation, etc. In addition, religions supported by the OS and communities to which the owner belongs can be arbitrarily added.

[0026] Here, the analysis service unit 303 provided by the OS can communicate with a cloud service for re-educating the model via the network I / F 205. The cloud service is a platform prepared to realize the processing in the learning stage for creating the model used by the analysis service unit 303. The learned model used in this embodiment is a model that outputs words that can be search words used in the search of an application operating on the OS, which is learned by preparing a large amount of data in a predetermined format from input sources supported by the OS. Input sources supported by the OS include the OS assistant function that searches for voice input from the microphone provided by the OS, the voice assistant terminal 102 and peripheral devices 103 provided by a predetermined vendor, etc. The profile information of the user managed by the service of the vendor providing the OS is also used in the learning. In this embodiment, in order to output search words that can derive appropriate search results for the aforementioned input, a model is generated by supervised learning using SVM (Support Vector Machine). However, various other algorithms are also applicable. For example, a neural network incorporating deep learning may be used.

[0027] The information providing unit 304 provides the result of the estimation process by the analysis service unit 303 to the cooperating applications. For applications operating on the OS, information is provided when there is consent from the user to permit receiving recommendations based on the estimation results from the analysis service unit 303 during installation or the like. Hereafter, it is assumed that this consent has already been given for applications with the cooperation function of the analysis service enabled. The information providing unit 304 publishes an API (Application Programming Interface) for providing the result of the estimation process to the application when the application is launched or in response to a request from the application.

[0028] The function control unit 305 receives user operations and launches applications and the like.

[0029] 311 to 313, 321 to 323 indicate modules corresponding to services provided by two applications (Application A, Application B) installed to operate on the OS.

[0030] The function units 311, 321 provide functions that the applications individually have. For a web browser, it analyzes HTML documents related to websites and generates display images or executes scripts. For applications for social networks or applications for message exchange, it acquires and processes messages and user information managed in time series. The display control units 312, 322 display the processing results by the function units 311, 321 and the like. The cooperation units 313, 323 function with the consent of the user and acquire information via the information providing unit 304. The acquired information is processed by the function units 311, 321 and displayed by the display control units 312, 322.

[0031] FIG. 5 is a flowchart for explaining a series of processes including processes using a learned model, which are realized by the OS of the mobile device 101. The program related to this process is assumed to be provided in a form incorporated into the OS in order to use the learned model. Note that this process may also be realized by a program that is obtained from the outside via a network or the like and additionally installed.

[0032] This process is periodically executed at intervals of several minutes or the like while the OS is operating. It may also be executed at other timings such as when a new operation history is recorded. And it is assumed to be automatically performed in the background asynchronously with the processes by the application targeted for information provision.

[0033] In S501, the collection unit 301 collects history data corresponding to manual operations or voice input operations by the user performed inside and outside the mobile device 101, and manages it as shown in FIG. 4(a). The data that can be collected is, as described above, user operations on the touch panel 206, voice input by the microphone 207, operation contents recorded by cooperative devices on the network (such as the voice assistant terminal 102 and peripheral devices 103), and the like.

[0034] In S502, the input control unit 302 extracts unprocessed records managed in FIG. 4(a), and vectorizes the data (contents) of the records. This is the process performed for input to the analysis service unit 303 by the input control unit 302 as described above. Note that the collected information may be data-converted or filtered in order to be input data. The information including the vectorized data is managed as shown in FIG. 4(b).

[0035] In S503, the input control unit 302 inputs the vectorized input data to the analysis service unit 303. At this time, in addition to inputting the input data of the latest record included in FIG. 4(b), a plurality of different inputs such as inputting a plurality of new record input data together may be performed.

[0036] In S504, the analysis service unit 303 executes an estimation process using the input data acquired from the input control unit 302 and the learned model. In S505, the analysis service unit 303 records the result of the estimation process in a table (Figure 6) in the storage device 204. In Figure 6, the result (output) of the estimation process obtained by the analysis service unit 303, the time at that time, and the content of the corresponding input data are recorded.

[0037] The data recorded in Figure 6 (especially the result of the estimation process as the output) will be provided to application A and application B by the information providing unit 304.

[0038] Figure 7 is a flowchart for explaining the process of providing the estimation result by the analysis service unit 303 realized by the OS of the mobile device 101 to the application. It is assumed that the program related to this process is provided in a form incorporated into the OS in order to use the learned model. Note that this process may also be realized by a program acquired from the outside via a network or the like and additionally installed.

[0039] Figure 7(a) is a process in the case where the application is started by the function control unit 305 in particular.

[0040] In S701, the function control unit 305 detects a start instruction for an application according to a user operation. In S702, the function control unit 305 determines whether the application for which the start instruction has been detected has an active analysis service cooperation function. If it is active, the process proceeds to S703; if it is not active, this process ends.

[0041] In S703, the information providing unit 304 obtains the estimation result from the table shown in FIG. 6. One or more records within a predetermined period from the current time are obtained as the records to be obtained. At this time, information that has not been provided to the corresponding application may be managed so that only the corresponding records are obtained. In S704, the information providing unit 304 provides the information of the records obtained in S703 to the application as the estimation result. When providing, only the content included in the output managed in FIG. 6 may be provided.

[0042] Note that in S704, the function control unit 305 may be designed to pass the information of the records obtained in S703 as parameters at the time of application startup.

[0043] FIG. 7(b) is a process in the case where the information providing unit 304 particularly cooperates with the application being started. In the mobile device 101, the information providing unit 304 operates resident as a service and executes this process.

[0044] In S711, the information providing unit 304 determines whether it has received a request for an estimation result from the application. If a request is received, the process proceeds to S712; if not, the process proceeds to S714.

[0045] In S712, the information providing unit 304 obtains the estimation result from the table shown in FIG. 6 in the same manner as S703. In S713, the information providing unit 304 provides the information of the obtained records to the application as the estimation result in the same manner as S704.

[0046] In S714, the information providing unit 304 determines whether it has received feedback from the application. The feedback indicates that the provided estimation result has been used. If feedback is received, the process proceeds to S715; if not, the process proceeds to S711.

[0047] In S715, when the estimated result used by the user included in the feedback is obtained, the information providing unit 304 identifies the input data input to the analysis service unit 303 with reference to FIG. 6. Further, the information providing unit 304 identifies the identification information of the function that is the input source of the identified input data with reference to FIG. 4(b). In S716, the information providing unit 304 associates the identified identification information of the function with the received feedback information and records it in the storage device 204.

[0048] This recorded data is provided to the cloud service for re-educating the above-described model via the OS. The re-educated learned model can be downloaded and installed at the timing of an OS update or the like.

[0049] FIG. 8 is a flowchart for explaining the processing realized by an application that receives information from the information providing unit 304. Here, the processing at the time of executing application A in which the cooperation function of the analysis service is effective will be described.

[0050] In S801, the display control unit 312 displays the home screen of application A in accordance with the startup instruction of the application according to the user operation.

[0051] In S802, the function unit 311 executes a search process using the estimated result provided from the information providing unit 304. Specifically, a search process using words such as "basket" and "super play collection" included in the estimated result is executed, and users who have made related posts or the like are automatically searched. The search by the function unit 311 may be performed by connecting all the provided estimated results with an AND condition or an OR condition. Further, the search may be performed after selectively extracting a part of the estimated result according to the search target of application A or the search result under the OR condition. The search target may be in an external area connected via a network.

[0052] In S803, the display control unit 312 receives the search result from the functional unit 311 and automatically updates the display content.

[0053] FIG. 9 is an example of a screen of application A provided by the updated mobile device 101. 901 displays an object that is currently attracting the attention of many other users, which application A has acquired via the network. In 902, as "AI search", the result of a search process using the estimation result provided by the information providing unit 304 is displayed. This display 902 can be the result of the estimation result output by the OS obtained from the history corresponding to the user's actions from before the activation of application A until immediately after, and the result of the process desired by the user for application A obtained from an action such as activating application A.

[0054] In S804, the functional unit 311 determines whether or not a predetermined time has elapsed since the process of S802 was last performed. If it has elapsed, the process proceeds to S805; if not, the process proceeds to S806. In S805, the cooperation unit 313 executes a request for an estimation result to the information providing unit 304. When the estimation result is received, the processes after S802 are executed again. If there is no difference in the estimation result, the functional unit 311 may skip the search process in S802 and the update process in S803. The latest estimation result by the analysis service unit 303 can be applied to the application by S804 and S805.

[0055] In S806, the functional unit 311 determines whether or not there has been a user operation on the screen provided by application A. If there has been a user operation, the process proceeds to S807; if not, the process returns to S804.

[0056] In S807, the functional unit 311 executes a process according to the user operation. Specifically, when an object, link, etc. included in the display 901 and the display 902 is specified, a display switch or process for providing information according to the specification is executed.

[0057] In S808, the functional unit 311 determines whether the user operation resulting from the process in S807 corresponds to the estimated result received from the information providing unit 304. For example, when an object or link included in the display 902 is specified, it is determined that there was a user operation corresponding to the estimated result. If it is a user operation corresponding to the estimated result, the process proceeds to S809; otherwise, it returns to S804. In S809, the cooperation unit 313 transmits feedback including the estimated result corresponding to the user operation to the information providing unit 304.

[0058] (Application Example 1) FIG. 10 shows an application example when applied to another application with respect to FIG. 8 of the above-described embodiment. In FIG. 10, the same processes as those in FIG. 8 are denoted by the same reference numerals, and the description thereof is omitted here. Hereinafter, the differences and their effects will be described.

[0059] In S1001, the functional unit 311 determines whether the process executed in S807 is a predetermined user operation. If it is a predetermined user operation, the process proceeds to S1002; if it is not a predetermined user operation, the process proceeds to S808.

[0060] Specific examples of the predetermined user operation include a scroll operation by flick and an operation on a display object corresponding to a search function.

[0061] Specifically, assume that the application being launched is an application that displays messages posted by the user on the timeline in chronological order, and the user performs a scroll by flick instead of a swipe. At this time, it is assumed that the user who performs the flick wants to refer to messages or image data posted in the relatively distant past with respect to the current time. When referring to data from several years or several decades ago, the scroll operation by flick by the user will be detected repeatedly many times.

[0062] Also, for any application, if it is an operation on a display object corresponding to a pre-prepared search function, the user intention to perform a search using a specific search function can be grasped.

[0063] In this application example, it is for using the estimation result provided from the information providing unit 304 including such user intention.

[0064] In S1002, the functional unit 311 executes a search process using the estimation result provided from the information providing unit 304, and automatically updates the display content so as to reflect the search result.

[0065] FIG. 11(a) is an example of a screen provided by an application that displays messages posted by users in chronological order on a timeline, in which the process of S1002 is reflected.

[0066] Reference numeral 1101 in this screen indicates an additional object for jumping to a message or time information corresponding to a search result in data included in the estimation result among a plurality of messages included in the scroll destination of the scroll bar. While the user is performing a flick operation, the user may be able to obtain a display of a desired message by selecting any one of the objects 1101 that automatically appear. Also, as a scroll destination by flicking, it may be possible to immediately proceed to the display of a message corresponding to the search result.

[0067] Also, when a message corresponding to the search result in S1002 is included in a plurality of messages to be displayed during the scroll of the screen by flicking, it is also possible to stop the scroll with the corresponding message being displayed so that the user does not skip it.

[0068] FIG. 11(b) is an example of a screen in which the process of S1002 is reflected when the user selects a specific search function in the application being launched.

[0069] In Fig. 11(b), in addition to the input field where the user can arbitrarily enter a search word, as shown in 1102, the result of "AI search" using the estimation result provided by the information providing unit 304 is displayed. Here, an automatic search is performed based on the word search and the period (from the current time to the time included in the estimation result). The user can perform additional operations such as arbitrarily adding a search word using the input field or deleting the search word by "AI search".

[0070] In this application example, in addition to the predetermined operation on the application performed by the user, by automatically applying the above-described estimation result, there is a possibility that the search target desired by the user can be quickly provided. In this application example, the user himself / herself may be able to omit operations such as visually finding the search target and thinking about and entering an appropriate search word.

[0071] (Application Example 2) The above-described OS also has a report function and a bias adjustment function.

[0072] The report function is a function for presenting the basis on which the learned model derives an output from the input data in the estimation process. For example, when one output is selected, the input data and parameters that are the basis for obtaining that output can be listed, and it can be shown to what extent each of the listed data has influenced the derivation of that output.

[0073] The bias adjustment function is a function for detecting and notifying that the learned model has a strong tendency to produce an output that the user does not desire for a predetermined input in the estimation process. Furthermore, this function also has a function of setting, for the input control unit 302, a conversion rule for reflecting an inverse bias for obtaining an estimation result desired by the user in accordance with the user's instruction in the predetermined input. When this setting is made, thereafter, when a predetermined input is received, at least a part of the information of the input is changed in accordance with the conversion rule and used by the learned model.

[0074] (Other Embodiments) The present invention also includes an apparatus or system and its method configured by appropriately combining the above-described embodiments.

[0075] Here, the present invention is an apparatus or system that serves as a main body for executing one or more software (programs) that realize the functions of the above-described embodiments. Also, a method for realizing the above-described embodiments executed by the apparatus or system is also one aspect of the present invention. Further, the program is supplied to the system or apparatus via a network or various storage media, and the program is read into one or more memories by one or more computers (such as a CPU or MPU) of the system or apparatus and executed. That is, as one aspect of the present invention, it also includes the program itself, or various storage media readable by a computer storing the program. Also, the present invention can be realized by a circuit (for example, ASIC) that realizes the functions of the above-described embodiments.

Explanation of Reference Numerals

[0076] 101 Mobile device 102 Voice assistant terminal 103 Peripheral device

Claims

A program incorporated with an operating system in an information processing apparatus in which an application for displaying messages posted among a plurality of users in chronological order is installed, in the information processing apparatus, a recording step of recording output information output by an estimation process executed using, as input data, data corresponding to an input based on a user's speech or action, the input data being used in a search process in the application, the model being trained to output information to be used in the search process in the application; a providing step of providing the output information recorded in the recording step to the application installed in the information processing apparatus for a search process of a post related to the output information in the application; causing the steps to be executed, wherein in the providing step, the output information is provided as a parameter at the time of activation of the detected application to the application for which an activation instruction following a user operation of the information processing apparatus is detected. A program characterized by this.

2. The program according to claim 1, wherein the search process based on the provided output information is executed by the application.

3. The program according to claim 1, wherein the device communicable with the information processing apparatus includes at least one of at least a voice assistant terminal, a digital home appliance device, and an in-vehicle terminal.

4. The program according to claim 1, wherein the user input includes an input by voice.

5. The program according to claim 1, wherein the application to which the output information is provided is an application in which the user has agreed to permit the use of the output information of the estimation process using the model.

6. An application for displaying messages posted among multiple users in chronological order and an operating system are installed. A program incorporated with the operating system, which is an information processing device on which a program for providing a service using a model learned to output information to be used in a search process in the application using, as input data, data corresponding to an input based on a user's behavior is executed. Recording means for recording output information output by an estimation process executed using, as input data, data corresponding to at least one of an input to the information processing device by a user and an input to a device communicable with the information processing device using the model. Providing means for providing the output information for a search process of a post related to the output information in the application to the application installed in the information processing device. The providing means provides the output information as a parameter at the time of activation of the detected application to the application for which an activation instruction following an operation of a user of the information processing device is detected. An information processing device characterized by this.

7. The information processing device according to claim 6, wherein the search process based on the provided output information is executed by the application.

8. The device communicable with the information processing device includes at least one of at least a voice assistant terminal, a digital home appliance device, and an in-vehicle terminal. The information processing device according to claim 6, characterized by this.

9. The input of the user includes an input by voice. The information processing device according to claim 6, characterized by this.

10. The application to which the output information is provided is an application in which the user has agreed to permit the use of the output information of the estimation process using the model. The information processing device according to claim 6, characterized by this.

11. The information processing apparatus according to claim 6, wherein in the application, display is performed using the result of processing executed based on the output information by the application.

12. An application for displaying messages posted among a plurality of users in chronological order and an operating system are installed, A method in an information processing apparatus in which a program incorporated with the operating system is executed to provide a service using a model learned to output information to be used in a search process in an application using, as input data, data corresponding to an input based on a user's speech and actions, A recording step of recording, as output information output in an estimation process executed using, as input data, data corresponding to at least any one of an input to the information processing apparatus by a user and an input to a device communicable with the information processing apparatus, using the model; A providing step of providing the output information for a search process of a post related to the output information in the application to the application installed in the information processing apparatus, the method comprising: In the providing step, the output information is provided as a parameter at the time of activation of the detected application to the application for which an activation instruction following an operation of a user of the information processing apparatus is detected.

Images