Information processing device, device inference system, program, and inference processing method

Abstract

Obtain appropriate inference results using equipment that you do not own. [Solution] The information processing device includes a communication unit, an inference model management unit that holds a machine learning model used for inference processing, an inference application that performs inference processing using the machine learning model held by the inference model management unit, and a device management application. The device management application has a first machine learning model connected by the communication unit and obtains a copy of the first machine learning model from a first device that is controlled based on the results of inference processing by the first machine learning model, and has the inference model management unit hold it. The inference application performs inference processing using the first machine learning model held by the inference model management unit in response to a request for inference processing, in order to control a second device.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a device inference system, a program, and an inference processing method.

Background Art

[0002] In recent years, the introduction of AI technology into devices has become an important means to enhance the value and competitiveness of devices. The range supported by AI is becoming increasingly diverse day by day, contributing to the automation, efficiency improvement, and quality improvement of work using devices.

[0003] Conventionally, since the execution of inference processing by AI required corresponding computing resources, it was often limited to remote execution via a network.

[0004] However, in addition to the evolution of tuning technologies such as lightweighting of machine learning models, the performance improvement of information devices has made it possible to execute inference processing by AI on a single edge device.

[0005] Furthermore, there are also devices that execute inference processing by AI in a local environment on an edge device when responsiveness is required and in a remote environment when higher accuracy and versatility are required by combining conventional remote execution and local execution of inference processing by AI on an edge device (see, for example, Non-Patent Document 1).

[0006] Some edge devices use a machine learning model installed in advance, and there are also devices that select and install an optimal model based on the environmental information where the edge device is installed and use it (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

[0008] [Non-Patent Document 1] Sharp Corporation, "Smooth Conversation with Home Appliances Now Possible?! ~What is Sharp's Human-Centered Edge AI Technology 'CE-LLM'?", [online], March 28, 2024, Sharp Blog, [Accessed September 13, 2024], Internet<URL:https: / / blog.sharp.co.jp / 2024 / 03 / 28 / 44007 / > [Overview of the project] [Problems that the invention aims to solve]

[0009] However, machine learning models change to suit the user's preferences through repeated learning. For example, if the edge device is a consumer electronics device, training it with user-specific training data will clearly reflect the user's individuality in the machine learning model.

[0010] The prior art inventions listed are characterized by the use of machine learning models suitable for the environment in which edge devices are installed, but the results of inference processing using the acquired models are not necessarily optimal for the users of those devices.

[0011] This invention has been made in view of the above-mentioned conventional examples, and aims to obtain inference processing results suitable for the user by making a machine learning model trained by the user available even on a device that does not have that machine learning model installed. [Means for solving the problem]

[0012] To achieve the above objective, according to one aspect of the present invention, a communication means and An inference model management means for holding machine learning models used for inference processing, An inference means that performs inference processing using the machine learning model held by the inference model management means, Device management means for acquiring a copy of the first machine learning model from a first device that has the first machine learning model connected by the communication means and is controlled based on the result of the inference process by the first machine learning model, and causing the inference model management means to hold it. In response to a request for an inference process, the inference means executes an inference process using the first machine learning model held by the inference model management means to control a second device. An information processing apparatus is provided, which is characterized by the above.

Effect of the Invention

[0013] According to the present invention, by using the machine learning model learned by the user even on a device that does not have the machine learning model, it becomes possible to obtain the result of an inference process suitable for the user.

Brief Description of the Drawings

[0014] [Figure 1] Overall system diagram [Figure 2A] Hardware configuration diagram [Figure 2B] Hardware configuration diagram [Figure 3] Software configuration diagram [Figure 4] Diagram showing an example of a device registration screen [Figure 5] Device registration sequence diagram [Figure 6A] Inference process execution sequence diagram [Figure 6B] Inference process execution sequence diagram [Figure 7] Diagram showing an example of an inference screen [Figure 8] Diagram showing an example of a model portable screen [Figure 9] Model portable sequence diagram [Figure 10] Diagram showing an example of a non-owned device registration screen [Figure 11] Non-owned device registration sequence diagram [Figure 12A]Inference processing execution sequence diagram for non-owned devices [Figure 12B] Inference processing execution sequence diagram for non-owned devices [Figure 13] A diagram showing an example of a screen for indirect registration of a non-owned device. [Figure 14A] Sequence diagram for indirect registration of non-owned devices [Figure 14B] Sequence diagram for indirect registration of non-owned devices [Figure 15] Indirect inference processing execution sequence diagram for non-owned devices [Figure 16] A diagram showing an example of the model deletion settings screen. [Figure 17] Flowchart for model deletion process [Modes for carrying out the invention]

[0015] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0016] [Embodiment 1] In this embodiment, for example, a system is described in which a machine learning model installed on a device owned by the user is carried on a terminal device, and when using a device of the same model as the device, inference processing can be performed using the machine learning model carried on the terminal device, depending on the user's selection.

[0017] ● Hardware of the device inference system Figure 1 shows the overall configuration of the device inference system (or information processing system) according to the present invention. Network 100 is a communication network that connects each component of this system. Network 100 is a communication network that can be realized by, for example, the Internet, a local area network (hereinafter referred to as LAN), a wide area network (hereinafter referred to as WAN), a telephone line, a dedicated digital line, an ATM or frame relay line, a cable television line, a wireless line for data broadcasting, etc. Network 100 can be of any type as long as it enables data transmission and reception between each component.

[0018] In this embodiment, the connection between the device management server 104 and the device application server 105 is described assuming that the network 100 is the Internet. Furthermore, the connection between the owned device 101, the unowned device 102 and the terminal 103 is described assuming that the network 100 is an intranet.

[0019] The owned device 101 and the unowned device 102 are devices that have the function of connecting to the network 100 and the function of independently executing inference processing using a machine learning model. Specifically, these are devices that have the functions described later in addition to the original functions of the device, such as home appliances and image forming machines. In this embodiment, the description will be based on the assumption that owned device 101 and unowned device 102 are home appliances, but the type of device is not limited as long as it has the two functions of network connectivity and the function of executing inference processing using a machine learning model. For example, it may be an image forming machine such as an inkjet printer or a digital multifunction device. With such devices, it is possible to use a machine learning model to suggest recommended settings such as print settings, scan settings, or data transmission destinations according to the print data or original document.

[0020] Furthermore, although the owned device 101 and the unowned device 102 are described separately in this embodiment, it is assumed that they are the same model with the same functions and specifications. The owned device 101 and the unowned device 102 may be referred to as the first device and the second device, respectively. Also, in the following description, the owned device 101 and the unowned device 102 may be referred to as device 101 and device 102, respectively.

[0021] Terminal 103 is a client terminal that has the function of connecting to the network 100 and connects to devices 101, 102, device management server 104, and device application server 105. In addition, terminal 103 is a terminal that has the function of independently executing inference processing using a machine learning model. Specifically, this is a mobile terminal such as a smartphone or a personal computer (PC). In this embodiment, the explanation assumes that terminal 103 is a smartphone, but the type of device is not limited as long as it has the same two functions as devices 101 and 102: the network connection function and the function of executing inference processing using a machine learning model.

[0022] The device management server 104 is a server computer that manages information on devices 101 and 102. The device owner registers the device information and the terminal 103 information with the server 104. This allows the device owner to execute the device's inference processing function. In this embodiment, devices registered in association with terminal 103 on the device management server 104 are referred to as owned devices. All other devices are referred to as unowned devices. In this system, there is always one device owner. Therefore, an owned device for one user in this system may be an unowned device for another user.

[0023] The device application server 105 is a server computer that has the function of connecting to the network 100 and the function of independently executing inference processing using machine learning models. A device application server 105 exists for each device. For example, if there are multiple devices of different types, such as cooking appliances and air conditioning appliances, there will be a device application server 105 corresponding to the cooking appliances and a device application server 105 corresponding to the air conditioning appliances, and each will provide services to the corresponding devices 101 and 102. In this embodiment, since devices 101 and 102 are assumed to be the same model, they will be provided with services by the same device application server 105. The device application server 105 also provides services to terminals 103 associated with the devices in the device management server 104. Note that the device application servers 105 corresponding to each of the multiple types of devices may be servers with separate hardware, or they may be logical or virtual servers that share a single piece of hardware.

[0024] In Figure 1, each component is shown as a single unit for the sake of simplicity in explaining this embodiment. However, there is no intention to limit or restrict the number of units of each element in terms of configuration. Each component may be composed of one or more units.

[0025] Figures 2A and 2B show the hardware configurations of devices 101 and 102, terminal 103, and servers 104 and 105, respectively. Figure 2A(a) is a block diagram showing the general hardware configuration of devices 101 and 102 according to this embodiment. Just as devices 101 and 102 may be distinguished as the first device and the second device, the resources and software modules of each device may also be distinguished by designating them as the first and second.

[0026] The CPU 200 starts the OS using the boot program stored in the ROM 201. The CPU 200 then executes various processes by running application programs stored in the external storage device 203 on the OS.

[0027] RAM 202 is used as the working area for the CPU 200. The external storage device 203 stores the aforementioned application program, machine learning models used by the inference execution unit 209 and the learning unit 210, and various data such as device settings and history information. The external storage device 203 can be an HDD or an SSD.

[0028] The network unit 204 connects to the network 100 and communicates with each element that makes up the device inference system. The CPU 200 is sometimes called the processor, control unit, or processing unit.

[0029] The CPU 200 is connected to both the ROM 201 and RAM 202 via the system bus 210, along with the operation interface (I / F) 205, display interface 207, device control unit 209, inference execution unit 211, and learning unit 212.

[0030] The control unit I / F 205 is an interface that connects the device to the control unit 206. The control unit 206 is an input unit that receives input from the user, such as from a keyboard, hardware keys, or microphone, and sends the received user input data to the CPU 200 via the control unit I / F 205.

[0031] The display unit I / F 207 is an interface that connects the device to the display unit 208. The display unit 208 is an output unit such as a display or speaker, and outputs data sent from the CPU 200 via the display unit I / F 207. There are also devices that combine the functions of both the operation unit 206 and the display unit 208, such as touch panels, in which case they are connected to both I / F 205 and 207. The operation unit 206 and the display unit 208 are sometimes collectively called the user interface (UI) unit.

[0032] The device control unit 209 controls the operation of the device in order to realize the operation of the device by the application program executed by the CPU 200. Assume that the device 101 is a microwave oven. For example, if the user selects the heating function, the CPU 200 sends control instructions to the components necessary to perform the heating function, specifically the heater, fan, sensors, etc., via the device control unit 209, to drive or control those components.

[0033] The inference execution unit 211 executes inference processing in the application program run by the CPU 200. The inference execution unit 211 is a device suitable for inference processing using machine learning models, such as an NPU (Neural Processing Unit) or GPU (Graphics Processing Unit). Depending on the scale of the machine learning model, the inference processing may be performed by the CPU 200 instead of the inference execution unit 211.

[0034] The learning unit 212 obtains the results of the inference processing performed by the inference execution unit 211 and performs retraining processing of the machine learning model stored in the external storage device 203. Retraining methods include, for example, supervised learning and unsupervised learning, and a method suitable for the device may be adopted. For example, if the user can change or adjust settings related to the device's output or operation controlled using the results obtained from the inference processing by the machine learning model, then the changed or adjusted settings can be used as training data, and supervised learning can be applied. If the device does not receive or cannot expect user feedback, unsupervised learning may be used for training.

[0035] In this embodiment, details of the inference process and the retraining process of the machine learning model will not be mentioned.

[0036] Figure 2A(b) is a block diagram showing the general hardware configuration of terminal 103 according to this embodiment. Since the basic components are the same as those of devices 101 and 102 described above, the explanation of overlapping components will be omitted. In this embodiment, terminal 103 is described assuming it is a mobile device such as a smartphone.

[0037] The CPU 220, ROM 221, RAM 222, external storage device 223, network unit 224, operation unit I / F 225, operation unit 226, display unit I / F 227, display unit 228, system bus 230, inference execution unit 231, and learning unit 232 are the same as the respective components of devices 101 and 102. The Global Positioning System module (GPS) 213 performs positioning to obtain the location information of terminal 103. Other positioning systems may be used instead of GPS.

[0038] Figure 2B(c) is a block diagram showing the general hardware configuration of the device management server 104 and device application server 105 according to this embodiment. The basic components are the same as those of the devices 101, 102, and terminal 103 described above, so the explanation of the overlapping components will be omitted.

[0039] The CPU 240, ROM 241, RAM 242, external storage device 243, network unit 244, operation unit I / F 245, operation unit 246, system bus 350, inference execution unit 251, and learning unit 252 are the same components as those of terminal 103. A key feature of the server is that it exchanges information with each element constituting the device inference system via communication over network 100. Therefore, display unit I / F 207 and display unit 208 are not essential for the server. Of course, a configuration with these components is also possible.

[0040] ● Software for device inference systems Figure 3 is a block diagram showing the software configuration of each component of the device inference system according to this embodiment. These are application programs, which are stored in either the RAM 201, ROM 202, or external storage device 203 of the respective devices shown in Figures 2A and 2B, and executed by the CPU 200.

[0041] Note that the block diagrams shown in Figure 3 only include the software relevant to this embodiment. If device 101 is a microwave oven, for example, there is software to implement device-specific functions such as heating and air blowing applications, but this is omitted here.

[0042] The device application 300 is an application program that has the function of performing a series of inference processes on devices 101 and 102. The device application 300 includes a communication unit 201, a determination application 302, an inference application 303, a setting management unit 304, and an inference model management unit 305.

[0043] The communication unit 301 transmits and receives data between the terminal 103, the device management server 104, and the device application server 105 via the network 100.

[0044] The determination application 302 determines whether it is optimal to perform the inference processing on device 101 or device application server 105 in response to the user request received by the operation unit 206, and sends an inference processing execution request to inference application 303 or inference application 332 on device application server 105 according to the determination result. In addition, if the device management application 314 of terminal 103 (described later) has configured the terminal 103 to perform the inference processing, the determination application 302 sends an inference processing execution request to inference application 313 on terminal 103. Furthermore, the determination application 302 receives the user response received by the operation unit 206 regarding the result of the inference processing performed by either device 101, terminal 103, or device application server 105, and decides whether to continue or interrupt the execution of the inference processing. The determination application is sometimes referred to as the determination unit or determination processing unit.

[0045] The inference application 303 performs inference processing using a machine learning model managed by the inference model management unit 305. The inference application 303 displays the results of the inference processing on the display unit 208. The inference application 303 performs inference processing in cooperation with or using the inference execution unit 211, but the inference application 303 itself may also perform inference processing independently. The inference application is sometimes referred to as the inference unit or inference processing unit.

[0046] The configuration management unit 304 manages the configuration information of device 101 and transmits and receives configuration information to and from terminal 103, device management server 104, and device application server 105 via the network 100. Specific examples of configuration information will be described later.

[0047] The inference model management unit 305 performs processes such as acquiring, sending, updating, and deleting machine learning models used by the inference application 303.

[0048] The terminal application 310 is an application on terminal 103 that has the function of instructing device 101 and device application server 105 to execute a series of inference processes, and the function of executing a series of inference processes using a machine learning model acquired from device 101. The terminal application 310 also has the function of sending and receiving information about device 101 to and from device management server 104. The terminal application 310 includes a communication unit 311, a judgment application 312, an inference application 313, a device management application 314, a configuration management unit 315, and an inference model management unit 316.

[0049] The communication unit 311 transmits and receives data between the device 101, the device management server 104, and the device application server 105 via the network 100.

[0050] The determination application 312 determines whether it is optimal to perform the inference processing on device 101 or device application server 105 in response to the user request received by the operation unit 226, and sends an inference processing execution request to inference application 313 or inference application 332 on device application server 105 according to the determination result.

[0051] When the inference application 313 receives an inference processing execution request, it performs inference processing using a machine learning model managed by the inference model management unit 316. The inference application 313 receives inference processing execution requests from the judgment application 312 or the judgment application 302 of device 101, etc.

[0052] The device management application 314 registers and manages the devices that the terminal 103 communicates with. Information about the registered devices (referred to as device information) is managed by the configuration management unit 315. The application 314 also transmits the device information managed by the management unit 315 to the device management server 104 (described later) to share the data.

[0053] The inference model management unit 316 copies and manages the machine learning models managed by the inference model management unit 305 for devices registered in the device management application 314. The machine learning models managed by the inference model management unit 316 are not normally used for inference processing. In the inference processing on the non-owned device 102, which will be described later, if it is not desired to use the machine learning models held by device 102, the machine learning models managed by the management unit 316 are used. Details of this inference processing will be described later.

[0054] The device management server application 320 is an application that has the function of executing a series of device management processes on the device management server 104. The device management server application 320 includes a communication unit 321, an authentication application 322, a device management application 323, and a data management unit 324.

[0055] The communication unit 321 transmits and receives data to and from devices 101, 102, terminal 103, and device application server 105 via the network 100.

[0056] The authentication application 322 identifies the device owner user and performs authentication and authorization processing. The device owner user information is stored and managed by the data management unit 324.

[0057] The device management application 323 registers and manages the device information of users authenticated by the authentication application 322 and their owned devices, associating them with each other. In this embodiment, the device information registration process is assumed to be performed by receiving data from the device management application 314 on terminal 103. However, device information may also be registered directly with the device management server 104 via a web browser or a dedicated application.

[0058] The data management unit 324 stores and manages user information handled by the authentication application 322 and device information handled by the device management application 323.

[0059] The device service application 330 is an application that has the function of performing inference processing for the corresponding device on the device application server 105. The device service application 330 includes a communication unit 331, an inference verification application 332, a data management unit 333, and an inference model management unit 334.

[0060] The communication unit 331 transmits and receives data between devices 101, 102, terminal 103, and device management server 104 via the network 100.

[0061] The inference application 332 receives a request from the device 101 determination application 302 and performs inference processing that is difficult to execute on the device. Inference processing that is difficult to execute on the device includes, for example, those that would take a long time to complete given the device's hardware performance. Generally, the server 105 has better hardware performance than the device 101.

[0062] The inference model management unit 332 performs processes such as acquiring, updating, and deleting machine learning models used by application 331.

[0063] ●Example of screen display during device registration Figure 4 shows a sequence of screens displayed on the display unit 208 of the terminal 103 during the device registration process according to this embodiment. This section describes the device list screen, device details screen, and device details input screen associated with device registration. The loading and deletion of machine learning models, registration of unowned devices, etc., will be described later by referring to the screens in Figures 8 and 10.

[0064] The device list screen 400 is a screen that displays a list of device information registered in application 314. The device list screen 400 is displayed, for example, on terminal 103, when an authenticated user launches or activates terminal application 310 and selects, for example, device registration or device list from the operation menu.

[0065] The device list display area 401 on the device list screen 400 displays a list of device information for devices registered in application 314. If no device information is registered, a blank space is displayed. Although not shown in Figure 4, as explained in detail in Figure 10, the device list display area 401 is divided into a display area for owned devices and a display area for unowned devices, allowing for the registration of owned and unowned devices. However, in the following explanation, the operations and display screens for unowned devices will be omitted, and the operations and display screens for registering a device as an owned device will be explained.

[0066] The device registration button 402 is a button used to initiate a series of device registration processes. When button 402 is pressed (or tapped, etc.), the device registration process begins and the screen transitions to the device search screen 410.

[0067] The device selection button 403 is used to confirm the selection of the desired device information from the device information displayed in the device list display area 401. When button 403 is pressed while any of the displayed device information is selected, the screen transitions to the device details screen 430.

[0068] The device delete button 404 is used to delete registered device information. When button 404 is pressed while a device is selected in the device list display area 401, the selected device information is deleted from the registered device information. The deleted device information is also removed from the device list display area 401.

[0069] The unowned device registration button 1004 is used to search for the device and register it as an unowned device of the logged-in user of terminal 103. The registration of unowned devices will be explained later, referring to Figure 10, so the explanation will be omitted here.

[0070] The device discovery screen 410 is a screen where instructions can be entered to search for devices that can be connected via the network 100. When the search button 411 is pressed, the device management application 314 sends a discovery request to all devices connected to the network 100 via the communication unit 311. Devices that receive a discovery request send their device information to the terminal 103 as the result of the discovery request (i.e., the discovery result).

[0071] The device discovery results screen 420 is a screen that displays a list of discovery requests received by application 314 from network 100.

[0072] The search result display area 421 displays a list of device information included in the search request results. In this embodiment, an example is shown in which the model name (device name), which is a fixed value for identifying the device, and the display name, which can be freely entered by the device owner user in application 314, are displayed in a list for each device found, i.e., for each device that responded with a search result.

[0073] The detailed information input button 422 is a button that transitions to the device details screen 430, which displays detailed information about the selected device. When the button 422 is pressed while device information displayed in the search results display area 421 is selected, the screen transitions to the device details screen 430.

[0074] The device details screen 430 is a screen where the device owner enters additional information to easily identify the device information registered in the device management application 314. An example of this additional information is the display name.

[0075] The display name input field 431 is a field for entering information that makes it easy for the user to identify what the device is. Information that makes the device easier to use on the device management application 314, such as who the device owner is, may also be entered. In this embodiment, only field 431 is shown, but there is no intention to limit the number of fields. For example, an input field for the device's location may be displayed.

[0076] The device registration button 432 is used to register the currently selected device information, including the setting value entered in the display name input field 431. Tapping the device registration button 432 transitions to the user authentication screen 440 for registering the selected device information.

[0077] The user authentication screen 440 is where the server 104 performs authentication to identify the user to whom device information is associated.

[0078] User ID input field 441 is for entering the user ID. Password input field 442 is for entering the password.

[0079] When the user enters their user ID and password and presses the authentication button 443, application 314 sends an authentication request to server 104. If authentication is successful, device management application 314 registers the device information in the settings management unit 315 and transitions to the device list screen 400.

[0080] Once the device registration is successful, the device information 405 will be displayed in the device list display area 401 of the device list screen 400.

[0081] In this way, the user can search for devices from terminal 103, select a desired device from the searched devices, and register it with terminal 103.

[0082] ●Device registration process Figure 5 is a sequence diagram showing a series of steps in the device registration process related to this embodiment. The devices involved in the processing sequence in Figure 5 are terminal 103, device management server 104, and device 101. The processing in each device is realized by the CPU of that device executing a program. Although Figure 5 mainly describes the software modules of each device, the hardware execution unit is the CPU of each device that realizes those software modules by executing a program. Prior to the sequence in Figure 5, the user completes authentication and logs in by entering authentication information, including the user ID, into terminal 103.

[0083] In step S501, the device management application 314 of terminal 103 displays the device list screen 400 on the display unit 208 in response to an operation such as selecting device registration or device list from the operation menu.

[0084] In step S502, when the registration button 402 on the device list screen 400 is pressed, the device management application 314 displays the device discovery screen 410.

[0085] In step S503, when the search button 411 on the device discovery screen 410 is pressed, the device management application 314 sends a device discovery request to all devices connected to the network 100.

[0086] In step S504, when the device 101 configuration management unit 304 receives a device discovery request, it acquires device information and transmits it to the device management application 314.

[0087] Table 1 is a table showing an example of device information that the configuration management unit 304 sends to the device management application 314 as a result of a discovery request.

[0088] [Table 1]

[0089] The serial column in Table 1 stores the serial number assigned to uniquely identify the device. The model name column stores the model name, which represents the type of device. The IP address column stores the device's IP address. Thus, the search results include the device identification information, model name, and address information such as the IP address from the acquired device information. If the device information of the device has been updated and new information such as a user ID or display name has been added, in addition to the contents of Table 1, device 101 may respond to terminal 103 with the user ID and display name as part of the device information.

[0090] In step S505, the device management application 314 displays at least the model name from the received device discovery result information on the device discovery screen 420.

[0091] When a device is selected on the device discovery screen 420 and the detailed information input button 422 is pressed, in step S506, the device management application 314 displays the device detailed information registration screen 430 and accepts the input of additional device detailed information for the selected device, such as the display name.

[0092] When the registration button 432 on the device details registration screen 430 is pressed, in step S507, the device management application 314 displays the authentication screen 440 and accepts user authentication. The device management application 314 sends the user authentication information entered on the authentication screen 440 to the authentication application 322.

[0093] Table 2 is a table showing an example of authentication information that the device management application 314 sends to the server 104.

[0094] [Table 2]

[0095] In Table 2, the User ID column stores the username, and the value from User ID field 441 is stored there. The Password column stores the password, and the value from Password field 442 is stored there.

[0096] In step S508, the authentication application 322 obtains the user's authentication information received by the communication unit 321 and executes the authentication process.

[0097] If the authentication process is successful, in step S509, the device management application 314 sends the device information of the device selected in step S505 and the additional device details entered in step S506 to the device management server 104. The registered device information includes the device information shown in Table 1.

[0098] Table 3 is a table showing an example of device registration information that the device management application 314 sends to the device management server 104.

[0099] [Table 3]

[0100] The serial column stores the serial number, or identification information, assigned to uniquely identify the device. The model name column stores the model name, which represents the type of device. The display name column stores the value entered in field 431. The IP address column stores the device's IP address. The user ID column stores the user ID of the user associated with the device. The user ID may be the user ID of the user logged into terminal 103.

[0101] In step S510, the device management application 323 of the device management server 104 stores the device information received by the communication unit 321 in the data management unit 324.

[0102] In step S511, the device management application 314 also sends the device registration information that it sent to the device management server 104 in step S509 to device 101. The contents are as shown in Table 3.

[0103] In step S512, the configuration management unit 304 of device 101 saves (or registers) the received device registration information. As a result, device 101 newly stores the user ID that performed the registration and the display name entered by that user. From this point onward, the device registration information shown in Table 3 may be sent as device information in response to a device discovery request.

[0104] In step S513, the device management application 314 of terminal 103 stores the device registration information that was sent to the device management server 104 in step S509 in the management unit 315.

[0105] Through the above process, the user who performed the device registration process on terminal 103 is registered with the device management server 104 as the owner of device 101. In other words, the user who performed the device registration process is registered with the server 104 in association with device 101.

[0106] ● Inference processing using owned devices or device application servers Figures 6A and 6B are sequence diagrams showing a series of steps in which inference processing is performed on the owned device 101 in this embodiment. Figures 6A and 6B show the sequence when inference processing is performed by device 101 or device application server 105.

[0107] The devices involved in the processing sequence in Figure 6A are terminal 103, device application server 105, and device 101, while the devices involved in the processing sequence in Figure 6B are device 101 and device application server 105. Processing in each device is realized by the execution of programs by the CPU of that device. Although Figures 6A and 6B mainly describe the software modules of each device, the hardware execution is carried out by the CPU of each device, which realizes those software modules by executing programs.

[0108] Figure 6A is a sequence diagram showing the flow of a series of processes when a request to execute inference processing is input, for example, by a user from the inference application 313 on terminal 103. Prior to Figure 6A, the device to execute the inference processing may be specified. For example, the process in Figure 6A may be performed when the device to be used is specified and the inference processing by that device is also utilized. The device may be specified, for example, by selecting a device from the device list 400 displayed on terminal 103.

[0109] In step S601, the determination application 312 on terminal 103 receives an inference processing request from the user.

[0110] In step S602, the determination application 312 obtains information about the device 101 that will perform the inference process from the device registration information table of the terminal 103's configuration management unit 315.

[0111] In step S603, the determination application 312 sends an inference processing request to the inference application 303 of device 101 based on the device information obtained in step S602. The inference processing request includes parameters such as a message entered by the user.

[0112] In step S604, the inference application 303 obtains device-specific information (device information) from the configuration management unit 304.

[0113] In step S605, the inference application 303 performs inference processing using the inference processing request received in step S603 and the device information acquired in step S604. The inference application 303 sends the inference result to the determination application 312 on terminal 103. The inference result received by the determination application 312 is displayed on the display unit 208 of terminal 103.

[0114] If the displayed inference result is not the result the user desires, the determination application 312 on terminal 103 sends an inference processing request to the inference application 332 on device application server 105. This inference processing request may include the same parameters (message) as the one sent in S603. Whether or not the inference result is the result the user desires may be determined based on the user's input values ​​for the displayed inference result.

[0115] In step S607, the inference application 332 of the device application server 105 performs inference processing and sends the result to application 312. The received inference result is displayed on the display unit 208 of the terminal 103, in the same way as in step S605.

[0116] If the inference result derived by the inference application 303 of device 101 or the inference application 332 of the device application server 105 is the result desired by the user, in step S608, the determination application 312 sends a device operation instruction corresponding to the estimation result to the inference application 303 of device 101. The device operation instruction received by the inference application 303 of device 101 is processed by the control unit 209. The device operation instruction may include, for example, the inference result accepted by the user and the instruction for it as parameters. When device 101 receives the device operation instruction, it may perform control according to the parameters. For example, if device 101 is a microwave oven and the received parameters are the name of the dish and the temperature setting instruction, device 101 may perform a temperature setting appropriate for the instructed dish.

[0117] Through the above sequence, device 101 is controlled based on the estimation results of its own inference application in response to operations from terminal 103. Furthermore, if the estimation results of device 101's inference application are not what the user desires, device 101 is controlled based on the estimation results of an inference application provided by the server.

[0118] Figure 6B is a sequence diagram showing the flow of a series of processes when an inference processing request is input to the determination application 302 of device 101. The request to execute the inference processing may be input, for example, from the operation unit 206 of device 101 by user operation, or from the network unit 204 or the like via communication. Input parameters (or messages) for the inference processing may also be input from the operation unit 206 of device 101.

[0119] In step S609, the determination application 302 of device 101 receives an inference processing request from the user.

[0120] In step S610, the determination application 302 obtains device-specific information from the configuration management unit 304.

[0121] In step S611, the determination application 302 passes the inference processing request received in step S609 and the device information acquired in step S610 to the inference application 303 and executes the inference processing. The inference result derived by application 303 is displayed on the display unit 208 of device 101 by the determination application 302.

[0122] If the displayed inference result is not the result the user desired, in step S612, application 302 sends an inference processing request to application 332. Whether or not the inference result is the result the user desired may be determined based on the user's input values ​​for the displayed inference result.

[0123] In step S613, application 332 performs inference processing and sends the result to inference application 302 on device application server 105. Inference application 302 sends the inference result to judgment application 320 on device 101. Judgment application 302 displays the received inference result on display unit 208 of device 101, in the same manner as in step S611.

[0124] If the inference result derived by inference application 303 or inference application 332 is the result desired by the user, in step S614, application 302 executes a device operation instruction corresponding to the estimation result. The device operation instruction is processed by the control unit 209.

[0125] Through the above sequence, in response to operations performed by device 101, device 101 is controlled based on the estimation results from its own inference application. Furthermore, if the estimation results from device 101's inference application are not what the user desired, device 101 is controlled based on the estimation results from the server's inference application.

[0126] ●Example of device operation using terminal 103 Figure 7 shows an example of a screen displayed on the display unit 208 of terminal 103 during the inference process shown in Figure 6A. The explanation assumes that device 101 is a microwave oven. While the explanation assumes terminal 103, the screen shown in Figure 7 may also be displayed on the display unit 208 of device 101 according to the sequence in Figure 6B. Furthermore, the screen in Figure 7 is a screen that is commonly displayed in this embodiment when terminal 103 uses device 101 or device 102, regardless of which device is the primary component of the inference process.

[0127] Figure 7(a) shows the screen of terminal 103 when only the inference processing in the inference application 303 of device 101 is executed during steps S601 to S608 shown in Figure 6, specifically when steps S606 and S607 are not executed. The portion that cannot be displayed on the display unit 228 is displayed by scrolling or other operations.

[0128] The judgment application screen 700 is a screen that displays both the inference processing request and the inference result during the execution of the inference process. Screen 700 is the screen that the judgment application 312 displays on the display unit 208.

[0129] The message display area 701 is an area that displays inference processing requests input by the user of terminal 103 and inference results received from inference applications 303, 313, and 332 of device 101, terminal 103, and device application server 105, respectively, in message format. In this embodiment, area 701 is represented in the form of a general chat application, but there is no intention to limit its display format.

[0130] The message input field 702 is a field for inputting responses to inference processing requests and inference results as messages.

[0131] The message send button 703, when pressed, sends the message entered in field 702 to the inference application 312.

[0132] Request message areas 704 and 706 are areas that display the message entered by the user of terminal 103 in field 701. This message may be used as a parameter for the inference request.

[0133] The response message area 705 displays the results of the inference processing performed by the inference application 312 on terminal 103 for the message that was input and displayed in the request message area 704. Specifically, it mainly displays the inference results performed by inference applications 303 and 332.

[0134] The response message area 707 displays the response to the message entered in area 706. Specifically, it indicates that an operation instruction has been sent to device 101 in response to the user's instructions.

[0135] The sequence shown in Figure 6A and the content displayed in the message display area 701 in Figure 7(a) will be explained in detail by relating them.

[0136] First, when an inference processing execution request is entered into the message input field 702 and the message send button 703 is pressed, step S601 is executed and the request message area 704 containing the entered message is displayed.

[0137] In step S605, when the determination application 312 on terminal 103 receives the inference result from the inference application 303 on device 101, the content is displayed in the response message area 705.

[0138] If the user of terminal 103 is satisfied with the inference result displayed in response message area 705, the user enters a device control instruction in message input field 702 and presses message send button 703. The contents of the device control instruction are displayed in request message area 706.

[0139] When step S608 is executed, the determination application 312 displays a message in the response message area 707 indicating that a control instruction has been received.

[0140] Figure 7(b) shows the screen of terminal 103 when the inference processing in application 332 is executed in steps S601 to S608 as shown in Figure 6, specifically when steps S601 to S608 are also executed, including steps S606 and S607. In Figure 7(b), the request message area 708 is the same as the request message area 704, so its explanation is omitted. The response message area 709 is the same as the response message area 705, so its explanation is omitted. The request message area 710 displays a message given by the user agreeing to the inference result, although the content of the displayed message differs, similar to the request message area 706.

[0141] The sequence shown in Figure 6 and the content displayed in the message display area 701 will be explained in detail, showing their correspondence.

[0142] When an additional inference processing execution request is entered into the message input field 702 and the message send button 703 is pressed, step S606 is executed and the request message area 708 is displayed.

[0143] In step S607, when the determination application 312 receives the inference result from the inference application 332, the content is displayed in the response message area 709.

[0144] As explained in Figures 1-7, according to the system of this embodiment, when a user uses a device owned by the user, that is, a device registered in association with the owner, the user can utilize inference processing using the machine learning model installed on that device. Furthermore, in response to user instructions, the system can also utilize inference processing using the machine learning model of the device application server. Since the inference results obtained using either of these inference processes can be used to control the device accordingly, the user can use the device more easily, more effectively, and more appropriately.

[0145] ● Terminal 103 carries the machine learning model of device 101. The following describes a configuration in which terminal 103 acquires a trained machine learning model from device 101, carries it with it, and performs inference processing using the carried machine learning model. In this embodiment, device 101 has a learning unit 212, and the learning of the machine learning model in device 101 can be advanced by the user's use of the machine learning model. Therefore, the machine learning model acquired by terminal 103 can be said to be the latest machine learning model that reflects the learning on device 101 at the time of acquisition.

[0146] Figure 8 shows an example of a screen when the machine learning model of device 101 is carried on terminal 103 in this embodiment. Note that in Figure 8, the transition starts from the device list screen 400 shown in Figure 4. Therefore, the explanation of duplicate screens and their components is omitted.

[0147] The device details screens 800 and 810 are screens that the user transitions to after selecting a device on the device list screen 400 and pressing the select button 403. However, the selected device is an owned device, and is selected from the owned devices display area within the device list display area 401.

[0148] The device details screen 800 is displayed when the machine learning model for device 101, selected on the device list screen 400, is not saved in the inference model management unit 316 of terminal 103.

[0149] When the user presses the detailed information editing button 801, the device details input screen 430 is accessed. When the user presses the model carrying button 802, the authentication screen 440 is displayed, and the user of terminal 103 is prompted to authenticate. If the authentication process is successful, terminal 103 retrieves the machine learning model stored in the inference model management unit 305 of device 101. Although authentication is performed in this embodiment, it is not mandatory. When the user presses the back button 803, the device returns to the device list screen 400.

[0150] The device details screen 810 is displayed when the machine learning model for device 101, selected on the device list screen 400, is saved in the inference model management unit 316 of terminal 103. Pressing the model cancellation button 811 deletes the machine learning model for device 101 that is saved in the inference model management unit 316 of terminal 103.

[0151] The device list screen 820 is an example of a device list screen that displays information about devices on which machine learning models are stored in the inference model management unit 316. In this example, in addition to the display name of the registered device, a model carrying label 821 is displayed for each device, or it is not displayed. The model carrying label 821 is an additional label that is displayed when displaying information about devices on which machine learning models are stored in the inference model management unit 316 in the device list. By whether or not the label is displayed, the user of terminal 103 can easily check whether or not a machine learning model is carried. When a device with a model carrying label 821 displayed is selected on the device list screen 820 and the select button 403 is pressed, the device details screen 810 is displayed. When a device without a model carrying label 821 displayed is selected on the device list screen 820 and the select button 403 is pressed, the device details screen 800 is displayed.

[0152] ● Processing to carry the device's machine learning model onto the terminal. Figure 9 is a sequence diagram showing the process of transferring the machine learning model from device 101, shown in Figure 8, to terminal 103. The devices involved in the processing sequence in Figure 9 are terminal 103, device management server 104, and device 101. The processing in each device is realized by the execution of programs by the CPU of that device. Although Figure 9 mainly describes the software modules of each device, the hardware execution is carried out by the CPU of each device, which realizes those software modules by executing programs.

[0153] In step S901, the device management application 314 displays the device list screen 400 or 820 on the display unit 208. When the selection button 403 is pressed while the target device information is selected, the application transitions to the device details screen 800 or the device details screen 810.

[0154] If the machine learning model for device 101 selected in step S901 is not saved in the inference model management unit 316, the process proceeds to step S902.

[0155] In step S902, the device management application 314 displays the device details screen 800 on the display unit 208. When the model phone button 802 is pressed, in step S903, the device management application 314 sends an authentication request to the authentication application 322 of the device management server 104.

[0156] If the authentication process is successful in the authentication application 322, in step S904 the device management application 314 retrieves the machine learning model from the inference model management unit 305 of device 101 and stores it in the inference model management unit 316 of terminal 103. The retrieved machine learning model may be stored associated with information that identifies which device it was obtained from, such as the device information of the source device (e.g., serial number and model name). The machine learning model stored in the inference model management unit 316 can be used for inference processing by the inference application 313.

[0157] On the other hand, if the machine learning model for the device 101 selected in step S901 is stored in the inference model management unit 316, the device details screen 810 is displayed on the terminal 103 when the selection button 403 on the device list screen 400 or 820 is pressed, and the process proceeds to step S905.

[0158] In step S905, the device management application 314 displays the device details screen 810 on the display unit 208.

[0159] When the model cancellation button 811 is pressed, in step S906, the device management application 314 deletes the machine learning model of the selected device 101 that was stored in the inference model management unit 316.

[0160] In step S907, the device management application 314 updates the device list screen by displaying the latest information on the display unit 208. The device details screen 820 is an example of how device information is displayed on terminal 103 when a machine learning model is obtained from device 101 during the series of processes from steps S902 to S904. For devices where a machine learning model is stored in the inference model management unit 316, the model label 821 is displayed along with the device information.

[0161] Through the above procedure, terminal 103 can obtain a copy of the machine learning model of device 101 that is registered in association with its user. The obtained machine learning model can be used, for example, for inference processing when using other devices.

[0162] ● Registration screen for unowned devices Figure 10 shows a series of screens for the device registration process, which allows the machine learning model of device 101, copied to terminal 103, to be used for inference processing on device 102. Note that the screens in Figure 10 share many similarities with the device list screen shown in Figure 4 in terms of elements and screen transitions. Therefore, redundant explanations will be omitted, and only the elements and transitions unique to Figure 10, which were omitted in the explanations of Figures 4 and 8, will be explained.

[0163] The device list screen 1000 is a screen that displays a list of device information registered in the device management application 314. The device list screen 1000 displays information on both devices owned and not owned by the user currently logged into terminal 103.

[0164] The device information display area 1001 is a screen that displays device information stored in the settings management unit 315, and consists of an owned device display area 1002 and an unowned device display area 1003.

[0165] The owned device display area 1002 is an area that displays device information registered by the owner of terminal 103. Device information registered using the procedure shown in Figures 4 and 5 is displayed in the owned device display area 1001.

[0166] The unowned device display area 1003 displays device information registered by users other than the owner of terminal 103 using the same procedure shown in Figures 4 and 5. In this embodiment, the owned device display area 1002 and the unowned device display area 1003 are displayed separately within the device information display area 1001, but it is not always necessary to display them separately. In this embodiment, they are displayed separately for the sake of clarity in the explanation, but they may be displayed together in the device information display area 1001. Even in that case, it is desirable that the owned device information and unowned device information be displayed in a way that allows for identification.

[0167] Pressing the unowned device registration button 1004 starts the registration process for device 102, which is an unowned device. The device search screen displayed in response to pressing the unowned device registration button 1004 is the same as the device search screen 410, so its explanation is omitted. When the search button 411 is pressed on the device search screen 410, the device details screen 1030 for the unowned device is displayed.

[0168] The device details screen 1030 for unowned devices is accessed when the user selects an unowned device on the device list screen 400 or device list screen 1000 and presses the select button. The device details screen 1030 displays the device name and display name, and also shows an edit details button and a back button. However, since the target is an unowned device, there are no buttons for adding or deleting machine learning models, as seen on the device details screens 800 and 810 in Figure 8, which are for owned devices. When the edit details button is pressed on the device details screen 1030, the user is redirected to the device details input screen 1020.

[0169] The Unowned Device Search Results Screen 1010 displays the devices 102 found as a result of the search process in response to pressing the search button 411 on the Device Search Screen 410. The device search process for unowned devices may be the same as that shown in Figure 5. The difference between the unowned device search and the device search described in Figures 4 and 5 is that the search results include devices that are the same model as devices registered in association with the logged-in user of terminal 103, but are not registered in association with the logged-in user. In other words, the Device Search Results Screen 1010 displays only device information where the model name matches the value in the model name column of a device registered as an owned device in the device registration information table managed by the data management unit 324 of the device management server 104, and where the user ID does not match the logged-in user's user ID. This is the difference from the Device Search Screen 420. Note that device information from devices associated with a user includes the user ID and display name, as shown in Table 4. However, if the device information does not include a user ID, the device information obtained from the device information that matches the model name of a device registered as owned, and that includes a serial number that does not match any of the registered devices, may be used as the device information for a device not owned.

[0170] In order for the device information transmitted by the device to include the username and display name, the device itself must refer to the device registration information table in which it is registered. Therefore, a device that receives a device discovery request may request the device information of the device managed by the device management server 104 from the device management server 104, and if the device information is returned, it may return it to the sender of the device discovery request. Alternatively, terminal 103 that receives the device information may request and obtain the device information of the device managed by the device management server 104 based on the received device information.

[0171] When a device is selected on the Unowned Device Search Results screen 1010 and the detailed information input button 1011 is pressed, the user is redirected to the Unowned Device Information Registration screen 1020 for the selected device.

[0172] The Unowned Device Information Registration Screen 1020 is displayed when a device 101 of the same model as the selected device 102 is registered in the settings management unit 315 of terminal 103, and a machine learning model has been copied from device 101 to the inference model management unit 316. The Unowned Device Information Registration Screen 1020 displays the model name and display name of the selected device, as well as a message and options regarding the use of the machine learning model held in terminal 103. If the display name is not included in the device information, a display name may be displayed according to user input, such as a name with a predetermined name (e.g., shared device) plus a number based on the number of registered unowned devices.

[0173] The user can use the non-owned device information registration screen 1020 to select whether to use the machine learning model copied from device 101 or the machine learning model held by device 102 for inference processing on the device 102 to be registered.

[0174] When the "Use Mobile Model" button 1021 is selected, the machine learning model stored in the inference model management unit 316 of terminal 103 is used for inference processing on device 102. When the "Use Non-Owned Device Model" button 1022 is selected, the machine learning model in the inference model management unit 305 of device 102 is used for inference processing on device 102.

[0175] When the registration button 1023 is pressed, the device management application 314 on terminal 103 registers the device information of device 102 as unowned device information in the settings management unit 315, and transitions to the device list screen 1000.

[0176] Registered unowned devices are displayed in the unowned device display area 1003, along with a usage model setting label 1005 indicating which device the machine learning model used to perform inference processing belongs to.

[0177] ● Registration process for non-owned devices Figure 11 is a sequence diagram showing the sequence of steps in the unowned device registration process performed through the screen shown in Figure 10. The devices involved in the processing sequence in Figure 11 are terminal 103, device management server 104, and unowned device 102. The processing in each device is realized by the execution of a program by the CPU of that device. Although Figure 11 mainly describes the software modules of each device, the hardware execution entity is the CPU of each device, which realizes those software modules by executing the program.

[0178] In step S1101, the device management application 314 of terminal 103 displays the device list screen 1000 on the display unit 208.

[0179] When the "Register Unowned Device" button 1004 is pressed on the device list screen 1000, in step S1102, the device management application 314 displays the device discovery screen 410.

[0180] When the device discovery screen 410 is accessed, and the discovery button 411 is pressed, in step S1103, the device management application 314 sends a device discovery request to a device connected to the network 100 via the communication unit 311.

[0181] In step S1104, the configuration management unit 304 of device 102, which received the device discovery request, retrieves device registration information from the device registration information table, creates a device discovery response, and sends it to the device management application 314.

[0182] Table 4 is a table showing an example of a search result that an unowned device sends to the communication unit 311 as a response to a device search request from the communication unit 301.

[0183] [Table 4]

[0184] The serial number column stores the serial number assigned to uniquely identify the device. The model name column stores the model name, which represents the type of device. The display name column stores the value entered in field 431. The IP address column stores the device's IP address. The user ID column stores the user ID of the user associated with the device.

[0185] Since unowned devices are devices that have already been registered by a different user, unlike in Table 1, the search results will include additional columns for display name and user ID. As mentioned above, terminal 103 may receive device information from device 102 that does not include the user ID and display name, and obtain the device information for that device registered with the device management server 104.

[0186] In step S1105, the device management application 314 displays the discovery result response received from the unowned device on the unowned device discovery result screen 1010. When the registration button 1011 is pressed on the unowned device discovery result screen 1010 with the device information to be registered as an unowned device selected, the device management application 314 displays the device details input screen 1020 and accepts input of device details.

[0187] When the registration button 1023 is pressed on the device details input screen 1020, in step S1106, the device management application 314 transitions to the authentication screen 440 and sends an authentication request to the authentication application 322.

[0188] Upon receiving a successful authentication result from the authentication application 322, in step S1107, the device management application 314 creates unowned device registration information from the device information of the unowned device selected in step S1105 on the unowned device discovery result screen 1010 and the device details entered on the device details input screen 1020, and sends it to the device management server 104.

[0189] Table 5 is a table showing an example of unowned device registration information that the communication unit 311 of terminal 103 sends to the device management server 104.

[0190] [Table 5]

[0191] The serial number column stores the serial number assigned to uniquely identify the device. The shared user ID column stores the ID of a non-owner user who shares a non-owned device that is already associated with a user. If multiple users register a non-owned device, the shared user ID column will store multiple user IDs.

[0192] The "Model Used" column stores information about the model used for inference processing on the device by the user whose value corresponds to the "Shared User ID" column. If the "Use" button 1021 is selected on the device details input screen 1020, the value "Terminal" is entered, indicating that the machine learning model stored in the inference model management unit 316 of terminal 103 will be used. If the "Do Not Use" button 1022 is selected, the value "Device" is entered, indicating that the machine learning model stored in the inference model management unit 305 of device 102 will be used.

[0193] In step S1108, when the device management application 323 of the device management server 104 receives the registration information for an unowned device, the data management unit 324 saves the received information to the unowned device registration information table.

[0194] Furthermore, the data management unit 324 of the device management server 104 adds the value of the shared ID column to the device registration information record whose serial column value matches the value of the unowned device registration information.

[0195] Table 6 is a table showing an example of device registration information after the data management unit 324 of the device management server 104 receives and updates the registration information for unowned devices.

[0196] [Table 6]

[0197] The basic elements are the same as in Table 3. In addition, for records where the serial column of the device registration information managed in the device registration information table matches the serial column of the unowned device registration information, the value of the shared user ID column of the unowned device registration information is added. The shared user ID column stores the user ID value of the user who registered that device as an unowned device.

[0198] In step S1109, the device management application 314 registers unowned device registration information and updates device registration information to the configuration management unit 315, similar to step S1108.

[0199] In step S1110, the device registration information for device 102 is requested from the configuration management unit 304 of device 102 and the data management unit 324 of the device management server 104 and obtained.

[0200] In step S1111, the configuration management unit 304 of device 102 updates the device registration information it holds based on the acquired device registration information.

[0201] Steps S1110 and S1111 may be performed periodically by each device. Alternatively, instead of device 102 voluntarily updating the device registration information, the device management server 104 may send the updated device registration information to the corresponding device when an update is available, causing the device registration information to be updated. Furthermore, terminal 103 may send the shared user ID from the device registration information of device 102 to device 102 to request an update, causing the device registration information to be updated.

[0202] Through the above process, the owner of device 101 completes the setup so that when using the inference function of device 102, which is the same model as device 101, the machine learning model of device 101 on terminal 103 will be used instead of the machine learning model of device 102. With this setup, when a user of terminal 103 uses device 102, even if it is a device they do not own, they can perform inference processing using the machine learning model of device 101 that they have on terminal 103. Furthermore, since the identification information of shared users held by unowned devices can be updated, it becomes possible to identify the sharers of a device from the device registration information obtained from the device.

[0203] ● Inference processing on non-owned devices Figures 12A and 12B are sequence diagrams showing a series of steps in executing inference processing on the non-owned device 102 in this embodiment. The devices involved in the processing sequences in Figures 12A and 12B are the terminal 103 and the non-owned device 102. Processing in each device is realized by the CPU of that device executing a program. Although Figures 12A and 12B mainly describe the software modules of each device, the hardware execution entity is the CPU of each device that realizes those software modules by executing a program.

[0204] Figure 12A is a sequence diagram showing the flow of a series of processes when an inference processing execution request is input from the inference application 313 on terminal 103. Note that Figure 12A shows the case where one inference processing is performed for one input message. If the input of messages and inference processing are repeated multiple times, steps S1203 to S1205 or steps S1206 to S1208 will be executed repeatedly.

[0205] In step S1201, the inference application 313 receives an inference processing request from the user. This inference request includes a message entered in the message input field 702.

[0206] In step S1202, the inference application 313 obtains information about the device 102 on which to perform the inference process from the device registration information table of the configuration management unit 315. Next, the inference application 313 obtains the values ​​of the serial column and shared user ID column of the obtained device registration information. Then, the inference application 313 obtains records from the unowned device registration information table that match the obtained serial column and shared user ID column values. The inference application 313 refers to the value of the usage model column of the obtained unowned device registration information record to determine whether to use terminal 103 or device 102 for the machine learning model used in the inference process on device 102.

[0207] If the value of the "Use Model" column in the acquired non-owned device registration information record is "Terminal," the inference process uses the machine learning model copied from device 101 and stored in the inference model management unit 316. In this case, execution starts from S1203.

[0208] In step S1203, the inference application 313 sends a request to the inference application 303 on device 102 to obtain device 102-specific information necessary for executing the inference process. Device 102-specific information refers to data that exists only on device 102, such as device configuration information stored in the configuration management unit 304.

[0209] In step S1204, the inference application 303 on device 102 obtains device-specific information necessary for inference processing from the management unit 304 and transmits it to the inference application 313 on terminal 103.

[0210] In step S1205, the inference application 313 performs inference processing using the data acquired from device 102 in step S1204 and the inference processing request received in step S1201.

[0211] If the value of the "Model Used" column is "Device", the machine learning model stored in the inference model management unit 305 of device 102 is used for the inference process. Furthermore, in S1202, if a record matching the serial number of device 102 and the user ID logged into terminal 103 cannot be obtained from the unowned device registration information table, the machine learning model stored in the inference model management unit 305 of device 102 is also used for the inference process. In other words, unless it is determined that the machine learning model copied from device 101 and stored in the inference model management unit 316 of terminal 103 should be used for the inference process, the machine learning model stored in the inference model management unit 305 of device 102 is used for the inference process. In these cases, execution starts from S1206.

[0212] In step S1206, the inference application 313 on terminal 103 transmits the inference processing request received in step S1201 to the inference application 303 on device 102.

[0213] In step S1207, the inference application 303 of device 102 obtains device-specific information necessary for inference processing from the configuration management unit 304.

[0214] In step S1208, the inference application 303 performs inference processing using, for example, the device-specific information obtained in step S1207 and the inference processing request received in step S1206 as input parameters. Then, the inference application 303 sends the inference result to the inference application 313 on terminal 103.

[0215] In step S1209, the determination application 312 of terminal 103 displays the inference result in the response message area 705 of the screen 700 displayed on the display unit 208.

[0216] As described above, when terminal 103 initiates inference processing using a non-owned device 102, it becomes possible to perform inference processing on the non-owned device using the machine learning model of the device owned and normally used by the user, depending on the settings of device 102. Therefore, a trained model trained on an owned device can be used on a non-owned device as well, and even when using a device at a remote location, for example, inference results that reflect the user's training can be obtained.

[0217] Figure 12B is a sequence diagram showing the flow of a series of processes when an inference processing execution request is input from the inference application 303 of device 102. Note that Figure 12B shows the case where one inference processing is performed for one input message. If the input of messages and inference processing are repeated multiple times, steps S1213 to S1214 or step S1215 will be executed repeatedly.

[0218] In step S1210, the inference application 303 receives an inference processing request from the user. In device 102, as in terminal 103, a message is input via the operation unit 206, and the inference processing request is accompanied by the input message.

[0219] In step S1211, the inference application 303 obtains information about the device 102 on which to perform the inference process from the device registration information table of the configuration management unit 304. Next, the inference application 303 obtains the values ​​of the serial column and the shared user ID column of the obtained device registration information. Then, the inference application 303 obtains records from the unowned device registration information table where the values ​​of the serial column and the shared user ID column match. The inference application 303 refers to the value of the usage model column of the records in the obtained unowned device registration information table to determine whether to use terminal 103 or device 102 for the machine learning model to be used in the inference process on device 102.

[0220] In step S1212, the inference application 303 of device 102 obtains device-specific information necessary for inference processing from the configuration management unit 304.

[0221] If the value of the "Model Used" column is "Terminal," the inference process uses the machine learning model copied from device 101 and stored in the inference model management unit 316. In this case, execution starts from S1213.

[0222] In step S1213, the inference application 303 on device 102 sends the device-specific information obtained in step S1212 and the inference processing request received in step S1210 to the inference application 313 on terminal 103.

[0223] In step S1214, application 313 performs inference processing and sends the result to application 303.

[0224] On the other hand, if the value of the "Model Used" column is "Device", the machine learning model stored in the management unit 305 is used for the inference process. Furthermore, even if, in S1211, a record matching the serial number of device 102 and the user ID logged into terminal 103 cannot be obtained from the unowned device registration information table, the machine learning model stored in the inference model management unit 305 of device 102 is used for the inference process. In other words, unless it is determined that the machine learning model copied from device 101 and stored in the inference model management unit 316 of terminal 103 is to be used for the inference process, the machine learning model stored in the inference model management unit 305 of device 102 is used for the inference process. In these cases, execution starts from S1215.

[0225] In step S1215, the inference application 303 performs an inference process.

[0226] In step S1216, the determination application 302 displays the inference result in the response message area 705 of the screen 700 displayed on the display unit 208.

[0227] As a result, even when an inference processing request is directly input to device 102, it becomes possible to perform inference processing on a non-owned device using the machine learning model of the device owned and originally used by the user.

[0228] Through the series of processes described above, the machine learning model from the owned device 101 is carried on the terminal 103 and used for inference processing on a device 102 that is located outside the home and owned by a different person. As a result, users of this system can enjoy inference processing results that are adapted to them, even if the device is owned by a different person, as long as it is of the same model.

[0229] [Embodiment 2] Embodiment 1 assumes that terminal 103, which has a copy of the machine learning model of owned device 101, and a non-owned device 102 are directly connected via a network. As a result, terminal 103 can use the machine learning model of device 101 held by terminal 103 for inference processing on device 102, and the user of terminal 103 can obtain the same results on device 102 as on device 101.

[0230] However, even though there is the advantage of being able to use device 102 with the same feel as a device you normally use, connecting terminal 103 to a third-party network, such as when you are out and about, may be avoided due to security concerns.

[0231] In this embodiment, a method is shown for using the machine learning model of device 101 held by terminal 103 for inference processing on device 102, without directly connecting terminal 103 and device 102 to a network.

[0232] ● Registration operation that does not connect to a device you do not own Figure 13 shows a series of screens displayed on terminal 103 when registering device 102 to the device management application 314 of terminal 103 without directly connecting to the network, in this embodiment. Elements that overlap with Figures 4 and 10 are omitted from the explanation.

[0233] Figure 13(a) shows a series of screens for issuing a device registration code, which is used when the owner of device 102 registers device 102 as an unowned device to another user.

[0234] The device list screen 1300 is a screen that displays a list of device information registered in the device management application 314 in this embodiment.

[0235] When the device registration code issuance button 1301 is pressed while device information is selected, the device management application 314 displays the authentication screen 440 and requests user authentication to access the device management server 104.

[0236] Upon successful user authentication, the device management application 314 receives a device registration code issued by the device management application 323 on the device management server 104 and displays the device registration code confirmation screen 1310 shown in Figure 13(c).

[0237] The device registration code display area 1311 on the device registration code confirmation screen 1310 is an area that displays the device registration code issued by the device management application 323 of the device management server 104.

[0238] Pressing the back button 1312 will take you to the device list screen 1300.

[0239] If device information exists for which a device registration code has been issued, the device management application 314 displays the device registration code label 1303 alongside that device information in the device list 1300 (see Figure 13(b)). The device registration code label 1303 includes a display of the device registration code. Here, "device information for which a device registration code has been issued" refers to the device information that was selected when the issuance of the device registration code was instructed.

[0240] Figure 13(d) is the unowned device registration screen 1320 for the owner of device 101 to register device 102 as an unowned device using the device registration code of device 102 in the device management application 314 on terminal 103.

[0241] When the device registration application 314 is displaying the device list screen 1300 and the "Register Unowned Device" button 1302 is pressed, the device management application 314 displays the "Register Unowned Device" screen 1320 shown in Figure 13(d).

[0242] The device registration code input field 1321 on the unowned device registration screen 1320 is a field that accepts the input of a device registration code. When a device registration code is entered in the device registration code input field 1321 and the registration button 1322 is pressed, the device management application 314 performs the registration of the unowned device.

[0243] ● Registration process sequence that does not connect to unowned devices Figures 14A and 14B are sequence diagrams illustrating the procedure for registering device 102 with the device management application 314 on terminal 103 without directly connecting device 102 to the network, using the screen shown in Figure 13. The devices involved in the processing sequence in Figures 14A and 14B are terminal 103, device management server 104, and device 101. The processing in each device is realized by the execution of a program by the CPU of that device. While Figures 14A and 14B mainly describe the software modules of each device, the hardware execution entity is the CPU of each device, which realizes those software modules by executing a program.

[0244] Figure 14A is a sequence diagram showing the series of processes by which the owner of device 102 issues a device registration code, using the screen shown in Figure 13(a). In other words, the person instructing the process in Figure 14A is the owner of device 102. Furthermore, device 102 has already been registered in the device registration information table held by the device management server 104 by its owner.

[0245] In step S1401, the device management application 314 of terminal 103 displays the device list screen 1300 in response to user operation.

[0246] When the device registration code issuance button 1301 is pressed, in step S1402, the device management application 314 displays the authentication screen 440. When the user ID and password are entered on the authentication screen 440, the application 314 sends a user authentication request to the authentication application 322.

[0247] Upon receiving a successful authentication result from the authentication application 322, the device management application 314 on terminal 103 sends a request to the device management application 323 on device management server 104 to issue a device registration code, along with the value of the serial column of the device registration information managed by the configuration management unit 315.

[0248] In step S1404, the device management application 323 issues a device registration code. The device registration code has a unique value, and its uniqueness only needs to be unique within a single model, for example. That is, it only needs to be unique for each model among devices managed as unowned devices that may be used by users other than the owner. The device management application 314 of terminal 103 registers the issued device registration code in the device registration information table managed by the data management unit 324, in a device registration information record that matches the value of the received serial column.

[0249] Table 7 is a table showing an example of the data structure of the device registration information table managed by the data management unit 324 of the device management server 104 in this embodiment.

[0250] [Table 7]

[0251] Since the basic items in Table 7 are the same as in Table 4, only the differences will be explained. The device registration code column is the column that stores the device registration code issued by the device management application 323 in step S1404.

[0252] In step S1405, the device management application 314 of terminal 103 displays the device registration code confirmation screen 1310 and displays the device registration code received from the device confirmation application 323 in the device registration code display area 1311.

[0253] In this way, a device registration code can be assigned to a non-owned device 102 for the owner of device 101, and it can be registered with the device management server 104. The device registration code is made known to the non-owner user who will use the device to which it is assigned, and device 102 is registered as a non-owned device using the procedure shown in Figure 14B with that code.

[0254] Figure 14B is a sequence diagram showing the process by which the owner of device 101 registers device 102 as an unowned device using a device registration code issued by the owner of device 102. There are no particular restrictions on how the device registration code is passed from the owner of device 102 to the owner of device 101. However, it is assumed that device 102 and terminal 103 do not belong to the same network, so terminal 103 receives the device registration code either offline or in a way that does not access device 102. For example, it could be passed directly using email, or a sticker with the device registration code printed on it could simply be attached to device 102. The person who instructs the execution of the process in Figure 14B is the owner of device 101.

[0255] In step S1406, the device management application 314 of terminal 103 displays the device list screen 1300 in response to the operation.

[0256] When the "Register Unowned Device" button 1302 is pressed on the device list screen 1300, in step S1407, the device management application 314 displays the "Search for Unowned Devices" screen 1320.

[0257] In step S1408, when the registration button 1322 is pressed with the device registration code entered in the device registration code input field 1321, the device management application 314 transmits the device registration code entered as a device search request to the device management application 323 of the device management server 104. The device management application 323 refers to the device registration information table managed by the data management unit 324, obtains a device information record whose value in the device registration code column matches the value of the received device registration code, and transmits that record to the device management application 314 of the terminal 103.

[0258] In step S1409, the device management application 314 displays the unowned device search result screen 1010 and displays the device information received in step S1408. The items to be displayed include the model name and the set display name.

[0259] The processing from step S1410 to step S1414 is omitted from the description because it is the same as the processing from step S1105 to S1109 in FIG. 11. In these steps, the unowned device registration information is registered in the unowned device registration information tables of the terminal 103 and the device management server 104. Also, the device registration information of the devices registered in the unowned device registration information table is updated with the shared user ID.

[0260] Table 8 is a table showing an example of the data structure of the device registration information table after the device 102 is registered as an unowned device of the owner of the device 101 in step S1413. The unowned device registration information table may be as shown in Table 5.

[0261]

Table 8

[0262] The value of the user ID of the user of terminal 103, which was authenticated in step S1411 for the value of the shared user column, is stored. Note that as a result of step S1413, the updated device registration information is transmitted to terminal 103 and registered in management unit 315.

[0263] In step S1415, the setting management unit 304 of device 101 transmits the serial number of device 101 to the device management application 323 of the device management server 104, acquires the latest device information of its own device managed by the data management unit 324, and updates the device registration information table held by the setting management unit 315 with that information. Step S1415 may be executed, for example, periodically.

[0264] Through the above series of processes, terminal 103 can be registered as a non-owned device without directly connecting to device 102 via a network using the device registration code of device 102.

[0265] ● Inference processing on non-owned devices FIG. 15 is a sequence diagram showing a series of processes for executing inference processing without directly connecting to device 102 registered as a non-owned device in terminal 103 by the non-owned device registration process shown in FIG. 14 via a network.

[0266] In step S1501, in response to an operation, the determination application 312 receives an inference processing request from the user of terminal 103. Note that in this operation, the non-owned device 102 is selected and the operation is performed on it.

[0267] In step S1502, the determination application 312 acquires the device information of device 102, which is the operation target, stored in the setting management unit 315.

[0268] In step S1503, the inference application 313 performs inference processing using a machine learning model stored in the inference model management unit 316, which is used by a shared user of device 102 and is the same model as device 102, for device 101. Since the machine learning model is stored in association with information that identifies the device on which it was stored, the user and model can be identified from the device information.

[0269] In step S1504, the judgment application 312 displays the inference result on the operation screen 700 and primarily receives device operation instructions based on the inference result from the user of terminal 103. Then, the judgment application 312 on terminal 103 sends the device operation instructions to the device management application 323 on the device management server 104.

[0270] Table 9 is a table showing an example of a data structure for device operation instructions.

[0271] [Table 9]

[0272] The serial column stores the serial value of the device on which the device operation instruction is to be executed. The value of the serial column in the record for device 102 in the device registration information table in Table 8 is stored there. The user ID column stores the user ID of the user who sent the device operation instruction. The value of the shared user ID column in the record for device 102 in the device registration information table in Table 8 is stored there. The operation instruction content column stores the value of the device operation instruction entered into the operation screen 700 in step S1504.

[0273] In step S1505, the device management application 323 retrieves a record in which the serial column of the device registration information table stored in the data management unit 324 matches the value of the serial column of the received device operation instruction record. The device management application 323 then determines whether the value of the user ID column of the received device operation instruction record is included in the shared user ID column of the device registration information record. If it is included, the device management application 323 saves the device operation instruction in the data management unit 324.

[0274] In step S1506, the configuration management unit 304 of device 102 retrieves a device operation instruction record from the device management application 323 of device management server 104, where the serial column value matches its own serial number in the device operation instruction table stored by the data management unit 324 of device management server 104. Note that step S1506 is executed asynchronously and periodically by device 102. After obtaining the operation instruction record, device 102 controls the device according to the operation instruction.

[0275] As a result, device 102 can indirectly receive device operation instructions input from the user via the device management server 104, without directly connecting to terminal 103 via the network, after the terminal 103 has gone through inference processing.

[0276] [Embodiment 3] In embodiments 1 and 2, the machine learning model copied from device 101 is stored in the inference model management unit 316 by terminal 103 and used for inference processing. The data size of the machine learning model varies depending on the type of device 101 and the scale of the inference processing performed. However, continuously storing the machine learning model can put a strain on the storage area of ​​the terminal 103's external storage device 203, potentially affecting other applications running on terminal 103.

[0277] In this embodiment, a method for automatically deleting a machine learning model from the inference model management unit 316 when the machine learning model copied to the terminal 103 becomes unnecessary will be described. Note that this embodiment is implemented in combination with Embodiment 1 or 2.

[0278] FIG. 16 is a diagram showing an example of a machine learning model portable setting screen in this embodiment. In this embodiment, when the model portable button 802 is pressed on the device details information screen 800, the machine learning model portable setting screen 1600 is displayed.

[0279] When the date and time setting checkbox 1601 is selected, the user of the terminal 103 can set the date and time to delete the machine learning model copied from the device 101 and carried in the date and time setting input field 1602. When the location setting checkbox 1603 is selected, the user of the terminal 103 can set the reference location from which the machine learning model copied from the device 101 is deleted when it is at a certain distance from the reference position in the location setting input field 1604. Note that the position coordinates may be directly input into the location setting input field 1604. For example, when a map application using the GPS 213 is installed in the terminal 103, the position coordinates of the reference position may be input via the map application started by pressing the map start button 1605, for example, by touching a desired position. The reference position input into the location setting input field 1604 is assumed to be the location where the device 102 is installed.

[0280] The back button 1607 is the same as the back button 803, so the description is omitted.

[0281] When the model portable button 1606 is pressed, the device management application 314 acquires a copy of the machine learning model from the device 101 in the procedure described in FIG. 9 and stores it in the inference model management unit 316. At the same time, the model portable settings set on the model portable setting screen 1600 are stored in the setting management unit 315.

[0282] Table 10 is a table showing an example of the data structure of device mobile settings.

[0283] [Table 10]

[0284] The Date and Time Setting column stores the model deletion date and time value entered in the Date and Time Setting Input Field 1602. The Location Setting column stores the location information of the reference location, which is entered in the Location Setting Input Field 1604, indicating that the model will be deleted when it reaches a certain distance from that reference location. The Arrival Flag column stores a value indicating whether or not the location indicated by the value in the Location Setting column has been reached. A value of FALSE indicates that the location has not been reached. A value of TRUE indicates that the location has been reached.

[0285] ● Deletion process for machine learning models Figure 17 is a flowchart showing a series of processes for deleting machine learning models stored in the management unit 316 of terminal 103 based on the model mobile settings configured on screen 1600. Note that the model deletion process shown in Figure 17 is performed asynchronously and periodically by the device management application 314 of terminal 103. In other words, the process in Figure 17 is executed primarily by the CPU 220 of terminal 103.

[0286] Figure 17(a) is a flowchart showing the sequence of processes by which application 314 deletes a machine learning model based on the values ​​in the date and time setting column shown in Table 10.

[0287] In step S1701, the device management application 314 obtains the current time.

[0288] In step S1702, the device management application 314 retrieves the value of the date and time setting column of the model mobile settings stored in the settings management unit 315, and determines whether the current time obtained in step S1701 has exceeded the value of the date and time setting column.

[0289] If the current time exceeds the value in the date and time setting column, in step S1703, the device management application 314 deletes the machine learning model from the inference model management unit 316.

[0290] If the current time has not exceeded the value in the date and time setting column, return to step S1701.

[0291] Figure 17(b) is a flowchart illustrating the sequence of processes by which the device management application 314 deletes a machine learning model based on the location setting column values ​​shown in Table 10.

[0292] In step S1704, the device management application 314 obtains location information from the GPS 213.

[0293] In step S1705, the device management application 314 obtains the model mobile settings from the management unit 315.

[0294] In step S1706, the device management application 314 checks the value of the reach flag column. If the value of the reach flag column is TRUE, i.e., if the configured location has been reached, the application proceeds to step S1707.

[0295] In step S1707, the device management application 314 calculates the distance between the location information obtained in step S1704 and the value of the location setting column obtained in step S1705. If the calculated distance is greater than a certain value, that is, if the terminal 103 has reached the reference location and then moved a predetermined distance from the reference location, the process proceeds to step S1708. In step S1708, the device management application 314 deletes the machine learning model from the inference model management unit 316.

[0296] On the other hand, if the value of the arrival flag column is not TRUE in step S1706, that is, if the set location has not been reached, the process proceeds to step S1709.

[0297] In step S1709, if the location information obtained in step S1704 matches the value in the location setting column obtained in step S1705, the value in the arrival flag column is updated to TRUE in step S1710. Note that the match here does not need to be exact, and a predetermined error may be allowed.

[0298] With the above configuration and procedure, the machine learning model of device 101, which is duplicated and carried on terminal 103, can be automatically deleted from terminal 103 after a specified time has passed or when it moves a predetermined distance from a specified location.

[0299] Furthermore, the deletion conditions described above may be combined. That is, if either a specified time is reached, or a predetermined distance is moved from a reference position after reaching that position, the machine learning model on device 101 held by terminal 103 may be deleted.

[0300] In addition, the model mobile phone settings screen 1600 in Figure 16 may allow specifying the distance along with the coordinates of the reference position. In this case, when the terminal 103 moves a specified distance away from the reference position, the machine learning model of device 101 held by the terminal 103 is deleted.

[0301] Alternatively, instead of deleting the machine learning model on device 101 held by terminal 103 when it has moved a predetermined or specified distance after reaching the reference position, it may be deleted when a predetermined or specified time has elapsed after reaching the reference position.

[0302] [Differentiation] In the above embodiments 1-3, since the terminal 103 also has a learning unit 232, when the terminal 103 uses the machine learning model of the device 101 that it carries, the terminal 103 can also proceed with learning. Therefore, the parameters learned on the terminal 103 may be saved and sent to the learning unit 212 of the device 101 to train the machine learning model of the device 101. This allows the learning results from the form 103 to be reflected in the machine learning model of the device 101 while retaining the learning results of the machine learning model on the device 101 after the mobile phone 103 has carried the machine learning model of the device 101.

[0303] [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0304] ●Summary of Embodiments The above embodiments can be summarized as follows: (Item 1) Communication methods, An inference model management means for holding machine learning models used for inference processing, An inference means that performs inference processing using the machine learning model held by the inference model management means, The system includes a device management means that has a first machine learning model connected by the aforementioned communication means, and acquires a copy of the first machine learning model from a first device that is controlled based on the results of inference processing by the first machine learning model, and maintains it in the inference model management means, In response to a request for inference processing, the inference means executes inference processing using the first machine learning model held in the inference model management means in order to control the second device. An information processing device characterized by the following: (Item 2) The information processing device described in item 1, The inference means acquires information specific to the second device and uses the specific information to perform inference processing using the first machine learning model. An information processing device characterized by the following: (Item 3) An information processing device as described in item 1 or 2, It further includes an input means for user input, The request for the aforementioned inference process is based on input from the user via the input means. An information processing device characterized by the following: (Item 4) An information processing device as described in item 1 or 2, The inference processing request is based on the inference processing request received from the second device via the communication means. An information processing device characterized by the following: (Item 5) An information processing device described in any one of items 1 to 4, It further has output means for outputting to the user, The device management means further registers the first device, outputs information about the registered first device using the output means, and the information about the first device includes information indicating that the first machine learning model is held by the inference model management means. An information processing device characterized by the following: (Item 6) An information processing device described in any one of items 1 to 5, item 1, The second device is a device of the same model as the first device, and is registered in association with a second user who is different from the first user who is registered in association with the first device. The second device is registered in association with the first user as a device available to the first user. An information processing device characterized by the following: (Item 7) The information processing device described in item 6, The device management means can further configure whether to use the second machine learning model possessed by the second device, or to use the first machine learning model held by the inference model management means, for the registered second device. If the second device is configured to use the second machine learning model, then, in response to a user input request for the inference process, the second device will be controlled to execute the inference process using the second machine learning model. An information processing device characterized by the following: (Item 8) An information processing device as described in item 6 or 7, The device management means further registers a device selected from among the devices connected by the communication means as the second device. An information processing device characterized by the following: (Item 9) An information processing device described in any one of items 6 to 7, The device management means receives a device registration code issued by a server connected via the communication means in association with the information of the second device, receives the information of the second device associated with the received device registration code from the server, and registers it as a device available to the first user. An information processing device characterized by the following: (Item 10) An information processing device described in any one of items 1 to 9, The first machine learning model held by the inference model management means is deleted by the device management means. An information processing device characterized by the following: (Item 11) The information processing device described in item 10, The first machine learning model is deleted by the device management means in accordance with the user's instructions. An information processing device characterized by the following: (Item 12) An information processing device as described in item 10 or 11, The first machine learning model is deleted by the device management means at a specified time. An information processing device characterized by the following: (Item 13) An information processing device described in any one of items 10 to 12, It further has positioning means, The first machine learning model is deleted by the device management means when the position determined by the positioning means moves a predetermined distance away from the specified position. An information processing device characterized by the following: (Item 14) A first device having a first machine learning model and capable of performing inference processing using the first machine learning model, An information processing device as described in any one of items 1 to 13, including A device inference system characterized by this feature. (Item 15) A device inference system as described in item 14, The present invention further comprises a second device having a second machine learning model and capable of performing inference processing using the second machine learning model. A device inference system characterized by this feature. (Item 16) A program for causing a computer to function as an information processing device as described in any one of items 1 through 13.

[0305] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0306] 101 Owned devices, 102 Unowned devices, 103 Terminals, 104 Device management server, 105 Device application server

Claims

1. Communication methods, An inference model management means for holding machine learning models used for inference processing, An inference means that performs inference processing using the machine learning model held by the inference model management means, The system includes a device management means that has a first machine learning model connected by the communication means and is controlled based on the results of inference processing by the first machine learning model, and acquires a copy of the first machine learning model from the first device and maintains it in the inference model management means, In response to a request for inference processing, the inference means executes inference processing using the first machine learning model held in the inference model management means in order to control the second device. An information processing device characterized by the following:

2. An information processing apparatus according to claim 1, The inference means acquires information specific to the second device and uses the specific information to perform inference processing using the first machine learning model. An information processing device characterized by the following:

3. An information processing apparatus according to claim 1, It further includes an input means for user input, The request for the aforementioned inference process is based on input from the user via the input means. An information processing device characterized by the following:

4. An information processing apparatus according to claim 1, The inference processing request is based on the inference processing request received from the second device via the communication means. An information processing device characterized by the following:

5. An information processing apparatus according to claim 1, It further has output means for outputting to the user, The device management means further registers the first device, outputs information about the registered first device using the output means, and the information about the first device includes information indicating that the first machine learning model is held by the inference model management means. An information processing device characterized by the following:

6. An information processing apparatus according to claim 1, The second device is a device of the same model as the first device, and is registered in association with a second user who is different from the first user registered in association with the first device. The second device is registered in association with the first user as a device available to the first user. An information processing device characterized by the following:

7. An information processing apparatus according to claim 6, The device management means can further set whether to use the second machine learning model possessed by the second device, or to use the first machine learning model held by the inference model management means, for the registered second device. If the second device is configured to use the second machine learning model, then, in response to a user input request for the inference process, the second device will be controlled to execute the inference process using the second machine learning model. An information processing device characterized by the following:

8. An information processing apparatus according to claim 6, The device management means further registers a device selected from among the devices connected by the communication means as the second device. An information processing device characterized by the following:

9. An information processing apparatus according to claim 6, The device management means receives a device registration code issued by a server connected via the communication means in association with information of the second device, receives information of the second device associated with the received device registration code from the server, and registers it as a device available to the first user. An information processing device characterized by the following:

10. An information processing apparatus according to claim 1, The first machine learning model held by the inference model management means is deleted by the device management means. An information processing device characterized by the following:

11. An information processing apparatus according to claim 10, The first machine learning model is deleted by the device management means in accordance with the user's instructions. An information processing device characterized by the following:

12. An information processing apparatus according to claim 10, The first machine learning model is deleted by the device management means at a specified time. An information processing device characterized by the following:

13. An information processing apparatus according to claim 10, It further has positioning means, The first machine learning model is deleted by the device management means when the position determined by the positioning means moves a predetermined distance away from the specified position. An information processing device characterized by the following:

14. A first device having a first machine learning model and capable of performing inference processing using the first machine learning model, The information processing apparatus according to any one of claims 1 to 13, including A device inference system characterized by this feature.

15. A device inference system according to claim 14, The present invention further comprises a second device having a second machine learning model and capable of performing inference processing using the second machine learning model. A device inference system characterized by this feature.

16. A program for causing a computer to function as an information processing device according to any one of claims 1 to 13.

17. An inference processing method using an information processing device having communication means, device management means, inference model management means for holding machine learning models used for inference processing, and inference means, The device management means has a first machine learning model connected by the communication means, and obtains a copy of the first machine learning model from a first device controlled based on the results of inference processing by the first machine learning model, and has the inference model management means maintain it. The inference means, in response to a request for inference processing, executes inference processing using the first machine learning model held in the inference model management means in order to control the second device. An inference processing method characterized by the following.

Images