Information processing system and information processing method

Abstract

The present invention enables flexible change of functions according to various use cases. An information processing system according to an embodiment includes a sensor data acquisition unit that acquires sensing data acquired by a sensor, a processing unit that includes an inference unit that performs inference using the sensing data as input, and a switching unit that switches the inference unit according to a use case.

Description

Technical Field

[0001] This disclosure relates to information processing systems and information processing methods. Background Technology

[0002] For example, one could conceive of using multiple sensor devices to construct a surveillance system, such as deploying surveillance cameras in various locations throughout a town and using the images to perform monitoring.

[0003] However, traditionally, in this type of surveillance system, it is assumed that the sensing data acquired by sensor devices such as cameras is only used by the individual users who installed the sensor devices, and from a security point of view, and because there is no public interface, it is difficult for other users to easily use the sensing data.

[0004] Therefore, the following patent document 1 proposes a system for constructing a framework in which various users can easily use information obtained from sensing data acquired by various sensor devices.

[0005] Specifically, in Patent Document 1, sensor devices of different specifications and user devices of different specifications can be used within the framework by providing a common interface related to the transmission of sensed data. Furthermore, Patent Document 1 discloses a method to reduce the load associated with data transmission by not continuously transmitting sensed data from the sensor device to the user device, but only transmitting data to the user device when a condition requested from the user device side is met (e.g., when a specific person appears in a monitored image, only the data portion showing the person is transmitted).

[0006] Here, as disclosed in Patent Document 1, specifications called the Smart Camera Ecosystem (NICE) are formulated to enable the use of sensor devices with different specifications and user devices with different specifications within the framework, and to reduce the load related to data transmission by sending data to the user device side only when predetermined conditions are met.

[0007] As a NICE specification, the NICE Data Pipeline Specification v1.0.1 (10.8.2.JSON object) defines the transmission data format used by a sensor device to transmit sensed data (“SceneData”) when predetermined conditions are met. Specifically, this format specifies that “SceneData” is sent as the actual data portion of the sensed data, and data called “SceneMark” is an additional data portion of “SceneData” that includes information about “SceneDataType” indicating the type of “SceneData”.

[0008] Reference List

[0009] Patent documents

[0010] Patent Document 1: WO 2020 / 100922 A

[0011] Patent Document 2: JP 2020-129390 A

[0012] Patent Document 3: JP 2020-123379 A Summary of the Invention

[0013] Technical issues

[0014] However, the current NICE specification has not defined a specific mechanism for replacing or updating models pre-integrated as inference units on the device side. As a result, the use of devices with models already incorporated is limited, and there is a difficulty in flexibly changing the functionality provided to users according to various use cases.

[0015] Therefore, this disclosure proposes an information processing system and information processing method that can flexibly change functions according to various usage scenarios.

[0016] Solution to the problem

[0017] To address the aforementioned problems, an information processing system according to embodiments of this disclosure includes: a sensor data acquisition unit for acquiring sensing data obtained by a sensor; a processing unit including an inference unit that uses the sensing data as input to perform inference; and a switching unit for switching the inference unit according to usage. Attached Figure Description

[0018] Figure 1 This is a system diagram illustrating the schematic functional configuration of an information processing system according to the first embodiment.

[0019] Figure 2 This is a block diagram illustrating an example of the functional configuration of the sensor device 10 according to the first embodiment.

[0020] Figure 3 This is a block diagram illustrating an example of the functional configuration of the service server 20 according to the first embodiment.

[0021] Figure 4 This is a block diagram illustrating an example of the functional configuration of the aggregate server 50 according to the first embodiment.

[0022] Figure 5 This is an explanatory diagram showing an example of generating the recognition model 610 according to the first embodiment.

[0023] Figure 6 This is a sequence diagram illustrating an example of an information processing method according to a first embodiment.

[0024] Figure 7 A block diagram illustrating a schematic example of a system architecture for an apparatus capable of downloading an AI model into an information processing system according to a first embodiment.

[0025] Figure 8 This is a block diagram illustrating a schematic example of the node architecture of the device according to the first embodiment.

[0026] Figure 9 This is a schematic diagram showing an overview of the overall operation of the information processing system according to the first embodiment.

[0027] Figure 10 This is a sequence diagram illustrating an example of the basic operation sequence according to the first embodiment.

[0028] Figure 11 This is a diagram illustrating an example of a SceneMode object according to the first embodiment.

[0029] Figure 12 This is a diagram illustrating an example of a RefDataList according to the first embodiment.

[0030] Figure 13 This is a diagram illustrating an example of RefData according to the first embodiment.

[0031] Figure 14 This is a diagram illustrating an example of a processing stage according to the first embodiment.

[0032] Figure 15 This is a diagram showing an example of a StartScene object according to the first embodiment.

[0033] Figure 16 This is a diagram illustrating an example of model data in the reference data according to the first embodiment.

[0034] Figure 17 This is a sequence diagram illustrating an example of an operation sequence when using a pre-set AI model according to the first embodiment.

[0035] Figure 18 This is a sequence diagram illustrating an example of the operation sequence when downloading an AI model according to the first embodiment.

[0036] Figure 19 This is a block diagram illustrating a schematic example of the node architecture of a device according to a variation of the first embodiment.

[0037] Figure 20 This is a sequence diagram illustrating an example of the operation sequence when downloading an AI model according to a variation of the first embodiment.

[0038] Figure 21 This is a diagram illustrating an example of an encrypted object according to the second embodiment.

[0039] Figure 22 This is a diagram illustrating an example of a privacy object according to the second embodiment.

[0040] Figure 23 This is a sequence diagram illustrating an example of an operation sequence according to the second embodiment.

[0041] Figure 24 This is a block diagram illustrating an implementation example of an information processing system according to a third embodiment.

[0042] Figure 25 A hardware configuration diagram illustrating an example of a computer implementing the functions of an information processing apparatus according to the present disclosure. Detailed Implementation

[0043] In the following description, embodiments of the present disclosure will be illustrated with reference to the accompanying drawings. Furthermore, in the following embodiments, the same reference numerals will be used to refer to the same parts, and repeated descriptions will be omitted.

[0044] This disclosure will be described in the following order of items.

[0045] 0. Introduction

[0046] 1. First Implementation Method

[0047] 1.1 Illustrative Configuration of Information Processing System

[0048] 1.2 Detailed Configuration of Sensor Device

[0049] 1.3 Detailed Configuration of the Service Server

[0050] 1.4 Detailed Configuration of the Total Server

[0051] 1.5 Identification according to this embodiment

[0052] 1.6 Information Processing Methods

[0053] 1.7 System architecture capable of downloading AI models

[0054] 1.8 Node Architecture in the Device

[0055] 1.9 Overall Operational Overview

[0056] 1.10 Basic Operation Sequence

[0057] 1.11 Example of a SceneMode object

[0058] 1.12 Example of the StartScene object

[0059] 1.13 Example of model data

[0060] 1.14 Example of operation sequence when using a preset AI model

[0061] 1.15 Example of an operation sequence when downloading and using an AI model

[0062] 1.16 Variation Example

[0063] 1.17 Other variations

[0064] 1.18 Summary

[0065] 2. Second Implementation Method

[0066] 2.1 Example of an encrypted object

[0067] 2.2 Examples of Privacy Objects

[0068] 2.3 Example of operation sequence when encrypting model data

[0069] 2.4 Summary

[0070] 3. Third Implementation Method

[0071] 3.1 Implementation Example

[0072] 3.2 Operation Example

[0073] 4. Hardware Configuration

[0074] 0. Introduction

[0075] As mentioned above, it is difficult to replace an already implemented model in a device that pre-integrates a model for performing authentication or inference. Therefore, since specific results can only be obtained from devices that already incorporate models and whose applications are limited, it is difficult to flexibly change the functionality provided to users according to various use cases. Furthermore, replacing functionality requires updating firmware, etc., and it is difficult to implement appropriate functionality in a timely manner according to the scenarios, services, etc., required by users, etc.

[0076] Therefore, the following embodiments propose an information processing system and method capable of timely switching of functions according to usage. Since various detections and analyses can be performed by switching functions, functions can be switched with a high degree of flexibility according to usage.

[0077] Furthermore, the following implementation method enables the replacement of functions in a secure state. Therefore, the information processing device can be protected from information leakage, tampering, etc., and thus, functions can be reliably provided to the user based on usage.

[0078] For example, in the following embodiments, the device performing AI processing (e.g., machine learning) is free to acquire and implement AI models, parameters, etc., for performing AI processing from an application, server, or another device while maintaining a secure state. This enables the provision of a mechanism that can immediately perform arbitrary AI processing when necessary. It should be noted that, for example, the device may be equipped with various sensors such as cameras, microphones, thermometers, and barometers, or may have the ability to receive information obtained from sensors from another device, perform AI processing on the data obtained therefrom, and transmit the processing results to an application, server, or another device.

[0079] By providing this mechanism, device functionality can be freely added or switched not only based on usage scenarios or situations, but also based on data required by applications, servers, or other devices. Therefore, it can also collaborate with AI processing on the application, server, or other device side, depending on usage scenarios and situations, and provide the target service to the user in the optimal form as a whole system.

[0080] Furthermore, when performing relearning of AI models, the relearned AI models can be easily relocated to various devices.

[0081] 1. First Implementation Method

[0082] In the following, the information processing system and information processing method according to the first embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0083] 1.1 Illustrative Configuration of Information Processing System

[0084] First, refer to Figure 1 A configuration example of an information processing system according to this embodiment is described. Figure 1 This is a system diagram illustrating the schematic functional configuration of the information processing system according to this embodiment.

[0085] like Figure 1As shown, the information processing system 1 according to this embodiment may mainly include multiple sensor devices 10a, 10b and 10c, a service server 20, multiple user devices 30a, 30b and 30c, an authentication server 40, and a aggregation server 50. For example, each of these devices is connected to a network (not shown) via a base station (not shown) or the like (e.g., a base station for a mobile phone and an access point for a wireless local area network (LAN)), thereby constructing the information processing system 1. Note that as a communication system used in the network, any method can be applied, whether wired or wireless (e.g., fifth-generation communication systems, WiFi (registered trademark), and Bluetooth (registered trademark)), but it is desirable to use a communication system capable of transmitting large amounts of data at high speed and stably. Furthermore, the number of sensor devices 10 and user devices (request sources) 30 included in the information processing system 1 are not limited to, for example, Figure 1 The three, and three or more, sensor devices and three or more user devices shown may be included in the information processing system 1 according to this embodiment. In other words, the information processing system 1 according to this embodiment can manage multiple sensor devices 10, receive requests from multiple user devices 30, and send data to user devices 30. Hereinafter, an outline of each device included in the information processing system 1 according to this embodiment will be described.

[0086] (Sensor device 10)

[0087] Sensor device 10 can acquire sensing data (e.g., images and sounds) of the surrounding environment on which it is mounted, and send distribution data (predetermined data) obtained from the acquired sensing data to an external device such as user device 30. Furthermore, sensor device 10 is expected to have AI capabilities and can identify whether the acquired sensing data corresponds to a request (distribution request) from a user based on a machine learning model (hereinafter referred to as an AI model) sent from service server 20. Note that in this specification, the identification model that performs recognition processing on the data is exemplified as an AI model; however, the AI ​​model is not limited to this, and various AI models learned for various purposes can be applied.

[0088] For example, sensor device 10 can be an imaging device (camera) mounted on a mobile body such as a car, an imaging device mounted on a smartphone carried by a user, or an imaging device such as a surveillance camera mounted in a residence, shop, etc. In this case, the sensing data is an image. These imaging devices acquire images by collecting light from around the subject on which the imaging device is mounted, forming an optical image on an imaging surface, and converting the optical image formed on the imaging surface into an electrical image signal. In the following description, unless otherwise specified, the mobile body can be a car, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot (mobile robot), construction machinery, agricultural machinery (tractor), etc.

[0089] Furthermore, in this embodiment, the sensor device 10 is not limited to the imaging device described above. In this embodiment, for example, the sensor device 10 may be a depth sensor that measures the distance (depth) to the subject, a sound collection device (such as a microphone) that collects sounds from the surrounding environment, a temperature sensor and a humidity sensor that measure the temperature and humidity of the surrounding environment, and a water level sensor that measures the water level of a river, etc.

[0090] Note that in this embodiment, the internal configuration of the sensor device 10 is essentially unrestricted, as long as the sensor device 10 has an interface (e.g., data transmission format and data transmission method) that is common to the information processing system 1. Therefore, the information processing system 1 according to this embodiment can include various sensor devices 10 with different specifications. Note that the detailed configuration of the sensor device 10 will be described later.

[0091] (Server 20)

[0092] Service server 20 is a computer that receives distribution requests from user device 30 for distributing distribution data generated from sensing data. Furthermore, service server 20 can integrate multiple distribution requests, generate an identification model based on the distribution requests, and transmit the generated identification model to the aforementioned sensor device 1. Additionally, service server 20 can receive distribution data from sensor device 10 and send the received distribution data to the user device 30 corresponding to the distribution request as needed. Service server 20 is implemented using hardware such as a central processing unit (CPU), read-only memory (ROM), and random access memory (RAM). Note that the detailed configuration of service server 20 will be described later.

[0093] (User device 30)

[0094] User device 30 is a terminal carried or attached to the user, capable of receiving user input, sending the received information as a distribution request to service server 20, and receiving distribution data related to the distribution request. For example, user device 30 can be a mobile terminal such as a tablet PC, smartphone, smartwatch, mobile phone, laptop PC, or notebook PC, or a wearable device such as a head-mounted display (HMD). Furthermore, user device 30 can be an electronic control unit (ECU) installed in a vehicle, or a controller for remotely controlling drones, robots, etc. Additionally, user device 30 may include a display unit (not shown) that performs a display facing the user, an operation unit (not shown) that receives user operations, or a speaker (not shown) that outputs voice to the user.

[0095] It should be noted that, for example, the user device 30 may install applications common to the information processing system 1 or applications with specifications common to the service server 20. When such applications are installed, the user device 30 may generate and send distribution requests with formats common to the information processing system 1, and receive distribution data.

[0096] Note that in this embodiment, the user is considered not only as an individual, but also as someone who can be a map maker, a store development strategy planner, a road management agency, or a staff member responsible for social infrastructure development. More specifically, by using the services provided in this embodiment, map makers can create detailed maps without expending manpower, and store development strategy planners can easily collect information when surveying potential store locations. Furthermore, by using the aforementioned services, road management agencies can easily collect information for planning road maintenance programs based on estimates of road conditions, vehicle types, etc. Additionally, by using the aforementioned services, staff responsible for social infrastructure development planning can consider preventative measures and telematics insurance based on statistics and analysis of driving tendencies and accident causes.

[0097] Then, the user sends a distribution request to the service server 20 via user device 30. The distribution request includes information specifying the data content (data type) the user requests to be distributed. For example, the distribution request may include object information about the object to which the requested distribution is targeted (e.g., face, person, animal, moving object, text, road (sidewalk, sidewalk, road width or sign), sign, and barcode), and information specifying the type of information about the requested object (e.g., quantity, speed, location, status, age, gender, name of the identifier, and estimation result) (data type). Furthermore, the distribution request may include data format information specifying the data format of the distributed data (e.g., image and text). Additionally, the distribution request may include the user's or user device 30's identification information (ID). Note that in this embodiment, the distribution request may include various types of data used by the service server 20 to generate the identification model.

[0098] In this embodiment, the distribution request has a common data format to the information processing system 1. For example, in this embodiment, the distribution request may include a string or numeric string corresponding to object information or data format information. In this case, for object information and data format information requested frequently by the user, the corresponding string or numeric string is predetermined and stored as a database (not shown) in a storage unit (not shown) maintained by the service server 20. Then, the service server 20 can identify the object information and data format information corresponding to the string, etc., contained in the distribution request by referring to the database. Furthermore, when there is no string, etc., corresponding to the object information and data format information expected by the user in the database, the object information and data format information can be directly described in the distribution request. Alternatively, in this case, a new string corresponding to the object information and data format information expected by the user can be determined, and the determined string, etc., can be described in the distribution request and the database.

[0099] (Authentication Server 40)

[0100] The authentication server 40 is a computer that receives authentication information (ID) from each of the sensor devices 10 and the service server 20 and determines whether each of these devices has the authority to provide or receive services through the information processing system 1 according to this embodiment. Furthermore, the authentication server 40 sends a key to access the service, commands for providing or receiving services, etc., to the authorized device. The authentication information has a common data format of the information processing system 1. In other words, the authentication server 40 is used as an authentication application programming interface (API) and can authenticate the sensor devices 10 and the service server 20 and associate them with each other. By incorporating the authentication server 40, the information processing system 1 according to this embodiment can ensure the security of the information processing system 1 and associate each sensor device 10 with each user device 30 via the service server 20. Furthermore, the authentication server 40 can be implemented using hardware such as a CPU, ROM, and RAM, and can cooperate with the aggregate server 50. Note that in this embodiment, the authentication server 40 can perform authentication on the user device 30.

[0101] (Total of 50 servers)

[0102] The aggregate server 50 is a computer that collaborates with the aforementioned authentication server 40 to aggregate the amount of data distributed to each user or user device 30 and each sensor device 10, and to determine the data usage fee based on the aggregate distribution. Furthermore, the aggregate server 50 can transmit data usage fees between the administrator of the sensor device 10 and each user via the service server 20. For example, the aggregate server 50 can be implemented using hardware such as a CPU, ROM, and RAM.

[0103] Note that in the information processing system 1 according to this embodiment, each of the sensor device 10 and the service server 20 may not be implemented by a single device, but may be implemented by multiple devices connected to each other via various wired or wireless networks (not shown) to cooperate with each other. Furthermore, in this embodiment, two or all of the service server 20, authentication server 40, and aggregation server 50 may be implemented by an integrated device. Additionally, the information processing system 1 according to this embodiment may include another server (not shown).

[0104] 1.2 Detailed Configuration of Sensor Device

[0105] Next, we will refer to Figure 2 The detailed configuration of the sensor device 10 according to this embodiment is described. Figure 2 This is a block diagram illustrating an example of the functional configuration of the sensor device 10 according to this embodiment. Specifically, as... Figure 2As shown, the sensor device 10 mainly includes a sensor unit 100, a positioning unit 110, a processing unit 130, a storage unit 160, and a communication unit 170. Each functional block of the sensor device 10 will be described sequentially below.

[0106] (Sensor Unit 100)

[0107] The sensor unit 100 acquires sensing data and outputs the acquired sensing data to the processing unit 130. Specifically, when the sensor device 10 is an imaging device, the sensor unit 100 includes an imaging optical system (such as an imaging lens and a zoom lens that converge light emitted from the subject) and an imaging element (such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS)). Note that in this embodiment, the sensor unit 100 may be fixed in the sensor device 10 or may be detachably mounted on the sensor device 10.

[0108] Furthermore, in this embodiment, in addition to the imaging device, the sensor unit 100 may also include a time-of-flight (TOF) sensor (not shown) as a depth sensor. The TOF sensor can acquire shape information (depth information / image), such as the distance and unevenness between the TOF sensor and the subject, by directly or indirectly measuring the return time of reflected light from the subject. In addition, in this embodiment, the sensor unit 100 may include a sound collection device (microphone), a temperature sensor, a barometric pressure sensor, a humidity sensor, a wind direction / speed sensor, a sunlight sensor, a rainfall sensor, a water level sensor, etc., and is not particularly limited, as long as sensing data can be obtained from the surrounding environment.

[0109] (Positioning Unit 110)

[0110] When acquiring sensing data corresponding to a distribution request, the positioning unit 110 acquires the positioning data of the sensor device 10 and outputs the acquired positioning data to the processing unit 130 (specifically, to the data generation unit 144). For example, the positioning data is sent to the user device 30 along with distribution data via the distribution unit 146 of the processing unit 130. Specifically, the positioning unit 110 may be a Global Navigation Satellite System (GNSS) receiver. In this case, the positioning unit 110 may generate positioning data indicating the latitude and longitude of the sensor device 10 based on signals from GNSS satellites. Furthermore, in this embodiment, since the relative position of the user can be detected based on information from, for example, radio frequency identification (RFID), Wi-Fi access points, or radio base stations, these communication devices may also be used as the positioning unit 110. Note that in this embodiment, the positioning unit 110 may not be provided in the sensor device 10.

[0111] (Processing Unit 130)

[0112] The processing unit 130 has the function of processing the sensing data acquired by the sensor unit 100 and generating distribution data. Furthermore, the processing unit 130 sums the distribution amount of the distributed data and sends the summation information to the summation server 50. The processing unit 130 is implemented by, for example, processing circuitry (such as a CPU or graphics processing unit (GPU)), ROM, RAM, etc. Specifically, as... Figure 2 As shown, the processing unit 130 mainly includes an ID transmission unit 132, a key receiving unit 134, a sensor data acquisition unit 136, a preprocessing unit 138, a model acquisition unit 140, an identification unit 142, a data generation unit 144, a distribution unit 146, a aggregation unit 148, and an aggregation information transmission unit 150. Details of each functional unit included in the processing unit 130 will be described below.

[0113] ID transmission unit 132

[0114] ID transmission unit 132 sends the authentication information (ID) of sensor device 10 to authentication server 40 via communication unit 170. Authentication server 40 uses the authentication information when determining whether sensor device 10 has the authority to provide services through information processing system 1 according to this embodiment. Information processing system 1 according to this embodiment ensures its security through authentication.

[0115] Key receiving unit 134

[0116] The key receiving unit 134 receives, via the communication unit 170, a key that enables access to the service, commands for providing the service, etc., sent from the authentication server 40, and outputs the received key, etc., to the model acquisition unit 140. The model acquisition unit 140 can use the key, etc., received by the key receiving unit 134 to acquire an identification model from the service server 20.

[0117] Sensor data acquisition unit 136

[0118] The sensor data acquisition unit 136 controls the sensor unit 100 to acquire the sensing data output from the sensor unit 100, and outputs the acquired sensing data to the preprocessing unit 138 or the recognition unit 142.

[0119] Preprocessing unit 138

[0120] The preprocessing unit 138 preprocesses the sensing data output from the sensor data acquisition unit 136 based on information included in the recognition model acquired by the model acquisition unit 140 (e.g., information related to the supervision data used when generating the recognition model), and outputs the preprocessed sensing data to the recognition unit 142. Specifically, the recognition unit 142 identifies whether the sensing data corresponds to the distribution request by using a recognition model corresponding to the distribution request obtained through machine learning. Therefore, in this embodiment, by performing preprocessing, sensing data suitable for recognition can be provided to the recognition unit 142, making the sensing data have a form close to the recognition model. As a result, according to this embodiment, the recognition accuracy of the recognition unit 142 can be improved.

[0121] Model acquisition unit 140

[0122] The model acquisition unit 140 acquires the recognition model corresponding to the distribution request from the service server 20 via the communication unit 170, and outputs the acquired recognition model to the preprocessing unit 138 and the recognition unit 142.

[0123] Identification unit 142

[0124] The recognition unit 142, based on the recognition model output from the model acquisition unit 140, uses AI functions or the like to identify whether the sensing data output from the sensor data acquisition unit 136 or the sensing data preprocessed by the preprocessing unit 138 corresponds to a distribution request. More specifically, for example, the recognition unit 142 can identify whether an image of the object specified by the distribution request is included in the image as sensing data (e.g., object recognition). Then, the recognition unit 142 outputs the recognition result to the data generation unit 144. Note that the recognition model is obtained through machine learning in the service server 20, and may be, for example, feature information characterizing the object specified by the distribution request, or feature information obtained from data such as images or speech of the object. In this embodiment, since the above recognition is performed by the sensor device 10, recognition can be performed immediately after the sensing data is acquired.

[0125] Data generation unit 144

[0126] When the identification unit 142 identifies that the sensing data corresponds to the distribution request, the data generation unit 144 can generate distribution data by performing processing on the sensing data corresponding to the distribution request. For example, the data generation unit 144 can generate distribution data from the sensing data by extracting only data related to the object specified in the distribution request, abstracting the data, or converting the data into text data. More specifically, the distribution data may include at least one of the following: characteristic information, quantization information, location information, status information, operation information, surrounding environment information, and prediction information of the object specified in the distribution request. Furthermore, in this embodiment, the data format of the distribution data may be image data, audio data, text data, etc., and is not particularly limited. As described above, in this embodiment, the sensor device 10 processes the sensing data corresponding to the distribution request and generates distribution data, thus enabling real-time distribution. In addition, in this embodiment, if the sensing data does not correspond to the distribution request, the data generation unit 144 does not generate or distribute distribution data. Therefore, according to this embodiment, compared to the case where sensing data is sent regardless of whether the sensing data corresponds to the distribution request, the data transmission load can be reduced.

[0127] Furthermore, in this embodiment, for example, the data generation unit 144 can exclude privacy-related information (e.g., images that can identify people) included in the sensing data from the distributed data. Additionally, for example, the data generation unit 144 can mask privacy-related information in the distributed data. Thus, privacy is protected in this embodiment.

[0128] Distribution Unit 146

[0129] The distribution unit 146 distributes the distribution data generated by the data generation unit 144 to the user device 30 or the service server 20. Alternatively, the distribution unit 146 can distribute multiple different distribution data to the user device 30 or the service server 20. Furthermore, the distribution unit 146 outputs information such as the distribution amount of the distribution data distributed to the aggregation unit 148. For example, the distribution unit 146 outputs information such as the date and time information of the sensing data corresponding to the distribution data, the date and time information of the distribution data, the data type, the data format, the distribution amount, and the distribution destination (e.g., the identification information of the user device 30).

[0130] Total of 148 units

[0131] For example, the aggregation unit 148 generates aggregation information by aggregating the distribution amount of each user or each user device 30, and outputs the aggregation information to the storage unit 160.

[0132] A total of 150 information transmission units.

[0133] The total information transmission unit 150 retrieves total information from the storage unit 160 at predetermined intervals (e.g., monthly) and sends the retrieved total information to the total server 50. As total information, the total information transmission unit 150 sends information such as the date and time that the sensing data corresponding to the distributed data was acquired, the date and time that the distributed data was distributed, the data type, the data format, the amount distributed, the distribution destination (e.g., identification information of the user device 30), the identification information of the sensor device 10, and information about the administrator of the sensor device 10.

[0134] (Storage unit 160)

[0135] Storage unit 160 stores programs, information, etc., for processing unit 130 to perform various types of processing, as well as information obtained through processing. For example, storage unit 160 stores aggregated information generated by aggregate unit 148. It should be noted that storage unit 160 is implemented by, for example, a storage device (e.g., a hard disk drive (HDD)).

[0136] (Communication Unit 170)

[0137] The communication unit 170 can send information to an external device (such as a service server 20) and receive information from an external device. In other words, the communication unit 170 is a communication interface with the function of sending and receiving data. It should be noted that the communication unit 170 is implemented by communication devices (not shown), such as a communication antenna, a transmitting / receiving circuit, and a port.

[0138] 1.3 Detailed Configuration of the Service Server

[0139] Next, we will refer to Figure 3 The detailed configuration of the service server 20 according to this embodiment is described. Figure 3 This is a block diagram illustrating an example of the functional configuration of the service server 20 according to this embodiment. Specifically, as... Figure 3 As shown, the service server 20 mainly includes a processing unit 230, a storage unit 260, and a communication unit 270. Each functional block of the service server 20 will be described sequentially below.

[0140] (Processing Unit 230)

[0141] The processing unit 230 has the function of acquiring a distribution request from the user device 30 via the communication unit 270, generating a recognition model according to the acquired distribution request, and transmitting the generated recognition model to the sensor device 10. The processing unit 230 is implemented by, for example, processing circuitry such as a CPU or GPU, ROM, or RAM. Specifically, as... Figure 3As shown, the processing unit 230 mainly includes an ID transmission unit 232, a request acceptance unit 234, a capability information acquisition unit 236, a model generation unit 238, a model transmission unit 240, a data acquisition unit 242, and a distribution unit 246. Details of each functional unit included in the processing unit 230 will be described below.

[0142] ID transmission unit 232

[0143] ID transmission unit 232 sends the authentication information (ID) of service server 20 to authentication server 40 via communication unit 270. This authentication information is used by authentication server 40 to determine whether service server 20 has the authority to receive services from information processing system 1 according to this embodiment. Information processing system 1 according to this embodiment ensures its security through authentication.

[0144] Request Acceptance Unit 234

[0145] The request receiving unit 234 receives distribution requests from one or more user devices 30 via the communication unit 270 and outputs the received distribution requests to the model generation unit 238. Alternatively, the request receiving unit 234 can also synthesize common distribution requests and output a synthesized distribution request.

[0146] Capability Information Acquisition Unit 236

[0147] The capability information acquisition unit 236 acquires capability information in advance via the communication unit 270, indicating the sensing capabilities (e.g., sensing type, accuracy, location, range, and granularity) and arithmetic capabilities of each sensor device 10, and outputs the acquired capability information to the model generation unit 238. In this embodiment, the capability information acquisition unit 236 preferably reacquires the capability information when the functions of the sensor devices 10 are updated. Then, in this embodiment, the model generation unit 238 generates a recognition model based on the capabilities of each sensor device 10 based on the capability information of each sensor device 10. Therefore, according to this embodiment, by reacquiring the capability information when the functions of the sensor devices 10 are updated, the recognition model can be updated according to the current capabilities of each sensor device 10.

[0148] Model generation unit 238

[0149] The model generation unit 238 can generate a recognition model corresponding to the distribution request based on the distribution request from the request acceptance unit 234 and the capability information from the capability information acquisition unit 236, using machine learning. Furthermore, the model generation unit 238 can output the generated recognition model to the model transmission unit 240. Note that the model generation unit 238 can acquire the data required for machine learning from the user device 30 or another server (not shown). In this embodiment, since the model generation unit 238 can generate a recognition model suitable for each sensor device 10, recognition within the sensor device 10 can be achieved. Furthermore, in this embodiment, when the functions of the sensor device 10 are updated, the capability information can be reacquired and the recognition model regenerated. Therefore, the recognition model can be dynamically changed to improve the recognition of the sensor device 10.

[0150] Furthermore, the model generation unit 238 may include information about the data used in the machine learning process of the model generation unit 238 (e.g., information about the supervision data used in generating the recognition model) in the recognition model. This information is used when the preprocessing unit 138 of the sensor device 10 performs preprocessing to make the sensed data have a form close to the recognition model. Additionally, in order to obtain the sensed data required to generate the distribution data specified by the distribution request based on the capability information of the sensor device 10, the model generation unit 238 may include setting information about the settings of the sensor unit 100 of the sensor device 10 in the recognition model. It should be noted that in this embodiment, the model generation unit 238 may be provided as a device separate from the service server 20 and is not particularly limited thereto.

[0151] Model transmission unit 240

[0152] The model transmission unit 240 transmits the recognition model obtained from the model generation unit 238 to the sensor device 10 corresponding to each recognition model via the communication unit 270.

[0153] Data acquisition unit 242

[0154] The data acquisition unit 242 acquires distribution data corresponding to the distribution request from the sensor device 10 via the communication unit 270, and outputs the acquired distribution data to the distribution unit 246. Note that in this embodiment, when the distribution data is sent directly from the sensor device 10 to the user device 30, the data acquisition unit 242 may not be provided.

[0155] Distribution Unit 246

[0156] The distribution unit 246 distributes the distribution data acquired by the data acquisition unit 242 to the user device 30 corresponding to the distribution request via the communication unit 270. It should be noted that in this embodiment, when the distribution data is directly transmitted from the sensor device 10 to the user device 30, the distribution unit 246 may not be provided.

[0157] (Storage unit 260)

[0158] Storage unit 260 stores programs, information, etc., for processing unit 230 to perform various types of processing, as well as information obtained through processing. It should be noted that storage unit 260 is implemented using, for example, a storage device (e.g., HDD).

[0159] (Communication Unit 270)

[0160] The communication unit 270 can send information to and receive information from external devices (such as sensor device 10 and user device 30). In other words, the communication unit 270 is a communication interface with the function of sending and receiving data. Furthermore, the communication unit 270 is implemented by communication devices (not shown), such as a communication antenna, a transmitting / receiving circuit, and a port.

[0161] 1.4 Detailed Configuration of the Total Server

[0162] Next, we will refer to Figure 4 The detailed configuration of the total server 50 according to this embodiment is described. Figure 4 This is a block diagram illustrating an example of the functional configuration of the aggregate server 50 according to this embodiment. Specifically, as... Figure 4 As shown, the aggregate server 50 mainly includes a processing unit 530, a storage unit 560, and a communication unit 570. Each functional block of the aggregate server 50 will be described sequentially below.

[0163] (Processing Unit 530)

[0164] The processing unit 530 has functions such as processing aggregated information transmitted from the sensor device 10, and performing processing such as calculating data usage fees based on the aggregated information. The processing unit 530 is implemented, for example, by processing circuitry such as a CPU, ROM, or RAM. Specifically, such as... Figure 4 As shown, the processing unit 530 mainly includes a total information acquisition unit 532 and a cost processing unit 534. Details of each functional unit included in the processing unit 530 will be described below.

[0165] Total information acquisition units: 532

[0166] The total information acquisition unit 532 acquires total information from each sensor device 10 via the communication unit 570, which will be described later, and outputs the acquired total information to the storage unit 560, which will be described later.

[0167] Fee processing unit 534

[0168] The fee processing unit 534 determines the data usage fee based on aggregate information stored in the storage unit 560 (described later), and can transmit the data usage fee between the aggregate server 50, the administrator of the sensor device 10, and the service server 20. Specifically, the fee processing unit 534 determines the data usage fee to be charged to the service server 20 based on the amount of distributed data and the fee per unit of distributed data (e.g., defined by the data format). Similarly, the fee processing unit 534 determines the data usage fee to be paid to each sensor device 10. In this embodiment, since the administrator of each sensor device 10 can receive the data usage fee, the reward for the service provided by the information processing system 1 according to this embodiment can also be distributed to the administrator of each sensor device 10. Thus, in this embodiment, since the party providing the sensing data can also receive reward based on the amount distributed and the value of the distributed data, the motivation of the information processing system 1 according to this embodiment to provide sensing data to the service can be increased.

[0169] (Storage Unit 560)

[0170] Storage unit 560 stores programs, information, etc., for processing unit 530 to perform various types of processing, as well as information obtained through processing. For example, storage unit 560 stores aggregated information transmitted from sensor device 10. It should be noted that storage unit 560 is implemented by, for example, a storage device (e.g., HDD).

[0171] (Communication Unit 570)

[0172] The communication unit 570 can send information to and receive information from external devices such as the sensor device 10. In other words, the communication unit 570 is a communication interface with the function of sending and receiving data. Note that the communication unit 570 is implemented by communication devices (not shown), such as a communication antenna, a transmitting / receiving circuit, and a port.

[0173] 1.5 Identification according to this embodiment

[0174] Next, we will refer to Figure 5 An example of identification according to this embodiment is described. Figure 5 This is an explanatory diagram showing an example of generating the recognition model 610 according to this embodiment.

[0175] (Generation of the recognition model)

[0176] First, the generation of the model used in the recognition according to this embodiment will be described. As described above, the recognition model is generated by the model generation unit 238 of the service server 20. Figure 5 As shown, the model generation unit 238 includes, for example, a supervised learning device 238a, such as support vector regression or a deep neural network. First, the learning device 238a receives, for example, multiple pieces of supervised data 602-1 to 602-n, which are information related to the object requested by the distribution specified by the distribution request. Then, the learning device 238a can generate a recognition model 610 used by the recognition unit 142 of the sensor device 10 for recognition by performing machine learning on the input multiple pieces of supervised data 602-1 to 602-n. In this embodiment, because the multiple sensor devices 10 have different sensing capabilities and arithmetic capabilities, i.e., different recognition capabilities, the learning device 238a preferably generates the recognition model 610 based on the capability information of each sensor device 10. Therefore, in this embodiment, even when multiple sensor devices 10 with various specifications are included, each sensor device 10 can perform recognition because the recognition model 610 can be generated based on the capabilities of each sensor device 10.

[0177] More specifically, for example, the learning device 238a receives input of supervision data 602-1 to 602-n related to an object requested by a distribution specified by a distribution request, the object being labeled by objects 600-1 to 600-n. The learning device 238a then extracts feature points and feature quantities of the object from the multiple supervision data 602-1 to 602-n using machine learning such as a recurrent neural network. Information such as feature points extracted in this way becomes a recognition model 610 used to identify whether information about the object is included in the sensing data acquired by each sensor device 10.

[0178] Here, the generation of the recognition model 610 according to this embodiment will be described by specific examples. For example, a case will be described where a search for a predetermined person (object) is requested using images (sensing data) acquired by the sensor device 10 based on a distribution request from a user (the distribution data is location information about the predetermined person). The service server 20 acquires multiple images of the predetermined person from the user device 30 that sent the distribution request or from a server (not shown), as multiple pieces of supervision data 602 for generating the recognition model 610. Then, as... Figure 5As shown, the service server 20 inputs multiple acquired images (supervision data) 602-1 to 602-n labeled with predetermined persons (objects) 600-1 to 600-n into the learning device 238a. Furthermore, the learning device 238a performs machine learning using the multiple images (supervision data) 602-1 to 602-n to extract feature points and feature quantities of the images of the predetermined persons (objects) 600 from the multiple images (supervision data) 602-1 to 602-n, and generates a recognition model 610 for recognizing the images of the predetermined persons (objects) 600 based on the images (sensing data).

[0179] Note that in this embodiment, similar to the example described above, when a search for a predetermined person (object) is requested based on a distribution request from a user using sensing data acquired by sensor device 10 (here, the type of sensing data is not particularly limited), the learning device 238a can generate a recognition model 610 based on the type of sensing data that can be acquired by each sensor device 10. More specifically, in this case, the learning device 238a generates a recognition model 610 for recognizing an image of the predetermined person from an image acquired by sensor device 10 capable of acquiring an image, and generates a recognition model 610 for recognizing the speech of the predetermined person from an ambient sound acquired by sensor device 10 capable of acquiring ambient sound. Therefore, in this embodiment, even when multiple sensor devices 10 with various specifications are included, each sensor device 10 can perform recognition because a recognition model 610 can be generated based on the capabilities of each sensor device 10.

[0180] Furthermore, in this embodiment, for example, when a user requests to search for a predetermined cat (here, a cat named "TAMA" (type assumed to be American Shorthair)) using images acquired by sensor device 10 according to a distribution request from the user, learning device 238a can generate a recognition model 610 based on the capabilities of each sensor device 10. More specifically, in this case, learning device 238a generates a recognition model 610 for recognizing images of "TAMA" from images of sensor devices 10 that have the sensing capability to acquire high-resolution images and high arithmetic capability. Furthermore, learning device 238a generates a recognition model 610 for recognizing images of American Shorthair (cat species) from images of sensor devices 10 that have the sensing capability to acquire medium-resolution images and high arithmetic capability. Additionally, learning device 238a generates a recognition model 610 for recognizing cat images from images of sensor devices 10 that have the sensing capability to acquire low-resolution images and low arithmetic capability. Therefore, in this embodiment, even when multiple sensor devices 10 with various specifications are included, each sensor device 10 can perform recognition because a recognition model 610 can be generated according to the capabilities of each sensor device 10.

[0181] Furthermore, in this embodiment, as described above, the recognition model 610 may include information about the supervision data 602 used in machine learning. Here, the information about the supervision data 602 may be the type of supervision data 602 (e.g., image and sound) or the quality of the supervision data (e.g., distortion compensation level, pixel defects, white balance, image size, saturation, brightness, gamma, contrast, edge enhancement level, focus, exposure level, resolution, dynamic range, and noise reduction level). Such information about the supervision data 602 can be used when the preprocessing unit 138 of the sensor device 10 performs preprocessing to make the acquired sensing data approximate the form of the recognition model (specifically, the supervision data 602). By doing so, in this embodiment, the accuracy of the recognition by the recognition unit 142 of the sensor device 10 can be improved. Details will be described later.

[0182] Furthermore, in this embodiment, as described above, the recognition model 610 may include setting information related to the setting of the sensor unit 100 of the sensor device 10 to obtain sensing data required to generate the distribution data specified by the distribution request. Here, the setting information may be the type of sensing data (e.g., image and sound) or the setting value of the sensor unit 100 based on the desired quality of the sensing data (e.g., distortion compensation level, white balance, image size, saturation, brightness, gamma, contrast, edge enhancement level, focus, exposure level, resolution, dynamic range, and noise reduction level). In this embodiment, the setting information is used when setting the sensor unit 100, and sensing data can be obtained according to the recognition model 610, ultimately improving the recognition accuracy of the recognition unit 142.

[0183] Note that in this embodiment, the learning device 238a can be located in a server independent of the service server 20, and is not particularly limited thereto. Furthermore, in this embodiment, the learning method in the learning device 238a is not limited to the method described above, and other methods can be used.

[0184] (Recognition using a recognition model)

[0185] Next, the recognition using the recognition model 610 described above will be described. As described above, the recognition model 610 is used when the recognition unit 142 of the sensor device 10 identifies whether the sensing data or preprocessed sensing data corresponds to a distribution request. Here, the recognition according to this embodiment will be described by way of specific examples. For example, the case where a request is made to search for a predetermined person (object) using an image (sensing data) acquired by the sensor device 10 based on a distribution request from a user (the distribution data is the location information of the predetermined person) will be described. The sensor device 10 acquires an image from the sensor unit 100. Then, the recognition unit 142 refers to the recognition model 610 acquired from the service server 20, specifically the feature points and feature quantities of the image of the predetermined person (object) 600, and identifies whether the image of the predetermined person is included in the image acquired from the sensor unit 100. In other words, in this embodiment, since the recognition described above is performed by the sensor device 10, the recognition can be performed immediately after the image is acquired.

[0186] In this embodiment, as described above, since the multiple sensor devices 10 have different sensing capabilities and arithmetic capabilities, i.e., different recognition capabilities, a recognition model 610 for performing recognition is generated based on the capability information of each sensor device 10. For example, in this embodiment, when a user requests to search for a predetermined person (object) using sensing data acquired by the sensor devices 10 (here, the type of sensing data is not particularly limited), the recognition unit 142 of the sensor device 10 capable of acquiring images recognizes the image of the predetermined person from the image based on the recognition model 610, and the recognition unit 142 of the sensor device 10 capable of acquiring ambient sound recognizes the speech of the predetermined person from the ambient sound based on the recognition model 610.

[0187] Furthermore, in another example, in this embodiment, when a user requests to search for a predetermined cat (here, a cat named "TAMA" (type considered American Shorthair)) using images acquired by sensor device 10, recognition can be performed as follows: The recognition unit 142 of sensor device 10, which has the sensing capability to acquire high-resolution images and high arithmetic capability, recognizes an image of "TAMA" from the image based on recognition model 610. Furthermore, the recognition unit 142 of sensor device 10, which has the sensing capability to acquire medium-resolution images and high arithmetic capability, recognizes an image of an American Shorthair (cat species) from the image based on recognition model 610. Furthermore, the recognition unit 142 of sensor device 10, which has the sensing capability to acquire low-resolution images and low arithmetic capability, recognizes a cat image from the image based on recognition model 610. As described above, in this embodiment, even when multiple sensor devices 10 with various specifications are included, each sensor device 10 can perform recognition.

[0188] Note that in this embodiment, the identification unit 142 can be located in a device separate from the sensor device 10, and is not particularly limited thereto. Furthermore, in this embodiment, the identification method in the identification unit 142 is not limited to the methods described above, and other methods can be used.

[0189] 1.6 Information Processing Methods

[0190] Next, we will refer to Figure 6 An information processing method according to embodiments of the present disclosure is described. Figure 6 This is a sequence diagram illustrating an example of an information processing method according to this embodiment. For example... Figure 6 As shown, the information processing method according to this embodiment may mainly include multiple steps from step S101 to step S111. Details of these steps of the information processing method according to this embodiment will be described below.

[0191] First, the user device 30 receives information input from the user and sends the received information as a distribution request to the service server 20 (step S101).

[0192] Next, the service server 20 receives a distribution request from the user device 30 (step S102). Then, the service server 20 generates an identification model based on the distribution request received in step S102 above, and sends the generated identification model to each sensor device 10 (step S103).

[0193] Next, sensor device 10 receives an identification model from service server 20 (step S104). Additionally, sensor device 10 performs sensing and acquires sensing data (step S105). Furthermore, based on the identification model received in step S104, sensor device 10 identifies whether the sensing data acquired in step S105 corresponds to a distribution request (step S106). Then, based on the identification in step S106 that the sensing data corresponds to a distribution request, sensor device 10 performs processing on the sensing data corresponding to the distribution request and generates distribution data. Furthermore, sensor device 10 directly sends the generated distribution data to user device 30 associated with the distribution request (step S107).

[0194] Next, the user device 30 receives the distribution data sent from the sensor device 10 (step S108). In this embodiment, since the distribution data can be sent directly from the sensor device 10 to the user device 30, the processing load of the service server 20 can be reduced, and the increase in the operating cost of the service server 20 can be avoided.

[0195] Furthermore, the sensor device 10 generates aggregate information based on the distribution of the distributed data in step S107 above, and sends the generated aggregate information to the aggregate server 50 (step S109).

[0196] Next, the totaling server 50 receives totaling information from the sensor device 10 (step S110). Furthermore, the totaling server 50 performs cost processing based on the totaling information received in step S110 (step S111). Then, the information processing according to this embodiment ends.

[0197] As described above, according to this embodiment, it is possible to construct various frameworks that allow users to easily use information obtained from sensing data acquired by various sensor devices 10.

[0198] 1.7 System architecture capable of downloading AI models

[0199] Next, the system architecture of the information processing system 1 according to this embodiment will be described in detail with reference to the accompanying drawings. This system is capable of downloading AI models to the device according to the NICE specification. Figure 7 This is a block diagram illustrating a schematic example of a system architecture capable of downloading AI models to an information processing system according to this embodiment. Note that in this specification, the case where the AI ​​model is downloaded to sensor device 10 is illustrated, but the invention is not limited to this and various modifications can be made, such as downloading to user device 30. Furthermore, compliance with the NICE specification is not necessary, and this embodiment can also be applied to cases where AI models, etc., are downloaded to the device via an interface that does not conform to the NICE specification.

[0200] exist Figure 7 In the example shown, two sensor devices 10A and 10B are exemplified as sensor device 10. In terms of system architecture, sensor device 10A is illustrated as having a processing unit 2200A as a processing unit and a sensor unit 2300A as a detection unit, and sensor device 10B is illustrated as having a processing unit 2200B as a processing unit and a sensor unit 2300B as a detection unit. Note that in the following description, when the processing units 2200A and 2200B are indistinguishable from each other, their reference numerals are "2200". Similarly, when the sensor units 2300A and 2300B are indistinguishable from each other, their reference numerals are "2300".

[0201] like Figure 7As shown, the architecture according to this embodiment includes an application 2001, a service unit 2100, a processing unit 2200, and a sensor unit 2300, and provides services to the sensor device 10 for using the application 2001. In this example, the sensor device 10 has, for example, a laminated chip structure in which a chip configuring the processing unit 2200 and a chip configuring the sensor unit 2300 are laminated, but the structure is not limited to this.

[0202] (Sensor Unit 2300)

[0203] The sensor unit 2300 includes, for example, a sensor 2301, a signal processing unit 2302, an identification unit 2303, and an inference unit 2304.

[0204] Sensor 2301

[0205] Sensor 2301 may be, for example, a sensor that generates image data, such as an image sensor that generates a color or monochrome image of the subject, an event-based vision sensor (EVS) that generates an image based on changes in the brightness of the subject, or a distance measurement sensor (also known as a depth sensor) that generates an image indicating the distance to the subject. However, the invention is not limited thereto, and various sensors may be applied to sensor 2301, including sound collection devices such as microphones that collect sounds from the surrounding environment, temperature sensors and humidity sensors that measure the temperature and humidity of the surrounding environment, and water level sensors that measure the water level of rivers, etc.

[0206] Signal processing unit 2302

[0207] The signal processing unit 2302 performs predetermined processing on the sensing data acquired by the sensor 2301 according to the configuration from the processing unit 2200. For example, when the sensor 2301 is an image sensor that generates image data, the signal processing unit 2302 performs predetermined processing such as white balance adjustment and distortion correction on the image data generated by the sensor 2301.

[0208] Identification unit 2303

[0209] The recognition unit 2303 performs predetermined recognition processing on the sensed data after signal processing. For example, when the sensor 2301 is an image sensor, the recognition unit 2303 performs predetermined recognition processing on the image data output from the signal processing unit 2302, such as motion detection and face region recognition.

[0210] Reasoning Unit 2304

[0211] The inference unit 2304 is, for example, an inference unit that uses an AI model to perform inference. The inference unit 2304 may include a programmable signal processing unit such as a digital signal processor (DSP) or a field-programmable gate array (FPGA), and uses the AI ​​model and preset weight parameters (weights) input from or from the processing unit 2200 to perform inference based on the sensed input output from the sensor 2301 and the recognition result output from the recognition unit 2303, and outputs the obtained inference result to the processing unit 2200. It should be noted that... Figure 7 The example shown is a convolutional neural network (CNN) for the AI ​​model included in the inference unit 2304, but the neural network is not limited to this and can be an AI model using various neural networks, such as deep neural networks (DNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or autoencoders.

[0212] (Processing Unit 2200)

[0213] The processing unit 2200 includes, for example, a central processing unit (CPU) 2201, a signal processing unit 2202, and an inference unit 2203.

[0214] CPU 2201

[0215] CPU 2201 is a component of control processing unit 2200 and sensor unit 2300 and is connected via communication unit 170. Figure 2 The CPU 2201 is an information processing unit that exchanges instructions and information with the service unit 2100. Furthermore, the CPU 2201 generates a SceneMark based on data (sensing data and inference results) input from the sensor unit 2300, according to the SceneMode set from the service unit 2100 (described later), and sends the SceneMark to the service unit 2100. Note that in this specification, SceneMode can be a general setting for the device, SceneData can be raw data such as images and sounds, and SceneMark can be an analysis result (e.g., metadata) associated with images, sounds, etc., through AI or the like.

[0216] Signal processing unit 2202

[0217] The signal processing unit 2202 performs predetermined processing (such as automatic exposure control, automatic focus, automatic white balance adjustment and format conversion) on the data (sensing data and inference results) input from the sensor unit 2300, and sends the results as part of the SceneData to the service unit 2100.

[0218] Inference Unit 2203

[0219] Inference unit 2203 is, for example, an inference unit that uses an AI model to perform inference and is configured with a programmable signal processing unit such as a DSP or a field-programmable gate array (FPGA). Inference unit 2203 uses an AI model and... (The rest of the text is missing). Figure 18 The resource server (AI asset service) 3200 acquires or presets weight parameters (weights), uses data (sensing data and inference results) output from the sensor unit 2300 as input to perform inference, and sends the data obtained through inference as part of SceneData to the service unit 2100. Note that in Figure 7 The example illustrates the case where the neural network of the AI ​​model included in the inference unit 2203 is a CNN, but the neural network is not limited to this and can be an AI model using various neural networks such as DNN, RNN, GAN or autoencoder.

[0220] (Service Unit 2100)

[0221] Service unit 2100 is a provider that provides services for using application 2001 to processing unit 2200 via a predetermined network, and corresponds to, for example... Figure 1 Server 20 in the middle.

[0222] Service unit 2100 includes, for example, an AI network unit 2101, an analysis unit 2102, and data pipeline units 2103A and 2103B. In the following description, when the individual data pipeline units 2103A and 2103B are not distinguished, their reference numerals are '2103'.

[0223] Data pipeline unit 2103

[0224] Data pipeline unit 2103 may be, for example, a set of tasks that sequentially perform predetermined processing (such as format conversion and category classification) on data (SceneMark and SceneData) input from processing unit 2200. The pre-processed data is then input to analysis unit 2102.

[0225] Analysis Unit 2102

[0226] The analysis unit 2102 may be a processing unit that performs analysis or simple analysis on the data (SceneMark and SceneData) processed by the data pipeline unit 2103. For example, when jointly analyzing AI-processed data acquired from multiple sensor devices 10, the analysis unit 2102 performs analysis or simple analysis on the collected AI-processed data. As a specific example, for instance, when analyzing the direction of vehicle movement, etc., as imaged by cameras (sensor units 100) mounted on each sensor device 10 and generating its statistics, the statistical results (AI-processed data) are transmitted from each sensor device 10 to the service unit 2100. On the other hand, the analysis unit 2102 of the service unit 2100 can generate comprehensive analysis results, such as heat maps, by analyzing the statistical results from each sensor device 10 based on the location information of each sensor device 10.

[0227] AI Network Unit 2101

[0228] AI network unit 2101 includes one or more AI models and is a processing unit that collects and organizes information required by a specific user from the network and provides curation services that can be shared with other users. AI network unit 2101 processes data and / or analysis results input from analysis unit 2102 by, for example, combining one or more AI models according to a curation configuration specified from application 2001.

[0229] 1.8 Node Architecture in the Device

[0230] Next, the node architecture of the device capable of replacing the AI ​​model conforming to the NICE specification will be described in detail with reference to the accompanying drawings. Figure 8 This is a block diagram illustrating a schematic example of the node architecture of the device according to this embodiment. It should be noted that in the following description, [the following text includes...]. Figure 7 The sensor device 10 of the processing unit 2200 and sensor unit 2300 is shown as a device capable of replacing the AI ​​model, and its reference numeral is '3000'. Furthermore, similar to reference numerals... Figure 7 The system architecture described does not necessarily need to conform to the NICE specification, and this implementation can also be applied to situations where AI models, etc., are downloaded to the device via an interface that does not conform to the NICE specification.

[0231] like Figure 8 As shown, the apparatus (also called an information processing apparatus) 3000 according to this embodiment includes a node 3010 configured to implement the inference unit 2203 or 2304 using an AI model. For example, when the apparatus 3000 is designed based on an architecture such as a multiprocessor computer system of the shared memory type, the node 3010 may be a pair of processors and memory.

[0232] Node 3010 can implement processing 3011, which includes an inference unit and an AI framework 3013. The inference unit includes model data of the AI ​​model and its weight parameters (hereinafter also collectively referred to as reference data) 3012. The AI ​​framework 3013 is used to perform inference using the reference data 3012, such as TensorFlow (registered trademark) or Keras. Therefore, this embodiment provides a configuration that allows the reference data 3012 in processing 3011 to be replaced. As a result, the functionality of device 3000 can be replaced in a timely manner depending on the usage.

[0233] 1.9 Overall Operational Overview

[0234] Next, an overview of the overall operation of the information processing system according to this embodiment will be described in detail with reference to the accompanying drawings.

[0235] Figure 9 This is a schematic diagram illustrating an overview of the overall operation of the information processing system according to this embodiment. (For example...) Figure 9 As shown, the information processing system 1 according to this embodiment includes nodes 3010A, 3010B, 3010C, etc., implemented in one or more devices 3000, application / service 3100 and resource server 3200.

[0236] (Step S1)

[0237] In step S1, for example, a user of application 2001 accesses application / service 3100 from his / her own device 3000 (e.g., device 3000 with node 3010A configured) and requests the use of application 2001. Here, as an example, it is assumed that the use of application 2001 is requested for the purpose of cat detection.

[0238] Application / Service 3100 maintains an application list 2002 that lists applications available to users. In the application list 2002, an algorithm ID is registered to uniquely identify the available application and the capabilities (service content) of each application.

[0239] (Step S2)

[0240] In step S2, the application / service 3100 that receives the request from the user using application 2001 specifies the algorithm ID (AppID) of the application that provides services to the user according to the requested content, and queries the device 3000 or node 3010A of the user who has requested to use the application (GetCapabilities).

[0241] (Step S3)

[0242] In step S3, in response to the query in step S2, node 3010A or device 3000 obtains information about its own processing capabilities. For example, it obtains device information such as the model of device 3000, CPU specifications and memory capacity, the version and / or parameter set of preset or downloaded AI models, information about the framework maintained by device 3000, and processing capabilities such as communication speed and bandwidth, and notifies application / service 3100 of this information.

[0243] (Step S4)

[0244] In step S4, the application / service 3100 notifies the resource server (AI asset service) 3200 of the accumulated reference data of the algorithm ID (AppID) specified in step S2 and the processing capability information (Capabilities) notified in step S3. The reference data is a collection of downloadable model data and / or weight parameters.

[0245] Resource server 3200 maintains a model list 3220 that lists AI models that can be provided to device 3000. In model list 3220, the available AI models are registered for each algorithm ID and processing capabilities, along with information related to usage limitations. In other words, the learned model data and its weight parameters are registered in resource server 3200 and managed by model list 3220.

[0246] (Step S5)

[0247] In step S5, resource server 3200 specifies the AI ​​models, usage restrictions, etc. (xxx, AAA, etc.) to be provided to the user based on the notified algorithm ID (AppID) and processing capability information (Capabilities), and downloads them to application / service 3100. In other words, resource server 3200 specifies and provides the best reference data from model list 3220 according to usage.

[0248] As a method for identifying the best reference data through the resource server 3200, information such as the content to be processed by the reference data to be downloaded, the model of the executable device, the processing capacity, the usage restrictions of the AI ​​model, the usage time of the AI ​​model, the validity period and usage area of ​​the AI ​​model, and the version of the model can be managed by the model list 3220, and reference data suitable for the device 3000 to be used according to the usage can be specified from the managed information.

[0249] (Step S6)

[0250] In step S6, application / service 3100 verifies whether the user requesting to use application 2001 has usage rights based on the usage restrictions (AAA) obtained from resource server 3200.

[0251] (Step S7)

[0252] In step S7, as a result of the verification in step S6, when the user has permission to use application 2001, application / service 3100 will download the reference data obtained from resource server 3200 in step S5 to the user.

[0253] Note that in this example, application / service 3100 can be deployed on the aforementioned service server 20 ( Figure 1 It can be placed in device 3000 (e.g., Figure 1 In the sensor device 10).

[0254] In this example, reference data is downloaded from resource server 3200 to node 3010 via application / service 3100. However, the present invention is not limited to this, and reference data can be downloaded directly from resource server 3200 to node 3010.

[0255] 1.10 Basic Operation Sequence

[0256] Next, the basic sequence of operations will be described when performing inference using an AI model for each node 3010 of device 3000 from the application 2001 or service unit 2100 side according to the NICE specification. Figure 10 This is a sequence diagram illustrating an example of a basic operation sequence according to this embodiment. Note that in the following description, the application 2001 and service unit 2100 that provide instructions to node 3010 are collectively referred to as application / service 3100. For example, application / service 3100 may have corresponding Figure 1 The configuration of service server 20 in the middle.

[0257] like Figure 10 As shown, the basic operations include: a capability acquisition phase P10, in which the application / service 3100 acquires the processing capabilities of the device 3000 and / or node 3010; a mode setting phase P20, in which the SceneMode is set for node 3010; an execution phase P30, in which node 3010 performs AI processing for each SceneMode; and a termination phase P40, in which node 3010 terminates AI processing.

[0258] (Skill Acquisition Stage, Page 10)

[0259] During the capability acquisition phase P10, firstly, application / service 3100 notifies node 3010 of an instruction (GetCapabilities) for reporting the processing capabilities of device 3000 and / or node 3010 to application / service 3100. In response, node 3010 notifies application / service 3100 of its own processing capability information (Capabilities) (N12→A12).

[0260] Note that the processing capabilities information of each device 3000 can be pre-managed in application / service 3100 through the pre-execution capability acquisition phase P10.

[0261] (Mode setup phase P20)

[0262] During the mode setup phase P20, the application / service 3100 notifies node 3010 of the instruction (SetSceneMode) regarding which SceneMode will be used (A21→N21).

[0263] (Execution phase, page 30)

[0264] In execution phase P30, firstly, application / service 3100 notifies node 3010 of an instruction (StartScene) to begin inference using the AI ​​model specified by SetSceneMode (A31→N31). In response, on node 3010 side, the setting of reference data specified by SceneMode in mode setting phase P20 is performed (N32→N33). Then, on node 3010 side, based on the data acquired by sensor unit 2300, SceneMark and SceneData are generated using the reference data specified by SceneMode, and the SceneMark and SceneData are sent to application / service 3100 (N34→A34). Note that the destination of the generated SceneMark and SceneData is not limited to application / service 3100, but can be another device 3000, etc.

[0265] (End phase, page 40)

[0266] In the termination phase P40, application / service 3100 notifies node 3010 of an instruction (StopScene) to terminate inference using the AI ​​model (from A41 to N41). In response, on the node 3010 side, inference using the AI ​​model specified by SceneMode is terminated.

[0267] 1.11 Example of a SceneMode object

[0268] The following will describe an example of a SceneMode object, which is based on the NICE specification. Figure 10 In the mode setting phase P20, the application / service 3100 sends the SceneMode to the application programming interface (API) object of node 3010.

[0269] Figure 11 This is a diagram illustrating an example of a SceneMode object according to this embodiment. For example... Figure 11 As shown, the SceneMode object 41 may include a LabelRefDataList 42. In addition, the LabelRefDataList 42 may include one or more objects 43, which include LabelName 441, RefDataList 442, RefData 443, processing stage 444, and additional attributes 445.

[0270] (LabelName 441)

[0271] LabelName 441 stores a string representing the label name of each object 43.

[0272] (RefDataList 442)

[0273] RefDataList 442 stores information used to download AI models and / or weight parameters from the network. Figure 12 This is a diagram illustrating an example of a RefDataList according to this embodiment. For example... Figure 12 As shown, RefDataList 442 may include object 452, which includes RefDataID 4621, RefDataEndpoint 4622, encryption 4623, and additional features 4624.

[0274] RefDataID 4621 stores an identifier used to uniquely identify the reference data to be downloaded.

[0275] RefDataEndPoint 4622 stores the address information (e.g., a Uniform Resource Locator (URL)) of reference data that is laid out on the network in a downloadable manner. Therefore, when an AI model is added, updated, or switched regarding node 3010, the reference data is downloaded from the endpoint specified by RefDataEndPoint 4622 and incorporated into node 3010.

[0276] Encryption 4623 stores reference information used to identify privacy objects used to encrypt reference data. Therefore, when downloaded reference data is encrypted, the encrypted reference data is decrypted using the privacy object specified by Encryption 4623.

[0277] Additional feature 4624 stores additional information. This additional feature 4624 can be used for various purposes.

[0278] (RefData 443)

[0279] RefData 443 stores information used to specify reference data preset in device 3000 or node 3010. Figure 13 This is a diagram illustrating an example of RefData according to this embodiment. For example... Figure 13 As shown, RefData 443 may include an object 453 containing RefDataID 4631, RefData 4632, encryption 4633, and additional features 4634.

[0280] RefDataID 4631 stores an identifier used to uniquely identify reference data preset in device 3000 or node 3010.

[0281] RefData 4632 stores information specifying preset reference data in device 3000 or node 3010. Therefore, when the SceneMode specified by SetSceneMode notified from application / service 3100 is the SceneMode specified by the identifier of RefDataID 4631, the preset reference data is incorporated into node 3010.

[0282] Encryption 4633 stores reference information used to identify the privacy object used to encrypt reference data. Therefore, when the preset reference data is encrypted, the reference data is decrypted using the privacy object specified by Encryption 4633.

[0283] Additional feature 4634 stores additional information. This additional feature 4634 can be used for a variety of purposes.

[0284] (Processing Phase 444)

[0285] Processing stage 444 stores information used to specify in which processing step the reference data is used. Figure 14 This is a diagram illustrating an example of a processing stage according to this embodiment. For example... Figure 14As shown, examples of processing steps defined in processing stage 444 include "CustomAnalysis", "Motion", "Detect", "Recognize", and "Characterize". However, the invention is not limited to this, and various processing steps can be defined.

[0286] (Additional Feature 445)

[0287] Additional feature 445 stores additional information. This additional feature 445 can be used for various purposes.

[0288] 1.12 Example of the StartScene object

[0289] Next, we will describe an example of a StartScene object, which is based on the NICE specification. Figure 10 The API object is sent in the execution phase P30 to enable node 3010 to execute SceneMode.

[0290] Figure 15 This is a diagram illustrating an example of a StartScene object according to this embodiment. Figure 15 As shown, the StartScene object 500 may include version 501, SceneModeID 502, RefDataID / RefDataListID 503, and additional attributes 504.

[0291] (Version 501)

[0292] Version 501 stores the version information of the SceneMode specified by SceneModeID, which will be described later.

[0293] (SceneModeID 502)

[0294] SceneModeID 502 stores an identifier used to uniquely identify the SceneMode.

[0295] (RefDataID / RefDataListID 503)

[0296] RefDataID / RefDataListID 503 stores an identifier used to identify the reference data to be executed. This RefDataID / RefDataListID 503 is a new field added to the existing NICE specification. Note that RefDataID / RefDataListID 503 can be omitted, for example, when including a single piece of reference data in node 3010.

[0297] (Additional Feature 504)

[0298] Additional feature 504 stores additional information. This additional feature 504 can be used for various purposes.

[0299] 1.13 Example of model data

[0300] Figure 16 This is a diagram illustrating an example of model data in the reference data according to this embodiment. For example... Figure 16 As shown, model data 70 includes a header 71 and a body 72 storing the model data. The header 71 includes version information of the model data, name of the model data, an identifier (AlgorithmID) for uniquely identifying the algorithm used in the model data, an identifier (PlatformID) for identifying the AI ​​platform that can incorporate the model data, access control information indicating the licensing and usage rules of the model data, contract expiry date indicating when the model data will be unavailable, information indicating the region where the model data will be available, and the capability of the model data.

[0301] 1.14 Example of an operation sequence when using a preset AI model

[0302] Next, the sequence of operations for performing inference using a preset AI model on each node 3010 of device 3000 from the application / service 3100 side according to the NICE specification will be described. Figure 17 This is a sequence diagram illustrating an example of the operation sequence when using a preset AI model according to this embodiment. Figure 17 In the following description, for the sake of simplicity, the capability acquisition phase P10 is omitted. Furthermore, the following description illustrates the case of switching and using two reference data points, but the invention is not limited to this; the same operational sequence can also be applied when using one reference data point or when switching and using three or more reference data points.

[0303] (Mode setup phase P20)

[0304] like Figure 17 As shown, in this operation, the instruction (SetSceneMode) regarding which SceneMode to use is communicated from application / service 3100 to node 3010 (A101→N101). For example, in this example using the preset AI model, SceneMode 41 with SceneModeID=1 is specified. Figure 11 The SetSceneMode method is notified from application / service 3100 to node 3010.

[0305] (Phase 1 of Implementation)

[0306] Next, application / service 3100 uses the preset reference data with RefDataID=1 in the SceneMode specified by SetSceneMode, which has SceneModeID=1. Figure 13 The instruction (StartScene) used to begin inference is sent to node 3010 (A102→N102).

[0307] In response, node 3010 performs the setting of the preset reference data with RefDataID=1 in the SceneMode specified by SetSceneMode, which has SceneModeID=1 (N103→N104).

[0308] Then, node 3010 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and transmits SceneMark and SceneData to application / service 3100 (N105→A105).

[0309] (First End Stage, P40)

[0310] Next, the instruction (StopScene) used to terminate the currently executing SceneMode is notified from application / service 3100 to node 3010 (A106→N106). In response, node 3010 terminates the currently executing SceneMode.

[0311] (Second Implementation Phase, P30)

[0312] Next, application / service 3100 uses the preset reference data with RefDataID=2 in the SceneMode specified by SetSceneMode, which has SceneModeID=1. Figure 13 The instruction (StartScene) used to begin inference is sent to node 3010 (A107→N107).

[0313] In response, node 3010 performs the setting of the preset reference data with RefDataID=2 in the SceneMode specified by SetSceneMode, which has SceneModeID=1 (N108→N109).

[0314] Then, node 3010 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and sends SceneMark and SceneData to application / service 3100 (N110→A110).

[0315] (Second End Stage, P40)

[0316] Next, application / service 3100 notifies node 3010 of the instruction (StopScene) to terminate the currently executing SceneMode (A111→N111). In response, node 3010 terminates the currently executing SceneMode.

[0317] 1.15 Example of an operation sequence when downloading and using an AI model

[0318] Next, the sequence of operations will be described on the application / service 3100 side when each node 3010 of device 3000 downloads the AI ​​model and performs inference according to the NICE specification. Figure 18 This is a sequence diagram illustrating an example of the operation sequence when downloading an AI model according to this embodiment. Figure 18 In the following description, for the sake of simplicity, the capability acquisition stage P10 is omitted. Furthermore, the following description illustrates the case of switching between two reference data points, but the invention is not limited to this; the same operational sequence can also be applied when using one reference data point, or when switching between three or more reference data points. And, in the following description, regarding... Figure 17 Operations with the same sequence of operations are labeled with the same reference numerals.

[0319] (Mode setup phase P20)

[0320] like Figure 18 As shown, in this operation, the instruction (SetSceneMode) regarding which SceneMode to use is communicated from application / service 3100 to node 3010 (A101→N101). For example, similar to... Figure 17 In this example of downloading and using the AI ​​model, SceneMode 41 (with SceneModeID=1) will be used to specify the scene. Figure 11 The SetSceneMode method is notified from application / service 3100 to node 3010.

[0321] (Phase 1 of Implementation)

[0322] Next, application / service 3100 notifies node 3010 that RefDataListID=1 will be used in the SceneMode specified by SetSceneMode with SceneModeID=1. Figure 12 The instruction to start inferring from the reference data (StartScene)(A102→N102).

[0323] In response, node 3010, based on the SceneMode with SceneModeID=1 specified by SetSceneMode, and the RefDataList with RefDataListID=1, 442( Figure 12 The RefDataEndpoint4622 stored in the resource server (AI Asset Service) 3200 accesses endpoint 3201 (N131→E131) and downloads reference data (E132→N132) stored in that endpoint 3201 with RefDataListID=1. It should be noted that, for example, this can be achieved by... Figure 2 The model acquisition unit 140 downloads reference data (which also applies later).

[0324] It should be noted that, for example, when node 3010 maintains the reference data RefDataListID=1 in the SceneMode with SceneModeID=1 through past downloads, etc. Figure 12 When downloading reference data from endpoint 3201 (N131→E131, E132→N132), the step of downloading reference data can be omitted.

[0325] Then, node 3010 performs the creation of reference data downloaded from resource server 3200 (N133→N134).

[0326] Subsequently, with Figure 17 Similarly, node 3010 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and transmits SceneMark and SceneData to application / service 3100 (N105→A105).

[0327] (First End Stage, P40)

[0328] Next, similar to Figure 17 The application / service 3100 notifies node 3010 (from A106 to N106) of the instruction (StopScene) to terminate the currently executing SceneMode, and in response, node 3010 terminates the currently executing SceneMode.

[0329] (Second Implementation Phase, P30)

[0330] Next, application / service 3100 notifies node 3010 that RefDataListID=2 will be used in the SceneMode specified by SetSceneMode with SceneModeID=1. Figure 12 The instruction to start inferring from the reference data (StartScene) (A107→N107).

[0331] In response, node 3010, based on RefDataList442 with RefDataListID=2, is configured in the SceneMode specified by SetSceneMode and has SceneModeID=1. Figure 12 The RefDataEndpoint 4622 stored in the resource server (AI Asset Service) 3200 accesses endpoint 3202 (N141→E141) and downloads reference data with RefDataListID=1 stored in that endpoint 3202 (E142→N142).

[0332] It should be noted that, similar to the above, for example, when node 3010 maintains the reference data RefDataListID=1 in a SceneMode with SceneModeID=2 through a previously performed download ( Figure 12 When downloading reference data from endpoint 3202 (N131→E131, E132→N132), the step of downloading reference data can be omitted.

[0333] Then, node 3010 performs the creation of reference data downloaded from resource server 3200 (N143→N144).

[0334] Subsequently, with Figure 17 Similarly, node 3010 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and transmits SceneMark and SceneData to application / service 3100 (N110→A110).

[0335] (First End Stage, P40)

[0336] Next, similar to Figure 17 The application / service 3100 notifies node 3010 (from A111 to N111) of the instruction (StopScene) to terminate the currently executing SceneMode, and in response, node 3010 terminates the currently executing SceneMode.

[0337] 1.16 Variation Example

[0338] In the above embodiments, a case has been described where a device 3000 includes a node 3010. However, as... Figure 19 As shown, each device 3000 may include multiple nodes 3010-1, 3010-2 to 3010-N (N is three or more integers). In this case, SceneMode can be set for each node 3010.

[0339] Figure 20 This is a sequence diagram illustrating an example of the operation sequence when downloading an AI model according to a variation of this embodiment, and an example of the operation sequence when setting different SceneModes for each node. Figure 20 In the following description, for the sake of simplicity, the ability acquisition phase P10 is omitted. In the following description, with... Figure 17 or Figure 18 Operations similar to those in the operation sequence shown are given the same reference numerals.

[0340] (Mode setup phase P20)

[0341] like Figure 20 As shown, in this example, a device 3000 includes two nodes 3010-1 and 3010-2. Therefore, in operation according to this variant, an instruction (SetSceneMode) regarding which SceneMode to use is notified from the application / service 3100 to node 3010-1 (A101a→N101a), and an instruction (SetSceneMode) regarding which SceneMode to use is notified from the application / service 3100 to node 3010-2 (A101b→N101b).

[0342] (Phase 30, First Implementation)

[0343] Next, application / service 3100 notifies node 3010-1 to use RefDataListID=1 in the SceneMode specified by SetSceneMode with SceneModeID=1. Figure 12 The instruction (StartScene)(A102a→N102a) is used to start the inference based on the reference data.

[0344] In response, node 3010-1, based on RefDataList442 with RefDataListID=1, is configured in the SceneMode specified by SetSceneMode. Figure 12The RefDataEndpoint 4622 stored in the resource server (AI asset service) 3200 accesses endpoint 3201 (N131a→E131a) in the resource server (AI asset service) 3200 and downloads the reference data (E132a→N132a) with RefDataListID=1 stored in that endpoint 3201.

[0345] It should be noted that, for example, when node 3010-1 retains reference data RefDataListID=1 in a SceneMode with SceneModeID=1 through past downloads, etc. Figure 12 When downloading reference data from endpoint 3201 (N131a→E131a, E132a→N132a), the step of downloading reference data can be omitted.

[0346] Then, node 3010-1 performs the creation of reference data downloaded from resource server 3200 (N133a→N134a).

[0347] Subsequently, with Figure 17 Similarly, node 3010-1 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and transmits SceneMark and SceneData to application / service 3100 (N105a→A105a).

[0348] (First End Stage, P40)

[0349] Next, similar to Figure 17 Application / service 3100 notifies node 3010-1 of the instruction (StopScene) to terminate the currently executing SceneMode, and in response, node 3010-1 terminates the currently executing SceneMode.

[0350] (Second Implementation Phase, P30)

[0351] In addition, a notification from application / service 3100 to node 3010-2 will use RefDataListID=2 in the SceneMode specified by SetSceneMode with SceneModeID=1. Figure 12 The instruction to start inferring from the reference data (StartScene)(A102b→N102b).

[0352] In response, node 3010-2, in the SceneMode specified by SetSceneMode with SceneModeID=1, is based on RefDataList442 with RefDataListID=2. Figure 12 The RefDataEndpoint 4622 stored in the resource server (AI Asset Service) 3200 accesses endpoint 3202 (N141b→E141b) and downloads reference data (E142b→N142b) with RefDataListID=1 stored in that endpoint 3202.

[0353] It should be noted that, similar to the above, for example, when node 3010-2 maintains RefDataListID = 1 in SceneMode through a previously executed download ( Figure 12 When downloading reference data from endpoint 3202 (N141b→E141b, E142b→N142b), the process can be omitted.

[0354] Then, node 3010-2 performs the creation of reference data downloaded from resource server 3200 (N143b→N144b).

[0355] Subsequently, with Figure 17 Similarly, node 3010-2 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and transmits SceneMark and SceneData to application / service 3100 (N110b→A110b).

[0356] (First End Stage, P40)

[0357] Next, similar to Figure 17 Application / service 3100 notifies node 3010-2 of the instruction (StopScene) to terminate the currently executing SceneMode, and in response, node 3010-2 terminates the currently executing SceneMode.

[0358] 1.17 Other variations

[0359] In the above embodiments, an example has been given of the application / service 3100 notifying node 3010 of the StartScene command for SceneMode each time an AI model is switched and a reference model is downloaded. However, the present invention is not limited thereto. The application / service 3100 may pre-notify node 3010 or device 3000 of reference data based on the identified usage situation, and node 3010 or device 3000 may download the pre-notified reference data, enabling rapid switching of AI models based on changes in usage situation.

[0360] Furthermore, in the above embodiments, the case where application / service 3100 determines whether to switch the AI ​​model to node 3010 and instructs node 3010 to download reference data when switching the AI ​​model has been exemplified, but the present invention is not limited thereto. For example, the instruction to download reference data can be notified by determining whether another device 3000 or node 3010 on the network has switched the AI ​​model. Alternatively, device 3000 or node 3010 itself can determine the switching of the AI ​​model and actively perform the download of reference data or the switching of the preset AI model. In other words, the configuration (switching unit) of the inference unit for switching node 3010 according to usage can be any one of device 3000 or node 3010 itself, application / service 3100, another device 3000 or node 3010 on the network, or a configuration combined in any one of device 3000 or node 3010 itself, application / service 3100, or another device 3000 or node 3010 on the network.

[0361] The determination to switch AI models can be based on, for example, location information acquired by the positioning unit 110 mounted on the sensor device 10, changes in temperature and humidity, information identified or extracted from image data, and recognition results from previous AI models. This determination can be performed in any of the following: the device 3000 that has received sensing data from the sensor unit 2300, the application / service 31000, the resource server 3200, and another device 3000.

[0362] Furthermore, to determine the output format of the data (SceneData) output from the AI ​​model, the format can be placed as a schema in a resource server 3200, and an identifier used to identify the output format can be assigned to the data output from the AI ​​model. For example, a field named Schema can be provided in the SceneMark associated with the SceneData, and the identifier used to identify the output format can be stored in the Schema field. In this case, the application / service 3100 or another device 3000 that has received the SceneMark and SceneData can specify the format of the SceneData based on the identifier stored in the Schema field of the SceneMark.

[0363] Furthermore, the data (SceneData) output from two or more, or all, AI models can have a common output format. Alternatively, a format conversion unit that uses the data (SceneData) output from the AI ​​models as a common output format can be installed on device 3000, application / service 3100, etc.

[0364] Furthermore, the reference data managed by resource server 3200 can be relearned using SceneMark and / or SceneData obtained from each node 3010. The relearned reference data can be managed in a downloadable manner on resource server 3200, for example, as reference data whose version has been updated.

[0365] Note that the above description has illustrated the scenario where resource server 3200 is the source for downloading model data and weight parameters, but the invention is not limited thereto. For example, another device 3000 on the network can be used. Furthermore, the functionality of the application / service 3100 described above can be configured to be performed by another device 3000 on the network.

[0366] Furthermore, in this embodiment, the aggregate server 50 can charge a data sample fee not only for the distribution of distribution data obtained from sensing data, but also for the distribution of model data and the weight parameters (reference data) of the AI ​​model. For example, the aggregate server 50 can cooperate with the authentication server 40 described above to aggregate the amount of reference data distributed for each user or user device 30 or each sensor device 10, and determine the data usage fee based on the aggregate amount, etc.

[0367] For example, the service server 20, sensor device, and user device 30, which may be the distribution source of reference data, may include a totaling unit and a totaling information transmission unit. The totaling unit totals the distribution amount of reference data for each requesting source to generate totaling information, and the totaling information transmission unit transmits the totaling information for each requesting source to the totaling server 50, in conjunction with... Figure 2 The aggregation unit 148 in the sensor device 10 shown is similar. On the other hand, the fee processing unit 534 in the aggregation server 50 can determine the data usage fee for each requesting source based on the aggregation information of the reference data, and exchange the data usage fee between the device manager, which is the distribution source, and each requesting source.

[0368] 1.18 Summary

[0369] As described above, according to this embodiment, the device 3000 can download model data and weight parameters at any time interval and implement the model data and weight parameters in node 3010. Therefore, since the function of the device 3000 can be replaced in a timely manner according to the usage situation, the function can be flexibly changed according to various usage situations.

[0370] Furthermore, for example, since the model list 3220 manages the available AI models for each algorithm ID and processing capability information for device 3000, optimal reference data can be provided based on usage or device 3000.

[0371] Furthermore, for example, by managing information about the limitations of AI model usage through model list 3220, various services can be affected, such as the replacement of AI models that comply with the laws and regulations of the countries using device 3000, and the provision of AI models during limited-cycle events.

[0372] Furthermore, for example, by managing the intended use of AI models through an application list 2002, it is possible to know in advance what each AI model can do. Therefore, appropriate AI models can be provided in response to user requests.

[0373] Furthermore, since various AI models can be managed in resource server 3200, etc., an appropriate AI model that can output the necessary data can be selected, and the selected AI model can be implemented in device 3000.

[0374] Furthermore, since reference data can be obtained not only from resource server 3200 but also from another device 3000 on the network, the degree of freedom regarding the location of the reference model can be increased.

[0375] In addition, since the reference data is not limited to model data, but can also be weight parameters, it is possible to switch AI models with minimal data replacement.

[0376] Furthermore, since the AI ​​model can be replaced and the required AI model can be determined based on sensing data, the replacement or switching of the AI ​​model can be determined in any of the following: its own device 3000, another device 3000, application / service 3100, and resource server 3200.

[0377] Furthermore, since the AI ​​model can be replaced and the required AI model can be determined based on location information obtained from GPS and other sources, it is also possible to determine which AI model is the required or optimal AI model near the location of device 3000.

[0378] In addition, by setting the output format of the data (SceneData) output from the AI ​​model to a common output format, the configuration of the later stages of data (SceneData) processing, analysis and recognition can be common, which can reduce the processing burden of data (SceneData).

[0379] In addition, because of the version of the management reference data, the latest AI model can be provided even when the AI ​​model is updated through relearning or other means.

[0380] Furthermore, when a device 3000 includes multiple nodes 3010, the AI ​​model can be replaced or switched for each node 3010. As a result, even when performing multilateral inference based on sensing data acquired by a sensor 2301, it is possible to freely switch to an appropriate combination of AI models depending on the scene, etc.

[0381] In addition, by adding a Schema field to SceneMark and managing the output format of AI model output data (SceneData), the output format can be easily determined in the configuration of subsequent stages of processing, analyzing and identifying the data (SceneData), thereby reducing the processing burden of the data (SceneData).

[0382] In addition, by adding the Schema field to SceneMark and managing the output format of the data (SceneData) from the AI ​​model, the output format can be easily known in the configuration of subsequent stages, even when replacing or switching AI models, thereby reducing the processing burden of the data (SceneData).

[0383] Furthermore, for example, by managing information regarding usage limits for AI models through model list 3220, it is possible to determine whether an AI model is used based on the user. Therefore, it is possible to control the use of AI models based on factors such as the owner of the camera or the person receiving the data.

[0384] Furthermore, by configuring the AI ​​model as replaceable, data (SceneData) can be generated for specific users through time-division or parallel processing, so that the same device 3000 can be shared even when users using the device at the same time have different purposes.

[0385] Furthermore, because the device 3000 can determine whether an AI model needs to be downloaded, it can switch AI models more promptly based on usage.

[0386] 2. Second Implementation Method

[0387] Next, the information processing system and information processing method according to the second embodiment of this disclosure will be described in detail with reference to the accompanying drawings.

[0388] In the above embodiments or variations thereof, reference data downloaded from resource server 3200 or another device 3000 can be protected by encryption, electronic signature, etc.

[0389] Here, when encrypting reference data, 3000 SceneEncryptionKeys pre-distributed to each device can be used for encryption. Furthermore, the encryption algorithm can conform to the SceneData and SceneMark encryption algorithms according to the NICE specification. And, as the encryption format, formats such as JSON Web Encryption (JWE) compressed serialization can be used.

[0390] 2.1 Example of an encrypted object

[0391] Information about the SceneEncryptionKey used for encryption is stored in, for example, encryption 4623 in object 452 of the SceneMode object. Figure 12 ) or Encryption 4633 in object 453 ( Figure 13 It is used as an encrypted object in ).

[0392] Figure 21 This is a diagram illustrating an example of an encrypted object according to this embodiment. For example... Figure 21 As shown, encryption 4623 ( Figure 12 The encrypted object 47 in encryption 4633 may include EncryptionOn 471, SceneEncryptionKeyID 472, PrivacyServerEndPoint 473, and additional features 474.

[0393] EncryptionOn 471 stores information indicating whether reference data is encrypted. For example, when "true" is stored in EncryptionOn 471, the reference data is encrypted, and when "false" is stored, the reference data is not encrypted.

[0394] SceneEncryptionKeyID 472 stores an identifier used to identify the privacy object used to encrypt reference data.

[0395] PrivacyServerEndpoint 473 stores an endpoint (e.g., a URL) as the address information of a privacy object arranged in a downloadable manner over the network. When decrypting reference data, device 3000 downloads the privacy object from the endpoint specified by PrivacyServerEndpoint 473 and decrypts the reference data. Note that... Figure 2 The model acquisition unit 140 in the model can download privacy objects.

[0396] Additional feature 474 stores additional information. This additional feature 474 can be used for various purposes.

[0397] 2.2 Examples of Privacy Objects

[0398] Figure 22 This is a diagram illustrating an example of a privacy object according to this embodiment. For example... Figure 22 As shown, the privacy object 48 may include version 481, endpoint ID 482, PrivacyObjectID 483, StartDateTime 484, EndDateTime 485, UsageCount 486, storage rule 487, ExportRule 488, MaskedItems 489, AnalysisRule 490, encryption 491, authentication 492, and additional features 493.

[0399] The encryption 491 may include a SymmetricKey object 4911, kty (key type) 4912, alg (algorithm) 4913, k (key instance) (also known as key object) 4914, kid (key ID) 4915 and additional features 4916.

[0400] In kty 4912, information indicating the key type (key type) of the SceneEncryptionKey is stored.

[0401] In alg 4913, information indicating the encryption algorithm used for encryption is stored.

[0402] In k 4914, the key instance (key instance) (also known as the key object) of SceneEncryptionKey is stored.

[0403] In kid 4915, the corresponding SceneMode object 41 is stored. Figure 11 Encryption 4623 () Figure 12 ) or encrypted 4633 ( Figure 13 SceneEncryptionKeyID 472 in ) Figure 21 The identifier (key ID) of the key.

[0404] Additional feature 4916 stores additional information. This additional feature 4916 can be used for various purposes.

[0405] 2.3 Example of operation sequence when encrypting model data

[0406] This describes the sequence of operations when encrypting reference data. Figure 23 This is a sequence diagram illustrating an example of an operational sequence according to this embodiment. Figure 23 In the following description, for the sake of simplicity, the ability acquisition phase P10 is omitted. In the following description, with... Figure 17 or Figure 18Operations similar to those in the operation sequence shown are given the same reference numerals.

[0407] (Mode setup phase P20)

[0408] like Figure 23 As shown, in this operation, with Figure 18 Similarly, the instruction (SetSceneMode) regarding which SceneMode to use is sent from application / service 3100 to node 3010 (A101→N101).

[0409] (Phase 30, First Implementation)

[0410] Next, from application / service 3100, RefDataListID=1 will be used in the SceneMode specified by SetSceneMode, where SceneModeID=1. Figure 12 The instruction (StartScene) to begin inference based on the reference data is sent to node 3010 (A102→N102).

[0411] On the other hand, node 3010 is based on SceneMode object 41 ( Figure 11 Encryption 4623 () Figure 12 PrivacyServerEndPoint 473 in ) Figure 21 Access the PrivacyServer endpoint 3211 (N211→E211) in resource server 3200 and download the privacy object 48, which stores the SceneEncryptionKeyID and encryption 4623 in the PrivacyServer endpoint 3211. Figure 12 SceneEncryptionKeyID 472 in ) Figure 21 Match (E212→N212).

[0412] Subsequently, in order to download the encrypted reference data, RefDataEndpoint4622 is stored in RefDataList442 with RefDataListID=1 in the SceneMode specified by SetSceneMode with SceneModeID=1. Figure 12 ), node 3010 accesses endpoint 3101 in resource server 3200 (hereinafter, to distinguish it from PrivacyServer endpoint 3211, this is referred to as the RefData endpoint) (N221→E221).

[0413] The accessed node 3010 to the RefData endpoint 3201 performs a signature on the reference data requested from node 3010 using the private key of resource server 3200, encrypts the reference data using SceneEncryptionKey (E222→E223), and transmits the encrypted reference data to node 3010 (E224→N224).

[0414] Node 3010, which has already obtained the encrypted reference data, decrypts the reference data using k 4914 (i.e., SceneEncryptionKey) in privacy object 48 and performs authentication using the certificate of resource server 3200 to verify the decrypted reference data (N231→N232).

[0415] Then, node 3010 performs the establishment of the verified reference data (N133→N134).

[0416] Subsequently, with Figure 17 Similarly, node 3010 generates SceneMark and SceneData using reference data established based on data acquired by sensor unit 2300, and transmits SceneMark and SceneData to application / service 3100 (N105→A105).

[0417] (First End Stage, P40)

[0418] Next, similar to Figure 17 The application / service 3100 notifies node 3010 (from A106 to N106) of the instruction (StopScene) to terminate the currently executing SceneMode, and in response, node 3010 terminates the currently executing SceneMode.

[0419] 2.4 Summary

[0420] As described above, according to this embodiment, the downloaded reference data is signed and encrypted. Therefore, reference data can be downloaded securely, and the information processing device can be protected from information leakage, tampering, etc., enabling more reliable provision of functions to users based on usage.

[0421] Other configurations, operations, and effects may be similar to those of the above-described embodiments or variations thereof, and therefore their detailed descriptions are omitted here.

[0422] 3. Third Implementation Method

[0423] In the third embodiment, an implementation example of the information processing system 1 according to the above-described embodiments or variations thereof will be described. It should be noted that in the following description, the case where the sensor device 10 is an in-vehicle camera and the information processing system 1 is applied to a driving support system performing level 2 driving support will be used as an example.

[0424] 3.1 Implementation Example

[0425] Figure 24 This is a block diagram illustrating an implementation example of an information processing system according to this embodiment. Figure 24 In the implementation example shown, sensor device 10, service server 20 and resource server 60 are connected via network 90, and application 11 that provides services to users is implemented in sensor device 10.

[0426] (Sensor device 10)

[0427] like Figure 24 As shown, in the implementation example, in addition to application 11, sensor device 10 also includes processing unit 12, application user interface (App UI) 14, operating system (OS) 15, sensor unit 16 and output unit 17.

[0428] In this configuration, application 11 is the driving support application corresponding to application 2001 described above. Processing unit 12 corresponds to processing unit 2200 described above and performs various recognition processes required for driving assistance, such as inter-vehicle distance detection, lane departure detection, rear-end collision detection, and pedestrian detection. Sensor unit 16 corresponds to sensor unit 2300 described above and generates image data obtained by capturing the front, rear, and sides of the vehicle.

[0429] In this embodiment, the processing unit 12 includes a device state machine 12a and a data pipeline 13a. The device state machine 12a includes an authentication sequence 12b, and user authentication is performed using the authentication unit 61 in the resource server 60 according to the authentication sequence 12b. The data pipeline 13a includes a node 13b capable of executing processing 13c and a converter 13f. Node 13b corresponds to the aforementioned node 3010. Processing 13c corresponds to the aforementioned processing 3011, and is capable of executing AI processing 13d and SceneMark generation 13e.

[0430] Furthermore, the operating system 15 is the operating system installed to operate the sensor device 10. The application user interface 114 is, for example, a user interface that receives instructions from user input and provides the user with various types of information, such as front-facing camera images and navigation, via the output unit 17. The output unit 17 includes a display, a speaker, etc., and outputs information from the application user interface 14 to the user.

[0431] (Resource Server 60)

[0432] Resource server 60 corresponds to resource server 3200 described above and includes authentication unit 61, MQTT broker 62, user interface 63, API gateway 64, control unit 65, and storage unit 67. Note that storage unit 67 corresponds, for example, to RefData endpoint 3201 and / or PrivacyServer endpoint 3211 described above.

[0433] 3.2 Operation Example

[0434] Next, we will describe Figure 24 The following is an example of the operation of the information processing system 1 shown. In this embodiment, user authentication is first performed after activation. During user authentication, device state machine 12a responds to a service start request from an API conforming to the NICE specification from application 11, accesses the authentication unit 61 of resource server 60 according to authentication sequence 12b, and thereby performs user authentication using authentication unit 61. When user authentication is successful and activation is complete, node 13b sends an update request for the AI ​​model to resource server 60. Note that since it is also assumed that the AI ​​model has not been updated, node 13b can start in parallel with the startup of application 11, regardless of whether user authentication is successful. Node 13b can then generate process 13c and transducer 13f after activation and transition to an event waiting state.

[0435] An update request for the AI ​​model sent to resource server 60 is input to control unit 65 via MQTT broker 62. Upon receiving the update request for the AI ​​model, control unit 65 checks, for example, the version of the reference data implemented in node 13b to determine if an updated version of the reference data exists. If an updated version of the reference data exists, control unit 65 instructs node 13b to update the reference data accordingly. Figure 18 The operation sequence shown downloads the updated version of the reference data (SetSceneMode, StartSceneMode), and node 13b retrieves the updated version of the reference data from storage unit 67 of resource server 60, and implements the retrieved reference data in process 13c. At this time, when encrypting the reference data, the following can be executed: Figure 23 The operation sequence shown.

[0436] Then, for example, when a change in device state is detected based on sensing data input from sensor unit 16, node 13b notifies the resource server of a device state conflict and requests resource server 60 to update reference data. In response, on the resource server 60 side, a request to update the AI ​​model is input to control unit 65 via MQTT broker 62, and control unit 65 specifies an AI model suitable for changing the device state. Then, for example, according to Figure 18 The operation sequence shown involves control unit 65 instructing node 13b to download reference data (SetSceneMode, StartSceneMode) suitable for changing the device state, and node 13b retrieving the specified reference data from storage unit 67 of resource server 60, and implementing the reference data in process 13c. At this time, when encrypting the reference data, the following can be executed: Figure 23 The operation sequence shown.

[0437] Note that this embodiment has illustrated the scenario where the sensor device 10 requests the resource server 60 to update reference data, but the invention is not limited thereto. For example, the service server 20 may request the resource server 60 to update the reference data in the sensor device 10.

[0438] 4. Hardware Configuration

[0439] For example, by having such Figure 25 The computer 1000 configured as shown can implement the sensor device 10, service server 20, user device 30, authentication server 40, and aggregate server 50 according to the above embodiments and variations. Figure 25 This is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of a sensor device 10, a service server 20, a user device 30, an authentication server 40, and a summary server 50. The computer 1000 includes a CPU 1100, RAM 1200, read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected via a bus 1050.

[0440] The CPU 1100 operates based on programs stored in ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 loads programs stored in ROM 1300 or HDD 1400 into RAM 1200 and executes processing corresponding to various programs.

[0441] ROM 1300 stores boot programs, such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 is activated, and programs that depend on the hardware of computer 1000.

[0442] HDD 1400 is a computer-readable recording medium that non-transitoryly records a program executed by CPU 1100, data used by the program, etc. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, exemplified as program data 1450.

[0443] Communication interface 1500 is an interface for connecting computer 1000 to an external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from another device or sends data generated by CPU 1100 to another device via communication interface 1500.

[0444] The input / output interface 1600 has a configuration including the aforementioned I / F unit 18 and serves as an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input / output devices such as a keyboard, mouse, or microphone via the input / output interface 1600. Furthermore, the CPU 1100 transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. Additionally, the input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium. For example, the medium may be an optical recording medium such as a Digital Universal Disc (DVD) or Phase Change Rewritable Disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), magnetic tape, magnetic recording media, semiconductor memory, etc.

[0445] For example, when the computer 1000 is used as the sensor device 10, service server 20, user device 30, authentication server 40, and aggregate server 50 according to the above embodiments, the CPU 1100 of the computer 1000 implements the functions of the sensor device 10, service server 20, user device 30, authentication server 40, and aggregate server 50 by executing programs loaded on the RAM 1200. Furthermore, the HDD 1400 stores programs according to this disclosure. Note that the CPU 1100 reads program data 1450 from the HDD 1400 and executes the program data. As another example, these programs can be obtained from another device via an external network 1550.

[0446] The scope of this disclosure is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of this disclosure. Furthermore, components from different embodiments and variations can be appropriately combined.

[0447] It should be noted that the effects of each embodiment described in this specification are merely illustrative and not limited thereto, and other effects may be provided.

[0448] This technology may also have the following configurations.

[0449] (1) An information processing system, comprising:

[0450] The sensor data acquisition unit acquires the sensing data obtained by the sensor;

[0451] The processing unit includes an inference unit that performs inference using the sensed data as input; and

[0452] The switching unit switches the inference unit according to the usage situation.

[0453] (2) The information processing system according to (1) further includes:

[0454] The model acquisition unit acquires reference data, where

[0455] The switching unit switches the inference unit by replacing the reference data of the inference unit with the reference data obtained by the model acquisition unit.

[0456] (3) According to the information processing system in (2), where,

[0457] The model acquisition unit acquires the reference data via a predetermined network.

[0458] (4) The information processing system according to (3) further includes:

[0459] Model holding units are arranged on the predetermined network and hold one or more reference data, wherein

[0460] The model acquisition unit acquires the reference data from the model holding unit via the predetermined network.

[0461] (5) According to the information processing system in (3), wherein,

[0462] The model acquisition unit acquires the reference data from another information processing device arranged on the predetermined network, which is different from the information processing device that includes the processing unit.

[0463] (6) An information processing system based on any one of (2) to (5), wherein

[0464] The reference data includes at least one of the model data configuring the inference unit and the set of weight parameters in the model data.

[0465] (7) The information processing system according to any one of (2) to (6) further includes:

[0466] The aggregation unit aggregates the distribution amount of reference data for each device and generates aggregate information; and

[0467] The fee processing unit determines the usage fee based on the total information.

[0468] (8) An information processing system based on any one of (1) to (7), wherein

[0469] The processing unit switches the inference unit according to the processing capability of the information processing device including the processing unit.

[0470] (9) An information processing system based on any one of (1) to (8), wherein

[0471] The switching unit switches the inference unit based on at least one of the reference dataset expiration date for the user of the information processing device including the processing unit and the area of ​​use.

[0472] (10) The information processing system according to any one of (1) to (9) further includes:

[0473] The processing capacity management unit manages the processing capacity of the information processing device including the processing unit, wherein...

[0474] The switching unit switches the inference unit based on the processing capacity of the information processing device, which includes the processing unit and is managed by the processing capacity management unit.

[0475] (11) An information processing system based on any one of (1) to (10), wherein

[0476] The sensor data acquisition unit further acquires the location information of the information processing device, including the processing unit, and

[0477] The switching unit switches the inference unit based on the location information.

[0478] (12) An information processing system based on any one of (1) to (11), wherein

[0479] The inference unit outputs data in an output format that is common to the output format of another inference unit that may be included in the processing unit.

[0480] (13) The information processing system according to any one of (1) to (11) further includes:

[0481] The conversion unit converts the output format of the data output by the inference unit into a predetermined output format.

[0482] (14) An information processing system based on any one of (1) to (13), wherein

[0483] The processing unit associates the identification information used to identify the output format of the data output by the inference unit with the data.

[0484] (15) An information processing system, comprising:

[0485] The model acquisition unit acquires reference data via a predetermined network, and the reference data is encrypted.

[0486] The key information acquisition unit acquires key information used to decrypt the reference data; and

[0487] The processing unit uses the key information to decrypt the reference data and uses the decrypted reference data to configure the inference unit.

[0488] (16) An information processing system, comprising a device, a service server, and a resource server connected via a predetermined network, wherein,

[0489] The service server sends a SetSceneMode instruction to the device's nodes to set the SceneMode.

[0490] The service server sends an instruction to the node to start SceneMode's StartScene.

[0491] The node retrieves reference data from the endpoint of the resource server specified by StartScene.

[0492] The node establishes an inference unit by using the reference data.

[0493] The node sends the inference result of the inference unit to the service server.

[0494] The service server sends an instruction to the node to terminate the SceneMode's StopScene, and

[0495] The node responds to StopScene by terminating SceneMode.

[0496] (17) An information processing system, comprising a device, a service server, and a resource server connected via a predetermined network, wherein,

[0497] The service server sends a SetSceneMode instruction to the device's nodes to set the SceneMode.

[0498] The service server sends an instruction to the node to start SceneMode's StartScene.

[0499] The node obtains the privacy object from the PrivacyServerEndpoint in the resource server specified by SceneMode.

[0500] The node retrieves the encrypted reference data from the RefDataEndpoint in the resource server specified by StartScene.

[0501] The node uses the privacy object to decrypt the reference data.

[0502] The node establishes an inference unit by using the decrypted reference data.

[0503] The node sends the inference result of the inference unit to the service server.

[0504] The service server sends an instruction to the node to terminate the SceneMode's StopScene, and

[0505] The node responds to StopScene by terminating SceneMode.

[0506] (18) An information processing method, comprising:

[0507] Acquire the sensing data obtained by the sensor;

[0508] Configure an inference unit that uses the sensed data as input to perform inference; and

[0509] The inference unit is switched according to usage.

[0510] (19) An information processing method, comprising:

[0511] Reference data is obtained via a predetermined network and is encrypted.

[0512] Obtain the key information used to decrypt the reference data;

[0513] The reference data is decrypted using the key information; and

[0514] The inference unit is configured by using the decrypted reference data.

[0515] (20) An information processing method executed in an information processing system, wherein devices, a service server, and a resource server are connected via a predetermined network, the method comprising:

[0516] The service server sends a SetSceneMode instruction to the node of the device to set the SceneMode.

[0517] The service server sends a StartScene instruction to the node to begin the SceneMode.

[0518] The node obtains reference data from the endpoint in the resource server specified by StartScene.

[0519] The node establishes an inference unit by using the reference data.

[0520] The node sends the inference result of the inference unit to the service server.

[0521] The service server sends an instruction to the node to terminate the StopScene of the SceneMode, and

[0522] The SceneMode is terminated by the node in response to the StopScene.

[0523] (21) An information processing method executed in an information processing system, wherein the information processing system comprises a device, a service server, and a resource server connected via a predetermined network, the method comprising:

[0524] The service server sends a SetSceneMode instruction to the node of the device to set the SceneMode.

[0525] The service server sends a StartScene instruction to the node to begin the SceneMode.

[0526] The node obtains the privacy object from the PrivacyServerEndpoint in the resource server specified by SceneMode.

[0527] The node obtains encrypted reference data from the RefDataEndpoint in the resource server specified by StartScene.

[0528] The node uses the privacy object to decrypt the reference data.

[0529] The node establishes an inference unit by using the decrypted reference data.

[0530] The node sends the inference result of the inference unit to the service server.

[0531] The service server sends an instruction to the node to terminate the StopScene of the SceneMode, and

[0532] The SceneMode is terminated by the node in response to the StopScene.

[0533] Reference number list

[0534] 1. Information Processing System

[0535] Sensor devices 10, 10a, 10b, and 10c

[0536] 20 service servers

[0537] User facilities 30, 30a, 30b, and 30c

[0538] 40 Authentication Servers

[0539] 50 Total Servers

[0540] 60 resource servers

[0541] 70 Model Data

[0542] 71. Header

[0543] 72 main body

[0544] 90 Network

[0545] 100, 2300A, 2300B sensor units

[0546] 110 positioning unit

[0547] Processing units 130, 230, 530, 2200A, and 2200B

[0548] 132, 232ID transmission units

[0549] 134 Key Receiving Unit

[0550] 136 Sensor Data Acquisition Unit

[0551] 138 Preprocessing Units

[0552] 140 Model Acquisition Units

[0553] 142 Identification Units

[0554] 144 Data Generation Units

[0555] Distribution Units 146 and 246

[0556] 148 Total Units

[0557] 150 Total Information Transmission Units

[0558] 160, 260, 560 memory units

[0559] 170, 270, and 570 communication units

[0560] 234 Request Acceptance Unit

[0561] 236 Capability Information Acquisition Unit

[0562] 238 model generation units

[0563] 238a Learning Device

[0564] 240 Model Transmission Units

[0565] 242 Data Acquisition Unit

[0566] 532 Total Information Acquisition Unit

[0567] 534 Fee Processing Unit

[0568] Objects 600-1 to 600-n

[0569] 602-1~602-n Supervision Data

[0570] 610 Recognition Model

[0571] 2001 Application

[0572] 2002 Application List

[0573] 2100 Service Unit

[0574] 2101AI Network Unit

[0575] 2102 Analysis Unit

[0576] 2103A and 2103B data pipeline units

[0577] 2201 CPU

[0578] 2202, 2302 Signal Processing Units

[0579] 2203 Inference Unit

[0580] 2301 sensor

[0581] 2303 Identification Unit

[0582] 2304 Inference Unit

[0583] 3000 devices

[0584] Processing of nodes 3010, 3010A, 3010B, 3010C, 3010-1 to 3010-N, and node 3011.

[0585] 3012 Reference Data

[0586] 3013 AI Framework

[0587] 3100 Applications / Services

[0588] 3200 resource server

[0589] 3201 endpoint (RefData endpoint)

[0590] 3211PrivacyServer endpoint

[0591] 3220 Model List

[0592] 41 SceneMode object

[0593] 42LabelRefDataList

[0594] Objects 43, 452, and 453

[0595] 47 Encrypted Objects

[0596] 48 Privacy Objects

[0597] 441LabelName

[0598] 442RefDataList

[0599] 443RefData444 Processing Stage

[0600] Additional features 445, 474, 493, 504, 4624, 4634, 4916: 471 EncryptionOn

[0601] 472SceneEncryptionKeyID

[0602] Version 473PrivacyServerEndPoint481

[0603] 482EndPointID

[0604] 483PrivacyObjectID

[0605] 484StartDateTime

[0606] 485EndDateTime

[0607] 486UsageCount

[0608] 487StorageRule

[0609] 488ExportRule

[0610] 489MaskedItems

[0611] 490 Analysis Rule 491 Encryption

[0612] 492 certification

[0613] 500StartScene object

[0614] Version 501

[0615] 502SceneModeID

[0616] 503RefDataID / RefDataListID

[0617] 4621RefDataID

[0618] 4622RefDataEndPoint4623 Encryption

[0619] 4631RefDataID

[0620] 4632RefData4633 Encryption

[0621] 4911 SymmetricKey object

[0622] 4912kty

[0623] 4913alg

[0624] 4914k

[0625] 4915kid

Claims

1. An information processing system, comprising: The sensor data acquisition unit acquires the sensing data obtained by the sensor; The processing unit includes an inference unit that uses the sensed data as input to perform inference. The switching unit switches the inference unit according to the usage situation; as well as Model acquisition unit, acquire reference data, The switching unit switches the inference unit by replacing the reference data of the inference unit with the reference data obtained by the model acquisition unit. The reference data includes at least one of the model data configuring the inference unit and the set of weight parameters in the model data. Wherein, when the reference data includes the weight parameter set, the switching unit switches the inference unit by replacing the weight parameters of the inference unit with the weight parameter set.

2. The information processing system according to claim 1, wherein, The model acquisition unit acquires the reference data via a predetermined network.

3. The information processing system according to claim 2, further comprising: Model holding units are arranged on the predetermined network and hold one or more reference data, wherein The model acquisition unit acquires the reference data from the model holding unit via the predetermined network.

4. The information processing system according to claim 2, wherein, The model acquisition unit acquires the reference data from another information processing device arranged on the predetermined network, the other information processing device being different from the information processing device that includes the processing unit.

5. The information processing system according to claim 1, further comprising: The aggregation unit aggregates the distribution amount of the reference data for each device and generates aggregation information. as well as The fee processing unit determines the usage fee based on the total information.

6. The information processing system according to claim 1, wherein, The processing unit switches the inference unit according to the processing capability of the information processing device including the processing unit.

7. The information processing system according to claim 1, wherein, The switching unit switches the inference unit based on at least one of the expiration date of a reference dataset for a user of an information processing device including the processing unit and the area of ​​use.

8. The information processing system according to claim 1, further comprising: The processing capacity management unit manages the processing capacity of the information processing device including the processing unit, wherein... The switching unit switches the inference unit based on the processing capacity of the information processing device, which includes the processing unit and is managed by the processing capacity management unit.

9. The information processing system according to claim 1, wherein, The sensor data acquisition unit further acquires the location information of the information processing device, including the processing unit, and The switching unit switches the inference unit based on the location information.

10. The information processing system according to claim 1, wherein, The inference unit outputs data in an output format that is common to the output format of another inference unit that can be included in the processing unit.

11. The information processing system according to claim 1, further comprising: The conversion unit converts the output format of the data output by the inference unit into a predetermined output format.

12. The information processing system according to claim 1, wherein, The processing unit associates the output format identification information used to identify the data output by the inference unit with the data.

13. An information processing system, comprising: The model acquisition unit acquires reference data via a predetermined network, and the reference data is encrypted. A key information acquisition unit acquires key information used to decrypt the reference data; as well as The processing unit decrypts the reference data using the key information and configures the inference unit by replacing the reference data of the inference unit with the decrypted reference data. The reference data includes at least one of the model data configuring the inference unit and the set of weight parameters in the model data. Wherein, when the reference data includes the weight parameter set, the processing unit configures the inference unit by replacing the weight parameters of the inference unit with the weight parameter set.

14. An information processing system, comprising a device, a service server, and a resource server connected via a predetermined network, wherein, The service server sends a SetSceneMode instruction to the node of the device to set the SceneMode. The service server sends an instruction to the node to start the StartScene of the SceneMode; The node obtains reference data from the endpoint in the resource server specified by the StartScene; The node establishes an inference unit by using the reference data; The node sends the inference result of the inference unit to the service server; The service server sends an instruction to the node to terminate the StopScene of the SceneMode; and The node responds to the StopScene by terminating the SceneMode.

15. An information processing system, comprising a device, a service server, and a resource server connected via a predetermined network, wherein, The service server sends a SetSceneMode instruction to the node of the device to set the SceneMode. The service server sends an instruction to the node to start the StartScene of the SceneMode; The node obtains a privacy object from the PrivacyServerEndPoint in the resource server specified by the SceneMode; The node obtains encrypted reference data from the RefDataEndPoint in the resource server specified by the StartScene; The node uses the privacy object to decrypt the reference data; The node establishes an inference unit by using the decrypted reference data; The node sends the inference result of the inference unit to the service server; The service server sends an instruction to the node to terminate the StopScene of the SceneMode; and The node responds to the StopScene by terminating the SceneMode.

16. An information processing method, comprising: Acquire the sensing data obtained by the sensor; Configure an inference unit that uses the sensed data as input to perform inference; The inference unit is switched according to usage; as well as Reference data is obtained through model acquisition units; Specifically, the inference unit is switched by replacing the reference data of the inference unit with the reference data obtained by the model acquisition unit. The reference data includes at least one of the model data configuring the inference unit and the set of weight parameters in the model data. Wherein, if the reference data includes the weight parameter set, the inference unit is switched by replacing the weight parameters of the inference unit with the weight parameter set.

17. An information processing method, comprising: Reference data is obtained via a predetermined network and is encrypted. Obtain the key information used to decrypt the reference data; The reference data is decrypted using the key information. as well as The inference unit is configured by replacing the reference data of the inference unit with the decrypted reference data. The reference data includes at least one of the model data configuring the inference unit and the set of weight parameters in the model data. Wherein, when the reference data includes the weight parameter set, configuring the inference unit includes configuring the inference unit by replacing the weight parameters of the inference unit with the weight parameter set.

18. An information processing method executed in an information processing system, wherein devices, a service server, and a resource server are connected via a predetermined network, the information processing method comprising: The service server sends a SetSceneMode instruction to the node of the device to set the SceneMode. The service server sends an instruction to the node to start the SceneMode's StartScene; The node obtains reference data from the endpoint in the resource server specified by StartScene; The node establishes an inference unit by using the reference data; The node sends the inference result of the inference unit to the service server; The service server sends an instruction to the node to terminate the StopScene of the SceneMode; and The SceneMode is terminated by the node in response to the StopScene.

19. An information processing method executed in an information processing system, wherein devices, a service server, and a resource server are connected via a predetermined network, the information processing method comprising: The service server sends a SetSceneMode instruction to the node of the device to set the SceneMode. The service server sends an instruction to the node to start the StartScene of the SceneMode; The node obtains the privacy object from the PrivacyServerEndpoint in the resource server specified by the SceneMode; The node obtains encrypted reference data from the RefDataEndpoint in the resource server specified by the StartScene; The node uses the privacy object to decrypt the reference data; The node establishes an inference unit by using the decrypted reference data; The node sends the inference result of the inference unit to the service server; The service server sends an instruction to the node to terminate the StopScene of the SceneMode; and The node terminates the SceneMode in response to the StopScene.

Images