Information processing device, terminal device, information processing system, and data generation method

Abstract

An information processing device according to the present disclosure comprises: a combination unit that combines two or more artificial intelligence (AI) models with added management information for managing each of the two or more AI models and generates a combined AI model; and a transmission unit that transmits the combined AI model to a terminal device serving as an execution environment for executing the two or more AI models included in the combined AI model.

Description

Information processing device, terminal device, information processing system, and data generation method

[0001] The present disclosure relates to an information processing device, a terminal device, an information processing system, and a data generation method.

[0002] In recent years, edge devices, which are information terminal devices connected to the Internet, have been used in a variety of fields. Examples of edge devices include surveillance cameras that transmit information such as image data via the Internet, and electronic devices that are installed in home appliances and have Internet communication functions.

[0003] Also, a technology for executing AI (Artificial Intelligence) models on edge devices is known. Furthermore, in recent years, there has been an increase in solutions that support various use cases by running multiple AI models in series or parallel within the same edge device.

[0004] Japanese Patent Application Laid-Open No. 2022-108450

[0005] For example, when handling multiple AI models in an IoT (Internet of Things) device, which is a type of edge device, it is necessary to import the required number of target AI models from an external device, which increases communication costs and the number of times the imported AI models are written to the memory of the edge device.

[0006] The present disclosure aims to provide an information processing device, a terminal device, an information processing system, and a data generation method that can reduce the load when using multiple AI models in an edge device.

[0007] The information processing device according to the present disclosure includes a combination unit that combines two or more AI (Artificial Intelligence) models by adding management information that manages each of the two or more AI models to generate a combined AI model, and a transmission unit that transmits the combined AI model to a terminal device that serves as an execution environment for executing the two or more AI models included in the combined AI model.

[0008] 1 is a schematic diagram illustrating an example of the configuration of an information processing system applicable to each embodiment. FIG. 1 is a functional block diagram illustrating an example of the configuration of an AI model management server according to each embodiment. FIG. 2 is a functional block diagram illustrating an example of the function of an edge device according to each embodiment. FIG. 3 is a schematic diagram illustrating a process of importing and executing multiple AI models into an edge device according to existing technology. FIG. 4 is a schematic diagram illustrating a process of importing and executing multiple AI models into an edge device according to a first embodiment. FIG. 5 is a schematic diagram illustrating an example of a use case of switching between multiple AI models applicable to the first embodiment. FIG. 6 is a functional block diagram illustrating the functions of an edge device and an AI model management server included in an information processing system according to the first embodiment. FIG. 7 is a block diagram illustrating an example of the hardware configuration of an AI model management server according to the first embodiment. FIG. 8 is a block diagram illustrating an example of the hardware configuration of an edge device 10b according to the first embodiment. FIG. 9 is a flowchart illustrating an example of a process of importing an AI model according to the first embodiment. FIG. 10 is a flowchart illustrating an example of a process of using an AI model according to the first embodiment. FIG. 11 is a schematic diagram illustrating an example of a data flow during deployment according to the first embodiment. FIG. 12 is a schematic diagram illustrating an example of the configuration of a file "network.fpk" according to the first embodiment. FIG. 1 is a schematic diagram showing an example of the configuration of a file "network_info.txt" according to the first embodiment. FIG. 2 is a timing chart showing an example of processing when an AI model is deployed to an edge device. FIG. 3 is a timing chart showing an example of processing when an AI model is executed in an edge device. FIG. 4 is a schematic diagram showing an example of a command notified to an edge device from an AI model management server. FIG. 5 is a schematic diagram for explaining a method of specifying an order for multiple AI models according to the first embodiment. FIG. 6 is a schematic diagram showing an example of the configuration of a file "network.fpk" according to the first embodiment. FIG. 7 is a schematic diagram for explaining a method of separating and selecting multiple AI models in an edge device according to the first embodiment. FIG. 8 is a schematic diagram showing an example of a screen using a UI for generating a combined AI model according to the first embodiment.1 is a schematic diagram showing an example of a manifest file for defining the order of multiple AI models, applicable to the first embodiment. FIG. 2 is a schematic diagram for generally explaining ensemble learning. FIG. 3 is a functional block diagram for generally explaining the function of an edge device according to the second embodiment. FIG. 4 is a block diagram showing the configuration of an example of an imaging device applicable to the second embodiment. FIG. 5 is a perspective view showing the configuration of an example of an imaging device applicable to the embodiment. FIG. 6 is a schematic diagram showing the configuration of a data flow during deployment according to the second embodiment. FIG. 7 is a schematic diagram for explaining a process of writing a combined AI model to a non-volatile memory according to the second embodiment. FIG. 8 is a schematic diagram showing the configuration of an example of a file "network.fpk" according to the second embodiment. FIG. 9 is a schematic diagram showing the configuration of an example of a download header and a download footer according to the second embodiment. FIG. 10 is a schematic diagram showing the configuration of an example of an image packet header according to the second embodiment. FIG. 11 is a schematic diagram showing the configuration of an example of an image header according to the second embodiment. FIG. 12 is a schematic diagram for explaining a first example of switching between multiple AI models according to the second embodiment. FIG. 13 is a schematic diagram for explaining a second example of switching between multiple AI models according to the second embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0010] Hereinafter, embodiments of the present disclosure will be described in the following order: 1. Configuration applicable to each embodiment of the present disclosure 1-1. Regarding existing technology 2. First embodiment of the present disclosure 2-1. Overview of processing and configuration according to the first embodiment 2-2. Configuration applicable to the first embodiment 2-3. Processing according to the first embodiment 2-4. Regarding the method for specifying / separating multiple AI models according to the first embodiment 2-5. Modified example of the first embodiment 3. Second embodiment of the present disclosure 3-1. Configuration according to the second embodiment 3-2. Processing according to the second embodiment 3-3. Example of data format according to the second embodiment 3-4. Model switching processing according to the second embodiment

[0011] Each embodiment of the present disclosure relates to a technology for executing multiple AI (Artificial Intelligence) models in an execution environment such as an edge device, combining the multiple AI models to be executed to generate a single combined AI model, and transferring and deploying this combined AI model to the execution environment.

[0012] In the execution environment, the transferred combined AI model is temporarily stored in non-volatile memory such as flash memory, and then deployed in volatile memory such as static random access memory (SRAM) as execution memory. In the execution environment, the specified AI model is read from the volatile memory and executed.

[0013] In this way, by combining multiple AI models to generate a single combined AI model and deploying this combined AI model in an execution environment, it is possible to reduce the transfer costs of the AI ​​models and also reduce the number of times data is written to memory in the execution environment.

[0014] (1. Configuration Applicable to Each Embodiment of the Present Disclosure) A configuration applicable to each embodiment of the present disclosure will be briefly described.

[0015] 1 is a schematic diagram illustrating an example of the configuration of an information processing system applicable to each embodiment. In FIG. 1, the information processing system 1 includes an edge device 10 and an AI model management server 30 connected via a communication network 2 such as the Internet.

[0016] The edge device 10 is an information device at a terminal connected to the communication network 2. Examples of the edge device 10 include a surveillance camera that transmits information such as image data via the Internet, and an electronic device that is provided in a home appliance and has a communication function for communicating with the Internet. In the present disclosure, the edge device 10 is an execution environment for an AI model in which an AI model is deployed.

[0017] The AI ​​model management server 30 manages multiple AI models. Furthermore, the AI ​​model management server 30 selects two or more AI models from the multiple AI models it manages, for example, in response to a user instruction, and combines the two or more selected AI models to generate a combined AI model. The AI ​​model management server 30 transmits the generated combined AI model to the edge device 10. The edge device 10 executes processing using the AI ​​models included in the combined AI model. In other words, the edge device 10 can be said to be a terminal device for the AI ​​model management server 30 in the information processing system 1.

[0018] 1, the AI ​​model management server 30 is shown as being configured in a cloud network 3 connected to a communication network 2. However, the AI ​​model management server 30 is not limited to this, and may be configured as a single computer, or may be configured as a distributed system across multiple computers.

[0019] 1, the user terminal 4 is, for example, a general computer device that accepts input operations by a user and provides a user interface (UI) that presents information to the user by screen display or the like. The user terminal 4 may, for example, transmit commands corresponding to the user's input operations to the AI ​​model management server 30 via the communication network 2. The user terminal 4 may also, for example, present information transmitted from the AI ​​model management server 30 to the user by a display or the like.

[0020] 2 is a functional block diagram for schematically illustrating the functions of the AI ​​model management server 30 according to each embodiment. In FIG. 2, the AI ​​model management server 30 includes an AI model storage unit 31, a processing unit 32, and a transmission unit 33.

[0021] The AI ​​model storage unit 31 stores multiple AI models in, for example, a storage device. The processing unit 32 reads one or more AI models from the AI ​​model storage unit 31 and performs predetermined processing on the read AI models. For example, the processing unit 32 reads two or more AI models from the AI ​​model storage unit 31 and combines the two or more read AI models to generate a combined AI model. The transmission unit 33 transmits the AI ​​model processed by the processing unit 32 and the combined AI model to the edge device 10 via the communication network 2.

[0022] The AI ​​model storage unit 31, processing unit 32, and transmission unit 33 are configured by running programs according to each embodiment on, for example, a CPU (Central Processing Unit). However, without being limited to this, some or all of the AI ​​model storage unit 31, processing unit 32, and transmission unit 33 may be configured by hardware circuits that operate in cooperation with each other.

[0023] In the following, the storing of the AI ​​model by the AI ​​model storage unit 31 in a storage device may be described as "the AI ​​model storage unit 31 stores the AI ​​model," etc.

[0024] 3 is a functional block diagram for explaining the functions of the edge device 10 according to each embodiment. In FIG. 3, the edge device 10 includes a receiving unit 11, a processing unit 12, a volatile memory 13, a non-volatile memory 14, and a device control unit 15.

[0025] Here, the nonvolatile memory 14 is a memory that stores data in a nonvolatile manner, and may be, for example, a flash memory. On the other hand, the volatile memory 13 is a memory that stores data in a volatile manner, i.e., temporarily, and may be, for example, an SRAM (Static Random Access Memory). Generally, the volatile memory 13 can be accessed faster than the nonvolatile memory 14.

[0026] The receiving unit 11 communicates via the communication network 2 and receives, for example, a combined AI model transmitted from the AI ​​model management server 30. The processing unit 12 performs predetermined processing on the combined AI model received by the receiving unit 11. For example, the processing unit 12 writes the combined AI model received by the receiving unit 11 to the non-volatile memory 14. When executing an AI model included in the combined AI model, the processing unit 12 reads the combined AI model stored in the non-volatile memory into the volatile memory 13 and deploys it. The processing unit 12 executes the AI ​​model deployed on the volatile memory 13 and outputs the results. The equipment control unit 15 controls the equipment to be controlled by the edge device 10 based on the output of the processing unit 12. There are no particular limitations on the equipment to be controlled by the equipment control unit 15.

[0027] (1-1. Regarding Existing Technology) Here, to facilitate understanding, an outline of existing technology will be explained. FIG. 4 is a schematic diagram for explaining the import and execution process of multiple AI models into an edge device according to existing technology. In the example of FIG. 4, in an information processing system 1a according to existing technology, two AI models, AI models #1 and #2, are imported from an AI model management server 30a to an edge device 10a.

[0028] Section (a) of Figure 4 shows an example of the process of deploying an AI model to the edge device 10a. In existing technology, when deploying an AI model, the AI ​​model management server 30a sends two AI models #1 and #2 to be executed on the edge device 10a, each individually, to the edge device 10a. The edge device 10a receives the AI ​​models #1 and #2 sent from the AI ​​model management server 30a, writes them to the non-volatile memory 14 each time it receives them, and imports the AI ​​models #1 and #2.

[0029] With existing technology, when deploying multiple AI models to the edge device 10a, the AI ​​models must be written sequentially to the non-volatile memory 14 for each AI model to be deployed. This increases communication costs and the number of write operations depending on the number of AI models to be deployed. Furthermore, when tuning an AI model deployed to the edge device 10a on the edge device 10a, re-importing occurs due to updating and relearning the AI ​​model. This also increases communication costs and the number of write operations.

[0030] Section (b) of Figure 4 shows the process of executing multiple deployed AI models on the edge device 10a. When executing an AI model, the edge device 10a loads the AI ​​model written in the non-volatile memory 14 into the volatile memory 13. When the edge device 10a executes multiple AI models #1 and #2 serially or in parallel, the edge device 10a loads each of the AI ​​models #1 and #2 to be executed into the volatile memory 13. Because the edge device 10a loads each AI model into the volatile memory 13 in advance, even when handling multiple AI models of sufficiently small size, the AI ​​models must be loaded multiple times from the non-volatile memory 14, resulting in large overhead related to latency and power consumption.

[0031] (2. First Embodiment of the Present Disclosure) Next, a first embodiment of the present disclosure will be described.

[0032] (2-1. Overview of Processing and Configuration According to First Embodiment) FIG. 5 is a schematic diagram for explaining the import and execution processing of multiple AI models into an edge device according to the first embodiment.

[0033] 5, the information processing system 1b according to the first embodiment includes an edge device 10b and an AI model management server 30b. Note that the cloud network 3 in which the AI ​​model management server 30b is configured, the communication network 2 connecting the cloud network 3 and the edge device 10b, and the user terminal 4 are omitted from FIG.

[0034] In FIG. 5, the AI ​​model management server 30b according to the first embodiment includes an AI model storage unit 310 and an AI model combination unit 320.

[0035] The AI ​​model storage unit 310 corresponds to the AI ​​model storage unit 31 in FIG. 2, and stores a plurality of AI models 40 1 , 40 2 , 40 3 , . . . that have been trained.

[0036] 5, the AI ​​models 401, 402, 403, ... are also respectively referred to as AI models #1, #2, #3, .... In the following, when there is no particular need to distinguish between the AI ​​models 401, 402, 403, ..., they will be collectively described as AI model 40.

[0037] The AI ​​model combination unit 320 is included in the processing unit 32 in Fig. 2. The AI ​​model combination unit 320 combines two or more specified AI models from the multiple AI models 401, 402, 403, ... stored in the AI ​​model storage unit 310 through a combination process 321 to generate one combined AI model 41.

[0038] The combined AI model 41 is configured by repeating pairs of a header for each AI model and an AI model, the number of which is equal to the number of AI models 401, 402, ... to be combined in the combined AI model 41, following a header for the entire combined AI model 41.

[0039] Furthermore, when the AI ​​model combination unit 320 switches between the AI ​​models 40 1, 40 2, ... on the edge device 10 b or executes them in parallel, it generates management information 42 1, 42 2, ... for managing the AI ​​models 40 1, 40 2, .... In the combination process 321, the AI ​​model combination unit 320 adds the generated management information 42 1, 42 2, ... to the combined AI model 41.

[0040] In the following description, when there is no need to distinguish between the management information 421, 422, . . . , they will be collectively referred to as management information 42.

[0041] The AI ​​model management server 30b transmits the combined AI model 41 with the management information 42, 42, ... added to the edge device 10b. The edge device 10b receives the combined AI model 41 with the management information 42, 42, ... added, sent from the AI ​​model management server 30b, and writes it to the non-volatile memory 14. As a result, multiple AI models 40, 40, ... are imported into the edge device 10b.

[0042] When AI models 40, 40, ... are executed serially or in parallel, edge device 10b reads the combined AI model 41 written in non-volatile memory 14 into volatile memory 13 and deploys it. Based on management information 42, 42, ... added to the combined AI model 41, edge device 10b separates each of the AI ​​models 40, 40, ... combined in the combined AI model 41.

[0043] In this way, in the first embodiment, when a combined AI model 41 is generated by combining multiple AI models 401, 402, ... in the AI ​​model management server 30b, it is sufficient to import only this combined AI model 41 into the edge device 10b, and it is possible to reduce communication costs and the number of writes to the non-volatile memory 14 compared to the case of the existing technology described using Figure 4.

[0044] Furthermore, in the first embodiment, when executing each of the AI ​​models 401, 402, ..., the edge device 10b only needs to read one combined AI model 41 from the non-volatile memory 14, which makes it possible to reduce overhead related to latency and power consumption compared to existing technologies.

[0045] In the above description, the combination process 321 that combines multiple AI models 40, 40, 40, ... is executed on the AI ​​model management server 30b, but this is not limited to this example. For example, the combination process 321 may be executed in advance in the same device or edge device 10b as the execution environment in which each of the AI ​​models 40, 40, ... to be combined is executed, or in an environment local to the AI ​​model management server 30b on the cloud network 3, and the combined AI model 41 generated as a result may be deployed to the edge device 10b.

[0046] 6 is a schematic diagram illustrating an example of a use case of switching between multiple AI models, which is applicable to the first embodiment. In this example, the edge device 10b is a camera having an execution environment for the AI ​​models, and the AI ​​models 40a and 40b are used in series to perform license plate detection and license plate recognition on a captured image 60.

[0047] In the following, AI model 40a is an AI model for detecting an area including a license plate from an image, and AI model 40b is an AI model for recognizing information indicated on a license plate included in an image.

[0048] 6, in response to an instruction from a user via the user terminal 4, for example, the AI ​​model management server 30b combines an AI model 40a (also referred to as model #a in the figure) used for license plate detection with an AI model 40b (also referred to as model #b in the figure) used for license plate recognition to generate a combined AI model 41. The AI ​​model management server 30b transmits the generated combined AI model 41 to the edge device 10b (step S10).

[0049] The edge device 10b deploys the combined AI model 41 transmitted from the AI ​​model management server 30b. That is, the edge device 10b receives the combined AI model 41 in the volatile memory 13, reads the combined AI model 41 from the volatile memory 13, and writes it to the non-volatile memory 14.

[0050] When executing the AI ​​model, the edge device 10b expands the combined AI model 41 written in the non-volatile memory 14 into the volatile memory 13 (step S11). The edge device 10b acquires a captured image 60 (step S12), performs license plate detection processing on the acquired captured image 60 using the AI ​​model 40a included in the combined AI model 41, and passes the detection result to the AI ​​processing unit 50 (step S13). The AI ​​processing unit 50 may be included in, for example, the processing unit 12 (see FIG. 3) of the edge device 10b.

[0051] In response to the detection result passed from the AI ​​model 40a, the AI ​​processing unit 50 instructs, for example, the processing unit 12 of the edge device 10b to crop the license plate area 61 in the captured image 60 by post-processing (step S14) (step S15). In addition, in order to perform recognition processing on the image 62 of the license plate area 61 cropped from the captured image 60 (step S16), the AI ​​processing unit 50 instructs, for example, the processing unit 12 to switch the AI ​​model to be executed from the AI ​​model 40a to the AI ​​model 40b.

[0052] In response to the instruction, the edge device 10b switches the AI ​​model to be executed from the AI ​​model 40a to the AI ​​model 40b. The edge device 10b executes recognition processing on the image 62 of the license plate area 61 using the AI ​​model 40b included in the combined AI model 41, and passes the recognition result to the AI ​​processing unit 50 (step S17).

[0053] Based on the recognition result passed from the AI ​​model 40b, the AI ​​processing unit 50 performs predetermined post-processing by post-processing (step S18) and outputs the result as license plate information 70 (step S19).

[0054] In this way, in the first embodiment, the edge device 10b can dynamically switch between the execution of the multiple AI models 40a and 40b included in the combined AI model 41.

[0055] In the above description, the multiple AI models to be combined are described as AI models that are used in series and succession, but this is not limited to this example. In other words, the multiple AI models to be combined may be AI models that are used in parallel, or may each be an AI model that is used independently.

[0056] An example of using multiple AI models in series is the case where a later AI model is executed based on the output of a previously executed AI model, such as the license plate detection AI model 40a and the license plate recognition AI model 40b described above. An example of using multiple AI models in parallel is a case where an AI model for detecting shopper attributes and an AI model for monitoring product shelves are operated in parallel in a retail store. Another example of using multiple AI models independently is a case where a retail store uses multiple independent AI models, such as an AI model for inventory detection used during the day and an AI model for intrusion detection used at night, as a single combined AI model 41, and uses each model independently.

[0057] Furthermore, the edge device 10b may update the deployed combined AI model 41 using as a trigger the recognition results and their accuracy at the time of execution of the AI ​​models 40 included in the combined AI model 41, communication conditions, etc. In this case, it is also possible to dynamically switch the combination of AI models included in the combined AI model 41 based on, for example, the recognition results of a predetermined AI model included in the combined AI model 41.

[0058] As an example, in an AI model whose detection accuracy changes depending on the scene (daytime, nighttime, etc.), if the detection accuracy becomes low, the AI ​​model executed in the subsequent stage may be switched to another model. As another example, the communication status (bandwidth, etc.) of the communication transmitting the output of the edge device 10b may be monitored, and the AI ​​model to be combined in the combined AI model 41 may be switched to make the output lighter.

[0059] (2-2. Configuration Applicable to First Embodiment) Next, a configuration applicable to the first embodiment will be described. Fig. 7 is a functional block diagram for explaining functions of the edge device 10b and the AI ​​model management server 30b included in the information processing system 1b according to the first embodiment.

[0060] In FIG. 7, the AI ​​model management server 30b according to the first embodiment includes an AI model storage unit 310, an AI model combination unit 320, and a data transmission unit 330.

[0061] 2, and stores a plurality of AI models 40, 40, 40, ... in, for example, a storage device. Furthermore, the AI ​​model storage unit 310 reads out one or more AI models 40 specified by, for example, the AI ​​model combination unit 320 from the plurality of stored AI models 40, 40, 40, ....

[0062] The AI ​​model combination unit 320 includes a memory calculation unit 322, an AI model combination processing unit 323, a data addition unit 324, and a data authentication unit 325. The AI ​​model combination unit 320 may correspond to the processing unit 32 in FIG.

[0063] The memory calculation unit 322 calculates the memory capacity required to deploy the combined AI model 41 generated by the AI ​​model combination unit 320. The memory calculation unit 322 may calculate the memory capacity based on the data size of each AI model 40 combined by the combined AI model 41, etc.

[0064] The AI ​​model combination processing unit 323 combines two or more specified AI models 40 from the AI ​​models 40, 40, 40, ... stored in the AI ​​model storage unit 310, to generate a combined AI model 41. The data addition unit 324 generates management information 42 based on information such as the AI ​​models 40 combined by the AI ​​model combination processing unit 323, and adds the generated management information 42 to the combined AI model 41.

[0065] The data authentication unit 325 performs authentication processing on the combined AI model 41 to which the management information 42 has been added.

[0066] The data transmission unit 330 includes a device connection unit 331 and a data import unit 332. The data transmission unit 330 may correspond to the transmission unit 33 in FIG.

[0067] The device connection unit 331 performs connection processing to connect to the edge device 10b to which the combined AI model 41 is to be imported via the communication network 2. The data import unit 332 transmits the combined AI model 41 generated in the AI ​​model combination unit 320 to the edge device 10b to which connection processing has been performed by the device connection unit 331, and imports the combined AI model 41 into the edge device 10b.

[0068] The AI ​​model storage unit 310, AI model combination unit 320, and data transmission unit 330 included in the AI ​​model management server 30b are realized, for example, by executing the information processing program according to the first embodiment on a CPU in the AI ​​model management server 30b. Without being limited to this, some or all of the AI ​​model storage unit 310, AI model combination unit 320, and data transmission unit 330 may be realized by hardware circuits that operate in cooperation with each other.

[0069] 7, the edge device 10b includes a data processing device 100, an input device 110, and an output device 120. The data processing device 100 may correspond to the processing unit 12 in Fig. 3. The input device 110 may correspond to the receiving unit 11 in Fig. 3. Furthermore, the output device 120 may be a device that is controlled by the device control unit 15 in Fig. 3.

[0070] The data processing device 100 includes an AI model control unit 101, a management information control unit 102, an equipment control unit 103, a volatile memory 13, and a non-volatile memory 14.

[0071] The AI ​​model control unit 101 performs processing on the combined AI model 41 acquired by the input device 110. For example, the AI ​​model control unit 101 writes the combined AI model 41 acquired by the input device 110 to the non-volatile memory 14 via the volatile memory 13. When each AI model included in the combined AI model 41 is executed, the combined AI model 41 is read from the non-volatile memory 14 into the volatile memory 13.

[0072] The management information control unit 102 acquires management information 42 from the combined AI model 41 on the volatile memory 13, and specifies the AI ​​model 40 to be executed in the combined AI model 41 on the volatile memory 13 based on the acquired management information 42. The AI ​​model control unit 101 executes processing by the AI ​​model 40 specified by the management information control unit 102. The processing results by the AI ​​model 40 may be output to, for example, the output device 120. Furthermore, the AI ​​model control unit 101 may switch the AI ​​model 40 to be executed based on the management information 42.

[0073] The AI ​​model control unit 101, management information control unit 102, and equipment control unit 103 included in the edge device 10b are realized by executing the terminal program according to the first embodiment in the edge device 10b. However, without being limited to this, some or all of the AI ​​model control unit 101, management information control unit 102, and equipment control unit 103 may be realized by hardware circuits that operate in cooperation with each other.

[0074] FIG. 8 is a block diagram showing an example of the hardware configuration of the AI ​​model management server 30b according to the first embodiment.

[0075] As described above, the AI ​​model management server 30b (AI model management server 30) is configured, for example, in the cloud network 3. In this case, the AI ​​model management server 30b is configured in a distributed manner by a plurality of computer devices and storage devices connected to each other via a communication network. For the sake of explanation, FIG. 8 shows the AI ​​model management server 30b as being configured by a single computer device.

[0076] In Figure 8, the AI ​​model management server 30b includes a CPU 3000, a ROM (Read Only Memory) 3001, a RAM (Random Access Memory) 3002, a storage device 3003, a data I / F (interface) 3004, and a communication I / F 3005, which are communicatively connected to each other via a bus 3010.

[0077] The storage device 3003 is a non-volatile storage medium such as a flash memory, a hard disk drive, etc. The storage device 3003 may be configured to be connected to the outside of the AI ​​model management server 30b.

[0078] The CPU 3000 operates in accordance with programs stored in the ROM 3001 and the storage device 3003, using the RAM 3002 as a work memory, and controls the overall operation of this AI model management server 30b.

[0079] The data I / F 3004 is an interface for communicating with external devices, and may be connected to an input device such as a keyboard or an external storage medium. The communication I / F 3005 is an interface for communicating via the communication network 2.

[0080] When the above-mentioned AI model storage unit 310, AI model combination unit 320, and data transmission unit 330 are realized by executing a program on CPU 3000, in the AI ​​model management server 30b, the CPU 3000 executes the information processing program related to the first embodiment, thereby configuring the above-mentioned AI model storage unit 310, AI model combination unit 320, and data transmission unit 330, for example, as modules, in the main memory area of ​​RAM 3002.

[0081] The information processing program can be obtained from the outside via the communication network 2, for example, by communication via the communication I / F 3005, or from a storage medium connected to the data I / F 3004, and installed on the AI ​​model management server 30b.

[0082] FIG. 9 is a block diagram illustrating an example of the hardware configuration of the edge device 10b according to the first embodiment.

[0083] In FIG. 9, edge device 10b includes a CPU 1000, a ROM 1001, a RAM 1002, a storage device 1003, a data I / F 1004, a communication I / F 1005, an equipment I / F 1006, and a data processing device 1008, which are communicatively connected to each other via a bus 1010.

[0084] The storage device 1003 is a non-volatile storage medium such as a flash memory. The CPU 1000 controls the overall operation of the edge device 10b in accordance with the programs stored in the storage device 1003 and the ROM 1001, and uses the RAM 1002 as a work memory.

[0085] The data I / F 1004 is an interface for communicating with external devices, and may be connected to an input device such as a keyboard or an external storage medium. The communication I / F 1005 is an interface for communicating via the communication network 2. The device I / F 1006 is an interface for communicating with a control target device 1007 that is the control target of the edge device 10b. The device I / F 1006 may be common to the data I / F 1004.

[0086] The data processing device 1008 is, for example, a hardware circuit for performing data processing, and an ISP (Image Signal Processor) may be applied as a DSP (Digital Signal Processor). However, the present invention is not limited to this, and the processing in the data processing device 1008 may be performed by the CPU 1000 in accordance with a program without providing the data processing device 1008 as a hardware circuit.

[0087] When the above-mentioned AI model control unit 101, management information control unit 102, and equipment control unit 103 are realized by executing a program on CPU 1000, in edge device 10b, CPU 1000 executes the terminal program related to the first embodiment, thereby configuring the above-mentioned AI model control unit 101, management information control unit 102, and equipment control unit 103, for example, as modules, in the main memory area of ​​RAM 1002.

[0088] The terminal program can be acquired from the outside via the communication network 2 by communication via the communication I / F 1005, or can be acquired from a storage medium connected to the data I / F 1004, and installed on the edge device 10b. Alternatively, the terminal program can be stored in advance in the ROM 1001, for example.

[0089] (2-3. Processing According to First Embodiment) Next, processing in the information processing system 1b according to the first embodiment will be described.

[0090] FIG. 10 is a flowchart illustrating an example of processing when an AI model is imported according to the first embodiment.

[0091] 10 , a user may specify one or more AI models to be imported into the edge device 10b to the AI ​​model management server 30b, for example, from the user terminal 4. For example, the user terminal 4 may specify one or more AI models 40 from the AI ​​models 40, 40, 40, ... stored in the AI ​​model storage unit 310 to the AI ​​model management server 30b.

[0092] 10 , the AI ​​model management server 30b determines, via the AI ​​model combination unit 320, whether or not the edge device 10b will use multiple AI models 40. If the AI ​​model combination unit 320 determines that multiple AI models 40 will not be used, for example, that only one AI model 40 will be used (step S100, "No"), the AI ​​model combination unit 320 proceeds to step S104. Although not shown in the figure, at this time, the AI ​​model combination unit 320 may obtain the one AI model 40 to be used from the AI ​​model storage unit 310.

[0093] If it is determined in step S100 that multiple AI models 40 will be used (step S100, "Yes"), the AI ​​model combination unit 320 proceeds to step S101. In step S101, the AI ​​model combination unit 320 acquires multiple AI models 40 to be deployed to the edge device 10b from the AI ​​model storage unit 310.

[0094] In the next step S102, the AI ​​model management server 30b determines, using the AI ​​model combination unit 320, whether the deployment size after combining multiple AI models 40 is within the range of the memory capacity of the edge device 10b to be deployed.

[0095] For example, the memory calculation unit 322 acquires the data size of each AI model acquired in step S101, and calculates the data size of the combined AI model 41 when each AI model 40 is combined based on the acquired data size. Meanwhile, the AI ​​model combination unit 320 acquires the data capacity available for deploying the AI ​​model 40 in the volatile memory 13 of the edge device 10b. The AI ​​model combination unit 320 may acquire this data capacity by querying the edge device 10b, or may store it in advance. The AI ​​model combination unit 320 compares the data size of the combined AI model 41 calculated by the memory calculation unit 322 with the data capacity available for deploying the AI ​​model 40 in the edge device 10b.

[0096] In step S102, if the AI ​​model combination unit 320 determines that the expansion size after combining multiple AI models 40 is not within the range of the memory capacity of the edge device 10b to be deployed (step S102, "No"), it transitions the processing to step S104.

[0097] In this case, the AI ​​model management server 30b may, for example, notify the user that combining multiple AI models 40 cannot be performed, and prompt the user to re-specify the AI ​​model 40 to be used.

[0098] In step S102, if the AI ​​model combination unit 320 determines that the expansion size after combining multiple AI models 40 is within the range of the memory capacity of the edge device 10b to be deployed (step S102, "Yes"), it transitions the processing to step S103.

[0099] In step S103, the AI ​​model combination unit 320 combines the multiple AI models 40 acquired in step S101 to generate a combined AI model 41. At this time, the AI ​​model combination unit 320 adds management information for each of the multiple AI models 40 to be combined to the combined AI model 41.

[0100] In the next step S104, the AI ​​model management server 30b transmits, via the data transmission unit 330, the combined AI model 41 that was combined in step S103 and to which management information 42 has been added, or one AI model 40 that was determined to be used in step S100, to the edge device 10b, and expands it into the volatile memory 13 of the edge device 10b.

[0101] In the next step S105, the edge device 10b writes the combined AI model 41 or one AI model 40 deployed in the volatile memory 13 in step S104 into the non-volatile memory 14 as a deployment target.

[0102] FIG. 11 is a flowchart illustrating an example of processing when an AI model is used according to the first embodiment.

[0103] Prior to the processing according to the flowchart of Figure 11, processing is performed according to the flowchart of Figure 10 described above, and a combined AI model 41, which is obtained by combining multiple AI models 40 and adding management information 42, is written to the non-volatile memory 14 of the edge device 10b.

[0104] 11 , in step S200, the edge device 10b causes the AI ​​model control unit 101 to read the combined AI model 41 from the non-volatile memory 14 into the volatile memory 13. In the next step S201, the AI ​​model control unit 101 specifies one AI model 40 to be executed from the multiple AI models combined by the combined AI model 41, based on the management information 42 included in the combined AI model 41.

[0105] In the next step S202, the AI ​​model control unit 101 executes processing using the AI ​​model 40 specified in step S201 and the output results from the AI ​​model 40. In the next step S203, the AI ​​model control unit 101 determines whether the processing executed in step S202 satisfies a termination condition for terminating the processing. If the AI ​​model control unit 101 determines that the termination condition is satisfied (step S203, "Yes"), it terminates the series of processing steps according to the flowchart in FIG. 11.

[0106] On the other hand, if the AI ​​model control unit 101 determines in step S203 that the processing executed in step S202 does not satisfy the termination condition for terminating the processing (step S203, "No"), it transitions the processing to step S204.

[0107] In step S204, the AI ​​model control unit 101 determines whether to switch the AI ​​model 40 to be executed to another AI model 40 included in the combined AI model 41. For example, the AI ​​model control unit 101 may determine whether to switch the AI ​​model 40 to be executed based on an external instruction according to the output result of the AI ​​model 40 executed in step S202.

[0108] If the AI ​​model control unit 101 determines that switching is not to be performed (step S204, "No"), it returns the processing to step S202 and continues the processing by the AI ​​model 40 that is currently being executed.

[0109] On the other hand, if the AI ​​model control unit 101 determines in step S204 to switch the AI ​​model 40 to be executed (step S204, "Yes"), it transitions the process to step S205. In step S205, the AI ​​model control unit 101 specifies another AI model 40 to be executed, from among the AI ​​models 40 included in the combined AI model 41 loaded into the volatile memory 13 in step S200, based on the management information 42. After processing in step S205, the AI ​​model control unit 101 returns the process to step S202 and executes processing using the AI ​​model 40 specified in step S205.

[0110] 12 is a schematic diagram illustrating a data flow during deployment according to the first embodiment. In FIG. 12, the part above the dotted line shows processing by the AI ​​model management server 30b, and the part below the dotted line shows processing by the edge device 10b.

[0111] In FIG. 12, the AI ​​model management server 30b combines two or more AI models 401, 402, . . . using the AI ​​model combination unit 320 to generate a combined AI model 41.

[0112] When combining to generate the combined AI model 41, the AI ​​model combination unit 320 generates management information 42 for managing each of the combined AI models 40, 40, ..., and adds it to the combined AI model 41. The AI ​​model combination unit 320 further adds to the combined AI model 41 an encryption key 44 for encrypting the combined AI model 41, and a network ID 43 as identification information for identifying the combined AI model 41.

[0113] Note that one network ID is assigned to one combined AI model 41, but this is not limited to this. A network ID for identifying each of the multiple AI models 401, 402, ... included in the combined AI model 41 may be assigned to each of the AI ​​models 401, 402, ....

[0114] The AI ​​models 401, 402, ... may be stored together in a single file "network.fpk", for example. Fig. 13 is a schematic diagram showing an example of the configuration of the file "network.fpk" according to the first embodiment. In Fig. 13, the file "network.fpk" shown in section (a) contains, as shown in section (b), various programs, data and parameters, as well as the AI ​​models 401, 402, ..., 40, sandwiched between a header and a footer. N Each includes weight information.

[0115] In the file "network.fpk", various programs, data, and parameters are placed at the beginning of the file, followed by each AI model 401, 402, ..., 40 N Each element is arranged sequentially. In the example of FIG. 13, each AI model 40, 40, ..., 40 N Each element is, for example, each AI model 401, 402, ..., 40 N The weight information of each AI model 401, 402, ..., 40 N As each element, each AI model 401, 402, ..., 40 N The setting information (config information) may be included.

[0116] In the example of FIG. 13, each AI model 40 1 , 40 2 , . . . , 40 N In this example, the elements of the combined AI model 41 are arranged consecutively, but this is not limited to this example. For example, in consideration of updating a part of the combined AI model 41, each of the AI ​​models 40, 40, ..., 40 in the combined AI model 41 is arranged consecutively. N may be arranged separately.

[0117] As shown in Figure 13, by making various programs, data, and parameters common to each piece of weight information corresponding to each AI model, it is possible to reduce the file size of the file "network.fpk."

[0118] The management information 42 is stored as text information in the file "network_info.txt." FIG. 14 is a schematic diagram showing an example of the configuration of the file "network_info.txt" according to the first embodiment. For the sake of explanation, it is assumed here that the combined AI model 41 is generated by combining three AI models #1, #2, and #3. It is also assumed that each of the AI ​​models #1, #2, and #3 performs processing on an image captured by a sensor (camera).

[0119] 14, section (a) shows information common to each of the AI ​​models 401, 402, ..., and sections (b) to (d) show management information for each of the AI ​​models 401, 402, .... In the example of Fig. 14, management information 42 summarizes the management information for three AI models, AI models #1, #2, and #3, included in the combined AI model 41.

[0120] In sections (a) to (d) of Figure 14, the numbers on the left side conveniently indicate the line numbers in the file "network_info.txt." As can be seen from the line numbers, the information in sections (a) to (d) is written consecutively in the file "network_info.txt," starting with the common information in section (a).

[0121] In the example of Figure 14, the common information shown in section (a) includes a network ID as identification information that identifies the combined AI model 41 corresponding to this file "network_info.txt", a networkType that indicates the type of each AI model #1 to #3, a ParamSize that indicates the total size of the management information for each AI model #1 to #3, and a networkNum that indicates the number of AI models managed in the management information 42.

[0122] The management information for each of AI models #1 to #3 shown in sections (b) to (d) has the order of each AI model #1 to #3 described as networkOrdinal in the first line, the sensor setting values ​​for the input image input to that AI model described from the next line onwards, and information indicating the output size of the output data by that AI model described from the next line onwards.

[0123] In Figure 12, the AI ​​model combination unit 320 stores the file "network.fpk" that compiles each AI model 401, 402, ... and the file "network_info.txt" in which management information 42 is written in a single file "network.pkg" to generate a combined AI model 41.

[0124] The AI ​​model management server 30b transmits the combined AI model 41 generated by the AI ​​model combination unit 320 to the edge device 10b.

[0125] In the edge device 10b, the data processing device 100 writes the combined AI model 41 received from the AI ​​model management server 30b to a memory 130, which is, for example, a non-volatile memory, that the data processing device 100 has.

[0126] In the data processing device 100, the AI ​​model control unit 101 writes the file "network.fpk" included in the combined AI model 41 written to the memory 130, i.e., each of the AI ​​models 401, 402, ..., to the non-volatile memory 14. At this time, the AI ​​model control unit 101 writes each of the AI ​​models 401, 402, ... and their respective header information to a continuous area of ​​the non-volatile memory 14 so that they can be read at high speed.

[0127] Alternatively, if the available capacity of the non-volatile memory 14 is small and it is difficult to secure a continuous area of ​​a certain size, the AI ​​model control unit 101 may write data in a distributed manner to non-contiguous areas. In this case, the AI ​​model control unit 101 may create a management table that indicates the address and order in which each AI model 40, 40, ... and its respective header information are written in the non-volatile memory 14. The AI ​​model control unit 101 may read data from the non-volatile memory 14 to the volatile memory 13 in order by referring to this management table.

[0128] In addition, the AI ​​model control unit 101 may manage the network ID contained in the management information 42 by corresponding it to the address of the non-volatile memory 14 to which it is written, so that it can simultaneously have multiple combined AI models 41 or a single AI model.

[0129] The AI ​​model control unit 101 writes each of the AI ​​models 40, 40, ... written in the non-volatile memory 14 to the volatile memory 13. The AI ​​model control unit 101 executes detection processing, recognition processing, etc. using the AI ​​model to be executed among the AI ​​models 40, 40, ... on the volatile memory 13.

[0130] Meanwhile, in the data processing device 100, the management information control unit 102 extracts and manages the management information 42 included in the combined AI model 41 written in the memory 130. The management information control unit 102 manages, for example, information indicating the storage location of the combined AI model 41 in the non-volatile memory 14, a valid flag for the combined AI model 41, a network ID, and the update date and time of the combined AI model 41 in the memory 130 or in a non-volatile memory (not shown) of the data processing device 100.

[0131] Next, the processing according to the first embodiment will be described in comparison with processing according to existing technology using Figures 15 and 16. Here, the description will be given assuming that the edge devices 10a and 10b are cameras. In Figures 15 and 16, the passage of time is shown to the right, but the length of each process on the diagram does not necessarily represent the time required for the actual processing.

[0132] 15 and 16, the area above the dotted line indicates processing by the AI ​​model management server 30a or 30b, and the area below the dotted line indicates processing by the edge device 10a or 10b. In addition, in sections (a) and (b) of Figures 15 and 16, each process is indicated by different filled areas, according to the legend shown in the upper left of each of Figures 15 and 16: processing in response to a user's UI (User Interface) operation, memory processing in the edge device 10a or 10b, and internal processing in the edge device 10a or 10b or the AI ​​model management server 30a and 30b.

[0133] (Regarding Deployment Processing) FIG. 15 is a timing chart showing an example of processing when the AI ​​model 40 is deployed to the edge devices 10a and 10b.

[0134] 15, the upper section (a) shows processing according to the existing technology, and the lower section (b) shows processing according to the first embodiment. That is, section (a) of Fig. 15 shows an AI model management server 30a according to the existing technology deploying two AI models #1 and #2 (simply shown as model #1 and model #2 in the figure) to an edge device 10a according to the existing technology, and section (b) of Fig. 15 shows an AI model management server 30b according to the first embodiment deploying two AI models #1 and #2 (simply shown as model #1 and model #2 in the figure) to an edge device 10b according to the first embodiment.

[0135] 15, in response to an instruction from a user terminal 4, for example, the AI ​​model management server 30a based on the existing technology selects model #1 as the AI ​​model 40 to be deployed in step S300-1, and starts execution of the deployment in step S301-1.

[0136] In response to the start of deployment execution in step S301-1, the AI ​​model management server 30a packages AI model #1 in step S302-1, and in step S303-1 notifies the edge device 10a, which is a camera using existing technology, of the start of deployment and commands it to execute the deployment process.

[0137] In step S304-1, the edge device 10a receives the command notified in step S303-1, and in step S305-1, the edge device 10a executes pre-deployment processing in response to the received command.

[0138] The pre-processing performed here may include, for example, checking whether the edge device 10a already has the same version of AI model #1 as the AI ​​model #1 to be executed (written in non-volatile memory 14), changing the status of the edge device 10a (such as stopping other processes), writing header information to the non-volatile memory 14, calculating the received size, determining the address in the non-volatile memory 14 to which AI model #1 will be written, etc.

[0139] When the edge device 10a receives the command to start the deployment process in step S304-1, it acquires the AI ​​model #1 sequentially transmitted from the AI ​​model management server 30a in step S306-1. The edge device 10a sequentially writes the acquired AI model #1 to the volatile memory 13 (SRAM) (step S307-1), and sequentially writes the AI ​​model #1 written to the volatile memory 13 to the non-volatile memory 14 (step S308-1).

[0140] When the edge device 10a has finished acquiring AI model #1 and writing it to the non-volatile memory 14, in step S309-1, it sends the deployment result to the AI ​​model management server 30a. The AI ​​model management server 30a sends the deployment result sent from the edge device 10a to, for example, the user terminal 4, and reflects it in the UI (step S310-1).

[0141] For example, when the user is notified by the UI that the deployment of AI model #1 has been completed, the user instructs the AI ​​model management server 30a to deploy the next AI model #2. That is, similar to steps S300-1 and S301-1 described above, the AI ​​model management server 30a selects model #2 as the AI ​​model 40 to be deployed in step S300-2 in response to an instruction from the user terminal 4 by the user, and starts the execution of the deployment in step S301-2.

[0142] Thereafter, the deployment process for AI model #2 is executed in steps S302-2 to S310-2 in the same manner as the processes in steps S302-1 to S310-1 described above.

[0143] (Regarding the deployment process according to the first embodiment) Next, the deployment process according to the first embodiment will be described. In section (b) of FIG. 15 , the AI ​​model management server 30b according to the first embodiment selects AI models #1 and #2 as the AI ​​models 40 to be deployed in step S320 in response to, for example, an instruction from a user via the user terminal 4. At this time, the execution order of the selected AI models #1 and #2 is also specified. The AI ​​model management server 30b starts the execution of the deployment in step S321.

[0144] In response to the start of deployment execution in step S321, the AI ​​model management server 30b packages AI models #1 and #2 using a combination process 321 in step S322 to generate a combined AI model 41, and in step S323 notifies the edge device 10b, which is a camera according to the first embodiment, of the start of deployment and commands it to execute the deployment process.

[0145] In step S324, the edge device 10b receives the command notified in step S323. In step S325, the edge device 10b executes pre-deployment processing in response to the received command.

[0146] The pre-processing performed here is the same as the processing described in step S305-1 of section (a), and may include, for example, checking whether the edge device 10b already has AI model #1 or #2 of the same version as the target AI model #1 or #2 (written to the non-volatile memory 14), changing the status of the edge device 10b (such as stopping other processes), writing header information to the non-volatile memory 14, calculating the received size, determining the addresses in the non-volatile memory 14 to which AI models #1 and #2 are to be written, and the like.

[0147] Upon receiving the command to start the deployment process in step S324, the edge device 10b acquires the AI ​​models #1 and #2 sequentially transmitted from the AI ​​model management server 30b in step S326. The edge device 10b sequentially writes the acquired AI models #1 and #2 to the volatile memory 13 (SRAM) (step S327), and sequentially writes the AI ​​models #1 and #2 written to the volatile memory 13 to the non-volatile memory 14 (step S328).

[0148] When the edge device 10b has finished acquiring AI models #1 and #2 and writing them to the non-volatile memory 14, it sends the deployment result to the AI ​​model management server 30b in step S329. The AI ​​model management server 30b sends the deployment result sent from the edge device 10b to, for example, the user terminal 4, and reflects it in the UI (step S330).

[0149] (Comparison between Processing According to Existing Technology and Processing According to First Embodiment) In the first embodiment, as shown in section (b) of Fig. 15 , when deploying multiple AI models, the AI ​​model management server 30b collectively selects and packages the multiple AI models to be deployed to generate a combined AI model 41, and deploys this combined AI model 41 to the edge device 10b. The edge device 10b writes the combined AI model 41 sent from the AI ​​model management server 30b to the non-volatile memory 14 and reflects the result in the UI, thereby completing the deployment of the multiple AI models.

[0150] On the other hand, with existing technology, as described above, when deploying multiple AI models, it is necessary to select an AI model, package it, and perform memory processing in the edge device 10a for each AI model. This results in overlapping portions in the deployment processing for each AI model, and these overlapping portions become overhead for the deployment processing according to the first embodiment. In the example of section (a) of FIG. 15 , as shown enclosed by a dotted line, the processing of steps S300-2 to S305-2 for the second AI model #2 becomes processing overhead for the deployment processing according to the first embodiment shown in section (b).

[0151] In the first embodiment, this processing overhead does not occur, so the processing time required for deployment processing can be shortened compared to existing technologies, and the number of writes to non-volatile memory 14 can be reduced.

[0152] Furthermore, with existing technology, for example, execution of a deployment process for the first AI model #1 is instructed (step S301-1), completion of the deployment process for that AI model #1 is confirmed (step S310-1), and then execution of a deployment process for the next AI model #2 is instructed (steps S300-2 and S301-2). This requires the user to monitor the deployment results presented by the UI on the user terminal 4, for example, and this may require time for user operations for the next deployment process. This may impose a heavy burden on the user if, for example, the number of edge devices 10a to be deployed increases.

[0153] In contrast, in the first embodiment, the deployment process can be executed for multiple AI models at once, eliminating waiting time for the user and reducing the burden on the user.

[0154] 16 is a timing chart showing an example of processing when the edge devices 10 a and 10 b execute processing using an AI model. In FIG. 16, the upper section (a) shows processing using existing technology, and the lower section (b) shows processing according to the first embodiment.

[0155] 16 shows edge devices 10a and 10b with AI models #1 and #2 already written to non-volatile memory 14. In edge device 10b, a combined AI model 41 in which AI models #1 and #2 are combined is written to non-volatile memory 14. Furthermore, processing by AI model #1 and processing by AI model #2 are executed serially, and AI model #2 executes processing using the processing results of AI model #1.

[0156] (Regarding Execution Processing Using Existing Technology) First, the execution processing using existing technology will be described. In section (a) of Figure 16, in step S400-1, for example, the user terminal 4 specifies AI model #1 in response to a user operation and sets parameters for performing inference using the AI ​​model #1. Next, in response to the user operation, for example, the user terminal 4 instructs the AI ​​model management server 30a using existing technology to perform inference using the AI ​​model #1 specified in step S400-1 and the set inference parameters, and transmits information indicating the AI ​​model #1 and the parameters to the AI ​​model management server 30a (step S401-1).

[0157] The AI ​​model management server 30a notifies the edge device 10a, which is a camera based on existing technology, of a command corresponding to this inference execution instruction, along with information indicating AI model #1 and inference parameters (step S402-1).

[0158] In step S403-1, the edge device 10a receives the command notified in step S402-1. In step S404-1, the edge device 10a performs initial settings of the edge device 10a in response to the received command. For example, in this example where the edge device 10a is a camera, the edge device 10a performs initial settings of various parameters such as image quality parameters, drive modes, and crop settings.

[0159] FIG. 17 is a schematic diagram showing an example of a command notified from the AI ​​model management server 30a to the edge device 10a in step S402-1 and step S402-2 (described later). Note that in FIG. 17, the numbers on the left side conveniently indicate the line number of each line of the command. The second line states that the data is a command, and the fourth line states the name of the command. In the example of FIG. 17, the fifth and subsequent lines describe various parameters for the edge device 10a (camera), and the sixth line describes identification information for identifying the AI ​​model to be executed by the edge device 10a.

[0160] The edge device 10a reads the AI ​​model #1 written in the non-volatile memory 14 (step S405-1) and writes the read AI model #1 to the volatile memory 13 (step S406-1). The edge device 10a, for example, uses the AI ​​model control unit 101 to execute a streaming start process for the captured image captured by the edge device 10a as a camera, using the AI ​​model #1 written in the volatile memory 13 (step S407-1).

[0161] In response to the streaming start process, the edge device 10a executes image processing, for example, in the data processing device 1008 (ISP) (step S408-1), and executes processing by a deep neural network (DNN) using AI model #1 on the image processing results (step S409-1). This DNN processing may be, for example, detection processing or recognition processing based on the processed image.

[0162] The results of the DNN processing are transmitted and received within the edge device 10a (steps S410-1 and S411-1) and subjected to predetermined post-processing (step S412-1). The results of the post-processing are stored in the edge device 10a for processing by AI model #2, for example. Furthermore, when the post-processing is completed, the edge device 10a notifies the AI ​​model management server 30a. The AI ​​model management server 30a may transmit a notification indicating the completion of the post-processing sent from the edge device 10a to the user terminal 4 and reflect the notification in the UI (not shown).

[0163] For example, when the user is notified via the UI that processing using AI model #1 has ended, the user instructs the AI ​​model management server 30a to perform processing using the next AI model #2. That is, similar to steps S400-1 and S401-1 described above, the AI ​​model management server 30a specifies AI model #2 in response to a user instruction from the user terminal 4 and sets parameters for performing inference using AI model #2 (step S400-2). Next, for example, the user terminal 4, in response to a user operation, instructs the AI ​​model management server 30a to perform inference using the set inference parameters using the AI ​​model #2 specified in step S400-2, and transmits information indicating AI model #2 and the parameters to the AI ​​model management server 30a (step S401-2).

[0164] The AI ​​model management server 30a notifies the camera side of a command corresponding to this inference execution instruction, together with information indicating AI model #2 and the inference parameters (step S402-2).

[0165] Thereafter, in the same manner as in steps S403-1 to S412-1 described above, processing using AI model #2 is performed using the processing results from AI model #1 in steps S403-2 to S412-2. In step S413, the edge device 10a transmits (uploads) the results of post-processing for the processing by AI model #2 to the AI ​​model management server 30a. The AI ​​model management server 30a may transmit the results of post-processing transmitted from the edge device 10a to, for example, the user terminal 4, and reflect them in a UI (not shown).

[0166] (Regarding the Execution Process According to the First Embodiment) Next, the execution process according to the first embodiment will be described. In section (b) of FIG. 16 , in step S420, for example, the user terminal 4 specifies AI models #1 and #2 in response to a user operation and sets parameters for performing inference using the AI ​​models #1 and #2, respectively. Next, for example, the user terminal 4 instructs the AI ​​model management server 30b according to the first embodiment to perform inference using the AI ​​models #1 and #2 specified in step S420 based on the set inference parameters, in response to the user operation, and transmits information indicating the AI ​​models #1 and #2 and the parameters to the AI ​​model management server 30b (step S421).

[0167] The AI ​​model management server 30b notifies the edge device 10b according to the first embodiment, which is a camera, of a command corresponding to the inference execution instruction, together with information indicating the AI ​​models #1 and #2 and their corresponding inference parameters (step S422). The command notified at this time may have the data structure described with reference to FIG. 17.

[0168] In step S423, the edge device 10b receives the command notified in step S422. In step S424, the edge device 10b performs initial settings of the edge device 10b in response to the received command. For example, in this example where the edge device 10b is a camera, the edge device 10b performs initial settings of various parameters such as image quality parameters, drive modes, and crop settings.

[0169] The edge device 10b reads out the AI ​​models #1 and #2 written in the non-volatile memory 14 (step S425), and writes the read out AI models #1 and #2 to the volatile memory 13 (step S426). At this time, the edge device 10b may also collectively set up registers for the AI ​​models #1 and #2 based on the parameters related to the AI ​​models #1 and #2, for example, based on the management information.

[0170] The edge device 10b, for example, uses AI model #1 of AI models #1 and #2 written to the volatile memory 13 by the AI ​​model control unit 101 to perform streaming start processing for the captured image captured by the edge device 10b as a camera (step S427-1).

[0171] In response to the streaming start process, the edge device 10b executes image processing, for example, in the data processing device 1008 (ISP) (step S428-1), and executes DNN processing using AI model #1 on the image processing results (step S429-1). This DNN processing may be, for example, detection processing or recognition processing based on the processed image.

[0172] The results of the DNN processing are transmitted and received within the edge device 10b (steps S430-1 and S431-1), and are subjected to predetermined post-processing (step S432-1). After the post-processing is completed, the edge device 10b, for example, uses the results of the post-processing of AI model #1 via the AI ​​model control unit 101 to execute streaming start processing using AI model #2 (step S427-2).

[0173] In response to the streaming start process, the edge device 10b performs image processing, for example, in the data processing device 1008 (ISP) (step S428-2), and performs processing using DNN using AI model #1 on the image processing results (step S429-2).

[0174] The results of the DNN processing are transmitted and received within the edge device 10b (steps S430-2 and S431-2) and subjected to predetermined post-processing (step S432-2). The results of the post-processing are transmitted from the edge device 10b to the AI ​​model management server 30b (uploaded) in step S433. The AI ​​model management server 30b may transmit the results of the post-processing transmitted from the edge device 10b to, for example, the user terminal 4, and reflect them in a UI (not shown).

[0175] (Comparison between Processing According to Existing Technology and Processing According to First Embodiment) In the first embodiment, as described using section (b) of Fig. 16, when processing using multiple AI models is executed, the AI ​​model management server 30b specifies inference parameters and the like and executes an inference execution instruction for the multiple AI models to be executed collectively. In response to the inference execution instruction from the AI ​​model management server 30b, the edge device 10b reads the multiple AI models collectively from the non-volatile memory 14, writes them to the volatile memory 13, and continuously executes processing using the multiple AI models.

[0176] On the other hand, with existing technology, as explained using section (a) of Figure 16, when executing processing using multiple AI models, it is necessary to specify inference parameters and execute inference execution instructions for each AI model. As a result, overlapping processing occurs when executing each AI model, and this overlapping processing becomes overhead for the deployment processing according to the first embodiment. In the example of section (a) of Figure 16, as shown surrounded by a dotted line, the processing of steps S400-2 to S406-2 for the second AI model #2 becomes processing overhead for the deployment processing according to the first embodiment shown in section (b).

[0177] In the first embodiment, this processing overhead does not occur, so the time required for execution by multiple AI models can be shortened compared to existing technologies, and the number of times data is written to non-volatile memory 14 can be reduced.

[0178] Furthermore, in the existing technology, as described using section (a) of FIG. 16 , for example, an instruction to execute inference for the first AI model #1 (step S401-1) is given, followed by post-processing of the processing results by AI model #1 (step S412-1), and then an instruction to execute inference for the next AI model #2 (steps S400-2 and S401-2). Therefore, the user must monitor the completion of processing by AI model #1, which is presented by the UI on the user terminal 4, for example, and the user operation to instruct inference execution by the next AI model #2 may require time. This may impose a heavy burden on the user if, for example, the number of edge devices 10a to be executed increases.

[0179] In contrast, in the first embodiment, as explained using section (b) of Figure 16, processing by multiple AI models can be executed with a single inference execution instruction, so there is no waiting time for the user and it is possible to reduce the burden on the user.

[0180] In the above description, in the execution process according to the first embodiment of section (b) of FIG. 16, when the streaming process and post-processing by AI model #1 are completed, the streaming process by the next AI model #2 is continuously executed. However, the execution timing of AI model #2 is not limited to this example. For example, after the processing by AI model #1 is completed, the processing may be temporarily handed over to a user program (e.g., the AI ​​processing unit 50 described in FIG. 6), and processing by AI model #2 (e.g., streaming processing) may be executed based on the output result of AI model #1. Furthermore, after the processing by AI model #1 is completed, the processing may be temporarily returned to the user program, and the processing by AI model #2 may be omitted, and processing by another AI model (e.g., AI model #3) may be executed.

[0181] (2-4. Method for specifying / separating multiple AI models according to the first embodiment) Next, a method for specifying / separating multiple AI models according to the first embodiment will be described. FIG. 18 is a schematic diagram for explaining a method for specifying the order of multiple AI models according to the first embodiment. In the example of FIG. 18, the order of multiple AI models is specified by description in a specified file.

[0182] In Figure 18, the numbers on the left side conveniently indicate the line numbers in the description of the order of multiple AI models. In this example, lines 4 to 6 and lines 8 to 10 each describe information indicating an AI model. Lines 4 and 5, and lines 8 and 9 each describe the name of the AI ​​model and its storage location using an absolute path. Lines 6 and 10 each describe the order of the AI ​​models.

[0183] More specifically, the description "ordinal":0 on line 6 and the description "ordinal":1 on line 10 indicate that the AI ​​model described on lines 4 to 6 is the first, and the AI ​​model described on lines 8 to 10 is the second. When executed serially, the AI ​​models may be executed in this order. The description shown in FIG. 18 may be included in, for example, management information 42 (network_info.txt).

[0184] In the description shown in Figure 18, for example, by changing the value of "ordinal" and repeatedly inserting the same description from "{" on the third line to "}" on the seventh line from the eleventh line onwards, it is possible to set the order for more than two AI models.

[0185] FIG. 19 is a schematic diagram showing an example of the configuration of the file "network.fpk" according to the first embodiment.

[0186] 19, section (a) shows an example list of files stored in the combined AI model 41 (network.pkg). Of this list, the file "I2staticData.bin" stores weight information for the AI ​​model. The file "deployManifest.json" is a management table that describes, for example, the address and size of each AI model included in the combined AI model 41.

[0187] In the list of section (a), the files "ap_params.bin", "cfgA.bin", "cfgC.bin", "dnn_params.bin", "sdpsA.bin" and "sdpsC.bin" store various programs, data, parameters and the like.

[0188] Section (b) of Figure 19 shows an example of the memory area cfgBank. The memory area cfgBank is a predetermined area in a working memory (e.g., SRAM) used by, for example, the AI ​​model combination unit 320. The memory area cfgBank writes input data, output data, and feature maps (intermediate results) as data common to multiple AI models, as well as weight information (Weights N#1, N#2, ...) for each of the multiple AI models. The header and footer described in section (b) of Figure 13 are removed from each piece of weight information, and the weight information is written to consecutive addresses in the memory area cfgBank.

[0189] Section (c) of Figure 19 shows an example of the data structure in the memory area cfgBank. Note that in this figure, the numbers on the left side conveniently indicate the line numbers in the figure. In this example, lines 8 to 14 show information about each weight information, line 9 shows the read address of that information in the memory area cfgBank, line 10 shows the file name "I2staticData.bin", line 11 shows the verification hash function (md5), and line 12 shows version information. Furthermore, line 13 shows the total size of each weight information in the memory area cfgBank.

[0190] FIG. 20 is a schematic diagram for explaining a method for separating and selecting multiple AI models in the edge device 10b according to the first embodiment.

[0191] In the example of FIG. 20 , the edge device 10b, for example, in the data processing device 100, issues a command DNN_EXEC_NETWORK to instruct the execution of processing by AI model #1 at the timing of the vertical synchronization signal VSYNC at the beginning of the nth frame (Frame #n) in accordance with the imaging operation of the camera. Furthermore, at the timing of the vertical synchronization signal VSYNC, the captured image of the nth frame is read, and image processing is performed on the read image signal by an ISP or the like. Once the image processing is completed, processing using AI model #1 is performed. Once this processing is completed, the processing results are transferred as output data. One frame period is defined as the time from the timing of the vertical synchronization signal VSYNC to the completion of output data transfer.

[0192] A command DNN_EXEC_NETWORK is issued at the timing of the vertical synchronization signal VSYNC at the beginning of the next frame (n+1) (Frame #n+1), instructing execution of processing by AI model #2. Thereafter, in the same manner as described above, image processing is performed on the captured image of the frame (n+1), and processing by AI model #2 is performed on the image processing results.

[0193] In this way, in the first embodiment, the command DNN_EXEC_NETWORK according to the timing of the vertical synchronization signal VSYNC may be used to separate and select the AI ​​model to be executed within the period of the vertical synchronization signal VSYNC.

[0194] Here, for example, before the streaming start process in step S427-1 shown in section (b) of Figure 16, a combined AI model 41 including multiple AI models is written to the volatile memory 13. The edge device 10b, for example, has registers for multiple AI models, and can specify normalization parameters for each AI model included in the combined AI model. Furthermore, when an AI model is specified by an ordinal value from a user program, the ordinal value can be passed as an argument to, for example, the command DNN_EXEC_NETWORK, thereby enabling inference processing and the like to be performed using the AI ​​model specified by the ordinal value.

[0195] (Specific Example of Setting the Order of Multiple AI Models) Next, a more specific description will be given of a method for setting the order of multiple AI models according to the first embodiment. Note that the order of multiple AI models here refers to the order in which the multiple AI models are executed on the edge device 10b.

[0196] There are three possible methods for setting the order of multiple AI models: (1) The user defines the order of multiple AI models on a UI screen and imports the multiple AI models; (2) The user collects multiple AI models into a single archive file and uploads it to the AI ​​model management server 30b; and (3) The user defines the order of multiple AI models using a manifest file.

[0197] First, (1) a method for setting the order of multiple AI models included in the combined AI model 41 on the UI screen will be described.

[0198] 21 is a schematic diagram showing an example of a screen by a UI for generating a combined AI model 41 according to the first embodiment. The UI screen 500 shown in Fig. 21 is generated by the AI ​​model combination unit 320 in, for example, the AI ​​model management server 30b, and is presented to the user using, for example, a browser application on the user terminal 4. Without being limited to this, the UI screen 500 may be presented to the user according to parameters acquired from the AI ​​model management server 30b by, for example, an application program installed on the user terminal 4.

[0199] 21 , UI screen 500 includes a menu area 510 for selecting a function and a command area 511 in which a command corresponding to the function selected in menu area 510 is displayed. In the example of FIG. 21 , the function "Create model" is selected in menu area 510, and in response to the selection of this function, a command "Import" is specified in command area 511, thereby displaying model setting screen 520. Note that the screen configuration of UI screen 500 is not limited to this example, and other configurations may also be used.

[0200] On the model setting screen 520, the input field "Model ID" is an area for inputting identification information of the AI ​​model to be imported into the edge device 10b, and the input field "Description" is an area for inputting a brief description of the AI ​​model.

[0201] In the model setting screen 520, an area 521 includes input areas for inputting information about the actual state of the AI ​​model to be imported. More specifically, the area 521 includes a designation section for designating whether the AI ​​model is to be imported from an address designated on the communication network 2 or in a local environment. In the example of FIG. 21 , the designation section designates a method for importing the AI ​​model from an address designated on the cloud network 3.

[0202] In addition, area 521 is provided with an input area for inputting a URL (Uniform Resource Locator) as an address to specify the AI ​​model on the cloud network 3, and an input area for uploading the AI ​​model in the local environment (e.g., user terminal 4) where the UI screen 500 is presented.

[0203] In the example of FIG. 21, the model setting screen 520 further includes an input area for inputting information such as the import source of the format parameters, and an input area for inputting the vendor name.

[0204] After entering the necessary information on the model setting screen 520, you can enter information about the next AI model to be imported by operating the "Next" button at the bottom.

[0205] In the method (2) of uploading multiple AI models included in the combined AI model 41 together in a single archive file, a method of specifying the order of each AI model is required. One possible method of specifying the order of each AI model is to include an index file describing the order of each AI model with the multiple AI models and archive them. Another possible method is to define the file names of each of the multiple AI models according to a naming rule that indicates a predetermined order (numerical order, alphabetical order, etc.).

[0206] An example of using a manifest file (3) will be described. A manifest file is generally used to define the basic components of an application. In the method (3), this manifest file is used to define the order of multiple AI models included in the combined AI model 41.

[0207] Fig. 22 is a schematic diagram showing an example of a manifest file for defining the order of multiple AI models, which is applicable to the first embodiment. Note that in Fig. 22, the numbers on the left side conveniently indicate the line numbers of each line shown in Fig. 22.

[0208] 22, the second line describes, as "modelName," information that identifies a single AI model that has been registered in, for example, the AI ​​model management server 30b (stored in the AI ​​model storage unit 310), such as the AI ​​model name. The third to fifth lines describe identification information (networkId) that identifies the combined AI model 41 that includes the AI ​​model described in the manifest file, and the version and metadata of the combined AI model 41.

[0209] In this data structure, "modelName" on the second line is written for each of the multiple AI models included in the combined AI model 41, and the models are added in order. For example, the AI ​​model combination unit 320 can obtain the number of AI models to be included in the combined AI model 41 and their names by looking at the manifest file from the second line to the line immediately before where "networkId" is written. The AI ​​model combination unit 320 can generate the combined AI model 41 based on this information obtained from the manifest file.

[0210] It is preferable to create the manifest file in advance, but the manifest file may also be created by the user manually entering the necessary information, or may be created from the user terminal 4 using a UI screen such as that shown in Figure 21.

[0211] (2-5. Modification of the First Embodiment) Next, a modification of the first embodiment will be described.

[0212] In the above description, the multiple AI models combined in the combined AI model 41 are described as being used in series and succession, or in parallel, but this is not limited to this example. Ensemble learning is known, in which multiple low-precision AI models are used to perform inference, and the inference results of each AI model are combined to obtain a highly accurate inference result. A modified example of the first embodiment of the present disclosure is an example in which the technology according to the first embodiment is applied to this ensemble learning.

[0213] 23 is a schematic diagram for explaining ensemble learning. The general machine learning model and the machine learning model using ensemble learning share the same overall processing flow: data is input into the model, inference processing is performed according to the model's algorithm, and a classification or regression result is output.

[0214] In a typical machine learning model, input data is input to, for example, a machine learning model that has been trained with high accuracy, and inference processing is performed to obtain a final prediction result.

[0215] On the other hand, in ensemble learning, input data is input to multiple AI models (1), (2), ..., (N) that have been trained with low accuracy, and inference processing is performed in each of the AI ​​models (1), (2), ..., (N). The prediction results (1), (2), ..., (N) obtained by the inference processing of each AI model (1), (2), ..., (N) are integrated or compared to obtain the final prediction result.

[0216] In a modification of the first embodiment, the AI ​​model management server 30b combines the AI ​​models (1), (2), ..., (N) used in ensemble learning using the method described in the first embodiment to generate a combined AI model 41. The AI ​​model management server 30b imports the combined AI model 41 thus generated into, for example, the edge device 10b. As described in the first embodiment, the edge device 10b writes the imported combined AI model 41 to the non-volatile memory 14 and reads it into the volatile memory 13 at runtime. The edge device 10b executes processing using each of the AI ​​models (1), (2), ..., (N) included in the combined AI model 41 read into the volatile memory 13, integrates or compares the results of each processing, and obtains a final prediction result.

[0217] The processes by each of the AI ​​models (1), (2), ..., (N) may be executed in parallel or at separate times.

[0218] By applying the technology related to the modified example of the first embodiment to the execution of ensemble learning, it is possible to reduce the number of writes to the non-volatile memory 14 of the edge device 10b, for example, when the number of low-precision AI models (1), (2), ..., (N) used in ensemble learning increases.

[0219] Note that, although two techniques, broadly classified as bagging and boosting, are known as ensemble learning techniques, the modified example of the first embodiment can be applied to either of them. Also, a method called stacking, in which processing using an AI model is performed hierarchically, is known as an application of ensemble learning, and the modified example of the first embodiment can be applied to this stacking as well.

[0220] (3. Second Embodiment of the Present Disclosure) Next, a second embodiment of the present disclosure will be described.

[0221] A second embodiment of the present disclosure is an example of applying, as an edge device, a CIS (CMOS (Complementary Metal Oxide Semiconductor) Image Sensor) in which an image sensor including a pixel array and a signal processing unit that performs predetermined signal processing such as image processing on an image captured by the image sensor are configured on a single chip.

[0222] (3-1. Configuration According to Second Embodiment) A configuration according to the second embodiment will be described. Fig. 24 is a functional block diagram for schematically explaining the functions of an edge device according to the second embodiment.

[0223] 24, the edge device 10c according to the second embodiment includes an imaging device 2100, a data processing unit 2200, a non-volatile memory 2201, an output unit 2202, and a receiving unit 2203.

[0224] The imaging device 2100 is the CIS described above and includes an imaging unit 2010, a volatile memory 2023a, and a non-volatile memory 2023b. Of these, the volatile memory 2023a and the non-volatile memory 2023b correspond to the volatile memory 13 and the non-volatile memory 14 shown in FIG. 3, etc., respectively. The imaging unit 2010 includes an imaging element and a drive circuit for the imaging element. The signal processing unit described above is omitted from FIG. 24.

[0225] The data processing unit 2200 includes a CPU, a memory, and the like, and performs predetermined data processing on the output of the image capturing device 2100. The data processing unit 2200 also outputs a control signal that controls the operation of the image capturing device 2100.

[0226] The receiving unit 2203 receives the combined AI model 41 transmitted from the AI ​​model management server 30b via the communication network 2. The data processing unit 2200 may write the combined AI model 41 received by the receiving unit 2203 to the non-volatile memory 2201. The non-volatile memory 2201 may also store programs and data for operating the data processing unit 2200. The output unit 2202 is an interface that enables the edge device 10c to communicate with external devices, and the output of the data processing unit 2200 is transmitted from the output unit 2202 to the outside.

[0227] The data processing unit 2200 in Fig. 24 can include the functions of the management information control unit 102 and the device control unit 103 in Fig. 7. The imaging device 2100 in Fig. 24 can include the functions of the AI ​​model control unit 101 in Fig. 7.

[0228] Fig. 25 is a block diagram showing an example of the configuration of an image pickup device 2100 applicable to the second embodiment. In Fig. 25, the image pickup device 2100 is configured as a CIS as described above, and includes an image pickup unit 2010 and a signal processing unit 2020. The image pickup unit 2010 and the signal processing unit 2020 are electrically connected by connection lines CL1, CL2, and CL3, which are internal buses, respectively.

[0229] The imaging unit 2010 has an imaging section 2011, an imaging processing section 2012, an output control section 2013, an output I / F 2014, and an imaging control section 2015, and captures an image of a subject to obtain a captured image.

[0230] The imaging unit 2011 includes a pixel array in which a plurality of pixels, which are light receiving elements that output signals according to received light through photoelectric conversion, are arranged in a matrix. The imaging unit 2011 is driven by the imaging processing unit 2012 to capture an image of a subject.

[0231] That is, light from an optical system (not shown) is incident on the imaging unit 2011. The imaging unit 2011 receives the incident light from the optical system at each pixel included in the pixel array, performs photoelectric conversion, and outputs an analog image signal corresponding to the incident light.

[0232] The size of the image generated by the image signal output by the imaging unit 2011 can be selected from a plurality of sizes, such as width x height, 3968 pixels x 2976 pixels, 1920 pixels x 1080 pixels, and 640 pixels x 480 pixels. The image sizes that can be output by the imaging unit 2011 are not limited to these examples. Furthermore, the image output by the imaging unit 2011 can be selected to be, for example, a color image of RGB (red, green, blue) or a black-and-white image of luminance only. These selections for the imaging unit 2011 can be made as one type of shooting mode setting.

[0233] Note that information based on the output of each pixel arranged in a matrix in the pixel array is called a frame. The imaging device 2100 repeatedly acquires information on the matrix of pixels in chronological order at a predetermined rate (frame rate) in the imaging unit 2011. The imaging device 2100 outputs the acquired information in a frame-by-frame manner.

[0234] The imaging processing unit 2012 performs imaging processing related to capturing an image in the imaging unit 2011, such as driving the imaging unit 2011, AD (Analog to Digital) conversion of the analog image signal output by the imaging unit 2011, and imaging signal processing, under the control of the imaging control unit 2015.

[0235] The imaging signal processing performed by the imaging processing unit 2012 includes, for example, a process of calculating the brightness of each small region of the image output by the imaging unit 2011 by calculating the average pixel value for each predetermined small region, an HDR (High Dynamic Range) conversion process of converting the image output by the imaging unit 2011 into an HDR image, defect correction, development, etc.

[0236] The imaging processing unit 2012 outputs, as a captured image, a digital image signal obtained by AD conversion or the like of the analog image signal output by the imaging unit 2011. The imaging processing unit 2012 can also output, as a captured image, a RAW image that has not been subjected to processing such as development. Note that an image in which each pixel has information on each of the RGB colors, obtained by subjecting a RAW image to processing such as development, is called an RGB image.

[0237] The captured image output by the image capture processing unit 2012 is supplied to the output control unit 2013 and also to the image compression unit 2025 of the signal processing unit 2020 via a connection line CL2.

[0238] The output control unit 2013 is supplied with the captured image from the imaging processing unit 2012, and also with the signal processing results of signal processing using the captured image and the like from the signal processing unit 2020 via a connection line CL3.

[0239] The output control unit 2013 performs output control to selectively output the captured image from the imaging processing unit 2012 and the signal processing result from the signal processing unit 2020 from (one) output I / F 2014 to the data processing unit 2200. In other words, the output control unit 2013 selects the captured image from the imaging processing unit 2012 or the signal processing result from the signal processing unit 2020 and supplies it to the output I / F 2014.

[0240] The output I / F 2014 is an I / F that outputs the captured image and the signal processing results supplied from the output control unit 2013 to the outside. As the output I / F 2014, for example, a relatively high-speed parallel I / F such as a Mobile Industry Processor Interface (MIPI) can be adopted.

[0241] The output I / F 2014 outputs to the outside the captured image from the imaging processing unit 2012 or the signal processing result from the signal processing unit 2020 in accordance with the output control of the output control unit 2013. Therefore, for example, when only the signal processing result from the signal processing unit 2020 is needed outside and the captured image itself is not needed, it is possible to output only the signal processing result, and it is possible to reduce the amount of data output from the output I / F 2014 to the outside.

[0242] Furthermore, the signal processing unit 2020 performs signal processing that produces the signal processing results required externally, and outputs the signal processing results from the output I / F 2014, thereby eliminating the need to perform signal processing externally and reducing the load on external blocks.

[0243] The imaging control unit 2015 includes a communication I / F 2016 and a register group 2017 .

[0244] The communication I / F 2016 is, for example, 2 The first communication I / F is a serial communication I / F such as an Inter-Integrated Circuit (C), and exchanges necessary information, such as information to be read and written to the register group 2017, with the outside (for example, a control unit that controls the device in which this imaging device 2100 is installed).

[0245] The register group 2017 has a plurality of registers and stores various types of information, such as imaging information related to image capture by the imaging unit 2011. For example, the register group 2017 stores imaging information received from the outside via the communication I / F 2016 and the results of imaging signal processing by the imaging processing unit 2012 (e.g., brightness of each small region of the captured image).

[0246] The imaging information stored in the register group 2017 includes, for example, ISO sensitivity (analog gain during AD conversion in the imaging processing unit 2012), exposure time (shutter speed), frame rate, focus, shooting mode, cropping range, etc. (information representing the same).

[0247] The shooting modes include a manual mode in which the exposure time, frame rate, etc. are manually set, and an automatic mode in which the settings are automatically set according to the scene. The automatic mode includes modes according to various shooting scenes, such as night scenes and human faces.

[0248] The cropping range refers to a range cropped from the image output by the imaging unit 2011 when the imaging processing unit 2012 crops a portion of the image output by the imaging unit 2011 and outputs the cropped image. By specifying the cropping range, it becomes possible to crop, for example, only a range in which a person appears from the image output by the imaging unit 2011. In addition to a method of cropping an image from the image output by the imaging unit 2011, there is also a method of reading out only the image (signal) of the cropping range from the imaging unit 2011.

[0249] The imaging control unit 2015 controls the imaging processing unit 2012 in accordance with the imaging information stored in the register group 2017 , thereby controlling the imaging of images by the imaging unit 2011 .

[0250] The register group 2017 can store imaging information, the results of imaging signal processing in the imaging processing unit 2012, and output control information related to output control in the output control unit 2013. The output control unit 2013 can perform output control to selectively output the captured image and the signal processing results in accordance with the output control information stored in the register group 2017.

[0251] Furthermore, in the imaging device 2100, the imaging control unit 2015 and the CPU 2021 of the signal processing unit 2020 are connected via a connection line CL1, and the CPU 2021 can read and write information from and to the register group 2017 via the connection line CL1. That is, in the imaging device 2100, reading and writing of information from and to the register group 2017 can be performed not only from the communication I / F 2016 but also from the CPU 2021.

[0252] The signal processing unit 2020 has a CPU 2021, a DSP (Digital Signal Processor) 2022, a memory 2023, a communication I / F 2024, an image compression unit 2025, and an input I / F 2026, and performs predetermined signal processing using captured images obtained by the imaging unit 2010. Note that the CPU 2021 is not limited to this, and may be an MPU (Micro Processor Unit) or an MCU (Micro Controller Unit).

[0253] The CPU 2021, DSP 2022, memory 2023, communication I / F 2024, and input I / F 2026 that constitute the signal processing unit 2020 are interconnected via a bus, and can exchange information as necessary.

[0254] The CPU 2021 executes programs stored in the memory 2023 to control the signal processing unit 2020, read and write information from and to the register group 2017 of the imaging control unit 2015 via the connection line CL1, and perform various other processes.

[0255] For example, by executing a program, the CPU 2021 functions as an imaging information calculation unit that calculates imaging information using the signal processing results obtained by signal processing in the DSP 2022, and feeds back and stores new imaging information calculated using the signal processing results to the register group 2017 of the imaging control unit 2015 via the connection line CL1.

[0256] Therefore, the CPU 2021 can ultimately control the image capturing in the image capturing unit 2011 and the image capturing signal processing in the image capturing processing unit 2012 according to the signal processing results of the captured image.

[0257] Furthermore, the imaging information stored in the register group 2017 by the CPU 2021 can be provided (output) to the outside from the communication I / F 2016. For example, focus information included in the imaging information stored in the register group 2017 can be provided from the communication I / F 2016 to a focus driver (not shown) that controls focus.

[0258] By executing a program stored in memory 2023, DSP 2022 functions as a signal processing unit that performs signal processing using the captured image supplied from the imaging processing unit 2012 to the signal processing unit 2020 via connection line CL2 and information received from the outside by input I / F 2026.

[0259] The memory 2023 is configured by a static random access memory (SRAM) or a dynamic RAM (DRAM), and stores data necessary for processing by the signal processing unit 2020.

[0260] For example, the memory 2023 stores programs received from outside via the communication I / F 2024, captured images compressed by the image compression unit 2025 and used for signal processing in the DSP 2022, the signal processing results of the signal processing performed by the DSP 2022, information received by the input I / F 2026, etc.

[0261] Furthermore, the memory 2023 may include the volatile memory 2023a and the non-volatile memory 2023b shown in Fig. 24. However, the volatile memory 2023a and the non-volatile memory 2023b may be connected to the bus as independent storage elements.

[0262] The communication I / F 2024 is a second communication I / F, such as a serial communication I / F such as an SPI (Serial Peripheral Interface), and exchanges necessary information such as programs executed by the CPU 2021 and DSP 2022 with the outside (for example, a memory or control unit not shown).

[0263] For example, the communication I / F 2024 downloads programs to be executed by the CPU 2021 and the DSP 2022 from an external device, and supplies and stores the programs in the memory 2023. Therefore, the programs downloaded by the communication I / F 2024 enable the CPU 2021 and the DSP 2022 to execute various processes.

[0264] The communication I / F 2024 can exchange any data, in addition to programs, with the outside. For example, the communication I / F 2024 can output to the outside the signal processing results obtained by the signal processing in the DSP 2022. The communication I / F 2024 can also output information according to instructions from the CPU 2021 to an external device, thereby controlling the external device according to instructions from the CPU 2021.

[0265] 9, the communication I / F 2024 may acquire, for example, a network distributed from the AI ​​model management server 30 and stored in the RAM 1002 or the storage device 1003 by communication with the bus 1010. The network acquired by the communication I / F 2024 may be stored, for example, in the memory 2023.

[0266] The signal processing results obtained by the signal processing in the DSP 2022 can be output to the outside from the communication I / F 2024, and can also be written to the register group 2017 of the imaging control unit 2015 by the CPU 2021. The signal processing results written to the register group 2017 can be output to the outside from the communication I / F 2016. The same applies to the processing results performed by the CPU 2021.

[0267] At least one of the CPU 2021 and the DSP 2022 may execute processing using an AI model.

[0268] The captured image is supplied from the imaging processing unit 2012 to the image compression unit 2025 via a connection line CL2. The image compression unit 2025 performs a compression process to compress the captured image, and generates a compressed image with a smaller amount of data than the captured image. The compressed image generated by the image compression unit 2025 is supplied to the memory 2023 via the bus and stored therein.

[0269] Here, the signal processing in the DSP 2022 can be performed using the captured image itself, or can be performed using a compressed image generated from the captured image by the image compression unit 2025. Since the compressed image has a smaller amount of data than the captured image, it is possible to reduce the load of signal processing in the DSP 2022 and save the storage capacity of the memory 2023 that stores the compressed image.

[0270] The compression process in the image compression unit 2025 can be, for example, scaling down a captured image of 3968 pixels x 2976 pixels to an image of 640 pixels x 480 pixels. Furthermore, if the signal processing in the DSP 2022 is performed on luminance and the captured image is an RGB image, the compression process can be YUV conversion, which converts the RGB image into a YUV image, for example.

[0271] The image compression unit 2025 can be realized by software or by dedicated hardware provided by an ISP or the like.

[0272] The DSP 2022 may further perform network processing using a network stored in the memory 2023. Alternatively, the DSP 2022 may refer to Fig. 9 and acquire a network stored in the RAM 1002 or the storage device 1003 via the communication I / F 2024 to perform network processing. In this way, the signal processing unit 2020 including the DSP 2022 can be said to realize the function of the AI ​​model control unit 101 in Fig. 7 described above.

[0273] The input I / F 2026 is an I / F that receives information from the outside. For example, the input I / F 2026 receives the output of an external sensor (external sensor output) from the external sensor, and supplies the output to the memory 2023 via a bus for storage. As with the output I / F 2014, for example, a parallel I / F such as MIPI can be used as the input I / F 2026.

[0274] Furthermore, as the external sensor, for example, a distance sensor that senses information regarding distance can be adopted.Furthermore, as the external sensor, for example, an image sensor that senses light and outputs an image corresponding to that light, that is, an image sensor separate from the imaging device 2100 can be adopted.

[0275] In addition to using captured images or compressed images generated from captured images, the DSP 2022 can perform signal processing using external sensor outputs received by the input I / F 2026 from external sensors such as those described above and stored in memory 2023.

[0276] In the imaging device 2100 configured as described above, signal processing including network processing is performed by the DSP 2022 using a captured image obtained by imaging in the imaging unit 2011 or a compressed image generated from the captured image, and the signal processing result of the signal processing and the captured image are selectively output from the output I / F 2014. Therefore, it is possible to configure a compact imaging device that outputs information required by the user.

[0277] Here, in the imaging device 2100, when signal processing by the DSP 2022 is not performed and therefore the imaging device 2100 outputs a captured image without outputting a signal processing result, that is, when the imaging device 2100 is configured as an image sensor that simply captures and outputs an image, the imaging device 2100 can be configured only with the imaging unit 2010 without the output control unit 2013. In this case, the network processing may be executed by the CPU 2021.

[0278] FIG. 26 is a perspective view schematically illustrating the structure of an example of an imaging device 2100 that can be applied to the embodiment described with reference to FIG.

[0279] The imaging device 2100 can be configured as a one-chip semiconductor device having a stacked structure in which a plurality of dies are stacked, for example, as shown in Fig. 26. In the example of Fig. 26, the imaging device 2100 is configured as a one-chip semiconductor device in which two dies, dies 2030 and 2031, are stacked.

[0280] A die refers to a small thin piece of silicon in which an electronic circuit is fabricated, and an individual in which one or more dies are sealed is called a chip.

[0281] 26 , an imaging unit 2011 is mounted on an upper die 2030. An imaging processing unit 2012, an output control unit 2013, an output I / F 2014, and an imaging control unit 2015 are mounted on a lower die 2031. As such, in the example of FIG. 26 , the imaging unit 2011 of the imaging unit 2010 is mounted on the die 2030, and the parts other than the imaging unit 2011 are mounted on the die 2031. A signal processing unit 2020 including a CPU 2021, a DSP 2022, a memory 2023, a communication I / F 2024, an image compression unit 2025, and an input I / F 2026 is further mounted on the die 2031.

[0282] The upper die 2030 and the lower die 2031 are electrically connected, for example, by forming a through-hole that penetrates the die 2030 and reaches the die 2031. Without being limited to this, the dies 2030 and 2031 may be electrically connected by metal-metal wiring such as Cu-Cu bonding that directly connects a metal wiring such as Cu exposed on the lower surface side of the die 2030 with a metal wiring such as Cu exposed on the upper surface side of the die 2031.

[0283] Here, the imaging processing unit 2012 may use, for example, a column-parallel AD method or an area AD method to perform AD conversion on the image signal output from the imaging unit 2011 .

[0284] In the column-parallel AD method, for example, an ADC (AD Converter) is provided for each column of pixels that constitutes the imaging unit 2011, and the ADC for each column is responsible for AD conversion of the pixel signals of the pixels in that column, thereby performing AD conversion of the image signals of the pixels in each column of one row in parallel. When the column-parallel AD method is adopted, part of the imaging processing unit 2012 that performs AD conversion in the column-parallel AD method may be mounted on the upper die 2030.

[0285] In the area AD system, the pixels constituting the imaging unit 2011 are divided into multiple blocks, and an ADC is provided for each block. The ADC in each block is responsible for AD conversion of the pixel signals of the pixels in that block, thereby performing AD conversion of the image signals of the pixels in multiple blocks in parallel. In the area AD system, the block is the smallest unit, and AD conversion (reading and AD conversion) of the image signals can be performed only on necessary pixels among the pixels constituting the imaging unit 2011.

[0286] If the area of ​​the imaging device 2100 is allowed to be large, the imaging device 2100 can be configured with a single die.

[0287] 26, two dies 2030 and 2031 are stacked to form the one-chip imaging device 2100, but three or more dies can be stacked to form the one-chip imaging device 2100. For example, when three dies are stacked to form the one-chip imaging device 2100, the memory 2023 mounted on the die 2031 in FIG. 26 can be mounted on a die different from the dies 2030 and 2031.

[0288] Here, in an imaging device in which the sensor chip, memory chip, and DSP chip are connected in parallel with multiple bumps (hereinafter also referred to as a bump-connected imaging device), the thickness increases significantly and the device becomes larger compared to the single-chip imaging device 2100 configured in a stacked structure.

[0289] Furthermore, in a bump-connected imaging device, due to signal degradation at the bump connection points, it may be difficult to ensure a sufficient rate for outputting captured images from the imaging processing unit 2012 to the output control unit 2013.

[0290] The imaging device 2100 having a stacked structure can prevent the device from becoming larger as described above, and can prevent a situation in which a sufficient rate cannot be ensured between the imaging processing unit 2012 and the output control unit 2013. Therefore, the imaging device 2100 having a stacked structure can make the imaging device that outputs information required for processing in a subsequent stage of the imaging device 2100 compact.

[0291] When the information required in the subsequent stage is a captured image, the imaging device 2100 can output the captured image (a RAW image, an RGB image, etc.). Furthermore, when the information required in the subsequent stage is obtained by signal processing using the captured image (such as processing by an AI model), the imaging device 2100 can obtain and output the signal processing result as information required by the user by performing the signal processing in the DSP 2022.

[0292] The signal processing performed by the imaging device 2100, i.e., the signal processing by the DSP 2022, may be, for example, recognition processing for recognizing a predetermined recognition target from a captured image. The DSP 2022 may perform this recognition processing using a network distributed from the AI ​​model management server 30.

[0293] Furthermore, for example, the image capturing device 2100 can receive, via the input I / F 2026, the output of a distance sensor such as a ToF (Time of Flight) sensor that is arranged to have a predetermined positional relationship with the image capturing device 2100. In this case, the signal processing of the DSP 2022 can employ, for example, fusion processing that integrates the output of the distance sensor with the captured image to determine a distance with high accuracy, such as processing that removes noise from a distance image obtained from the output of the distance sensor received via the input I / F 2026 using the captured image.

[0294] Furthermore, for example, the imaging device 2100 can receive, via the input I / F 2026, an image output by an image sensor disposed so as to have a predetermined positional relationship with the imaging device 2100. In this case, the signal processing of the DSP 2022 can employ, for example, a self-position estimation process (SLAM (Simultaneously Localization and Mapping)) using the image received via the input I / F 2026 and the captured image as a stereo image.

[0295] The DSP 2022 may perform the above-mentioned recognition processing, noise removal processing, self-position estimation processing, etc. using a network distributed from the AI ​​model management server 30.

[0296] The imaging device 2100 is not limited to the example configured as the CIS described above, but may also have a general camera configuration, for example, a configuration in which an image is captured by an imaging element and a captured image is output after predetermined signal processing such as noise removal and level adjustment has been performed. In this case, network processing for the captured image may be performed by, for example, a CPU included in the data processing unit 2200.

[0297] (3-2. Processing According to the Second Embodiment) Next, processing according to the second embodiment will be described.

[0298] Fig. 27 is a schematic diagram illustrating a data flow during deployment according to the second embodiment. Fig. 27 corresponds to Fig. 12 described above, in which the part above the dotted line shows processing by the AI ​​model management server 30b, and the part below the dotted line shows processing by the edge device 10c.

[0299] The AI ​​models 40, 40, ... to be combined are input to the AI ​​model conversion unit 340. The AI ​​model conversion unit 340 uses the memory 341 to interpret the contents of each of the input AI models 40, 40, ... and converts them into data in a format that can be processed by the edge device 10c. For example, the AI ​​model conversion unit 340 removes headers and footers from each of the input AI models 40, 40, ..., extracts weight information, and quantizes the extracted weight information.

[0300] Furthermore, the AI ​​model conversion unit 340 generates information indicating the order of the AI ​​models 40, 40, ... to be combined, which is specified, for example, on the UI screen 500. The information indicating this order may be written in a setting file of the AI ​​model conversion unit 340 in accordance with the data structure shown in FIG.

[0301] The AI ​​model conversion unit 340 passes converted data 342 obtained by converting each of the AI ​​models 40 1, 40 2, ... to the AI ​​model combination unit 320′. At this time, the AI ​​model conversion unit 340 adds information indicating the order of each of the AI ​​models 40 1, 40 2, ..., which is described in the setting file of the AI ​​model conversion unit 340, to the converted data 342 and passes the converted data to the AI ​​model combination unit 320′.

[0302] The AI ​​model combination unit 320′ combines the converted data 342 of each AI model 40 1 , 40 2 , ... passed from the AI ​​model conversion unit 340, and generates a combined AI model 41. The data structure of the management information 42 (file “network_info.txt”) included in the combined AI model 41 and the file “network.fpk” in which each AI model 40 1 , 40 2 , ... is stored will be described later.

[0303] The AI ​​model management server 30b transmits the combined AI model 41 generated by the AI ​​model combination unit 320' to the edge device 10c.

[0304] In the edge device 10c, the data processing unit 2200 may write the combined AI model 41 received from the AI ​​model management server 30b, for example, to a volatile memory (such as an SRAM) not shown that is connected to the data processing unit 2200. The data processing unit 2200 may write the management information 42 included in the combined AI model 41 written to the volatile memory to the non-volatile memory 2201. Alternatively, the data processing unit 2200 may directly write the combined AI model 41 received from the AI ​​model management server 30b to the non-volatile memory 2201.

[0305] The data processing unit 2200 sends the file "network.fpk" included in the combined AI model 41 written to the volatile memory (or non-volatile memory 2201) to the signal processing unit 2020. The signal processing unit 2020 writes the file "network.fpk" sent from the data processing unit 2200 to the non-volatile memory 2023b. For the sake of explanation, it is assumed here that the file "network.fpk" includes the two combined AI models 40a and 40b.

[0306] Here, the non-volatile memory 2023b is managed by dividing its storage area into multiple storage areas, each with a fixed data capacity. In the example of FIG. 27, the non-volatile memory 2023b is managed by dividing its storage area into four areas (Slots #1, #2, #3, and #4) called slots. The signal processing unit 2020 writes one file "network.fpk" to one slot. In the example shown, two AI models 40a and 40b included in the file "network.fpk" are shown written to the fourth slot, Slot #4, of the non-volatile memory 2023b.

[0307] 28 is a schematic diagram for explaining the process of writing the combined AI model 41 to the non-volatile memory 2023b according to the second embodiment. Note that the process in the data processing unit 2200 is omitted in FIG.

[0308] For example, in the flowchart of Figure 10 described above, in step S102, the AI ​​model combination unit 320' in the AI ​​model management server 30b determines whether the deployment size after combining multiple AI models 40a and 40b is within the capacity of one slot of the non-volatile memory 2023b in the edge device 10c to be deployed.

[0309] If the AI ​​model combination unit 320′ determines that the expansion size after combination is within the capacity of one slot (step S102, “Yes”), it combines these multiple AI models 40a and 40b to generate a combined AI model 41. This combined AI model 41 is written to a predetermined slot in the non-volatile memory 2023b.

[0310] The combined AI model 41 written in the non-volatile memory 2023b is read from the slot in the non-volatile memory 2023b and written into the volatile memory 2023a and deployed when the AI ​​models 40a and 40b are executed.

[0311] On the other hand, if the AI ​​model combination unit 320' determines that the expansion size after combination exceeds the capacity of one slot (step S102, "No"), the AI ​​model combination unit 320' does not combine the multiple AI models 40a and 40b. The multiple AI models 40a and 40b are each written to one slot in the non-volatile memory 2023b, as shown by the dotted lines in FIG.

[0312] In this case, when processing is executed by each of the AI ​​models 40a and 40b, the AI ​​models 40a and 40b are read out one by one from each slot and expanded into the volatile memory 2023a.

[0313] In the existing technology, multiple AI models are not combined, so that the AI ​​models 40a and 40b are each written to one slot in the non-volatile memory 2023b.

[0314] In other words, if multiple AI models are not combined, deployment and expansion processing to volatile memory 2023a will be required a number of times corresponding to the number of AI models to be executed, and multiple slots will be used.

[0315] 27 , the data processing unit 2200 extracts and manages management information 42 included in the combined AI model 41 written to the non-volatile memory 2201. The data processing unit 2200 manages, for example, slot information indicating the storage location in the non-volatile memory 2023b of the combined AI model 41, a valid flag for the combined AI model 41, a network ID, and the update date and time of the combined AI model 41 in the non-volatile memory 2201. This information is managed, for example, as fixed-length data.

[0316] (3-3. Example of Data Format According to Second Embodiment) Next, an example of the data format of the file "network.fpk" as the combined AI model 41 according to the second embodiment will be described in more detail with reference to Figs. 29 to 32. Note that the file "network_info.txt" as the management information 42 has the same format as that described with reference to Fig. 14, and therefore description thereof will be omitted here.

[0317] Fig. 29 is a schematic diagram showing an example of the configuration of the file "network.fpk" according to the second embodiment. In Fig. 29, sections (a) and (b) correspond to sections (a) and (b) in Fig. 13. In Fig. 29, section (c) showing the contents of section (b) in more detail is added to the configuration of Fig. 13.

[0318] In FIG. 29, the file "network.fpk" shown in section (a) contains various data (Download Data) sandwiched between a download header and a download footer, as shown in section (b).

[0319] As shown in section (c) of Figure 29, the data placed at the beginning of the file "network.fpk" between a download header and a download footer includes an image packet header at the beginning, followed by a set of an image header and data for each type of data.

[0320] Among the data items following the image packet header, the data "AP Params" may be AI model information for an application to analyze the processing results of the DNN (AI model) transmitted via MIPI. The data "Program / Data" may be, for example, a program and data used by the CPU 2021 included in the signal processing unit 2020. The data "DNN Params" is DNN (AI model) information for synchronizing various system cores. The data "CFG Blob1" and "CFG Blob2" may each be information regarding how to write data to the memory area cfgBank (see section (b) of Figure 19), which is a predetermined area in the working memory (e.g., SRAM) used by the AI ​​model combination unit 320'. The data "Conv-A Program" and "Conv-C Program" may be programs used by the AI ​​model conversion unit 340. The data "Post Process Params" is data that is placed when there is post-processing such as that shown in steps S14 and S18 in Fig. 6, and may be parameters related to the post-processing. The data "Network Weights_0" may be weight information for the first AI model.

[0321] Thereafter, pairs of an image header and each piece of weight information (Network Weights_1 to N) for the second and subsequent AI models are sequentially placed in each piece of download data sandwiched between a download header and a download footer.

[0322] In this way, the sets of weight information and image headers for each AI model are arranged consecutively via the download footer and download header, from the weight information for the first AI model "Network Weights_0" to the weight information for the Nth AI model "Network Weights_N."

[0323] FIG. 30 is a schematic diagram showing an example of the configuration of a download header and a download footer according to the second embodiment.

[0324] In Figure 30, section (a) shows the set of download header, download data, and download footer shown in section (b) of Figure 29. The data size including the download header and download footer is the data size of the download data. The file "network.fpk" can be divided into the number of files indicated by "total num" and sent.

[0325] Section (b) in Fig. 30 shows an example of the configuration of a download header. In section (b), the row-oriented fields "00" and "10" each have a data length of 16 bytes, and field "10" stores a UUID (Universally Unique Identifier).

[0326] In sections (b) and (c) of Figure 30, and sections (b) of Figures 31 and 32 described below, each byte in the row direction is indicated by hexadecimal notation, with the first byte at the left end and the fifteenth byte at the right end, such as "00", "01", ..., "0E", and "0F".

[0327] In field "00," the 0th to 3rd bytes store header identification information. The signal processing unit 2020 can determine the start of data transfer from this header identification information. The value of field "00" may be a fixed value that depends on the type of edge device 10c, for example.

[0328] In the field "00", the fourth to seventh bytes may be the data size of the encrypted download data related to the download data header. The encrypted download data may be aligned in 16 bytes.

[0329] The eighth and ninth bytes, and the tenth and eleventh bytes, may store the number of the divided file containing this download header and the total number of divided files, respectively, if the file "network.fpk" is divided and transmitted. If the file "network.fpk" is not divided, the value of these fields is set to "1."

[0330] The 12th byte may be a flag indicating whether an image packet header is included in the download data. The 13th and 14th bytes may be flags related to MAC (Media Access Control), with the 13th byte being a flag indicating whether an extended function for MAC authentication is available, and the 14th byte being a flag indicating whether MAC authentication is available. The 15th byte may be a flag indicating whether encryption is available.

[0331] Section (c) in Fig. 30 shows an example of the configuration of a download footer. In section (c), the row-oriented fields "00" to "30" each have a data length of 16 bytes. Field "00" may store a MAC value.

[0332] In field "10," bytes 0 and 1 may store the encryption generation number for the downloaded data. Bytes 2 to 11 of field "10" may be a reserved area, and bytes 12 to 15 may store footer identification information. The signal processing unit 2020 can determine the end of data transfer based on this footer identification information. The value of field "10" may be a fixed value that corresponds to the type of edge device 10c, for example.

[0333] Fields “20” and “30” may store a hash value used for MAC authentication. Fields “20” and “30” are valid when the value of the 13th byte of field “00” in the download header indicates enable.

[0334] In the download header shown in section (b) of Figure 30, the 0th to 3rd bytes of row field "00" and row field "10" each have values ​​written in big endian. Values ​​in the other parts are written in little endian. In the download footer shown in section (c), the 0th to 11th bytes of row field "10" have values ​​written in little endian, and values ​​in the other parts are written in big endian.

[0335] Fig. 31 is a schematic diagram showing an example of the configuration of an image packet header according to the second embodiment. In Fig. 31, section (a) shows an extracted image packet header shown in section (c) of Fig. 29.

[0336] Section (b) of Figure 31 shows an example of the structure of an image packet header. Row-oriented fields "00" and "10" each have a data length of 16 bytes. Field "00" describes a value in little endian, while field "10" describes a value in big endian. In field "00," bytes 0 to 3 may store the number of images contained in the file "network.fpk" in which this image packet header is located. In this case, for example, the number of image headers may be considered the number of images. Bytes 4 to 15 are reserved areas. Field "10" may store version information.

[0337] FIG. 32 is a schematic diagram showing an example of the configuration of an image header according to the second embodiment.

[0338] In Fig. 32, section (a) shows an extracted set of image header and data shown in section (c) of Fig. 29. The data paired with the image header consists of records 0 to N, each aligned to a data length of 16 bytes, and the total size of records 0 to N is the image size.

[0339] In Figure 32, section (b) shows an example of the structure of an image header. The row-oriented fields "00" and "10" each have a data length of 16 bytes. The value in field "00" is written in little endian, and the value in field "10" is written in big endian.

[0340] In the field "00", bytes 0 to 3 may store a target ID. The target ID is information indicating the type of data corresponding to this image header, and may use a 1-byte value to indicate which of the above-mentioned data "AP Params", "Program / Data", "DNN Params", "CFG Blob1", "CFG Blob2", "Conv-A Program", "Conv-C Program", "Post Process Params", and "Network Weights" the data is.

[0341] In field "00", the fourth to seventh bytes may store the address in memory of the data indicated by the target ID. The eighth to eleventh bytes may store the data size of the data indicated by the target ID, and the information indicating the data size is padded to 16 bytes. The twelfth to fifteenth bytes are a reserved area.

[0342] Field "10" may store version information of the data indicated by the target ID.

[0343] (3-4. Model Switching Process According to the Second Embodiment) In the second embodiment, the imaging device 2100 (edge ​​device 10c) can dynamically switch which of multiple deployed AI models is used to execute processing. Dynamic switching of multiple AI models will be described with reference to Figs. 33A and 33B .

[0344] 33A is a schematic diagram illustrating a first example of switching between multiple AI models according to the second embodiment. In Fig. 33A, sections (a) and (b) each represent a combined AI model 41, which combines two AI models 40 and 40, written in the volatile memory 2023a of the imaging device 2100, and in which processing is being performed by one of these AI models 40 and 40.

[0345] Section (a) of Fig. 33A shows an example in which processing by AI model 401 and processing by AI model 402, both written in the volatile memory 2023a of the imaging device 2100, are switched in response to an instruction from the data processing unit 2200 included in the edge device 10c. Similarly, section (b) shows an example in which processing by AI model 401 and processing by AI model 402 are switched in response to an instruction from the AI ​​model management server 30b.

[0346] In the imaging device 2100, each of the AI ​​models 40 and 40 is written to one slot in the non-volatile memory 2023 b, and is read from that slot and expanded into the volatile memory 2023 a during execution. The DSP 2022, for example, which executes processing using each of the AI ​​models 40 and 40 , switches the read address of the volatile memory 2023 a in response to an instruction to switch the AI ​​model, and reads the AI ​​model to be executed from the volatile memory 2023 a.

[0347] 33B is a schematic diagram illustrating a second example of switching between multiple AI models according to the second embodiment. In the example of Fig. 33B, the combined AI model 41 includes only one AI model 40, and similarly, the combined AI model 41 also includes only one AI model 40.

[0348] Section (a) of Figure 33B shows an example of switching between processing by AI model 401 and processing by AI model 402 in response to an instruction from the data processing unit 2200 included in edge device 10c. Similarly, section (b) shows an example of switching between processing by AI model 401 and processing by AI model 402 in response to an instruction from the AI ​​model management server 30b.

[0349] In either case of section (a) or (b), each of the combined AI models 41 and 41 is written to a different slot in the non-volatile memory 2023b. When the AI ​​model 40 or 40 is executed, the combined AI model that includes the AI ​​model to be executed, among the combined AI models 41 and 41, is deployed in the volatile memory 2023a.

[0350] The example in Figure 33B is also an example of AI model switching processing using existing technology.

[0351] 33A, there is a limit to the data capacity of one slot of the non-volatile memory 2023b for the data size of the combined AI model 41, which includes multiple AI models 40 and 40. On the other hand, because all of the AI ​​models 40 and 40 to be switched are simultaneously loaded on the volatile memory 2023a, faster switching processing is possible.

[0352] In contrast, in the example of Fig. 33B, the limit on the data capacity of one slot is applied to the data size of each of the AI ​​models 40 and 40, so there is more leeway in terms of the data size of the AI ​​models that can be used compared to the example of Fig. 33A. On the other hand, since the AI ​​models 40 and 40 (combined AI models 41 and 41) must be read from the non-volatile memory 2023b each time they are executed, the switching process is slower and the number of accesses to the non-volatile memory 2023b is greater compared to the example of Fig. 33A.

[0353] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0354] The present technology can also be configured as follows. (1) An information processing device comprising: a combination unit that combines two or more AI (Artificial Intelligence) models by adding management information that manages each of the two or more AI models to generate a combined AI model; and a transmission unit that transmits the combined AI model to a terminal device that serves as an execution environment for executing the two or more AI models included in the combined AI model. (2) The information processing device according to (1), wherein the combination unit combines the management information for each of the two or more AI models in accordance with an order set for each of the two or more AI models and adds the information to the two or more AI models. (3) The information processing device according to (1) or (2), wherein the combination unit determines whether to generate the combined AI model using the two or more AI models based on memory capacity in the execution environment. (4) The information processing device according to any of (1) to (3), wherein the combination unit generates the combined AI model using the two or more AI models that are intended to realize mutually different functions. (5) The information processing device according to (4), wherein the combination unit generates the combined AI model by combining the two or more AI models executed in series or in parallel. (6) The information processing device according to any of (1) to (3), wherein the combination unit generates the combined AI model using the two or more AI models used in one ensemble learning. (7) The information processing device according to any of (1) to (6), further comprising: a user interface unit that performs screen display and provides a user with a user interface that accepts user operations according to the screen display, wherein the combination unit generates the combined AI model using the two or more AI models selected from a plurality of AI models managed by an AI model management unit in accordance with the user operation on the user interface. (8) The information processing device according to (7), wherein the combination unit generates the combined AI model using the two or more AI models and the management information of each of the two or more AI models in accordance with an order set for the two or more AI models by the user operation on the user interface.(9) The information processing device according to any of (1) to (6), wherein the combination unit generates the combined AI model using the two or more AI models and the management information for each of the two or more AI models in accordance with an order described in a manifest file created in advance. (10) The information processing device according to any of (1) to (6), wherein the combination unit combines the two or more archived and acquired AI models with the management information for each of the two or more AI models in a pre-specified order to generate the combined AI model. (11) The information processing device according to any of (1) to (10), further comprising: a model conversion unit that converts each of the two or more AI models into a model according to the execution environment based on predetermined parameters, wherein the combination unit generates the combined AI model using a converted model obtained by converting each of the two or more AI models by the model conversion unit. (12) The information processing device according to (11), wherein the model conversion unit sets an order in which the two or more AI models are combined in the combination unit when performing the conversion for each of the two or more AI models. (13) The information processing device according to (11) or (12), wherein the predetermined parameters include parameters related to a model structure according to the execution environment. (14) A terminal device comprising: a non-volatile memory; a volatile memory used as an execution memory by a model execution unit that executes processing by an AI model; and a processing control unit that controls processing by each of the two or more AI models included in a combined AI model that is received via a network and is combined with management information that manages each of the two or more AI models, wherein the processing control unit stores the combined AI model received via the network in the non-volatile memory, and writes the combined AI model stored in the non-volatile memory to the volatile memory when at least one of the two or more AI models included in the combined AI model is executed.(15) The terminal device according to (14), wherein the processing control unit specifies, at the time of execution, an AI model to be executed from among the two or more AI models included in the combined AI model stored in the volatile memory based on the management information added to the combined AI model. (16) The terminal device according to (14) or (15), wherein the non-volatile memory has a storage area divided into a plurality of storage areas, each having a fixed storage capacity and address, and the processing control unit stores the combined AI model in one of the plurality of storage areas in the non-volatile memory. (17) The terminal device according to (16), wherein, when the data size of the combined AI model is larger than the storage capacity of each of the plurality of storage areas, the processing control unit stores each of the two or more AI models included in the combined AI model in each of the plurality of storage areas. (18) The terminal device according to (16) or (17), wherein the processing control unit associates the combined AI model or each of the two or more AI models included in the combined AI model with each of the plurality of storage areas based on the management information added to the combined AI model.(19) An information processing system including: an information processing device; and a terminal device connected to the information processing device via a network, wherein the information processing device comprises: a combination unit that combines two or more AI models by adding management information that manages each of the two or more AI models to generate a combined AI model; and a transmission unit that transmits the combined AI model to a terminal device that serves as an execution environment for executing the two or more AI models included in the combined AI model, wherein the terminal device comprises: a non-volatile memory; a volatile memory that is used as an execution memory by a model execution unit that executes processing by an AI model; and a processing control unit that controls processing by each of the two or more AI models included in the combined AI model received via the network, wherein the processing control unit stores the combined AI model received via the network in the non-volatile memory, and reads the combined AI model stored in the non-volatile memory into the volatile memory when at least one of the two or more AI models included in the combined AI model is executed. (20) A data generation method, comprising: a combining step of combining two or more AI models by adding management information for managing each of the two or more AI models, and generating a combined AI model.

[0355] 1, 1a Information processing system 2 Communication network 3 Cloud network 4 User terminal 10, 10a, 10b, 10c Edge device 11 Receiving unit 12, 32 Processing unit 13, 2023a Volatile memory 14, 2023b, 2201 Non-volatile memory 15, 103 Device control unit 30, 30a, 30b AI model management server 31, 310 AI model storage unit 33 Transmission unit 401, 402, 403, 40a, 40b AI model 41, 411, 412 Combined AI model 42, 421, 422 Management information 44 Encryption key 43 Network ID 50 AI processing unit 100 Data processing device 101 AI model control unit 102 Management information control unit 110 Input device 120 Output device 130, 341, 2023 Memory 320, 320' AI model combination unit 322 Memory calculation unit 323 AI model combination processing unit 324 Data addition unit 325 Data authentication unit 330 Data transmission unit 331 Device connection unit 332 Data import unit 340 AI model conversion unit 342 Converted data 500 UI screen 520 Model setting screen 2010 Imaging unit 2020 Signal processing unit 2022 DSP 2030, 2031 Die 2100 Imaging device 2200 Data processing unit 2202 Output unit 2203 Receiving unit

Claims

1. An information processing device comprising: a combination unit that combines two or more AI (Artificial Intelligence) models by adding management information that manages each of the two or more AI models to generate a combined AI model; and a transmission unit that transmits the combined AI model to a terminal device as an execution environment that executes the two or more AI models included in the combined AI model.

2. The information processing device according to claim 1, wherein the combination unit combines the management information of each of the two or more AI models in accordance with an order set for each of the two or more AI models, and adds the management information to the two or more AI models.

3. The information processing device according to claim 1, wherein the combination unit determines whether or not to generate the combined AI model using the two or more AI models based on the memory capacity in the execution environment.

4. The information processing device according to claim 1, wherein the combination unit generates the combined AI model using two or more AI models for realizing mutually different functions.

5. The information processing device according to claim 4, wherein the combination unit combines the two or more AI models executed in series or in parallel to generate the combined AI model.

6. The information processing device according to claim 1, wherein the combination unit generates the combined AI model using the two or more AI models used in one ensemble learning.

7. An information processing device as described in claim 1, further comprising a user interface unit that displays a screen and provides a user with a user interface that accepts user operations according to the screen display, wherein the combination unit generates the combined AI model using the two or more AI models selected from a plurality of AI models managed by an AI model management unit in response to the user operation on the user interface.

8. The information processing device according to claim 7, wherein the combination unit generates the combined AI model using the two or more AI models and the management information of each of the two or more AI models in accordance with an order set for the two or more AI models by the user operation on the user interface.

9. The information processing device according to claim 1, wherein the combination unit generates the combined AI model using the two or more AI models and the management information for each of the two or more AI models in accordance with an order described in a manifest file created in advance.

10. The information processing device according to claim 1, wherein the combination unit combines the two or more AI models that have been archived and acquired with the management information for each of the two or more AI models in a pre-specified order to generate the combined AI model.

11. The information processing device of claim 1, further comprising a model conversion unit that converts each of the two or more AI models into a model that corresponds to the execution environment based on predetermined parameters, wherein the combination unit generates the combined AI model using a converted model into which each of the two or more AI models is converted by the model conversion unit.

12. The information processing device according to claim 11, wherein the model conversion unit sets an order in which the two or more AI models are combined in the combination unit when performing the conversion for each of the two or more AI models.

13. The information processing device according to claim 11, wherein the predetermined parameters include parameters relating to a structure of a model according to the execution environment.

14. A terminal device comprising: a non-volatile memory; a volatile memory used as an execution memory by a model execution unit that executes processing by an AI model; and a processing control unit that controls processing by each of two or more AI models included in a combined AI model received via a network and combined with the addition of management information managing each of the two or more AI models, wherein the processing control unit stores the combined AI model received via the network in the non-volatile memory, and writes the combined AI model stored in the non-volatile memory to the volatile memory when at least one of the two or more AI models included in the combined AI model is executed.

15. The terminal device described in claim 14, wherein the processing control unit specifies, at the time of execution, an AI model to be executed from among the two or more AI models included in the combined AI model stored in the volatile memory based on the management information added to the combined AI model.

16. The terminal device according to claim 14, wherein the non-volatile memory has a memory area divided into a plurality of memory areas, each of which has a fixed memory capacity and address, and the processing control unit stores the combined AI model in one of the plurality of memory areas of the non-volatile memory.

17. The terminal device according to claim 16, wherein the processing control unit stores each of the two or more AI models included in the combined AI model in each of the multiple storage areas when the data size of the combined AI model is larger than the storage capacity of each of the multiple storage areas.

18. The terminal device described in claim 16, wherein the processing control unit associates the combined AI model or each of the two or more AI models included in the combined AI model with each of the multiple memory areas based on the management information added to the combined AI model.

19. An information processing system comprising: an information processing device; and a terminal device connected to the information processing device via a network, wherein the information processing device comprises: a combination unit that combines two or more AI models by adding management information for managing each of the two or more AI models to generate a combined AI model; and a transmission unit that transmits the combined AI model to a terminal device as an execution environment for executing the two or more AI models included in the combined AI model, wherein the terminal device comprises: a non-volatile memory; a volatile memory used as an execution memory by a model execution unit that executes processing by an AI model; and a processing control unit that controls processing by each of the two or more AI models included in the combined AI model received via the network, wherein the processing control unit stores the combined AI model received via the network in the non-volatile memory, and reads the combined AI model stored in the non-volatile memory into the volatile memory when at least one of the two or more AI models included in the combined AI model is executed.

20. A data generation method comprising: a combination step of combining two or more AI models by adding management information for managing each of the two or more AI models, to generate a combined AI model.

Images