Information processing apparatus, inference apparatus, control method, and program

Abstract

To achieve techniques of effectively utilizing, in a case where an expansion apparatus that expands the functionality of an inference apparatus is connected to the inference apparatus, hardware resources of the inference apparatus and the expansion apparatus.SOLUTION: An information processing apparatus includes: generation means which generates a program for executing inference processing using a trained inference model; and control means which divides the inference processing into a first process to be executed on a first inference apparatus and a second process to be executed on one or a plurality of second inference apparatuses which can be connected to the first inference apparatus. The generation means generates a program for executing the first process and a program for executing the second process on the basis of first information on hardware for inference processing on the first inference apparatus and second information on hardware for inference processing on the second inference apparatus(es).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for expanding the functionality of an inference device. [Background technology]

[0002] Deep learning technology using neural networks is known as machine learning using AI (artificial intelligence). Deep learning technology is used in image processing such as recognizing people's faces and expressions from captured images. In image processing using deep learning technology, captured images, etc. are applied as input data to a trained inference model to perform inference processing using machine learning, and the inference results are output.

[0003] Although the inference process may be executed by a single inference device, when performing more complicated processes, the hardware resources of the inference device alone may be insufficient, so a technique is also known in which an extension device capable of executing the inference process is connected to the inference device to extend the functions of the inference device. Patent Document 1 describes a technique in which a removable device connected to an imaging device executes an analysis process of a captured image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-133135 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, it is possible to execute the inference process by a removable device connected to the imaging device, but the hardware resources of the imaging device cannot be utilized for the inference process.

[0006] The present invention has been made in consideration of the above problems, and its purpose is to realize a technology that can effectively utilize the hardware resources of an inference device and an extension device that extends the functions of the inference device when the extension device is connected to the inference device. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the information processing device of the present invention has a creation means for creating a program for executing inference processing using a learned inference model, and a control means for distributing the inference processing into a first process to be executed in a first inference device and a second process to be executed in one or more second inference devices connectable to the first inference device, and the creation means creates a program for executing the first process and a program for executing the second process based on first information regarding hardware for inference processing in the first inference device and second information regarding hardware for inference processing in the second inference device. Effect of the Invention

[0008] According to the present invention, when an extension device that extends the functions of an inference device is connected to the inference device, the hardware resources of the inference device and the extension device can be effectively utilized. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a system configuration diagram including an inference device and an extension device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an inference device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a block diagram showing a configuration of an information processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating an inference program creation tool according to the present embodiment. [Diagram 5] 4 is a flowchart showing a control process of the information processing apparatus according to the first embodiment. [Figure 6] 4 is a flowchart showing a control process of the information processing apparatus according to the first embodiment. [Figure 7] 4 is a flowchart showing a control process of the inference device of the first embodiment. [Figure 8] 10 is a flowchart showing a control process of the information processing apparatus according to the second embodiment. [Figure 9] FIG. 11 is a system configuration diagram including an inference device and an extension device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In this embodiment, when an extension device that extends the function of an inference process is connected to an inference device that executes an inference process using a trained inference model, the calculation process of each processing layer in the inference model is appropriately distributed between the inference device and the extension device, thereby enabling effective use of hardware resources for executing the inference process of the inference device and the extension device.

[0012] [Embodiment 1] First, a first embodiment will be described with reference to FIGS.

[0013] <System configuration> First, with reference to FIG. 1, a system configuration including an inference device 100 and an extension device 120 according to the first embodiment will be described.

[0014] The inference device 100 is an imaging device such as a digital camera, and has a deep learning function using an AI (artificial intelligence) neural network. The inference device 100 has an image recognition function that recognizes, for example, a subject in an image and classifies it into a predetermined category (class) by deep learning using a captured image as input data. The inference device 100 includes a volatile memory 102, an inference unit 108, an interface (IF) control unit 109, and an image processing unit 113.

[0015] The inference device 100 is mechanically detachably and electrically connectable to an extension device 120 that extends the inference processing function of the inference device 100. The inference unit 108 executes image recognition processing by inference processing using a trained inference model, using image data output from the image processing unit 113 as input data. The interface control unit 109 performs control for connecting the extension device 120 to the inference device 100.

[0016] The trained inference model is configured by a neural network, and in this embodiment, is configured by a convolutional neural network (CNN). The CNN has multiple processing layers such as an input layer, intermediate layers (convolution layer, pooling layer, fully connected layer), and an output layer, and data input to each node of each processing layer is weighted by trained inference parameters (weighting coefficients and bias values) and output to a subsequent processing layer. In this embodiment, the calculation processing in each processing layer in the inference model is appropriately allocated to the hardware of the inference device 100 and the hardware of the extension device 120, so that both the hardware resources required for the inference processing of the inference device 100 and the extension device 120 can be effectively utilized.

[0017] It should be noted that the inference model of this embodiment is not limited to CNN, and may be configured using a Recurrent Neural Network (RNN) or the like.

[0018] The image processing unit 113 performs various image processing on the image data of the subject, and creates image data such as still images and videos. The inference unit 108 executes inference processing using the image data as input, and performs image recognition processing. The interface control unit 109 controls the connection of the extension device 120 to the inference device 100.

[0019] The extension device 120 is detachably attached to the inference device 100, and includes hardware for extending the inference processing function of the inference device 100. The extension device 120 is also capable of accessing a volatile memory 121. The volatile memory 121 stores input data and output data during the inference processing, intermediate data generated during the inference processing, and further stores inference parameters and information relating to the neural network structure, etc. (hereinafter, dictionary data).

[0020] <Configuration of inference device 100> Next, with reference to FIG. 2, a block diagram showing the configuration of the inference device 100 of the first embodiment is shown.

[0021] The inference device 100 has a control unit 101, a volatile memory 102, a non-volatile memory 103, an operation unit 104, a display unit 105, a recording unit 106, a communication unit 107, an inference unit 108, an interface control unit 109, a lens 111, an imaging unit 112, an image processing unit 113, an encoding processing unit 114 and an internal bus 115.

[0022] The control unit 101 has a processor (CPU) that performs arithmetic processing and control processing of the inference device 100 , and controls each of the components of the inference device 100 by executing a control program stored in the non-volatile memory 103 .

[0023] The volatile memory 102 is a RAM. Constants and variables for the operation of the control unit 101, and control programs and inference programs read from the non-volatile memory 103 are loaded into the volatile memory 102. The volatile memory 102 also stores information such as image data and inference programs received from an external device by the communication unit 116. The volatile memory 102 also stores image data acquired by the imaging unit 112, and image data processed by the image processing unit 113 and the encoding processing unit 114, etc. The volatile memory 102 has a sufficient storage capacity to hold this information.

[0024] The non-volatile memory 103 is an electrically erasable and recordable EEPROM, flash memory, etc. Constants and control programs for the operation of the control unit 101, and an inference program used by the inference unit 108 for inference processing, etc. are stored in the non-volatile memory 103. The inference program includes an inference model and a concealment model, inference parameters, decode information, etc. used by the inference unit 108 for inference processing.

[0025] The operation unit 104 is an operation member such as various switches, buttons, a touch panel, etc. that accepts various operations from a user and notifies the control unit 101. In addition, the operation unit 104 provides a user interface for the user to operate the inference device 100.

[0026] The display unit 105 renders the display data created by the image processing unit 113 and displays it on the display screen of the display device. The display unit 105 has an OSD (On Screen Display) function that displays setting information and operation information on the display screen by superimposing them on GUIs such as menus for various settings. The display unit 105 is composed of a liquid crystal display, an organic EL display, or the like.

[0027] The recording unit 106 is an interface that controls access to the recording medium 110. The recording unit 106 controls writing and reading of data to and from the recording medium 110 based on instructions from the control unit 101. The recording medium 110 records image data, a learning program used by the learning unit 208 for learning processing, and the like. The recording medium 110 is composed of a memory card, a hard disk, and the like.

[0028] The communication unit 107 controls communication with an external device based on instructions from the control unit 101. The communication unit 107 creates a modulated signal conforming to a wireless communication standard (wireless LAN) such as IEEE80111, outputs the modulated signal to the external device, and receives the modulated signal from the external device. Note that the communication method is not limited to wireless LAN, and may be a wired LAN connected by a wired cable or USB. The communication unit 107 is also capable of transmitting and receiving video signals conforming to communication standards such as HDMI (registered trademark) (High Definition Multimedia Interface) and SDI (Serial Digital Interface).

[0029] The inference unit 108 executes an inference process using a trained inference model and inference parameters according to an inference program. The inference unit 108 also has an area for temporarily storing the inference parameters. The inference unit 108 also has a function for decoding the encoded input data and inference parameters.

[0030] The inference processing in the inference unit 108 can be executed by a GPU or a DSP (Digital Signal Processor). The GPU or DSP is a processor capable of performing a large amount of product-sum calculations, and has the computational processing power to perform matrix calculations of a neural network in a short time. Note that the inference processing may be performed by the CPU of the control unit 101 and the GPU or DSP working together, or may be performed by either the CPU of the control unit 101 and the GPU or DSP.

[0031] The interface control unit 109 controls data communication with edge devices such as the extension device 120 connected to the inference device 100. For communication with edge devices, a PCI (Peripheral Component Interconnect), a USB (Universal Serial Bus), or other interfaces can be used.

[0032] The photographing lens (lens unit) 111 includes a group of lenses (not shown) including a zoom lens, a focus lens, etc., a lens control unit (not shown), an aperture (not shown), etc., and forms an optical image of a subject on an imaging surface of the imaging unit 112.

[0033] The imaging unit 112 has an image sensor configured with a CCD (charge-coupled device), a CMOS (complementary metal-oxide semiconductor) element, or the like, and converts an optical image of a subject into an electrical signal.

[0034] The image processing unit 113 includes a processor such as a GPU (Graphics Processing Unit) that executes various image processing operations on image data output from the imaging unit 112 or image data read from the volatile memory 102. The image processing unit 113 converts the image data that has been subjected to the image processing into an image file in a predetermined format (e.g., JPEG) and records the image file in the recording medium 110. The image processing unit 113 also creates display data for displaying an image on the display unit 105.

[0035] The encoding processing unit 114 encodes the image data processed by the image processing unit 113 by intraframe predictive coding (intra-screen predictive coding), interframe predictive coding (inter-screen predictive coding), or the like.

[0036] The above-mentioned components are connected via an internal bus 115 so as to be able to transmit and receive data to and from each other.

[0037] <Device configuration of information processing device 200> Next, the configuration and functions of the information processing apparatus 200 according to the first embodiment will be described with reference to FIG.

[0038] The information processing device 200 is a desktop or notebook personal computer (PC) or the like.

[0039] The information processing device 200 includes a control unit 201 , a volatile memory 202 , a non-volatile memory 203 , an operation unit 204 , a display unit 205 , a recording unit 206 , a communication unit 207 , a learning unit 208 , and an internal bus 209 .

[0040] The control unit 201 has a processor (CPU) that performs arithmetic processing and control processing of the information processing device 200 , and controls each component of the information processing device 200 by executing a control program stored in the non-volatile memory 203 .

[0041] The volatile memory 202 is a RAM. Constants, variables, programs, etc. for the operation of the control unit 201, and learning programs used in learning processing by the learning unit 208 are loaded into the volatile memory 202. The learning programs include information such as learning data, teacher data, inference parameters, and the neural network structure of an inference model.

[0042] The non-volatile memory 203 is an electrically erasable and recordable EEPROM, a flash memory, etc. The non-volatile memory 203 stores constants and control programs for the operation of the control unit 101, a learning program used by the learning unit 208 for learning processing, etc.

[0043] The non-volatile memory 203 also stores an OS (operating system), which is basic software executed by the control unit 201, and applications that work with the OS to realize applied functions. In this embodiment, the non-volatile memory 203 stores an application for providing an inference program creation tool, which will be described later with reference to FIG. 4. The inference program creation process of this embodiment is realized by reading software provided by the application. It is assumed that the application has software for utilizing the basic functions of the OS installed in the information processing device 200.

[0044] The operation unit 204 is an operation member such as various switches, buttons, a touch panel, or a remote controller that receives various operations from the user and notifies the control unit 201 of the operations.

[0045] The display unit 205 includes a liquid crystal panel or an organic EL panel, and displays various information and a GUI (Graphical User Interface) in response to instructions from the control unit 201.

[0046] The recording unit 206 is an interface that controls access to the recording medium 210. The recording unit 206 controls writing and reading of data to and from the recording medium 210 based on instructions from the control unit 201. A learning program and the like used by the learning unit 208 for learning processing are recorded in the recording medium 210. The recording medium 210 is composed of a memory card, a hard disk, and the like.

[0047] The communication unit 207 includes an interface for performing wireless or wired communication with an external device. The wireless communication interface is, for example, a wireless LAN (Local Area Network) conforming to the wireless communication standard IEEE802.21n / a / g / b. The communication unit 207 connects to an external access point by wireless LAN and performs wireless communication with the external device via the access point. Note that the wireless communication interface is not limited to wireless LAN, and infrared communication, Bluetooth (registered trademark), Bluetooth Low Energy, Wireless USB, etc. may be used. The wired communication interface is a USB cable, HDMI (registered trademark), IEEE1394, etc. The communication unit 207 communicates with external devices including the inference device 100 and the extension device 120, and exchanges image data, control data, learning programs, etc.

[0048] The learning unit 208 has a machine learning function including deep learning using a neural network. The learning unit 208 executes learning processing of an inference model and inference parameters according to a learning program including information such as learning data, teacher data, inference parameters, and a neural network structure of an inference model recorded in a recording medium 210. Note that various data used in the learning processing may be received from an external device by the communication unit 207. In this case, the amount of resource consumption of the information processing device 200 can be reduced.

[0049] The learning unit 208 includes a GPU or a DSP (Digital Signal Processor). The learning process in the learning unit 2082 can be executed by the GPU or the DSP. The GPU or the DSP is a processor capable of performing a large amount of product-sum operations, and has the computational processing power to perform matrix operations of a neural network in a short time. Note that the learning process may be performed by a CPU of the control unit 201 and a GPU of the learning unit 208 working together, or may be performed by either the CPU of the control unit 201 or the GPU of the learning unit 208.

[0050] Note that, similar to the inference unit 108, the learning unit 208 can execute inference processing by deep learning using a neural network on input data based on an inference model and inference parameters.

[0051] The above-mentioned components are connected via an internal bus 209 so as to be able to transmit and receive data to and from each other.

[0052] <Inference program creation tool> Next, with reference to FIG. 4, an inference program creation tool 400 provided by the information processing device 200 of the first embodiment will be described.

[0053] The inference program creation tool 400 is a support tool for creating program code of an inference program for executing inference processing in the inference device 100 and the extension device 120. The support tool 400 is provided by the control unit 201 of the information processing device 200 executing an application program of the support tool 400 stored in the non-volatile memory 203.

[0054] As shown in Fig. 4, the data used by the support tool 400 to create an inference program are first information 401 on the hardware of the inference device 100, which is information on the inference processing executed in the inference unit 108 of the inference device 100, second information 402 on the hardware of the extension device 120, which is information on the inference processing of the extension device 120, and third information 403 on the setting of the hardware at the time of program conversion. Note that, as in the third embodiment described later in Fig. 9, when a plurality of extension devices 120 and 130 are connected to the inference device 100, the second information 402 exists for each extension device. In addition, the support tool 400 converts an inference program including a trained inference model 404 based on the first information 401 on the inference device 100, the second information 402 on the extension device 120, and the third information 403 for program conversion, to create a program code that can be operated by the inference device 100 and the extension device 120.

[0055] First information 401 of inference device 100 includes items related to hardware such as the scale and operating frequency of the product-sum calculation circuit, which is the inference processing performance of inference unit 108, power consumption, processable data type (number of bits such as floating point, integer, 8 bits, 16 bits, etc.), data access amount during calculation processing, and internal memory held by the hardware. First information 401 of inference device 100 may include information that can be determined by actually executing a simulation, in which case it also includes information necessary for executing the simulation. Note that first information 401 of inference device 100 is not limited to the above-mentioned information and may include other information.

[0056] The second information 402 of the extension device 120, like the first information 401 of the inference device 100, includes items related to hardware such as the scale and operating frequency of the product-sum calculation circuit, which is the inference processing performance of the extension device 120, power consumption, processable data type (number of bits such as floating point, integer, 8 bits, 16 bits, etc.), data access amount during calculation processing, and internal memory held by the hardware. In addition, the second information 402 of the extension device 120 includes information related to the type of connection bus between the inference device 100 and the extension device 120 connected by the interface control unit 109 and the transfer speed of the bus. Furthermore, the second information 402 of the extension device 120 may include information that can be determined by actually executing a simulation, and in that case, information necessary for executing the simulation is also included. Note that the second information 402 of the extension device 120 is not limited to the above-mentioned information, and may include other information. In addition, the second information 402 of the extension device 120 may be text information or a file in a unique format that can be managed by the support tool 400.

[0057] The third information 403 for program conversion includes setting information for converting the trained inference model 404 into program code executable by the inference unit 108. The setting information includes, for example, information on the selection of items to be compared from the first information 401 of the inference device 100 and the second information 402 of the extension device 120, setting the priority of the items to be compared, setting a threshold value for each piece of information, and the like.

[0058] <Control process of the information processing device 200> Next, the control process of the information processing apparatus 200 according to the first embodiment will be described with reference to FIG.

[0059] The processing of FIG. 5 is realized when the information processing device 200 is powered on by the control unit 201 loading a control program stored in the non-volatile memory 203 into the volatile memory 202, executing it, and controlling each component of the information processing device 200.

[0060] In step S501, the control unit 201 determines whether or not it is necessary to set the second information 402 of the extension device 120. When performing conversion processing of an inference program, the control unit 201 determines whether or not data is included in the second information 402 of the extension device 120 recorded on the recording medium 210. If the control unit 201 determines that data is included in the second information 402 of the extension device 120 (YES in step S501), the process proceeds to step S502. If the control unit 201 determines that data is not included in the second information 402 of the extension device 120 (NO in step S501), the process proceeds to step S503.

[0061] In step S502, the control unit 201 displays a setting screen on the display unit 205, on which the second information 402 of the extension device 120 can be set. The user can input the second information 402 of the extension device 120 into the setting screen using the operation unit 204. The control unit 201 records the information input by the user via the operation unit 204 in the recording medium 210 as the second information 402 of the extension device 120, and proceeds to step S503.

[0062] In step S503, the control unit 201 reads the second information 402 of the extension device 120 from the recording medium 210 using the recording unit 206, and loads it into the volatile memory 202, and the process proceeds to step S504.

[0063] In step S504, the control unit 201 reads the first information 401 of the inference device 100 from the recording medium 210 by the recording unit 206 and loads it into the volatile memory 202, and the process proceeds to step S505.

[0064] In step S505, the control unit 201 reads the trained inference model 404 from the recording medium 210 via the recording unit 206 and loads it into the volatile memory 202, and the process proceeds to step S506.

[0065] In step S506, the control unit 201 reads the third information 403 for program conversion from the recording medium 210 using the recording unit 206, and loads it into the volatile memory 202, and the process proceeds to step S507.

[0066] In step S507, the control unit 201 determines whether to allocate the computational processing of each processing layer of the trained inference model 404 to the inference unit 108 of the inference device 100 or to the extension device 120, and proceeds to step S508. Details of the allocation process will be described later with reference to FIG.

[0067] In step S508, the control unit 201 creates program code for an inference program executable by the inference unit 108 and the extension device 120 of the inference device 100 in accordance with the allocation method determined in step S507, and ends this process. In this case, the control unit 201 may execute a quantization process to reduce the weight of the inference model, such as reducing the number of bits of the inference parameters, as necessary.

[0068] FIG. 6 is a flowchart showing the allocation process in step S507 of FIG.

[0069] In step S601, the control unit 201 determines whether the arithmetic processing of each processing layer in the trained inference model 404 can be executed by both the inference unit 108 and the extension unit 120 of the inference device 100. If the control unit 201 determines that the arithmetic processing of each processing layer in the trained inference model 404 can be executed by both the inference unit 108 and the extension unit 120 of the inference device 100 (YES in step S601), the control unit 201 proceeds to step S602. If the control unit 201 determines that the arithmetic processing of each processing layer in the trained inference model 404 cannot be executed by both the inference unit 108 and the extension unit 120 of the inference device 100 (NO in step S601), the control unit 201 proceeds to step S606.

[0070] In step S602, the control unit 201 selects items to compare between the first information 401 of the inference device 100 and the second information of the extension device 120 based on the third information 403 for program conversion, stores the selected items in the volatile memory 202, and proceeds to step S603.

[0071] In step S603, the control unit 201 sets a priority among the items selected in step S602, and proceeds to step S604. For example, when the control unit 201 determines that the highest priority is given to the calculation performance calculated from the circuit scale and frequency of the product-sum calculation and the performance value of the inference process including the connection bus and data access speed, the control unit 201 selects the setting value of the inference process as the item to be compared.

[0072] In step S604, the control unit 201 compares the selected items with the threshold in the order of priority set in step S603, and determines whether or not both the first information 401 of the inference device 100 and the second information 402 of the extension device 120 exceed the threshold. If the control unit 201 determines that both the first information 401 of the inference device 100 and the second information 402 of the extension device 120 exceed the threshold (YES in step S604), the control unit 201 advances the process to step S605. If the control unit 201 determines that neither the first information 401 of the inference device 100 nor the second information 402 of the extension device 120 exceed the threshold (NO in step S604), the control unit 201 returns the process to step S603, and compares the comparison item of the next priority with the threshold.

[0073] In step S605, the control unit 201 determines whether the inference unit 108 of the inference device 100 or the extension device 120 should execute the processing for the item determined to exceed the threshold in step S604, and proceeds to step S607. If the control unit 201 determines in step S604 that either the first information 401 of the inference device 100 or the second information 402 of the extension device 120 exceeds the threshold, the control unit 201 selects the hardware that exceeds the threshold. If the control unit 201 determines in step S604 that both the first information 401 of the inference device 100 and the second information 402 of the extension device 120 exceed the threshold, the control unit 201 selects the superior hardware. For example, if the control unit 201 determines to compare the performance values ​​of the inference processing calculated from the circuit scale, frequency, and data communication speed of the product-sum operation, the control unit 201 selects the one with the higher performance value of the inference processing.

[0074] In step S606, the control unit 201 determines that the computational processing of each processing layer in the trained inference model 404 cannot be processed by either the inference unit 108 of the inference device 100 or the extension device 120, and therefore that the processing should be performed by the CPU of the control unit 101, and proceeds to step S607.

[0075] In step S607, the control unit 201 determines whether or not hardware has been selected to perform arithmetic processing for all processing layers of the trained inference model 404. If the control unit 201 determines that selection of hardware to perform arithmetic processing for all processing layers of the trained inference model 404 has been completed (YES in step S607), the control unit 201 ends this process. If the control unit 201 determines that selection of hardware to perform arithmetic processing for all processing layers of the trained inference model 404 has not been completed (NO in step S607), the control unit 201 returns the process to step S601.

[0076] In setting the priority in step S603, if power consumption is prioritized, processing is allocated so that power consumption takes precedence, whereas if inference processing performance is prioritized, processing is allocated so that processing is executed at high speed. For example, it is possible to create appropriate program code according to the priority setting.

[0077] <Control process of the inference device 100> Next, the control process of the inference device 100 of the first embodiment will be described with reference to FIG.

[0078] The process of Fig. 7 is realized by the control unit 101 loading a control program stored in the non-volatile memory 103 into the volatile memory 202 and executing it while the power supply of the inference device 100 is on, and controlling each component of the inference device 100. Note that the program code of the inference process executed by the inference unit 108 and the extension device 120 of the inference device 100 is created according to a plurality of states and recorded in the recording medium 210. For example, the program code in which the process is assigned so as to reduce power consumption and the program code in which the process is assigned so as to increase the inference process performance are recorded in the recording medium 210. In addition, the program code for executing the inference process by the inference unit 108 alone when the extension device 120 is not connected to the inference device 100 is also recorded in the recording medium 210.

[0079] In step S701, the control unit 101 determines whether or not the expansion device 120 is connected via the interface control unit 109. If the control unit 101 determines that the expansion device 120 is connected (YES in step S701), the process proceeds to step S702. If the control unit 101 determines that the expansion device 120 is not connected (NO in step S701), the process proceeds to step S705.

[0080] In step S702, the control unit 101 obtains information about the operation mode loaded into the volatile memory 102, and determines the operation mode of the inference device 100. If the control unit 101 determines that the operation mode is the normal mode (normal mode in step S702), the process proceeds to step S703. If the control unit 101 determines that the operation mode is a power saving mode that operates with less power than normal (power saving mode in step S702), the process proceeds to step S704.

[0081] In step S703, the control unit 101 loads into the volatile memory 102 the program code of the inference program recorded on the recording medium 110, the program code being assigned to processes so as to improve the inference processing performance, the inference unit 108 executes the inference processing, and the process is terminated.

[0082] In step S704, the control unit 101 loads into the volatile memory 102 the program code of the inference program recorded on the recording medium 110, in which processing is allocated so as to reduce power consumption, and the inference unit 108 executes the inference processing, and the process is terminated.

[0083] In step S705, the control unit 101 loads into the volatile memory 102 program code, which is an inference program recorded on the recording medium 110 and which executes the inference processing only by the inference unit 108, and the inference unit 108 executes the inference processing, thereby terminating this process.

[0084] In this way, inference device 100 and extension device 120 execute the inference processing assigned to inference device 100 and extension device 120 according to predetermined conditions such as the operation mode of inference device 100 .

[0085] As described above, according to the first embodiment, when the extension device 120 is connected to the inference device 100, the computational processes of each processing layer in the inference model are appropriately allocated to the inference device 100 and the extension device 120. This allows the hardware resources for executing the inference processes of the inference device 100 and the extension device 120 to be effectively utilized.

[0086] [Embodiment 2] Next, a second embodiment will be described with reference to FIG.

[0087] In the first embodiment, the allocation of processing between the inference unit 108 of the inference device 100 and the extension device 120 is performed by comparing items with high priority with a threshold value. In the second embodiment, an example will be described in which the allocation of processing is performed by comparing all items with a threshold value.

[0088] The configurations of the inference device 100 and the information processing device 200 of the second embodiment are the same as those of the first embodiment. The control process of the information processing device 200 of the second embodiment is the same as that of the first embodiment shown in FIG. 5, and the inference process of the inference device 100 is the same as that of the first embodiment shown in FIG.

[0089] FIG. 8 is a flowchart showing the allocation process of step S507 in FIG. 5 according to the second embodiment.

[0090] Steps S801 and S802 are similar to steps S601 and S602 in FIG.

[0091] In step S803, the control unit 201 selects items that exceed a threshold from the items selected in step S802 based on the third information 403 for program conversion from the first information 401 of the inference device 100 and the second information 402 of the extension device 120, stores the items that exceed the threshold in the volatile memory 202, and proceeds to step S804.

[0092] In step S804, the control unit 201 determines the number of items exceeding the threshold selected in step S803. If the control unit 201 determines that the number of items exceeding the threshold is the same in the first information 401 of the extension device 120 and the second information 402 of the extension device 120 (YES in step S804), the control unit 201 proceeds to step S805. If the control unit 201 determines that the number of items exceeding the threshold is different in the first information 401 of the extension device 120 and the second information 402 of the extension device 120 (NO in step S804), the control unit 201 proceeds to step S806.

[0093] In step S805, it is determined whether the inference unit 108 of the inference device 100 or the extension device 120 will process the items exceeding the threshold selected in step S803, and the process proceeds to step S808. The control unit 201 compares the items with higher priorities from among the items selected in step S803, and selects the hardware with the superior priority. For example, when the control unit 201 determines that the comparison should be made based on the circuit scale of the product-sum operation, the frequency, and the data communication speed, it selects the one with the higher inference processing performance value.

[0094] In step S806, the control unit 201 selects either the inference unit 108 of the inference device 100 or the extension device 120, whichever has the greater number of items exceeding the threshold selected in S803, and proceeds to step S808.

[0095] Steps S807 and S808 are similar to steps S607 and S608 in FIG.

[0096] As described above, according to the second embodiment, when selecting either the inference device 100 or the extension device 120 as the hardware for executing the computational processing of each processing layer of the trained inference model 404, it is possible to make a decision based on a comparison of all processing performance items with threshold values.

[0097] [Embodiment 3] Next, a third embodiment will be described with reference to FIG.

[0098] In the first and second embodiments, an example has been described in which a single extension device 120 is connected to the inference device 100. In the third embodiment, an example in which a plurality of extension devices 120 and 130 are connected to the inference device 100 will be described.

[0099] The configurations of the inference device 100 and the information processing device 200 of the third embodiment are similar to those of the first embodiment shown in FIGS.

[0100] FIG. 9 is a diagram showing the configuration of a system including an inference device 100 and a plurality of extension devices 120 and 130 according to the third embodiment.

[0101] A plurality of extension devices 120, 130 can be mechanically detachably and electrically connected to the inference device 100. The configuration of the extension devices 120, 130 is the same as that of Fig. 1, and a volatile memory 131 is connected to the extension device 130. Note that, in the example of Fig. 9, for the sake of simplicity of explanation, a configuration in which two extension devices 120, 130 are connected is illustrated, but a configuration in which three or more extension devices are connected may also be used.

[0102] When multiple extension devices 120 and 130 are connected, the second information 402 of the extension device 120 shown in FIG. 4 exists for each extension device.

[0103] As a control process of the information processing device 200 of the third embodiment, in the process shown in Fig. 5, Fig. 6 or Fig. 8, the process is allocated to the inference device 100 and the multiple extension devices 120 and 130. For example, in steps S502, S503, S507 and S508 in Fig. 5, the process in Fig. 6 and Fig. 8 is performed for all items of all extension devices.

[0104] Furthermore, the control process of the inference device 100 of the third embodiment is executed on the premise that multiple extension devices are connected in the process shown in Fig. 7. For example, in Fig. 7, the control unit 101 determines whether multiple extension devices are attached in step S701, and executes the inference process by program codes in which the processes are allocated to the inference device 100 and the multiple extension devices 120 and 130 in steps S703 and S704.

[0105] As described above, according to the third embodiment, when selecting either the inference device 100 or the extension device 120 as the hardware for executing the computational processing of each processing layer of the trained inference model 404, it is possible to make an appropriate decision even when multiple extension devices are connected.

[0106] In each of the above-described embodiments, an example has been described in which the information processing device 200 automatically determines the method for allocating the inference process. However, for example, a function may be added to the support tool 400 shown in FIG. 4 that allows a user to manually determine the method for allocating the inference process.

[0107] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0108] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0109] The disclosure of this specification includes the following information processing device, inference device, control method, and program. [Configuration 1] A creating means for creating a program for executing an inference process using a trained inference model; a control means for distributing the inference processing to a first process executed in a first inference device and a second process executed in one or more second inference devices connectable to the first inference device; An information processing device characterized in that the creation means creates a program for executing the first processing and a program for executing the second processing based on first information regarding hardware for inference processing in the first inference device and second information regarding hardware for inference processing in the second inference device. [Configuration 2] The information processing device according to configuration 1, wherein the creation means converts a program for executing the inference processing into a program for executing the first processing and a program for executing the second processing based on the first information and the second information. [Configuration 3] 3. The information processing apparatus according to configuration 2, wherein the creating means further executes the conversion based on third information relating to settings for program conversion. [Configuration 4] The information processing device according to any one of configurations 1 to 3, characterized in that the control means selects an item to be compared from the first information and the second information, and determines a method of allocating the inference process by comparing the item with a threshold value. [Configuration 5] The information processing device according to configuration 4, wherein the control means selects an item to be compared from the first information and the second information, starting with an item with a high priority, and determines a method of allocating the inference process by comparing the selected item with a threshold value. [Configuration 6] The information processing device according to any one of configurations 1 to 3, characterized in that the control means determines a method for allocating the inference process by comparing all items of the first information and the second information with a threshold value. [Configuration 7] The information processing device according to configuration 6, wherein the control means determines a method for allocating the inference process depending on the number of items exceeding a threshold in the first information and the second information. [Configuration 8] when the number of items exceeding the threshold is the same in the first information and the second information, the control means determines a method of allocating the inference process according to an item with a higher priority; 8. The information processing device according to configuration 7, wherein, when the number of items exceeding the threshold differs, the inference process is assigned to the one with a larger number of items exceeding the threshold. [Configuration 9] the control means determines whether the inference process can be executed in both the first inference device and the second inference device; if the inference process can be executed by either the first inference device or the second inference device, determining a method for allocating the inference process; An information processing device as described in any one of configurations 1 to 8, characterized in that if the inference process cannot be executed by either the first inference device or the second inference device, a method for allocating the inference process is not determined, and it is decided to execute the inference process on hardware different from the first inference device and the second inference device. [Configuration 10] The information processing device according to any one of configurations 1 to 9, characterized in that the first information and the second information include at least one of inference processing performance, power consumption, processable data type, data access amount during calculation processing, and information regarding internal memory. [Configuration 11] 11. The information processing device according to configuration 10, wherein the second information includes at least one of information regarding the type and transfer speed of a connection bus between the first inference device and the second inference device. [Configuration 12] 12. The information processing device according to any one of configurations 1 to 11, further comprising a determination means for allowing a user to determine a method for allocating the inference processing. [Configuration 13] 13. The information processing device according to any one of configurations 1 to 12, further comprising setting means for enabling a user to set the second information. [Configuration 14] An information processing device described in configuration 2 or 3, characterized in that the conversion includes a quantization process for reducing the weight of the inference model. [Configuration 15] 15. The information processing device according to any one of configurations 1 to 14, wherein the second inference device is one or more extension devices for extending the inference processing function of the first inference device. [Configuration 16] The inference model includes a plurality of processing layers; The inference process includes calculation processes in each processing layer. 16. The information processing device according to any one of configurations 1 to 15, wherein the control means distributes arithmetic processing in each processing layer to the first processing or the second processing. [Configuration 17] 1. An inference device, comprising: An inference means for executing an inference process using a trained inference model; an interface means to which one or more extension devices for extending the functionality of the inference processing can be connected; a determination unit for determining whether the expansion device is connected by the interface unit; and a control means for executing the first process in accordance with a predetermined condition when the inference process is divided into a first process to be executed in the inference device and a second process to be executed in the extension device. [Configuration 18] the predetermined condition is an operating mode of the inference device; 18. The inference device according to configuration 17, wherein when the inference device is in a mode in which it operates with less power than normal, the inference process is allocated so as to reduce power consumption. [Configuration 19] the predetermined condition is an operating mode of the inference device; 18. An inference device according to configuration 17, characterized in that when the inference device is not in a mode in which it operates with less power than normal, the inference processing is allocated so as to increase inference processing performance. [Configuration 20] 20. The inference device according to any one of configurations 17 to 19, wherein the inference means executes the inference process when the extension device is not connected. [Configuration 21] A method for controlling an information processing device that executes an inference process using a trained inference model, comprising: distributing the inference process into a first process executed on a first inference device and a second process executed on one or more second inference devices connectable to the first inference device; A control method comprising the steps of: creating a program for executing the first processing and a program for executing the second processing based on first information regarding hardware for inference processing in the first inference device and second information regarding hardware for inference processing in the second inference device. [Configuration 22] A method for controlling an inference device, comprising: The inference device comprises: An inference means for executing an inference process using a trained inference model; and an interface means to which one or more extension devices for extending the function of the inference processing can be connected; The control method includes: determining whether the expansion device is connected by the interface means; and when the inference process is divided into a first process to be executed in the inference device and a second process to be executed in the extension device, executing the first process in accordance with a predetermined condition. [Configuration 23] 17. A program for causing a computer to function as an information processing device according to any one of claims 1 to 16. [Configuration 24] A program for causing a computer to function as the inference device described in any one of configurations 17 to 20. [Explanation of symbols]

[0110] 100... inference device, 101... control unit, 108... inference unit, 107... interface control unit, 200... information processing device, 201... control unit, 208... learning unit

Claims

1. A creating means for creating a program for executing an inference process using a trained inference model; a control means for distributing the inference processing to a first process executed in a first inference device and a second process executed in one or more second inference devices connectable to the first inference device; An information processing device characterized in that the creation means creates a program for executing the first processing and a program for executing the second processing based on first information regarding hardware for inference processing in the first inference device and second information regarding hardware for inference processing in the second inference device.

2. 2. The information processing apparatus according to claim 1, wherein the creation means converts the program for executing the inference processing into a program for executing the first processing and a program for executing the second processing based on the first information and the second information.

3. 3. The information processing apparatus according to claim 2, wherein said creating means executes said conversion further based on third information relating to settings for program conversion.

4. 2. The information processing apparatus according to claim 1, wherein the control means selects an item to be compared from the first information and the second information, and determines a method of allocating the inference process by comparing the selected item with a threshold value.

5. The information processing device according to claim 4, characterized in that the control means selects an item to be compared from the first information and the second information, starting with an item with a high priority, and determines a method of allocating the inference process by comparing the selected item with a threshold value.

6. 2. The information processing apparatus according to claim 1, wherein the control means determines a method of allocating the inference process by comparing all items of the first information and the second information with a threshold value.

7. 7. The information processing apparatus according to claim 6, wherein the control means determines a method of allocating the inference process depending on the number of items exceeding a threshold value in the first information and the second information.

8. the control means determines a method of allocating the inference process according to an item with a higher priority when the number of items exceeding the threshold is the same in the first information and the second information; 8. The information processing apparatus according to claim 7, wherein, when the number of items exceeding the threshold differs, the inference process is assigned to the one having a larger number of items exceeding the threshold.

9. the control means determines whether the inference process can be executed in both the first inference device and the second inference device; if the inference process can be executed by either the first inference device or the second inference device, determining a method for allocating the inference process; The information processing device according to claim 1, characterized in that if the inference process cannot be executed by either the first inference device or the second inference device, a method for allocating the inference process is not determined, and it is decided to execute the inference process on hardware different from the first inference device and the second inference device.

10. 2. The information processing device according to claim 1, wherein the first information and the second information include at least one of inference processing performance, power consumption, processable data types, data access amount during calculation processing, and information regarding internal memory.

11. 11. The information processing apparatus according to claim 10, wherein the second information includes at least one of information on a type and a transfer rate of a connection bus between the first inference apparatus and the second inference apparatus.

12. 2. The information processing apparatus according to claim 1, further comprising a determination means for allowing a user to determine a method for allocating the inference processing.

13. 2. The information processing apparatus according to claim 1, further comprising setting means for enabling a user to set the second information.

14. The information processing device according to claim 2 , wherein the conversion includes a quantization process for reducing the weight of the inference model.

15. 2. The information processing apparatus according to claim 1, wherein the second inference device is one or more extension devices for extending the inference processing function of the first inference device.

16. The inference model includes a plurality of processing layers; The inference process includes calculation processes in each processing layer.

2. The information processing apparatus according to claim 1, wherein the control means distributes arithmetic processing in each processing layer to the first processing or the second processing.

17. 1. An inference device, comprising: An inference means for executing an inference process using a trained inference model; an interface means to which one or more extension devices for extending the functions of the inference processing can be connected; a determination unit for determining whether the expansion device is connected by the interface unit; and a control means for executing the first process in accordance with predetermined conditions when the inference process is divided into a first process to be executed in the inference device and a second process to be executed in the extension device.

18. the predetermined condition is an operating mode of the inference device; 18. The inference device according to claim 17, wherein when the inference device is in a mode in which it operates with less power than normal, the inference process is allocated so as to reduce power consumption.

19. the predetermined condition is an operating mode of the inference device; 18. The inference device according to claim 17, wherein when the inference device is not in a mode in which it operates with less power than normal, the inference process is allocated so as to increase inference process performance.

20. 18. The inference device according to claim 17, wherein the inference means executes the inference process when the extension device is not connected.

21. A method for controlling an information processing device that executes an inference process using a trained inference model, comprising: distributing the inference process into a first process executed in a first inference device and a second process executed in one or more second inference devices connectable to the first inference device; A control method comprising the steps of creating a program for executing the first processing and a program for executing the second processing based on first information regarding hardware for inference processing in the first inference device and second information regarding hardware for inference processing in the second inference device.

22. A method for controlling an inference device, comprising: The inference device comprises: An inference means for executing an inference process using a trained inference model; and an interface means to which one or more extension devices for extending the function of the inference processing can be connected, The control method includes: determining whether the expansion device is connected by the interface means; and when the inference processing is divided into a first processing to be executed in the inference device and a second processing to be executed in the extension device, executing the first processing in accordance with a predetermined condition.

23. A program for causing a computer to function as the information processing device according to any one of claims 1 to 16.

24. A program for causing a computer to function as the inference device according to any one of claims 17 to 20.

Images