Information processing system and method for controlling information processing system

Abstract

To reduce the time required for inference processing in a more suitable mode according to the degree of congestion of radio resources, in a situation where a plurality of apparatuses co-operate to execute the inference processing.SOLUTION: An inference processing system includes an imaging apparatus 101 and a server 103. A radio congestion determination unit 218 determines the degree of congestion of radio resources on the basis of the communication state of another apparatus connected to a radio network. A CPU 212 of the imaging apparatus 101 sequentially executes processing on a plurality of intermediate layers until the degree of congestion becomes smaller than a threshold and generates intermediate data. The imaging apparatus 101 transmits the generated intermediate data to the server 103 via the radio network. A CPU 202 of the server 103 sequentially executes, with the intermediate data as input, processing on a series of intermediate layers on which the processing is not executed by the imaging apparatus 101, of the plurality of intermediate layers.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and a control method for an information processing system. [Background technology]

[0002] Inference processing systems that perform calculations using neural networks are known. In particular, so-called convolutional neural networks (CNNs) are often used in inference processing systems that perform image recognition. In a convolutional neural network, input image data is processed in the intermediate layer and then in the fully connected layer, resulting in a final inference result that recognizes the object contained in the image. In the intermediate layer, multiple feature extraction processing layers are hierarchically connected, and in each processing layer, convolutional calculations, activation processing, and pooling processing are performed on the input data input from the previous layer. By repeating the processing in each processing layer in this way, the intermediate layer extracts high-dimensional features contained in the input image data. In the fully connected layer, the calculation results data of the intermediate layers are combined to obtain the final inference result. To extract features in high dimensions, the number of intermediate layers is important for the accuracy of the final inference result.

[0003] However, increasing the number of intermediate layers can impose a heavy computational load on the neural network inference process, resulting in a long processing time, especially in devices with relatively low computational processing power, such as an imaging device. One way to solve this problem is to send the input image to a server with relatively high computational power and have the server perform the neural network inference. Patent Document 1 discloses a technology in which, when learning in a neural network is performed on a server, in order to maintain confidentiality of privacy-related information, some intermediate layer processing is performed on the imaging device side before being sent to the server, and the remaining intermediate layers and output layers are processed on the server. Furthermore, Patent Document 2 discloses a technology that explicitly creates a layer during learning that minimizes the amount of data traffic required for the intermediate layer sent from the imaging device to the server, and then sends that layer to the server during inference, thereby shortening the time required for communication. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 011842 [Patent Document 2] Patent Publication No. 2021-124832 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, when the intermediate layer data is transmitted from the imaging device to the server using wireless communication, wireless resources may become congested depending on the number of terminals using wireless communication on the same channel and the radio wave environment, which may increase the time required for communication. Therefore, even if the time required for inference processing is reduced, the time required to obtain the final inference result may not be reduced.

[0006] In consideration of the above problems, the present invention aims to shorten the time required for inference processing in a more suitable manner depending on the degree of congestion of wireless resources in a situation where inference processing is performed by multiple devices working together. [Means for solving the problem]

[0007] The information processing system of the present invention includes a first device and a second device connected via a wireless network, and causes an inference process to be performed by a neural network in which an input layer, a plurality of intermediate layers that extract features contained in data input from a previous layer, and an output layer are hierarchically connected, wherein the first device has a determination means for determining the congestion level of wireless resources in communication via the wireless network based on the communication status of other devices connected to the wireless network, a first control means for sequentially executing processing of each of the plurality of intermediate layers until the congestion level becomes smaller than a threshold value to generate intermediate data, and a first communication means for transmitting the intermediate data generated by the first control means to the second device via the wireless network, and the second device has a second communication means for receiving the intermediate data from the first device via the wireless network, and a second control means for sequentially executing processing of each of a series of intermediate layers among the plurality of intermediate layers that have not been processed in the first device, using the intermediate data received by the second communication means as input. [Effects of the Invention]

[0008] According to the present invention, in a situation where inference processing is performed by multiple devices working together, it is possible to shorten the time required for the inference processing in a more suitable manner depending on the degree of congestion of wireless resources. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an inference processing system. [Figure 2] FIG. 1 is a diagram showing an example of the configuration of an inference processing system. [Figure 3] FIG. 1 is a conceptual diagram of a mechanism for sharing inference processing among multiple devices. [Figure 4] 10 is a flowchart showing an example of processing of the inference processing system. [Figure 5] FIG. 1 is a diagram illustrating an example of a configuration of an inference processing system. [Figure 6] 10 is a flowchart showing an example of processing of the inference processing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] First Embodiment An inference processing system using a neural network will be described below as an example of an information processing system according to a first embodiment of the present disclosure with reference to FIGS. 1 is a diagram showing an example of the system configuration of an inference processing system according to this embodiment. The neural network used in the inference processing system according to this embodiment is configured by hierarchically connecting an input layer, multiple intermediate layers that extract features contained in data input from the previous layer, and an output layer. With this configuration, various operations are sequentially performed by each of the multiple intermediate layers on data input via the input layer, and the results are output via the output layer.

[0012] 1, the inference processing system according to this embodiment includes an imaging device 101, a server 103, and a wireless communication device 102. The imaging device 101 and the server 103 are connected to each other via the wireless communication device 102 so that they can transmit and receive information to and from each other. Note that this embodiment will be described focusing on a case where the inference processing system is configured to include the imaging device 101 and the server 103, but the system is not limited to this configuration, and for example, a mobile phone, a tablet terminal, or the like may be used instead of the imaging device 101. Furthermore, in the information processing system according to this embodiment, the server 103 is assumed to have higher processing power than other devices (hereinafter also referred to as first terminals) such as the imaging device 101 connected to the server 103 via the wireless communication device 102. In addition, in this embodiment, an example in which a wireless LAN is applied as a wireless communication method will be described, but the wireless communication method to be applied is not limited thereto. As a specific example, cellular communication (3G, 4G, 5G, etc.) or millimeter wave communication may be applied.

[0013] An example of the configuration of each of the imaging device 101, server 103, and wireless communication device 102 that constitute the inference processing system according to this embodiment will be described with reference to Fig. 2. The imaging device 101 and server 103 are communicably connected to the wireless communication device 102 via a wired communication network 231. In the inference processing system according to this embodiment, the imaging device 101 corresponds to an example of a first device, and the server 103 corresponds to an example of a second device.

[0014] The imaging device 101 includes a CPU (Central Processing Unit) 212, a ROM (Read Only Memory) 213, a memory 214, an input unit 215, a display unit 216, an imaging unit 217, a wireless congestion level determination unit 218, and an imaging device wireless communication unit 219. These components of the imaging device 101 are connected via a system bus 211 so as to be able to send and receive data to and from each other.

[0015] The CPU 212 controls the operations of the display unit 216, the imaging unit 217, the wireless congestion level determination unit 218, and the imaging device wireless communication unit 219 in accordance with a program based on a control signal supplied from the input unit 215 in response to a user operation received by the input unit 215. This makes it possible to cause the display unit 216, the imaging unit 217, the wireless congestion level determination unit 218, and the imaging device wireless communication unit 219 to operate in accordance with the user operation. The ROM 213 stores various programs and the like for the operation of the CPU 212. Note that the various programs for the operation of the CPU 212 are not limited to being stored in the ROM, and may be stored in other storage areas such as a hard disk. The memory 214 is realized by, for example, a random access memory (RAM), and is a storage area for temporarily holding data. The CPU 212 uses the memory 214 as a work memory in accordance with a program stored in the ROM 213, for example.

[0016] The input unit 215 accepts a user operation, generates a control signal in response to the user operation, and supplies the generated control signal to the CPU 212. For example, the input unit 215 may have physical operation buttons, a touch panel, or the like as an input device for accepting a user operation. Note that the touch panel is an input device configured to output coordinate information corresponding to a position touched on a planar input unit, for example. The display unit 216 is realized by, for example, a display, and includes a mechanism for outputting a display signal for displaying an image on a screen. When a touch panel is used as the input unit 215, the input unit 215 and the display may be integrally configured. In this case, for example, the touch panel may be configured so that its light transmittance does not interfere with the display on the display, and may be attached to the upper layer of the display surface of the display. Then, input coordinates on the touch panel may be associated with display coordinates on the display.

[0017] The imaging unit 217 is a mechanism that performs a series of photographing processes, including an imaging unit that is composed of a lens, a shutter with an aperture function, and a CCD or CMOS element that converts an optical image into an electrical signal, and an image processing unit that performs various image processes such as exposure control and distance measurement control based on the signal from the imaging unit. The operation of the imaging unit 217 is controlled by the CPU 212, which makes it possible to capture an image of a subject in response to a user operation via the input unit 215.

[0018] The wireless congestion level determination unit 218 determines the congestion level of the wireless communication resources (hereinafter also referred to as wireless communication congestion level) by observing (intercepting) wireless radio waves of data transfers by wireless communication of other terminals under the control of the CPU 212. Then, the wireless congestion level determination unit 218 notifies the CPU 212 of the determination result of the wireless communication congestion level. Specifically, by observing the wireless radio waves during communication of each terminal using wireless communication, it is possible to detect the extent to which wireless resources of the same channel are being used, and therefore it is possible to determine the wireless communication congestion level by using the detection result. When a wireless LAN is applied as in this embodiment, wireless communication between the image capture device 101 and the wireless communication device 102 is controlled in a time-division manner, and therefore, when the wireless communication is highly congested, a waiting time occurs, which may result in a slower communication speed. On the other hand, when the wireless communication is not highly congested, no waiting time occurs, or even if a waiting time occurs, it is for a shorter period of time, which results in faster data transfer. When observing (eavesdropping) the radio waves of other terminals, noise radio waves from devices such as microwave ovens may be included in the observation results, but by analyzing the observed radio waves, it is possible to determine whether the radio waves are radio waves for a wireless LAN or noise radio waves. On the other hand, such noise radio waves may affect the communication efficiency of wireless communication, so the influence of the noise radio waves may be taken into account when determining the congestion level of wireless communication.

[0019] The imaging device wireless communication unit 219 performs wireless communication via the wireless communication device 102 and transmits and receives data to and from the server 103 based on the control of the CPU 212 and in accordance with the determination result of the wireless communication congestion level by the wireless congestion level determination unit 218. In this embodiment, the CPU 212 determines the timing of transmitting and receiving data to and from the server 103 based on the determination result of the wireless communication congestion level by the wireless congestion level determination unit 218. The imaging device wireless communication unit 219 corresponds to an example of a first communication unit.

[0020] The server 103 includes a CPU 202, a memory 203, a GPU (Graphics Processing Unit) 204, and a server wired communication unit 205. These components of the server 103 are connected via a system bus 201 so as to be able to send and receive data to and from each other.

[0021] The CPU 202 controls the operations of the GPU 204 and the server wired communication unit 205 in accordance with a program. The memory 203 is realized by, for example, a RAM, and is used as a work memory for the CPU 202 and the GPU 204 . It should be noted that the programs for operating the CPU 212 are stored in a storage area such as a hard disk or ROM (not shown). The GPU 204 is configured to perform efficient calculations by processing a large amount of data in parallel. When performing learning multiple times using a learning model such as a neural network, it is effective to have the GPU 204 perform the processing. The GPU 204 can generally perform neural network operations such as matrix operations in a shorter time than the CPU 202.

[0022] In this embodiment, an example in which the CPU 202 and the GPU 204 are configured in the server 103 will be described, but the configuration of the server 103 is not necessarily limited thereto. As described above, any configuration in which the arithmetic processing capability of the server 103 is superior to that of the imaging device 101 is applicable, not limited to the configuration illustrated in FIG. 2 . Note that in this embodiment, the arithmetic processing capability of the server 103 corresponds to the combined arithmetic processing capability of the CPU 202 and the GPU 204, and the arithmetic processing capability of the imaging device 101 corresponds to the arithmetic processing capability of the CPU 212. Furthermore, the arithmetic processing capability in this disclosure corresponds to processing capability based on an index of how much neural network inference (which corresponds to matrix operations, etc.) can be processed. Furthermore, the GPU 204 does not need to be one, and multiple GPUs 204 may be configured.

[0023] Under the control of the CPU 202, the server wired communication unit 205 communicates with the imaging device 101 via a wired communication network 231 such as a LAN or the Internet and the wireless communication device 102 that performs wireless communication. In this embodiment, the CPU 202 of the server 103 receives a communication request from the image capturing device 101, generates a control signal in response to the communication request, and operates the GPU 204. Specific details of the communication between the image capturing device 101 and the server 103 will be described separately later with reference to FIG.

[0024] The wireless communication device 102 has a CPU 222, a memory 223, a wireless communication device wired communication unit 225, and a wireless communication device wireless communication unit 229. These components of the wireless communication device 102 are connected via a system bus 221 so as to be able to transmit and receive data to and from each other. In this embodiment, as described above, a wireless LAN is used for wireless communication between the wireless communication device 102 and the imaging device 101. In this case, a wireless router corresponds to the wireless communication device 102. Furthermore, a communication device such as the wireless communication device 102, which enables the server 103 to communicate with other devices such as the imaging device 101 via a wireless network, corresponds to an example of a second communication unit.

[0025] The CPU 222 controls the operations of the wired communication unit 225 of the wireless communication device and the wireless communication unit 229 of the wireless communication device in accordance with a program. The memory 223 is realized by, for example, a RAM, and is used as a work memory for the CPU 222. It should be noted that the programs for operating the CPU 222 are stored in a storage area such as a hard disk or ROM (not shown).

[0026] The wireless communication device wired communication unit 225 communicates with the server 103 via a wired communication network 231 such as a LAN or the Internet under the control of the CPU 222. In this embodiment, the wireless communication device 102 receives data from the imaging device 101 via the wireless communication device wireless communication unit 229 and transmits the data to the server 103 via the wired communication network 231. The wireless communication device wireless communication unit 229 communicates with the imaging device wireless communication unit 219 of the imaging device 101 via a wireless network under the control of the CPU 222. In this embodiment, the wireless communication device wireless communication unit 229 accepts a wireless communication request from the imaging device wireless communication unit 219 of the imaging device 101, and transmits and receives data to and from the imaging device wireless communication unit 219.

[0027] A mechanism for sharing and executing inference processing among a plurality of devices will be described with reference to Fig. 3. In the information processing system according to this embodiment, the neural network applied to the inference processing by the image capture device 101 has an input layer 301, a first hidden layer 302, and a second hidden layer 303. The processing of each layer is realized by the CPU 212 of the image capture device 101 operating in accordance with a program. The CPU 212, which is responsible for the processing of each layer on the image capture device 101 side, corresponds to an example of a first control unit. The neural network applied by the server 103 to the inference process has an input layer 301, a third hidden layer 305 to an Nth hidden layer 306, and an output layer 307. The processing of each layer is realized by the CPU 202 and GPU 204 of the server 103 operating in accordance with a program. The CPU 202 and GPU 204, which are the main processors of the processing of each layer on the server 103 side, correspond to an example of a second control unit. The server 103 receives intermediate data corresponding to the result of the inference processing performed by the imaging device 101 from the imaging device 101 by wireless communication with the imaging device 101 via the wireless communication device 102. The server 103 inputs the intermediate data received from the imaging device 101 to the input layer 304, and outputs the results of sequential calculations performed in each layer from the output layer 307 as the inference result of the information processing system.

[0028] Assuming the above-described configuration, in the information processing system according to this embodiment, the imaging device 101 determines up to which layer the imaging device 101 will perform processing, depending on the result of the wireless communication congestion level determination by the wireless congestion level determination unit 218. As a specific example, the imaging device 101 may perform processing up to the second intermediate layer 303, and then, depending on the wireless communication congestion level, have the server 103 perform subsequent processing. By applying such control, the imaging device 101 can proceed with processing of the intermediate layer until the wireless communication congestion level is low and wireless communication can be performed relatively quickly, and after that point in time, have the server 103 perform the remaining processing. This minimizes the time required for wireless communication between the imaging device 101 and the server 103, making it possible to perform inference processing more efficiently.

[0029] With reference to Fig. 4, an example of the processing of the inference processing system according to this embodiment will be described below, which is an example of a series of processing flows when the imaging device 101 and the server 103 cooperate to realize the calculation processing of the inference model. In the example shown in Fig. 4, the processing of S401 to S405 and S410 is executed by the imaging device 101, and the processing of S406 to S409 is executed by the server 103. Therefore, hereinafter, the processing of the imaging device 101 and the processing of the server 103 will be described separately.

[0030] First, an example of processing by the image capture device 101 will be described. In S401, the imaging unit 217 acquires image data to be input to the inference model. The CPU 212 inputs the image data acquired by the imaging unit 217 to the inference model, thereby starting a series of inference processes. In S402, the CPU 212 executes processing for the input layer 301 and the first hidden layer 302. If the processing for S402 is executed again as a result of the determination processing in S404 (described later), the CPU 212 executes processing for a predetermined number of hidden layers. For convenience, in this embodiment, the CPU 212 executes processing for one hidden layer. In S403, the wireless congestion level determination unit 218 determines the level of wireless communication congestion (that is, the level of wireless communication resource congestion) and notifies the CPU 212 of the determination result.

[0031] In S404, the CPU 212 determines whether the degree of congestion in wireless communication is lower than a predetermined threshold based on the determination result notified in S403. If the CPU 212 determines in S404 that the degree of congestion in wireless communication is lower than the predetermined threshold, the process proceeds to S405. On the other hand, if the CPU 212 determines in S404 that the degree of congestion in wireless communication is equal to or greater than the predetermined threshold, the process proceeds to S402. In this case, the processes from S402 onwards are executed again. In S405, the CPU 212 transmits the intermediate layer data that has been processed up to this point in time from the imaging device wireless communication unit 219 to the server 103 via the wireless communication device 102.

[0032] In S410 , the CPU 212 waits until an inference result is received from the server 103 . Specifically, the CPU 212 waits for the transmission of the inference result from the server 103 as long as it determines in S410 that the inference result has not been received from the server 103. If the CPU 212 determines in S410 that the inference result has been received from the server 103, the CPU 212 ends the series of processes shown in FIG.

[0033] Next, an example of the processing of the server 103 will be described. In S406 , the CPU 202 waits until the server wired communication unit 205 receives the intermediate layer data from the imaging device 101 . Specifically, the CPU 202 waits for transmission of intermediate layer data from the imaging device 101 as long as it determines in S406 that the server wired communication unit 205 has not received intermediate layer data from the imaging device 101. If the CPU 202 determines in S406 that the server wired communication unit 205 has received the intermediate layer data from the image capture device 101, the CPU 202 advances the process to S407.

[0034] In S407, the GPU 204 executes arithmetic processing of the neural network in accordance with the command of the CPU 202. Specifically, the GPU 204 executes arithmetic processing from the input layer 304 to the N-th hidden layer 306 shown in FIG. 3 using the hidden layer data (e.g., data of the second hidden layer 303) received from the image capture device 101 as input data. In S408, the GPU 204 executes the arithmetic processing of the output layer 307 (arithmetic processing of the output layer of the neural network) shown in Fig. 3. As described above, the calculations related to the inference processing of the neural network are completed, and it becomes possible to classify the image data acquired in S401 (for example, the imaging results by the imaging unit 217). In S409, the CPU 202 transmits the results of the calculation processing of the series of processes from the input layer 304 to the output layer 307 in S407 and S408, i.e., the results of the inference processing, from the server wired communication unit 205 to the imaging device 101 via the wireless communication device 102. Thereafter, the CPU 202 ends the series of processes shown in FIG.

[0035] As described above, according to the information processing system of this embodiment, intermediate layer data is transmitted from the image capture device 101 to the server 103 at a time when wireless communication is not congested, thereby further shortening the wireless communication time required for transmitting the data. Furthermore, part of the calculations in the neural network are executed by the server 103, which has a higher calculation capacity than the image capture device 101, which is expected to further shorten the processing time required for inference using the neural network. In this way, according to the information processing system of this embodiment, the image capture device 101 and the server 103 work together to more suitably shorten the processing time required for a series of processes required for inference using the neural network, depending on the situation at the time.

[0036] In this embodiment, an example has been described in which the wireless congestion level determination unit 218 determines the congestion level of wireless communication by observing (intercepting) data transfers from other terminals, but the method is not particularly limited as long as it is possible to determine the congestion level of wireless communication. As a specific example, the communication speed of wireless communication may be measured periodically, and the degree of wireless communication congestion may be determined based on the measurement result of the communication speed. As another example, the degree of wireless communication congestion may be determined based on the proportion of the number of channels in use among a series of channels used for wireless communication. Furthermore, the degree of wireless communication congestion may be determined based on the frequency with which retransmissions occur during wireless communication. Specifically, if retransmissions occur frequently during wireless communication, it may be determined that there is a hidden terminal that is difficult to observe, and the result of this determination may be taken into account when determining the degree of wireless communication congestion. Furthermore, in the present embodiment, an example has been described in which the congestion level of wireless communication is determined by the wireless congestion level determination unit 218 on the imaging device 101 side, but the congestion level of wireless communication may be determined on the wireless communication device 102 side. In this case, the wireless communication device 102 may be configured to notify the imaging device 101 of the determination result of the congestion level of wireless communication.

[0037] Furthermore, the imaging device wireless communication unit 219 may be able to use multiple different wireless frequency bands. For example, when a wireless LAN is applied for wireless communication, it may be possible to use the 2.4 GHz band and the 5 GHz band. In such a case, the imaging device wireless communication unit 219 may switch the wireless frequency band used for wireless communication depending on the degree of congestion. By applying such control, wireless communication can be performed using a less congested wireless frequency band. In this case, a configuration of the imaging device wireless communication unit 219 that performs wireless communication using some of the multiple wireless frequency bands corresponds to an example of a first communication unit, and a configuration that performs wireless communication using the other wireless frequency bands corresponds to an example of a third communication unit. Furthermore, the configuration of the neural network illustrated in FIG. 3 is merely an example, and the number of intermediate layers is not limited and may be changed as appropriate depending on the inference model to be applied.

[0038] <Second embodiment> 1, 5, and 6, an inference processing system using a neural network will be described below as an example of an information processing system according to a second embodiment of the present disclosure. As described above, in the technology according to the present disclosure, it is assumed that the computing power of the server is higher than that of the imaging device. In this case, if the computing power of the server is significantly higher than that of the imaging device, it may be possible that the time required for a series of inference processes will be shorter by sending intermediate layer data to the server and having the server perform processing, even in a situation where wireless communication is highly congested and data transmission between devices takes a long time. Therefore, in this embodiment, an example of a mechanism for further shortening the time required for inference processing by controlling the timing of transmitting intermediate data from the imaging device to the server via wireless communication in accordance with the server's processing capacity and the amount of calculations performed by the neural network will be described. In this embodiment, the description will focus on the parts that are particularly different from the first embodiment described above, and detailed description of parts that are substantially the same as those in the first embodiment will be omitted. Also, with regard to the system configuration, the configuration shown in Fig. 1 is applied, as in the first embodiment.

[0039] An example of the configuration of an inference processing system according to this embodiment will be described with reference to Fig. 5. The configurations of the server 103 and the wireless communication device 102 shown in Fig. 5 are substantially similar to the configurations of the server 103 and the wireless communication device 102 described with reference to Fig. 2 in the first embodiment, and therefore detailed description thereof will be omitted. Also, an image capture device 510 according to this embodiment differs from the image capture device 101 described with reference to Fig. 2 in that it includes a congestion threshold setting unit 520.

[0040] The congestion threshold setting unit 520 sets a threshold VTH against which the congestion level of wireless communication determined by the wireless congestion level determination unit 218 is compared. The CPU 212 of the imaging device 510 continues processing the intermediate layer until the congestion level becomes lower than the threshold VTH, and transmits intermediate data corresponding to the results of this processing to the server 103. This threshold VTH may be a preset fixed value, or may be dynamically changed depending on the situation at any given time. An example of a method for dynamically changing the threshold VTH will be described below.

[0041] For example, if the computing power of the server 103 is significantly higher than that of the imaging device 510, control may be applied to set the threshold VTH lower so that more intermediate layer computing processing is performed on the server 103 side even when wireless communication is congested. As another example, even when the amount of calculation of the neural network is large, control may be applied to set the threshold VTH lower so that more calculation processing of the intermediate layer is executed on the server 103 side. Furthermore, the larger the volume of data in the intermediate layer to be transmitted, the greater the impact on the time required for data transmission when wireless communication is congested (in other words, the impact on a decrease in communication speed). Therefore, in such a case, control may be applied to set the threshold value VTH higher so that more calculation processing in the intermediate layer is performed on the imaging device 510 side.

[0042] A more suitable method for determining the threshold value VTH is to take into consideration the communication time between the image capture device 510 and the server 103 and the calculation time of the server 103. 5 may be provided in the imaging device 510. The throughput measurement unit calculates the communication speed at the current level of wireless communication congestion from the time required for wireless communication between the imaging device 510 and the server 103 and the amount of transmitted data. The CPU 212 of the imaging device 510 then adds the communication time calculated by the throughput measurement unit to the calculation time required for the server 103 to perform calculation processing from the third hidden layer 305 to the Nth hidden layer 306. The resulting time is used as the time required for the server 103 to perform calculation processing from the third hidden layer 305 to the Nth hidden layer 306, thereby taking into account the time required for wireless communication and enabling the calculation of a more accurate processing time that is in line with reality. As described above, by calculating the processing time for each degree of congestion of wireless communication, it becomes possible to determine an appropriate threshold value VTH for further shortening the inference processing time.

[0043] Regarding the threshold value VTH, for example, the value to be set may be determined each time, but multiple candidates may be stored in advance as setting targets, and the candidate to be set may be determined from the multiple candidates depending on the judgment result exemplified above. As a specific example, a plurality of candidates for the communication speed allowable in wireless communication between the image capture device 510 and the server 103 may be set in stages, and then the plurality of candidates to be applied as the threshold VTH may be set based on the upper limit of the congestion level at which wireless communication is possible at the communication speed. As a result, for example, the CPU 212 may identify the congestion level allowable according to the processing capacity of the server 103 (in other words, the communication speed allowable even when wireless communication is congested), and then identify a candidate according to the congestion level as an application target for the threshold VTH. As another example, the CPU 212 may identify the congestion level allowable according to the amount of data in the intermediate layer to be transmitted, and then identify a candidate according to the congestion level as an application target for the threshold VTH. As another example, the plurality of candidates may be set in stages according to the difference in the computational processing capabilities between the image capture device 510 and the server 103. In this case, the plurality of candidates may be set so that the higher the computational processing capability of the server 103 is compared to the computational processing capability of the image capture device 510, the smaller the value applied as the threshold VTH. As another example, the plurality of candidates may be set in stages according to the amount of calculation of the neural network. In this case, the plurality of candidates may be set so that the larger the amount of calculation of the neural network, the smaller the value applied as the threshold VTH.

[0044] With reference to FIG. 6, an example of the processing of the inference processing system according to this embodiment will be described, which illustrates an example of a series of processing steps when the imaging device 510 and the server 103 cooperate to realize calculation processing of an inference model. In the example illustrated in FIG. 6, the processing of S401 to S405, S410, S601, and S602 is performed by the imaging device 510, and the processing of S406 to S409 is performed by the server 103. Note that the processing of S401 to S410 is substantially the same as the processing of S401 to S410 described with reference to FIG. 4 in the first embodiment, and therefore detailed description thereof will be omitted. In other words, the processing of the server 103 is substantially the same as the example illustrated in FIG. 4. Therefore, in this embodiment, the processing of the imaging device 510 will be described, focusing particularly on parts that are different from those in the first embodiment.

[0045] In S601, the congestion threshold setting unit 520 of the imaging device 510 sets a threshold VTH to be compared with the congestion level of wireless communication determined by the wireless congestion level determination unit 218 in S403. In S602, the CPU 212 of the imaging device 510 determines whether the congestion level of wireless communication is lower than the threshold VTH set by the congestion threshold setting unit 520 in S601, based on the determination result notified by the congestion threshold setting unit 520 in S403. If the CPU 212 determines in S602 that the congestion level of wireless communication is lower than the threshold value VTH, the process proceeds to S405. On the other hand, if the CPU 212 determines in S602 that the congestion level of wireless communication is equal to or greater than the threshold VTH, the process proceeds to S402. In this case, the processes from S402 onwards are executed again. The processes in S405 and S410 are substantially the same as those in the example shown in FIG. 4, and therefore detailed description thereof will be omitted. The above describes an example of a series of processing flows when the imaging device 510 and the server 103 according to this embodiment work together to realize calculation processing of an inference model.

[0046] As described above, in the information processing system according to this embodiment, the timing at which intermediate layer data is transmitted from the image capture device 510 to the server 103 via wireless communication is controlled in a series of inference processes, depending on the computing power of the server 103 and the amount of computation of the neural network. By applying such control, it is expected that the time required for the inference processes can be further reduced.

[0047] <Other embodiments> The above describes preferred embodiments of the present invention using the first and second embodiments, but the present invention is not limited to these embodiments and various modifications and changes may be made within the scope of the gist of the present invention. The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having the computer of the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions. The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0048] The disclosure of the present embodiment also includes the following systems, methods, and configurations. (System 1) An information processing system including a first device and a second device connected via a wireless network, and causing an inference process to be performed by a neural network in which an input layer, a plurality of intermediate layers that extract features contained in data input from a previous layer, and an output layer are hierarchically connected, wherein the first device has a determination means for determining the degree of congestion of wireless resources in communication via the wireless network based on the communication status of other devices connected to the wireless network, a first control means for sequentially executing processing of each of the plurality of intermediate layers until the degree of congestion becomes smaller than a threshold value to generate intermediate data, and a first communication means for transmitting the intermediate data generated by the first control means to the second device via the wireless network, and the second device has a second communication means for receiving the intermediate data from the first device via the wireless network, and a second control means for sequentially executing processing of each of a series of intermediate layers among the plurality of intermediate layers that have not been processed in the first device, using the intermediate data received by the second communication means as input. (System 2) The information processing system according to System 1, further comprising a setting means for setting one of a plurality of candidates as the threshold value. (System 3) An information processing system described in System 2, characterized in that the multiple candidates are set in stages according to the upper limit of congestion level at which communication via the wireless network is possible at or above a predetermined communication speed. (System 4) An information processing system described in System 2 or 3, characterized in that the multiple candidates are set in stages so that the higher the computing processing capacity of the second device is compared to the computing processing capacity of the first device, the smaller the value applied as the threshold. (System 5) An information processing system described in any one of Systems 2 to 4, characterized in that the multiple candidates are set in stages so that the greater the amount of calculation required by the neural network, the smaller the value applied as the threshold. (System 6) An information processing system described in any one of Systems 1 to 5, characterized in that the determination means determines the degree of congestion by observing data transfers of other terminals via the wireless network. (System 7) An information processing system described in any one of Systems 1 to 6, characterized in that the determination means determines the congestion degree according to the communication speed of communication via the wireless network. (System 8) An information processing system described in any one of Systems 1 to 7, characterized in that the determination means determines the degree of congestion based on the proportion of the number of channels in use out of the series of channels used for communication via the wireless network. (System 9) An information processing system described in any one of Systems 1 to 8, characterized in that the determination means determines the degree of congestion based on the frequency at which retransmissions occur in communication via the wireless network. (System 10) An information processing system described in any one of Systems 1 to 9, characterized in that the first device has a third communication means for communicating via the wireless network using a radio frequency band different from that of the first communication means, and a switching means for switching the communication means applied to communication via the wireless network between the first communication means and the third communication means, and the switching means switches the communication means used for transmitting the intermediate data between the first communication means and the third communication means depending on the congestion degree of the radio frequency bands used by each of the first communication means and the third communication means. (Method 1) A control method for an information processing system including a first device and a second device connected via a wireless network, causing a neural network to execute inference processing, the neural network having an input layer, a plurality of intermediate layers that extract features included in data input from a previous layer, and an output layer hierarchically connected, the method including a determination step in which the first device determines a congestion level of wireless resources in communication via the wireless network based on communication statuses of other devices connected to the wireless network, and a step in which the first device sequentially executes processing of each of the plurality of intermediate layers until the congestion level becomes smaller than a threshold value, thereby generating intermediate data. a first control step of generating a plurality of intermediate layers from the plurality of intermediate layers, a first communication step of the first device transmitting the intermediate data generated in the first control step to the second device via the wireless network, a second communication step of the second device receiving the intermediate data from the first device via the wireless network, and a second control step of the second device using the intermediate data received in the second communication step as input to sequentially execute processing of each of a series of intermediate layers among the plurality of intermediate layers that have not been processed in the first device. (Configuration 1) An information processing device comprising: a determination means for determining a degree of congestion of wireless resources in communication via a wireless network based on the communication status of other devices connected to the wireless network; a first control means for sequentially executing processing of each of a plurality of intermediate layers of a neural network in which an input layer, a plurality of intermediate layers for extracting features contained in data input from a previous layer, and an output layer are hierarchically connected, until the degree of congestion becomes smaller than a threshold, thereby generating intermediate data; and a first communication means for transmitting the intermediate data generated by the first control means to an external device connected via the wireless network, wherein the external device uses the intermediate data as input and sequentially executes processing of each of a series of intermediate layers out of the plurality of intermediate layers that have not been processed by the first control means. (Configuration 2) An information processing device characterized by comprising: a second communication means for receiving intermediate data from an external device via the wireless network, intermediate data generated by sequentially executing processing of each of a series of intermediate layers of a neural network in which an input layer, a plurality of intermediate layers that extract features contained in data input from a previous layer, and an output layer are hierarchically connected, until congestion of wireless resources in communication via the wireless network, determined based on communication conditions of other devices connected to the wireless network, becomes smaller than a threshold; and a second control means for using the intermediate data received by the second communication means as input and sequentially executing processing of each of a series of intermediate layers among the plurality of intermediate layers that have not been processed in the first device. [Explanation of symbols]

[0049] 101 Imaging device 103 Server 202 CPU 212 CPU 218 Wireless Congestion Determination Unit

Claims

1. An information processing system including a first device and a second device connected via a wireless network, and causing an inference process to be performed by a neural network in which an input layer, a plurality of intermediate layers that extract features included in data input from a previous layer, and an output layer are hierarchically connected, The first device is a determining means for determining a congestion level of wireless resources in communication via the wireless network based on communication statuses of other devices connected to the wireless network; a first control means for sequentially executing the processing of each of the plurality of intermediate layers until the congestion degree becomes smaller than a threshold value to generate intermediate data; a first communication means for transmitting the intermediate data generated by the first control means to the second device via the wireless network; and The second device is a second communication means for receiving the intermediate data from the first device via the wireless network; a second control means for sequentially executing processing of a series of intermediate layers among the plurality of intermediate layers, the intermediate data received by the second communication means being input, the series of intermediate layers not being processed by the first device; have An information processing system comprising:

2. 2. The information processing system according to claim 1, further comprising a setting unit for setting one of a plurality of candidates as the threshold value.

3. 3. The information processing system according to claim 2, wherein the plurality of candidates are set in stages according to an upper limit value of congestion level at which communication via the wireless network is possible at or above a predetermined communication speed.

4. 3. The information processing system according to claim 2, wherein the plurality of candidates are set in stages so that the higher the computing power of the second device is compared to the computing power of the first device, the smaller the value applied as the threshold.

5. 3. The information processing system according to claim 2, wherein the plurality of candidates are set in stages such that the larger the amount of calculation of the neural network, the smaller the value applied as the threshold value.

6. 2. The information processing system according to claim 1, wherein the determining means determines the congestion level by observing data transfers of other terminals via the wireless network.

7. 2. The information processing system according to claim 1, wherein the determining means determines the congestion level according to a communication speed of communication via the wireless network.

8. 2. The information processing system according to claim 1, wherein the determining means determines the congestion level based on the proportion of the number of channels in use among a series of channels used for communication via the wireless network.

9. 2. The information processing system according to claim 1, wherein the determining means determines the congestion level according to a frequency of retransmissions occurring in communications via the wireless network.

10. The first device is a third communication means for performing communication via the wireless network using a radio frequency band different from that of the first communication means; a switching means for switching a communication means to be applied to communication via the wireless network between the first communication means and the third communication means; and The switching means switches the communication means used for transmitting the intermediate data between the first communication means and the third communication means according to the congestion levels of the radio frequency bands used by the first communication means and the third communication means.

2. The information processing system according to claim 1, wherein:

11. A control method for an information processing system including a first device and a second device connected via a wireless network, the method causing a neural network to execute inference processing, the neural network having an input layer, a plurality of intermediate layers that extract features included in data input from a previous layer, and an output layer hierarchically connected, the method comprising: a determination step in which the first device determines a congestion level of wireless resources in communication via the wireless network based on communication statuses of other devices connected to the wireless network; a first control step in which the first device sequentially executes the processing of each of the plurality of intermediate layers until the congestion degree becomes smaller than a threshold value to generate intermediate data; a first communication step in which the first device transmits the intermediate data generated in the first control step to the second device via the wireless network; a second communication step in which the second device receives the intermediate data from the first device via the wireless network; a second control step in which the second device sequentially executes processing of a series of intermediate layers among the plurality of intermediate layers, the intermediate data received in the second communication step being used as input, the series of intermediate layers not yet executed in the first device; 10. A method for controlling an information processing system, comprising:

Images