Information processing device, control method for information processing device, and program

Abstract

This improves the accuracy of inference using neural networks that accept images and audio as input. [Solution] An information processing device that performs inference processing using a neural network has an inference processing means that performs inference processing as input to a plurality of input data including images and sound, and an acquisition means that acquires information about the zoom state in image capture, and the inference processing means controls the inference processing for at least one of the plurality of input data based on the acquired information about the zoom state.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] Deep learning technology using neural networks has been applied in a wide range of technical fields including class classification for image recognition and classification. The widely used convolutional neural network (CNN) realizes high-precision deep learning processing by repeatedly performing convolutional operations on images.

[0003] In addition, a technique for performing deep learning processing using multiple types (multimodal) of information has been proposed. In Patent Document 1, an image and voice associated with the speech of one or more speakers are received, and two types of information, namely, a lip feature amount calculated from the image and a voice feature amount calculated from the voice, are input to infer the speech content of a specific speaker among the one or more speakers.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology disclosed in Patent Document 1 is proposed as a configuration that assumes the sound source (speaker) is at close range to the device that acquires images and sound, such as in a small-group meeting. However, in situations where images are taken from close range to long range, such as in surveillance camera photography or bird photography, the distance from the device to the subject varies depending on the subject selected by the user. In such situations, if the subject is at a distance from the device, Patent Document 1 will make inferences based on sound emitted from an object other than the subject, resulting in a deterioration of inference accuracy. The present invention aims to improve the accuracy of inference using a neural network that takes images and sound as input. [Means for solving the problem]

[0006] The information processing apparatus according to the present invention is an information processing apparatus that performs inference processing using a neural network, and comprises an inference processing means that performs the inference processing with respect to a plurality of input data including images and sound, and an acquisition means that acquires information regarding the zoom state in the capture of the image, wherein the inference processing means controls the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the accuracy of inference using a neural network that takes images and audio as input. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the hardware configuration of an information processing device. [Figure 2] This figure shows an example of the functional configuration of an information processing device. [Figure 3] This figure shows an example of the configuration of the inference processing unit in Embodiment 1. [Figure 4] This diagram illustrates the selection of an inference model for a state in Embodiment 1. [Figure 5]This is a flowchart showing an example of processing in the information processing device according to Embodiment 1. [Figure 6] This figure shows an example of the configuration of the inference processing unit in Embodiment 2. [Figure 7] This figure illustrates the bond weights for states in Embodiment 2. [Figure 8] This is a flowchart showing an example of processing by the information processing device in Embodiment 2. [Figure 9] This figure shows an example of the configuration of the inference processing unit in Embodiment 3. [Figure 10] This diagram illustrates the delay value for the state in Embodiment 3. [Figure 11] This is a flowchart showing an example of processing by the information processing device in Embodiment 3. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. In the following, we will describe an example of a use case to which the information processing device in each embodiment is applied, assuming the determination of whether or not dangerous behavior is occurring in a surveillance camera. However, use cases are not limited to the determination of whether or not dangerous behavior is occurring. Other use cases may include, for example, the determination of whether or not nuisance behavior is occurring or the detection of noise problems. Instead of using an inference model that outputs an inference result for the presence or absence of dangerous behavior, an inference model that has been trained to output an inference result appropriate to the use case can be used.

[0010] <Embodiment 1> Figure 1 shows an example of the hardware configuration of the information processing device in this embodiment. The information processing device 100 in this embodiment includes a CPU 101, ROM 102, RAM 103, storage device 104, input unit 105, display unit 106, communication unit 107, and system bus 108. The CPU 101, ROM 102, RAM 103, storage device 104, input unit 105, display unit 106, and communication unit 107 are connected to each other via the system bus 108. The information processing device 100 in this embodiment may have other configurations.

[0011] The CPU (Central Processing Unit) 101 controls the entire information processing device 100 using computer programs and data stored in the ROM 102 and RAM 103, thereby realizing, for example, the various functions of the information processing device described later. The CPU 101 reads a program from the ROM 102, etc., loads it into the RAM 103, and executes it, thereby realizing the inference processing means, acquisition means, and characteristic value output means. The information processing device 100 may have one or more dedicated hardware components different from the CPU 101, and at least a portion of the processing performed by the CPU 101 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), DSPs (Digital Signal Processors), and GPUs (Graphics Processing Units).

[0012] The ROM (Read Only Memory) 102 stores programs and the like that do not require modification. The RAM (Random Access Memory) 103 temporarily stores programs and data supplied from the storage device 104, as well as data and the like supplied from the outside via the communication unit 107. Further, the RAM 103 functions as the main memory, work area, etc. of the CPU 101. The storage device 104 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc., and stores various data. The storage device 104 stores, for example, various data necessary when the CPU 101 performs processing related to a program, and various data obtained by the CPU 101 performing processing related to a program.

[0013] The input unit 105 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user and inputs various instructions to the CPU 101. The display unit 106 is composed of, for example, a liquid crystal display, an LED, etc., and presents a display image to the user. The display unit 106 displays, for example, a GUI (Graphical User Interface) for the user to operate the information processing device 100, and processing results by the CPU 101. Note that the input unit 105 and the display unit 106 may exist outside the information processing device 100 as separate devices. The communication unit 107 connects the information processing device 100 to a network and controls communication with other devices and the like.

[0014] FIG. 2 is a diagram showing an example of the functional configuration of the information processing device in the present embodiment. The information processing device 100 in the present embodiment performs integrated inference processing based on a plurality of inputs and outputs an inference result. In this example, image data is input as the first input 211, and voice data is input as the second input 212, and an inference result is output as the output 214. Hereinafter, the first input and the second input will be described as image data and voice data, respectively.

[0015] The information processing apparatus 100 includes a state acquisition unit 201, a characteristic value output unit 202, and an inference processing unit 203. The state acquisition unit 201 acquires, as a zoom state, the state of the shooting angle in shooting with a monitoring camera by a zoom operation such as zooming in or zooming out when the user appropriately operates the shooting angle with respect to the subject of interest based on the state data input as the third input 213. The state acquisition unit 201 acquires information regarding the zoom state by a zoom operation that enlarges (zooms in) or reduces (zooms out) the fixation point with respect to the captured image. Here, the fixation point is, for example, the center of the shooting angle in optical zoom and the center of the image after zooming in electronic zoom (digital zoom). The acquired information regarding the zoom state is output to the characteristic value output unit 202.

[0016] The characteristic value output unit 202 controls the inference processing performed by the inference processing unit 203 based on the zoom state. In the present embodiment, the characteristic value output unit 202 determines which model to select as the inference model used in the inference processing performed by the inference processing unit 203 based on the zoom state, and outputs the determination result to the inference processing unit 203 as a characteristic value. The determination method will be described later.

[0017] The inference processing unit 203 receives the first input (image data) 211, the second input (audio data) 212, and the output (characteristic value) of the characteristic value output unit 202, performs inference processing based on the image data and the audio data, and outputs the inference result as an output 214. The inference processing unit 203 performs arithmetic processing of inference on the image data and the audio data using an inference model corresponding to the output of the characteristic value output unit 202, and outputs the inference result. The output destination of the inference result may be output to the storage via the bus in the apparatus, or may be output to an external device via a communication network such as the Internet. In the present embodiment, the inference processing unit 203 is assumed to be a processing unit capable of performing multiplication-accumulation operations and non-linear processing, which is adopted in a general neural network model.

[0018] Figure 3 shows an example of the configuration of the inference processing unit 203 in Embodiment 1. The inference processing unit 203 is configured to include, for example, a convolutional neural network (CNN). Generally, CNN processing includes bias addition and nonlinear processing, but details are omitted in this embodiment. However, this embodiment is not limited to CNNs as the neural network configuration, and may include operations such as fully connected processing. The neural network operation method itself is a well-known technology, so its explanation is omitted here.

[0019] The inference processing unit 203 includes a first coupling layer 301, a second coupling layer 302, an image / sound selection unit 303, and a result selection unit 304. The inference processing unit 203 receives a first input (image data) 311, a second input (sound data) 312, and a characteristic value 313, performs inference processing based on the image data and sound data, and outputs the inference result as output 314. The first input (image data) 311 and the second input (sound data) 312 correspond to the first input (image data) 211 and the second input (sound data) 212 input to the information processing device 100, and the characteristic value 313 is the output of the characteristic value output unit 202.

[0020] The first connected layer 301 and the second connected layer 302 are inference models that take image data 311 and audio data 312 as input, perform inference through neural network calculations, and output an inference result regarding the presence or absence of dangerous behavior. These inference models can be created, for example, by training them using image data and audio data as input data and the presence or absence of dangerous behavior as training data. The first connected layer 301 is an inference model trained to emphasize audio data when the distance between the imaging surface and the subject is far, and the second connected layer 302 is an inference model trained to emphasize audio data when the distance between the imaging surface and the subject is close. In other words, the first connected layer 301 is trained to emphasize audio at positions far from the imaging surface, and the second connected layer 302 is trained to emphasize audio at positions close to the imaging surface. Note that the computational load for the inference processing of the first connected layer 301 and the second connected layer 302 may differ, and the layer configuration (number of layers) and number of nodes in the neural network may also differ. In other words, the first connected layer 301 and the second connected layer 302 do not need to have the same layer configuration or number of nodes, and may be configured with a layer configuration (number of layers) and number of nodes that enable optimal inference under each respective condition.

[0021] The image / sound selection unit 303 outputs the input image data 311 and sound data 312 to either the first coupling layer 301 or the second coupling layer 302 based on the characteristic value 313 output from the characteristic value output unit 202. The result selection unit 304 outputs the output of the coupling layer selected from the first coupling layer 301 or the second coupling layer 302 based on the characteristic value 313 output from the characteristic value output unit 202 as the inference result. In other words, the image / sound selection unit 303 and the result selection unit 304 select the inference model of the same coupling layer based on the characteristic value 313. Note that only one of the first coupling layer 301 and the second coupling layer 302 selected by the image / sound selection unit 303 and the result selection unit 304 needs to operate.

[0022] Next, referring to Figure 4, we will explain how the characteristic value output unit 202 determines the inference model to be selected by the image / sound selection unit 303 and the result selection unit 304 based on the zoom state information input from the state acquisition unit 201. Figure 4 is a diagram illustrating the selection of an inference model using an evaluation value for the zoom state pattern in Embodiment 1 and a predetermined threshold.

[0023] The state acquisition unit 201 calculates an evaluation value for the distance between the imaging surface and the subject of interest, ranging from 0 to 4, based on the input zoom state, with the smallest zoom magnification state of the surveillance camera being the minimum value and the largest zoom magnification state being the maximum value. Therefore, the evaluation value is calculated such that a larger value indicates a greater distance to the subject of interest, and a smaller value indicates a closer distance to the subject of interest.

[0024] In this embodiment, the characteristic value output unit 202, as an example, sets a predetermined threshold of 3 and compares the evaluation value with the threshold to determine which coupling layer to select in the inference processing unit 203. The characteristic value output unit 202 determines that if the evaluation value is 3 or greater, it will select the first coupling layer 301, and if it is less than 3, it will select the second coupling layer 302, and outputs a characteristic value 313 according to the determination result. In the example shown in Figure 4, in the case of zoomed-in pattern 401, the evaluation value is 4 and is greater than or equal to the threshold of 3, so the first coupling layer 301 is selected, and in the case of zoomed-in pattern 402, the evaluation value is 1 and is less than the threshold of 3, so the second coupling layer 302 is selected.

[0025] In the use case of determining whether or not dangerous behavior is occurring in a surveillance camera, as described in this embodiment, it is desirable that the sound source be located near the subject. Therefore, it is desirable that the first coupled layer 301 selects an inference model that emphasizes distant sound for determination, and the second coupled layer 302 selects an inference model that emphasizes nearby sound for determination. Furthermore, the determination method is not limited to the method described above; any method that can determine the distance between the imaging surface and the subject is acceptable.

[0026] Figure 5 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 1. The processing shown in the flowchart in Figure 5 is realized, for example, by the CPU 101 of the information processing device 100 executing a program or the like that is loaded into the RAM 103. The information processing device 100 starts the processing shown in the flowchart in Figure 5 by receiving image data, which is the first input 211, and audio data, which is the second input 212.

[0027] In step S501, the CPU 101 obtains an evaluation value for the distance between the imaging plane and the subject of interest, based on the zoom state of the surveillance camera. As mentioned above, the evaluation value is between a minimum value of 0 and a maximum value of 4.

[0028] In step S502, the CPU 101 determines whether the evaluation value obtained in step S501 is equal to or greater than a predetermined threshold. That is, by comparing the evaluation value with the threshold, the CPU 101 determines the inference model to be used for inferring whether or not there is dangerous activity in the surveillance camera, according to the evaluation value obtained in step S501. If the CPU 101 determines that the evaluation value is equal to or greater than the threshold (YES in step S502), the process proceeds to step S503. On the other hand, if the CPU 101 determines that the evaluation value is less than the threshold (NO in step S502), the process proceeds to step S504.

[0029] In step S503, the CPU 101 selects the first coupling layer 301 as the output destination for the image data and audio data, and outputs characteristic values ​​accordingly. In other words, the CPU 101 selects the first coupling layer 301 as the output destination for the input image data and audio data, based on the evaluation value of the distance between the imaging plane and the subject. After executing the processing in step S503, the process proceeds to step S505.

[0030] In step S504, the CPU 101 selects the second coupling layer 302 as the output destination for the image data and audio data, and outputs characteristic values ​​accordingly. In other words, the CPU 101 selects the second coupling layer 302 as the output destination according to the evaluation value of the distance between the imaging plane and the subject for the input image data and audio data. After executing the processing in step S504, the process proceeds to step S505.

[0031] In step S505, the CPU 101 performs neural network calculations (inference processing) using the first connected layer 301 or the second connected layer 302 selected in step S503 or S504, and outputs the inference result regarding the presence or absence of dangerous behavior in the surveillance camera.

[0032] In this embodiment, the information processing device 100 uses the zoom state of the surveillance camera as an evaluation value to select an inference model for inference processing. This enables inference that emphasizes audio near the subject selected by the user, resulting in inference based on images and audio that are highly related to the subject, and improving the accuracy of the inference.

[0033] In the above explanation, an example was shown in which the inference processing unit 203 has two inference models as the first coupling layer 301 and the second coupling layer 302, but it is not limited to this. The inference processing unit 203 may have three or more inference models, and may select one appropriate inference model from among the three or more inference models according to the evaluation value of the distance between the imaging plane and the subject of interest based on the zoom state to perform inference processing.

[0034] <Embodiment 2> Embodiment 2 will now be described. The hardware configuration and functional configuration of the information processing device 100 in Embodiment 2 are the same as those of the information processing device 100 in Embodiment 1 shown in Figures 1 and 2, so their description will be omitted. The differences from Embodiment 1 described above will be explained below.

[0035] Figure 6 shows an example of the configuration of the inference processing unit 203 in Embodiment 2. The inference processing unit 203 in Embodiment 2 is configured to include a convolutional neural network (CNN), similar to Embodiment 1 described above. However, this embodiment is not limited to a CNN for the neural network configuration, and may include operations such as fully connected processing.

[0036] The inference processing unit 203 in Embodiment 2 has a first layer 601, a second layer 602, and a coupling layer 603. The inference processing unit 203 receives a first input (image data) 611, a second input (audio data) 612, and a characteristic value 613, performs inference processing based on the image data and audio data, and outputs the inference result as output 614. The first input (image data) 611 and the second input (audio data) 612 correspond to the first input (image data) 211 and the second input (audio data) 212 input to the information processing device 100, and the characteristic value 613 is the output of the characteristic value output unit 202.

[0037] The first layer 601 is an inference model that takes image data 611 as input, infers whether or not a dangerous act occurred through neural network calculations, and outputs the inference result to the connected layer 603. This inference model can be created, for example, by training with image data as input data and the presence or absence of dangerous acts as training data.

[0038] The second layer 602 is an inference model that takes characteristic values ​​613 output from the characteristic value output unit 202 and audio data 612 as input, infers the presence or absence of dangerous behavior through neural network calculations, and outputs the inference result to the connected layer 603. This inference model can be created, for example, by training with audio data as input data and the presence or absence of dangerous behavior as training data.

[0039] The coupled layer 603 is a coupled layer of the inference model that takes the inference results from the first layer 601 and the inference results from the second layer 602 as input, infers whether or not there is a dangerous act based on the image data and audio data, and outputs the inference result as the final inference result of the inference processing unit 203.

[0040] Note that the first layer 601, the second layer 602, and the connected layer 603 may be configured together as a single inference model. When configured as a single inference model, the outputs of the first layer 601 and the second layer 602, and the inputs of the connected layer 603 may be in the form of inputs and outputs necessary for connecting multiple nodes, which are the intermediate layers of a neural network.

[0041] In this embodiment, we will describe the case where a microphone, which is a means for acquiring audio data (not shown), is located near the imaging surface of the surveillance camera. Referring to Figure 7, the characteristic value output unit 202 outputs characteristic values ​​to the second layer 602 in accordance with the zoom state information input from the state acquisition unit 201, and the method by which the second layer 602 performs inference using audio data in accordance with those characteristic values ​​will be explained. Figure 7 is a diagram illustrating the relationship between the evaluation value for the zoom state pattern in Embodiment 2 and the characteristic value input to the second layer 602 in accordance with the evaluation value. Here, in Embodiment 2, the characteristic value is the coefficient of the coupling weight performed between the input layer and the hidden layer nodes of the second layer 602, and refers to the weighting coefficient shown in Figure 7.

[0042] The state acquisition unit 201 calculates an evaluation value for the distance between the imaging surface and the subject of interest, ranging from 1 to 5, based on the input zoom state, with the smallest zoom magnification state of the surveillance camera being the minimum value and the largest zoom magnification state being the maximum value. Therefore, the evaluation value is calculated such that a larger value indicates a greater distance to the subject of interest, and a smaller value indicates a closer distance to the subject of interest.

[0043] As illustrated in Figure 7, in the case of zoom pattern 701, where the evaluation value is 5, the distance from the imaging plane to the subject is far, so the audio data acquired near the imaging plane is likely to contain a lot of audio that is not near the subject. Therefore, the characteristic value output unit 202 sets the weighting coefficient, which is a characteristic value, to 0.2 and adjusts the weighting of the audio data so that the audio data is not given much importance in the inference process in the inference processing unit 203. On the other hand, in the case of zoom pattern 702, where the evaluation value is 1, the distance from the imaging plane to the subject is close, and the subject and microphone are close together, so the audio data input to the second layer 602 is likely to contain a lot of audio emitted from the subject. Therefore, the characteristic value output unit 202 determines that the audio data is useful information for inference and sets the weighting coefficient, which is a characteristic value, to 1.0 and adjusts the weighting of the audio data so that the audio data is used as is in the inference process in the inference processing unit 203. Furthermore, in the case of the zoom state pattern 703, where the evaluation value is 3, the distance from the imaging plane to the subject is intermediate, so the characteristic value output unit 202 sets the weighting coefficient, which is a characteristic value, to 0.6 and adjusts the weighting of the audio data in the inference processing unit 203.

[0044] Figure 8 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 2. The processing shown in the flowchart in Figure 8 is realized, for example, by the CPU 101 of the information processing device 100 executing a program loaded into the RAM 103. The information processing device 100 starts the processing shown in the flowchart in Figure 8 by receiving image data, which is the first input 611, and audio data, which is the second input 612.

[0045] In step S801, the CPU 101 obtains an evaluation value of the distance between the imaging plane and the subject of interest based on the zoom state of the surveillance camera. As mentioned above, the evaluation value is between a minimum value of 1 and a maximum value of 5.

[0046] In step S802, the CPU 101 obtains weighting coefficients as coefficients for the connection weights for the nodes between the input layer and the hidden layer in the second layer 602, based on the evaluation values ​​obtained in step S801.

[0047] In step S803, the CPU 101 performs inference processing using the weighting coefficients obtained in step S802 and outputs the inference result regarding the presence or absence of dangerous behavior at the surveillance camera. Specifically, the CPU 101 performs inference in the second layer 602 using the input audio data and the weighting coefficients obtained in step S802. In parallel with the inference in the second layer 602, the CPU 101 also performs inference in the first layer 601 using the input image data. Furthermore, the CPU 101 performs inference using the inference results from the first layer 601 and the second layer 602 as input to the connected layer 603, and outputs the obtained inference result as the final inference result regarding the presence or absence of dangerous behavior at the surveillance camera.

[0048] In this embodiment, the information processing device 100 controls the connection weights (weighting coefficients) between nodes in the neural network for audio data according to the zoom state of the surveillance camera. This prevents a decrease in the accuracy of inference due to audio data that is considered to have little relevance to the subject, and improves the accuracy of inference using both images and audio.

[0049] In this embodiment, the connection weights between nodes of two layers in the second layer 602 were described as being applied to the nodes between the input layer and the hidden layer, but this is not limited to this. For example, the connection weights based on evaluation values ​​may be applied to nodes of other layers, or to nodes of any layer in the entire second layer 602.

[0050] Furthermore, during the training of the second layer 602, an inference model may be constructed that uses audio data and evaluation values ​​as inputs. For example, in the case of determining the presence or absence of dangerous behavior in a surveillance camera, the model may be trained to emphasize audio near the subject during inference, depending on the zoom level. By training in this way, the second layer 602 can always perform inference using audio data that is highly relevant to the subject, thereby improving the accuracy of the inference results for the entire inference processing unit 203.

[0051] <Embodiment 3> Embodiment 3 will now be described. The hardware configuration and functional configuration of the information processing device 100 in Embodiment 3 are the same as those of the information processing device 100 in Embodiment 1 shown in Figures 1 and 2, so their description will be omitted. The differences from Embodiment 1 described above will be explained below.

[0052] Figure 9 shows an example of the configuration of the inference processing unit 203 in Embodiment 3. The inference processing unit 203 in Embodiment 3 is configured to include, for example, a recurrent neural network (RNN). Generally, RNN processing includes bias addition and nonlinear processing, but details are omitted in this embodiment. However, this embodiment is not limited to RNNs only, and may include operations such as fully connected processing.

[0053] The inference processing unit 203 in Embodiment 3 has a first layer 901, a second layer 902, and a coupling layer 903. The inference processing unit 203 receives a first input (image data) 911, a second input (audio data) 912, and a characteristic value 913, performs inference processing based on the image data and audio data, and outputs the inference result as output 914. The first input (image data) 911 and the second input (audio data) 912 correspond to the first input (image data) 211 and the second input (audio data) 212 input to the information processing device 100, and the characteristic value 913 is the output of the characteristic value output unit 202.

[0054] The first layer 901 is an inference model that takes characteristic values ​​913 output from the characteristic value output unit 202 and image data 911 as input, infers the presence or absence of dangerous behavior through neural network calculations, and outputs the inference result to the connected layer 903. This inference model can be created, for example, by training with image data as input data and the presence or absence of dangerous behavior as training data.

[0055] The second layer 902 is an inference model that takes audio data 912 as input, infers whether or not a dangerous act occurred through neural network calculations, and outputs the inference result to the connected layer 903. This inference model can be created, for example, by training it using audio data as input data and the presence or absence of dangerous acts as training data.

[0056] The coupled layer 903 is a coupled layer of the inference model that takes the inference results from the first layer 901 and the inference results from the second layer 902 as input, infers whether or not there is a dangerous act based on the image data and audio data, and outputs the inference result as the final inference result for the information processing device 100.

[0057] Note that the first layer 901, the second layer 902, and the connected layer 903 may be configured together as a single inference model. When configured as a single inference model, the outputs of the first layer 901 and the second layer 902, and the inputs of the connected layer 903 may be in the form of inputs and outputs necessary for connecting multiple nodes, which are the intermediate layers of a neural network.

[0058] In this embodiment, we will describe the case where a microphone, which is a means for acquiring audio data (not shown), is located near the imaging surface of the surveillance camera. Referring to Figure 10, the characteristic value output unit 202 outputs a characteristic value to the first layer 901 in accordance with the zoom state information input from the state acquisition unit 201, and the method by which the first layer 901 performs inference using image data in accordance with that characteristic value will be explained. Figure 10 is a diagram illustrating the evaluation value for a zoom state pattern in Embodiment 3 and the delay value obtained according to the evaluation value. Here, the delay obtained according to the evaluation value is the time difference between when the light information of the subject is received on the imaging surface and when the audio data emitted from the subject at the same time is input to the microphone. Since the light information is received on the imaging surface almost instantaneously, while the speed of sound propagation is approximately 340 meters / second, a time difference occurs between the acquired image and the sound. In other words, when performing inference on a subject far away from the imaging surface, if inference is performed using image data and audio data input at the same time, the inference result will be output using image data and audio data with a low temporal relationship, resulting in a decrease in inference accuracy.

[0059] The state acquisition unit 201 calculates an evaluation value for the distance between the imaging surface and the subject of interest, ranging from 0 to 4, based on the input zoom state, with the smallest zoom magnification state of the surveillance camera being the minimum value and the largest zoom magnification state being the maximum value. Therefore, the evaluation value is calculated such that a larger value indicates a greater distance to the subject of interest, and a smaller value indicates a closer distance to the subject of interest.

[0060] As illustrated in Figure 10, in the case of zoom state pattern 1001, where the evaluation value is 4, the distance from the imaging plane to the subject is far, so there is a large delay in the audio data emitted from near the subject, and the characteristic value output unit 202 sets the characteristic delay value to 10. Conversely, in the case of zoom state pattern 1002, where the evaluation value is 1, the distance from the imaging plane to the subject is close, so there is a small delay in the audio data emitted from near the subject, and the characteristic value output unit 202 sets the characteristic delay value to 1.

[0061] In this embodiment, the first layer 901 recursively performs neural network calculations according to the delay value output as a characteristic value from the characteristic value output unit 202. When the delay value is large, the number of recursive calculations increases, and when the delay value is small, the number of recursive calculations decreases.

[0062] Figure 11 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 3. The processing shown in the flowchart in Figure 11 is realized, for example, by the CPU 101 of the information processing device 100 executing a program or the like that is loaded into the RAM 103. The information processing device 100 starts the processing shown in the flowchart in Figure 11 by receiving image data, which is the first input 911, and audio data, which is the second input 912.

[0063] In step S1101, the CPU 101 calculates an evaluation value for the distance between the imaging plane and the subject of interest based on the zoom state of the surveillance camera. As mentioned above, the evaluation value is between a minimum value of 0 and a maximum value of 4.

[0064] In step S1102, the CPU 101 obtains a delay value based on the evaluation value obtained in step S1101.

[0065] In step S1103, the CPU 101 performs inference processing based on the delay value obtained in step S1102 and outputs an inference result regarding the presence or absence of dangerous activity at the surveillance camera. Specifically, the CPU 101 recursively performs image data in the first layer 901 using the input image data based on the delay value obtained in step S1102. In parallel with the inference in the first layer 901, the CPU 101 also performs inference in the second layer 902 using the input audio data. Furthermore, the CPU 101 performs inference using the inference results of the first layer 901 and the second layer 902 as input to the connected layer 903, and outputs the obtained result as the final inference result regarding the presence or absence of dangerous activity at the surveillance camera.

[0066] Thus, the information processing device 100 in this embodiment can perform inference on image data using image data and audio data with high temporal affinity by recursively calculating only the delay value, thereby improving inference accuracy. Furthermore, by performing inference processing using an RNN, it is possible to make predictions in the temporal direction for image data inference, thus improving the inference accuracy of predictions in the temporal direction as well.

[0067] Furthermore, although the first layer 901 was explained using a recursive inference method, this method is not limited to any other method that improves the temporal affinity between the image data of the subject and the audio data generated from the vicinity of the subject. For example, the input image data could be used for inference after a delay. With this method, prediction in the temporal direction is difficult, so the deep learning technique using a neural network can be constructed as a CNN.

[0068] (Other embodiments of the present invention) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0069] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0070] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) An information processing device that performs inference processing using a neural network, An inference processing means that performs the inference processing using multiple input data, including images and audio, as input. The system includes an acquisition means for acquiring information regarding the zoom state during the capture of the aforementioned image, The information processing apparatus is characterized in that the inference processing means controls the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state. (Configuration 2) The system includes characteristic value output means that outputs characteristic values ​​corresponding to the distance between the imaging plane and the subject of interest in the image, based on the acquired information regarding the zoom state. The information processing apparatus according to configuration 1, characterized in that the inference processing means controls the inference processing for at least one of the plurality of input data based on the characteristic value. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the inference processing means is controlled to perform the inference processing using an image including a subject of interest and the sound emitted from the vicinity of the subject. (Composition 4) The information processing apparatus according to any one of configurations 1 to 3, wherein the inference processing means has a plurality of inference models that perform the inference processing, and selects one of the inference models based on the acquired information regarding the zoom state to perform the inference processing. (Composition 5) The information processing apparatus according to configuration 4, characterized in that the plurality of inference models have different computational loads related to the inference process. (Composition 6) The information processing device according to configuration 4, characterized in that the plurality of inference models differ in at least one of the number of layers and the number of nodes in the computation of the neural network. (Composition 7) The information processing device according to any one of configurations 1 to 3, characterized in that the inference processing means controls the connection weights between nodes in the neural network that performs the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state. (Composition 8) The information processing apparatus according to configuration 7, characterized in that it controls the connection weights between nodes in the neural network according to the distance between the imaging plane of the image and the subject of interest in the image. (Composition 9) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the inference processing means controls the inference processing for the first input data in accordance with the delay of the second input data to the first input data based on the acquired information regarding the zoom state. (Composition 10) The information processing apparatus according to configuration 9, characterized in that the inference processing means performs calculations on the first input data using a recurrent neural network in accordance with the delay of the second input data relative to the first input data. (Method 1) A control method for an information processing device that performs inference processing using a neural network, An inference processing step that performs the inference processing using multiple input data including images and audio as input, The process includes an acquisition step for acquiring information regarding the zoom state during the capture of the aforementioned image. A control method for an information processing device, characterized in that the inference processing step controls the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state. (Program 1) A program to cause a computer to function as one of the information processing devices described in any one of the configurations 1 to 10. [Explanation of Symbols]

[0071] 100: Information processing unit 101: CPU 102: ROM 103: RAM 104: Storage device 105: Input unit 106: Display unit 107: Communication unit 201: Status acquisition unit 202: Characteristic value output unit 203: Inference processing unit

Claims

1. An information processing device that performs inference processing using a neural network, An inference processing means that performs the inference processing using multiple input data, including images and audio, as input. The system includes an acquisition means for acquiring information regarding the zoom state during the capture of the aforementioned image, The information processing apparatus is characterized in that the inference processing means controls the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state.

2. The system includes characteristic value output means that outputs characteristic values ​​corresponding to the distance between the imaging plane and the subject of interest in the image, based on the acquired information regarding the zoom state. The information processing apparatus according to claim 1, characterized in that the inference processing means controls the inference processing for at least one of the plurality of input data based on the characteristic value.

3. The information processing apparatus according to claim 1, characterized in that the inference processing means is controlled to perform the inference processing using an image including a subject of interest and the sound emitted from the vicinity of the subject.

4. The information processing apparatus according to claim 1, wherein the inference processing means has a plurality of inference models for performing the inference processing, and selects one of the inference models based on the acquired information regarding the zoom state for the inference processing.

5. The information processing apparatus according to claim 4, characterized in that the plurality of inference models have different computational loads related to the inference process.

6. The information processing apparatus according to claim 4, characterized in that the plurality of inference models differ in at least one of the number of layers and the number of nodes in the computation of the neural network.

7. The information processing apparatus according to claim 1, wherein the inference processing means controls the connection weights between nodes in the neural network that performs the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state.

8. The information processing apparatus according to claim 7, characterized in that it controls the connection weights between nodes in the neural network according to the distance between the imaging plane of the image and the subject of interest in the image.

9. The information processing apparatus according to claim 1, characterized in that the inference processing means controls the inference processing on the first input data in accordance with the delay of the second input data to the first input data based on the acquired information regarding the zoom state.

10. The information processing apparatus according to claim 9, characterized in that the inference processing means performs calculations on the first input data using a recurrent neural network in accordance with the delay of the second input data relative to the first input data.

11. A control method for an information processing device that performs inference processing using a neural network, An inference processing step that performs the inference processing using multiple input data including images and audio as input, The process includes an acquisition step for acquiring information regarding the zoom state during the capture of the aforementioned image. A control method for an information processing device, characterized in that the inference processing step controls the inference processing for at least one of the plurality of input data based on the acquired information regarding the zoom state.

12. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 10.

Images