Signal processing device, processing method for signal processing device, and program

Abstract

Even if some of the input data among multiple input data is delayed, the system suppresses the degradation of the inference results of the neural network model and completes processing within a specific time. [Solution] The signal processing device includes a generation means for generating one or more first input data, a receiving means for receiving one or more second input data from an external device, a calculation means for calculating a delay value of the second input data relative to the first input data, and a determination means for determining the use of one of a plurality of neural network models that input both or either the first input data and the second input data, according to the delay value calculated by the calculation means.

Description

Technical Field

[0001] The present invention relates to a signal processing device, a processing method of the signal processing device, and a program.

Background Art

[0002] Deep learning technology using neural networks has been applied in a wide range of technical fields. In particular, class classification for recognizing and classifying images is said to exceed human recognition ability. Among them, the convolutional neural network (CNN), which is particularly widely used, realizes high-precision deep learning processing by recursively applying convolutional operations to images.

[0003] In recent years, such deep learning processing has been applied to emotion recognition processing for recognizing the emotions of faces included in captured images. The emotion recognition processing mainly improves the accuracy of recognizing emotions from information such as the unevenness, texture, and contour of the face extracted from the image. However, since it is recognized only with single-modal information such as a captured image, the improvement in accuracy is not sufficient.

[0004] Therefore, in recent years, attention has been focused on AI technology called multimodal AI that can process multiple types of information such as text, images, voice, and video at once. Reports have been made that the accuracy of inference processing is improved by using multimodal AI technology compared to single-type AI processing.

[0005] Patent Document 1 discloses a technique for performing deep learning processing using a plurality of modal information. According to Patent Document 1, by integrally learning a plurality of inference models using a plurality of modal information, it is possible to improve the accuracy of the inference result as compared with the case of learning with single-modal information.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-2023 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, even when using the technology described in Patent Document 1, if some of the modal information among multiple modal information is delayed as input data, the start of deep learning processing will be delayed. Therefore, in real-time systems where processing results need to be obtained within a specific time, there is a problem that the processing results cannot be obtained in time. In addition, if deep learning processing is started before the delayed input data is ready, there is a problem that the accuracy of the processing results will decrease.

[0008] The present invention aims to suppress the degradation of the accuracy of the inference results of a neural network model, even when some of the input data among multiple input data is delayed, and to complete the processing within a specific time. [Means for solving the problem]

[0009] The signal processing device includes a generation means for generating one or more first input data, a receiving means for receiving one or more second input data from an external device, a calculation means for calculating a delay value of the second input data relative to the first input data, and a determination means for determining the use of one of a plurality of neural network models that input both or either the first input data and the second input data, according to the delay value calculated by the calculation means. [Effects of the Invention]

[0010] According to the present invention, even if some of the input data among multiple input data is delayed, it is possible to suppress the degradation of the accuracy of the neural network model's inference results and complete the processing within a specific time. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of an overall configuration diagram of the signal processing device in the first embodiment. [Figure 2] This figure shows an example of a block diagram within the signal processing device in the first embodiment. [Figure 3] This flowchart shows an example of a series of processes in the first embodiment, from the signal processing device taking an image captured and an audio signal acquired from an external source as input to a neural network model and performing neural network processing. [Figure 4] This figure shows an example of a communication processing sequence for synchronizing the time between devices in the first embodiment. [Figure 5] This flowchart shows an example of a series of processes performed by the signal processing device in the first embodiment until it determines the neural network model. [Figure 6] This figure shows an example of the configuration of the neural network model used by the signal processing device in the first embodiment. [Figure 7] This timing chart shows the timing of a series of processes in the first embodiment, from when the signal processing device takes an image captured and an audio signal acquired from an external source as input to a neural network model, until it performs neural network processing. [Modes for carrying out the invention]

[0012] Hereinafter, an example of a preferred embodiment of the present invention will be described in detail with reference to the drawings.

[0013] [First Embodiment] Figure 1 shows an example configuration of the signal processing system 102 in the first embodiment. The signal processing system 102 includes a digital camera 100 and a wireless microphone 101. The wireless microphone 101 is an external microphone that is wirelessly connected to the digital camera 100.

[0014] The digital camera 100 is an example of a signal processing device and can be wirelessly connected to a wireless microphone 101 according to the Bluetooth (registered trademark) standard. In this wireless connection according to the Bluetooth standard, in synchronous communication, the digital camera 100 can receive audio data and the like from the wireless microphone 101.

[0015] Also, in the wireless connection according to the Bluetooth standard, in asynchronous communication, the digital camera 100 can transmit control data such as output instructions to the wireless microphone 101. By wirelessly connecting the wireless microphone 101 to the digital camera 100, the user can enable the digital camera 100 to receive sound information from a sound source far away via the wireless microphone 101.

[0016] <Configuration of Digital Camera 100> FIG. 2 is a block diagram showing a configuration example of a digital camera 100, which is an example of a signal processing device of this embodiment. Here, as an example of the signal processing device, the digital camera 100 will be described, but the signal processing device is not limited thereto. For example, the signal processing device may be a smartphone, a personal computer, a smartwatch, a tablet terminal, or the like.

[0017] The digital camera 100 includes a control unit 201, an imaging unit 202, a non-volatile memory 203, a working memory 204, an operation unit 205, a display unit 206, a microphone 207, a speaker 208, a power supply unit 209, a recording medium 210, a communication unit 211, a connection unit 212, and a neural network processing unit 213.

[0018] The control unit 201 controls each part of the digital camera 100 according to the input signals and the execution of the program described later. The control unit 201 controls the time synchronization with external devices in cooperation with the communication unit 211. The control unit 201 synchronizes the time with external devices by periodically exchanging time information with external devices. Also, the control unit 201 determines a model to be used from the neural network models for neural network processing recorded in the non-volatile memory 203 and the recording medium 210 described later. Instead of the control unit 201 controlling the entire digital camera 100, the entire digital camera 100 may be controlled by a plurality of hardware sharing the processing.

[0019] The imaging unit 202 is composed of, for example, an optical system that controls an optical lens unit and an aperture, zoom, focus, etc., and an imaging element for converting the light (video) introduced through the optical lens unit into an electrical video signal. The imaging unit 202, under the control of the control unit 201, converts the subject light imaged by the lens included in the imaging unit 202 into an electrical signal by the imaging element, performs noise reduction processing, etc., and outputs digital data as image data or video data. Also, the imaging unit 202 has a shutter that can freely control the exposure time of the imaging element under the control of the control unit 201.

[0020] The non-volatile memory 203 is an electrically erasable and recordable non-volatile memory, and stores the program etc. described later executed by the control unit 201. Also, a plurality of neural network models are recorded in the non-volatile memory 203. This neural network model is, for example, a neural network model corresponding to multimodal processing that takes two types of data, voice and image, as input, or a neural network model corresponding to single-modal processing that takes only an image as input.

[0021] The working memory 204 is used as a buffer memory for temporarily holding image data and video data captured by the imaging unit 202, as well as as an image display memory for the display unit 206 and a work area for the control unit 201. The working memory 204 is also used as a temporary storage location when the neural network processing unit 213 performs neural network calculations.

[0022] The control unit 205 is a user interface (UI) for receiving instructions from the user for the digital camera 100. The control unit 205 may include, for example, a power switch for the user to turn the digital camera 100 on or off, a shutter release switch for taking a picture, and a playback button for playing back image data. A touch panel formed on the display unit 206 can also be included in the control unit 205.

[0023] The release switch has two switches, SW1 and SW2. When the release switch is half-pressed, SW1 turns on. This allows the camera to receive preparatory instructions for image capture, such as AF (autofocus), AE (automatic exposure), AWB (auto white balance), and EF (flash pre-flash). When the release switch is fully pressed, SW2 turns on. This allows the camera to receive imaging instructions for image capture.

[0024] The display unit 206 displays the viewfinder image during shooting, displays captured image data, and displays text for interactive operation. The display unit 206 does not necessarily have to be built into the digital camera 100, and may be configured to be externally connected to the digital camera 100. The digital camera 100 can be connected to an internal or external display unit 206 and only needs to have a display control function to control the display of the display unit 206.

[0025] Microphone 207 is used to input sound waves, such as sound or voice, into the digital camera 100. Microphone 207 converts sound or voice into electrical signals and inputs them into the digital camera 100.

[0026] The control unit 201 generates audio data from the input electrical signal. For example, the control unit 201 can record this audio data in synchronization with video data captured by the imaging unit 202. Alternatively, the control unit 201 can record this audio data in association with image data captured by the imaging unit 202.

[0027] The microphone 207 may be configured to be detachable from the digital camera 100, or it may be built into the digital camera 100. In other words, the digital camera 100 only needs to have a means for receiving electrical signals from the microphone 207. Furthermore, when connecting the wireless microphone 101 using the communication unit 211, it is possible to record video data synchronized with the captured video data using the audio input from the wireless microphone 101, without using the audio input from the microphone 207.

[0028] The speaker 208 is an electroacoustic converter capable of outputting electronic sound. In this embodiment, the control unit 201 converts the audio data recorded in the non-volatile memory 203 into an audio signal, and the speaker 208 can output that audio signal.

[0029] The power supply unit 209, controlled by the control unit 201, can supply power to each element of the digital camera 100. The power supply unit 209 is, for example, a lithium-ion battery or an alkaline manganese dry cell battery.

[0030] The recording medium 210 can record, for example, image data output from the imaging unit 202. The recording medium 210 is, for example, a memory card. The recording medium 210 may be configured to be detachable from the digital camera 100, or it may be built into the digital camera 100. In other words, the digital camera 100 only needs to have means to access the recording medium 210.

[0031] The communication unit 211 is an interface for wireless connection with external devices. The digital camera 100 of this embodiment can exchange data with external devices via the communication unit 211. For example, the control unit 201 can transmit image data generated by the imaging unit 202 and audio data recorded in the non-volatile memory 203 to external devices via the communication unit 211. External devices include, for example, information devices such as smartphones and PCs, external speakers such as earphones and headphones, and strobes.

[0032] In this embodiment, the communication unit 211 includes an interface for communicating with external devices in accordance with the Bluetooth® standard. Hereafter, wireless communication compliant with the Bluetooth standard will be referred to as Bluetooth communication.

[0033] The control unit 201 enables wireless communication with external devices by controlling the communication unit 211. The communication unit 211 receives voice data from the wireless microphone 101 via Bluetooth communication. The communication unit 211 also performs wireless LAN communication with the wireless microphone 101 in accordance with the IEEE 802.11 standard.

[0034] The communication unit 211 performs time synchronization communication with the wireless microphone 101 using wireless LAN communication. Time synchronization communication is a communication method that synchronizes the time between devices using the PTP (Precision Time Protocol) standard. This allows the signal processing device 100 and the wireless microphone 101 to have common time information. Based on this synchronized time information, the digital camera 100 can then estimate the delay time of data from external devices.

[0035] The connection unit 212 is an interface for wired connection to external devices. The digital camera 100 of this embodiment can exchange data with external devices via the connection unit 212. For example, the control unit 201 can transmit image data generated by the imaging unit 202 or moving image data recorded in the non-volatile memory 203 to an external device via the connection unit 212. Also, for example, the control unit 201 can receive audio signals and audio data from an external device such as a microphone via the connection unit 212.

[0036] Furthermore, when the digital camera 100 connects to external devices such as microphones or headphones, the control unit 201 can detect the type of device after establishing a connection with the external device. In Bluetooth communication via the communication unit 211, the control unit 201 can use the Service Discovery Protocol (SDP) to detect whether the external device can function as, for example, headphones or a microphone. Also, in wireless LAN communication via the communication unit 211, for example, the control unit 201 can detect the type of device of the external device by receiving the device type from the external device.

[0037] The above describes one example of the configuration of the digital camera 100.

[0038] Next, using Figure 3, we will explain an example of a series of processes from the point in time when the signal processing device 100 takes the captured image generated by the imaging unit 202 and the audio information received from the wireless microphone 101 as input to a neural network model and performs neural network processing. The series of processes starts when the user turns on the power to the signal processing device 100 using the operation unit 205. The processing method of the signal processing device 100 will be explained below.

[0039] In step S301, the control unit 201 uses the communication unit 211 to check if there is an external device capable of wireless communication, and establishes wireless communication with the confirmed external device. In this embodiment, the control unit 201 communicates voice information with the wireless microphone 101 using Bluetooth communication and performs time synchronization communication between devices using wireless LAN communication. Step S301 is started not only when the power is turned on, but also when the user performs an operation to check for the existence of an external device that can be connected. Step S301 is also started when the connection partner changes.

[0040] In step S302, the control unit 201 performs time synchronization communication using PTP (Precision Time Protocol) to synchronize the time with the wireless microphone 101. Details of the communication procedure will be described later with reference to Figure 4. In this embodiment, the signal processing device 100 becomes the primary device for time synchronization, and the wireless microphone 101 becomes the secondary device. The primary and secondary devices synchronize their time by communicating periodically, with the secondary device adjusting to the time of the primary device. The control unit 201 performs time synchronization communication with the wireless microphone 101 to synchronize the time.

[0041] In step S303, the control unit 201 begins receiving one or more audio data wirelessly from the wireless microphone 101. The wireless microphone 101 is an example of an external device. The control unit 201 calculates the delay information of the audio data received by the communication unit 211 from the wireless microphone 101 based on the synchronized time via time-synchronized communication. The wireless microphone 101 records the timing of its own sound pickup using its own time synchronization time, adds this time information, and transmits it to the signal processing device 100. The control unit 201 compares the time information of the audio data received by the communication unit 211 from the wireless microphone 101 with its own time synchronization time and calculates the difference from the sound pickup time as the delay value.

[0042] In other words, the control unit 201 calculates the delay value of the audio data relative to the captured image data in step S304, which will be described later. Specifically, the communication unit 211 receives time information from the wireless microphone 101 that the audio data was generated. The control unit 201 calculates the difference between the time information that the audio data was generated and the time on the signal processing device 100 as the delay value.

[0043] In step S304, the control unit 201 controls the imaging unit 202 to start capturing moving images and generates one or more captured image data. The control unit 201 records the generation time of the images generated by the imaging unit 202 using its own time synchronization time. The control unit 201 can manage the acquisition times of images and sound by comparing the time of the images captured by the signal processing device 100 with the time of the sound picked up by the wireless microphone 101.

[0044] In step S305, the control unit 201 determines the neural network model to be used for neural network processing based on the calculated delay value of the audio data. Details of the processing will be described later with reference to Figure 5.

[0045] In step S306, the control unit 201 transmits the determined neural network model data to the neural network processing unit 213. Next, the control unit 201 controls the neural network processing to start processing by inputting the captured image generated by the imaging unit 202 and the audio data acquired from the wireless microphone 101 via the communication unit 211 to the two inputs of the neural network model.

[0046] In step S307, the control unit 201 checks whether a termination operation has been performed by the user via the operation unit 205. If a termination operation has been performed (step S307, YES), the control unit 201 terminates the process. If a termination operation has not been performed (step S307, NO), the control unit 201 returns to step S304 and continues the process.

[0047] The above processing sequence enables the execution of optimal neural network processing by determining and using a neural network model based on the time difference between the captured image and the audio data. Details of the neural network model will be described later with reference to Figure 6.

[0048] Next, Figure 4 will be used to explain the details of the packet exchange for time synchronization between the primary and secondary devices. Through this packet exchange, the delay of the network path is estimated, and taking that delay into account, the secondary device can synchronize its time with that of the primary device. The method described in this embodiment is an example of what is called the TWO-STEP method; other methods such as the ONE-STEP method also exist.

[0049] In this embodiment, the primary device is a signal processing device 100, and the secondary device is a wireless microphone 101.

[0050] In sequence S401, the primary device sends a Sync packet to the secondary device. The Sync packet contains information indicating that the synchronization method for this instance is the TWO-STEP method. Upon receiving the Sync packet, the secondary device stores the time of receipt.

[0051] In sequence S402, the primary device sends a Follow-Up packet. The Follow-Up packet contains the transmission time of the Sync packet sent immediately before it. The secondary device can calculate the delay time in the communication path from the primary device to the secondary device from the difference between the transmission time of the Sync packet contained in the Follow-Up packet and the reception time of the Sync packet it has stored.

[0052] In sequence S403, the secondary device sends a Delay-req packet to the primary device. The secondary device stores the time the Delay-req packet was sent. The primary device receives the Delay-req packet and stores the time it was received.

[0053] In sequence S404, the primary device sends a Delay-resp packet to the secondary device. The Delay-resp packet contains information about the time the primary device received the Delay-req packet. The secondary device can calculate the delay time in the communication path from the secondary device to the primary device from the difference between the transmission time of the Delay-req packet and the reception time of the Delay-req packet stored in the Delay-resp packet.

[0054] By periodically exchanging the packets described above, the secondary device can periodically correct its time to match that of the primary device, thereby enabling periodic time synchronization.

[0055] Figure 5 illustrates an example of the process for determining the neural network model to be used. Figure 5 is a flowchart detailing step S305 in Figure 3.

[0056] In step S501, the control unit 201 checks the mode that the user has previously set on the operation unit 205. There are two types of modes.

[0057] The first mode is a multimodal neural network model that uses a model trained on data where a predetermined input data is delayed by a certain amount. This mode is defined as the delayed data learning model mode. In this case, for example, in a multimodal neural network model that takes captured image data and audio data as input, the neural network model is trained on audio data from a different time than when the image was captured. When performing inference processing, the control unit 201 decides to use a model trained with a similar delay difference based on the delay value of the input data.

[0058] The second mode is a multimodal neural network model that uses a model with fewer layers for processing delayed data when a predetermined input data is delayed by a certain amount among multiple input data. The control unit 201 decides to use a model with fewer layers for processing audio data when audio data is input with a delay relative to the image.

[0059] If the selected mode in step S501 is the learning model mode using delayed data, proceed to step S502.

[0060] If the selected mode in step S501 is not the learning model mode using delayed data, proceed to step S505.

[0061] In step S502, the control unit 201 determines whether the delay value calculated in step S303 in Figure 3 is less than or equal to a first predetermined value. If the calculated delay value is less than or equal to the first predetermined value (step S502, YES), the control unit 201 proceeds to step S503.

[0062] If the calculated delay value is not less than or equal to the first predetermined value (step S502, NO), the control unit 201 proceeds to step S504.

[0063] In step S503, the control unit 201 decides to use a pre-trained multimodal neural network model corresponding to the delay value.

[0064] In step S504, the control unit 201 decides to use a single-modal neural network model.

[0065] In step S505, the control unit 201 determines whether the delay value calculated in step S303 in Figure 3 is less than or equal to a second predetermined value. If the calculated delay value is less than or equal to the second predetermined value (step S505, YES), the control unit 201 proceeds to step S506.

[0066] If the calculated delay value is not less than or equal to the second predetermined value (step S505, NO), the control unit 201 proceeds to step S504.

[0067] In step S506, the control unit 201 decides to use a multimodal neural network model that has a processing hierarchy for audio data corresponding to the delay value.

[0068] As described above, the use of a neural network model is determined based on the delay value of the audio data received from the wireless microphone 101.

[0069] Next, I will explain neural network models.

[0070] First, let's explain the trained multimodal neural network model corresponding to the delay value in step S503. In the trained multimodal neural network model corresponding to the delay value, the audio data that is ready at the time the image data generated by the imaging unit 202 is input to the neural network model is input. For example, in the case of audio data with a 1-second delay, the audio that was recorded 1 second earlier will be input, and the data input to the multimodal neural network model will have a 1-second difference between the image capture time and the audio recording time. By using a model that has been trained with data that is 1 second behind as training data, it is possible to suppress the decrease in inference accuracy and complete the processing within the real-time time limit.

[0071] Next, using Figure 6, we will explain a multimodal neural network model that has a processing hierarchy for audio data corresponding to delay values.

[0072] Figure 6(A) shows an example of a multimodal neural network model for performing neural network processing using multiple types of data as shown in step S503 of Figure 5. It takes two types of data, a first input data and a second input data, as input and outputs the result. In this embodiment, a multimodal neural network model with two types of input data is used for explanation, but it is not limited to this, and a multimodal neural network model with three or more types of input data may also be used.

[0073] In this embodiment, captured image data is input as the first input data, and audio data acquired from the wireless microphone 101 is input as the second input data. There is a layer that processes a single type of data for each of the first and second input data. This is defined as the single-type data processing layer. When the first input data is captured image data, this layer processes only the captured image data. The processing of the single-type data processing layer has a different configuration depending on the input data. In other words, the configuration and processing content of the single-type data processing layer that processes audio data are different from those of the single-type data processing layer that processes captured image data.

[0074] Next, after the single-type data processing layer is complete, there is a layer that processes multiple types of data. This is defined as the multi-type processing layer. This layer is the core processing of a multimodal neural network, and by processing multiple types of data using a neural network, it becomes possible to output highly accurate inference results.

[0075] Figure 6(B) shows an example of a multimodal neural network model in step S506 of Figure 5, which has a configuration with less processing in the layer that processes the second input data. Because there is less processing in the single-type data processing layer, the accuracy of the final inference result will decrease to some extent, but in return, the processing time will be reduced. In a real-time system where the inference result from the neural network must be output within a specific time, the neural network model in Figure 6(B) is used when the preparation of the second input data is delayed. This makes it possible to synchronize the processing completion times of the single-type data processing layer for the first input data and the second input data which is input later. This prevents delays in the timing of data input to the multi-type processing layer. As a result, even if the preparation of the second input data is delayed, it is possible to complete the processing within a specific time. The timing of each process will be described later using the timing chart in Figure 7.

[0076] Figure 6(C) shows an example of a single-modal neural network model for step S504 in Figure 5. If preparation is significantly delayed with the second input data, it is necessary to complete the processing using only the first input data in order to maintain real-time performance. This is the neural network model used in such cases. While it does not improve the accuracy of the inference results using a multimodal neural network, it makes it possible to complete the processing within a specific time.

[0077] As described above, in Figure 5, the control unit 201 determines, based on the delay value calculated in step S303 of Figure 3, to use one of the multiple neural network models in steps S503, S504, and S506, which input both or one of the first and second input data.

[0078] The neural network models in steps S503 and S506 are neural network models that take both the first and second input data as input, as shown in Figures 6(A) and (B).

[0079] The neural network model in step S504 is a neural network model that takes the first input data but does not take the second input data, as shown in Figure 6(C).

[0080] The control unit 201 decides to use the neural network in step S503 or S506 if the delay value is less than or equal to a predetermined value, and decides to use the neural network in step S504 if the delay value is not less than or equal to a predetermined value.

[0081] The neural network models in Figures 6(A) and (B) include a first single-type data processing layer that processes the first input data, a second single-type data processing layer that processes the second input data, and a multi-type processing layer that processes the output results of the first and second single-type data processing layers.

[0082] For example, the first input data is captured image data, and the second input data is audio data.

[0083] The second single-type data processing layer of the neural network model in Figure 6(B) requires less computation than the second single-type data processing layer of the neural network model in Figure 6(A).

[0084] The neural network model in step S503 is a neural network model that has been trained based on the input time difference between the first input data and the second input data.

[0085] Next, using Figure 7, we will explain an example of a timing chart for a series of processes related to neural network processing.

[0086] Figure 7(A) is a timing chart when the reception delay of audio data from the wireless microphone 101 is small and the neural network model in Figure 6(A) is used.

[0087] T701a marks the timing when the imaging unit 202 begins capturing an image. The wireless microphone 101 continuously picks up external sound, but for clarity, the diagram only shows the audio data pickup interval for which the same multimodal processing is applied to the captured image. The wireless microphone 101 transmits the audio data picked up at T701a to the signal processing device 100.

[0088] At T702a, the communication unit 211 of the signal processing device 100 begins receiving the first data of the voice data transmitted from the wireless microphone 101. In other words, the section from T701a to T702a corresponds to the delay time due to communication delays, etc. This delay time is determined by the processing performance of the wireless microphone 101, the communication protocol used, and the network congestion state.

[0089] In T703a, the imaging unit 202 completes the capture of one screen. Once the capture is complete, the control unit 201 sends the captured image data to the neural network processing unit 213, and neural network processing begins. In Figure 7(A), multimodal processing is performed because the delay time of the audio data is short. The control unit 201 inputs the captured image data into the first input data and starts single-type data processing hierarchical processing.

[0090] In the T704a, once the reception of audio data for an interval equivalent to the image acquisition interval of the captured image is complete, the control unit 201 sends the received audio data to the neural network processing unit 213, and neural network processing begins. The audio data is input to the second input data, and single-type data processing hierarchical processing begins.

[0091] In T705a, the neural network processing unit 213 completes the processing of the single-type data processing layer for the first and second input data, passes the processing results to the multi-type data processing layer, and starts processing in the multi-type data processing layer.

[0092] In T706a, the neural network processing unit 213 completes the processing of the multi-data processing layer and completes the neural network processing.

[0093] T707a indicates the time limit from the start of imaging to the completion of processing required to maintain the real-time system. Neural network processing must be completed by this time. In Figure 7(A), because the delay in receiving audio data from the wireless microphone 101 is small, even if multimodal processing with many processing layers is performed on the audio data received from the wireless microphone 101, the neural network processing can be completed by the time T707a is reached.

[0094] Figure 7(B) shows a timing chart when the reception delay of audio data from the wireless microphone 101 is greater than in Figure 7(A), and the neural network model shown in Figure 6(B), which has less processing load for audio data, is used.

[0095] T701b, like T701a, is the timing when the imaging unit 202 starts capturing images.

[0096] T703b operates at the same timing as T703a, and the imaging unit 202 completes the imaging of one screen.

[0097] T702b is the time when the communication unit 211 of the signal processing device 100 begins receiving the first audio data from the wireless microphone 101. However, this is significantly delayed compared to T702a, and the data is received while the processing of the single-type data processing layer for the image has progressed only partially. If the neural network model in Figure 6(A) is used here, the processing of the single-type data processing layer for the audio data will be delayed, and the neural network inference process cannot be completed by T707b. Therefore, the neural network model in Figure 6(B), which has less processing of the single-type data processing layer for audio, is used.

[0098] T704b is the time when the voice data reception is complete. The time taken to receive the voice data, T702b to T704b, is equivalent to T702a to T704b.

[0099] T705b represents the completion time of the single-type data processing layer. Although there is a significant delay in acquiring audio data, a neural network model with minimal processing of audio data is used, allowing for the completion of processing of both captured image data and audio data at roughly the same time.

[0100] T706b, like T706a, is the time when the processing of the multi-data processing layer is completed and the neural network processing is finished.

[0101] T707b, like T707a, is a time limit for maintaining the real-time system, and since T706b completes its processing before T707b, no problem occurs.

[0102] Next, Figure 7(C) shows a situation where the reception delay of audio data from the wireless microphone 101 is significantly delayed, and the multimodal neural network processing cannot complete within the real-time time limit. In this situation, using the multimodal neural network model would cause the real-time system to fail, so it is necessary to use the single-modal neural network model shown in Figure 6(C), and Figure 7(C) is the timing chart for this situation.

[0103] The T701c is the same as the T701a.

[0104] In T702c, there is a significant delay in receiving audio data from the wireless microphone 101. Even if multimodal neural network processing is started at this point, it will not be possible to complete the processing by the real-time time limit T707c, so the received audio data will not be used for neural network processing.

[0105] In the T703c, the control unit 201 inputs the image data generated by the imaging unit 202 into the input data of a single-modal neural network model and starts neural network processing.

[0106] In the T706c, the neural network processing unit 213 completes single-modal neural network processing on the input image data.

[0107] Although T707c has a real-time time limit, T707c completes before T707c, so no problem occurs.

[0108] As described above, in a real-time system that completes neural network processing using multiple types of data within a specific time frame, using a neural network model that has a single-type data processing layer with a processing amount corresponding to the input data delay makes it possible to complete processing within a specific time frame while suppressing a decrease in the accuracy of the inference results.

[0109] In deep learning processing that takes multiple input data into account, the signal processing device 100 can suppress the degradation of the accuracy of the inference results and complete the processing within a specific time by using a neural network model that takes into account the processing of delayed input data when some of the input data are delayed.

[0110] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0111] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0112] This embodiment includes the following configuration. (Item 1) A generation means for generating one or more first input data, A receiving means for receiving one or more second input data from an external device, A calculation means for calculating the delay value of the second input data relative to the first input data, A determination means that determines the use of one of a plurality of neural network models that take both or either the first input data and the second input data as input, in accordance with the delay value calculated by the calculation means. A signal processing device characterized by having (Item 2) The signal processing apparatus according to item 1, characterized in that the determination means determines, according to the delay value calculated by the calculation means, to use one of the following: a first neural network model that inputs both the first input data and the second input data, and a second neural network model that inputs the first input data but does not input the second input data. (Item 3) The signal processing apparatus according to item 2, characterized in that the determination means determines to use the first neural network when the delay value is less than or equal to a predetermined value, and determines to use the second neural network when the delay value is not less than or equal to a predetermined value. (Item 4) The first neural network model described above is A first single-type data processing hierarchical unit that processes the first input data, A second single-type data processing hierarchical unit that processes the second input data, The signal processing apparatus according to item 2 or 3, characterized by having a first single-type data processing layer and a multi-type processing layer for processing the output results of the second single-type data processing layer. (Item 5) The aforementioned determination means is In the first mode, and when the delay value is less than or equal to a first predetermined value, it is decided to use a first neural network model that takes both the first input data and the second input data as inputs. If it is the first mode and the delay value is not less than or equal to a first predetermined value, it is decided to use a second neural network model that takes the first input data but does not take the second input data. In the second mode, and when the delay value is less than or equal to the first predetermined value, it is decided to use a third neural network model that takes both the first input data and the second input data as inputs. If it is the second mode and the delay value is not less than or equal to the first predetermined value, then it is decided to use the second neural network model. The first neural network model and the third neural network model are each, A first single-type data processing hierarchical unit that processes the first input data, A second single-type data processing hierarchical unit that processes the second input data, It comprises a first single-type data processing hierarchy and a multi-type processing hierarchy that processes the output results of the first single-type data processing hierarchy, The signal processing apparatus according to item 1, characterized in that the second single-type data processing layer of the third neural network model has less computational complexity than the second single-type data processing layer of the first neural network model. (Item 6) The signal processing device according to item 5, characterized in that the first neural network model is a neural network model that has been trained based on the input time difference between the first input data and the second input data. (Item 7) The receiving means receives time information from the external device that generated the first input data, The signal processing apparatus according to any one of items 1 to 6, characterized in that the calculation means calculates the difference between the time information on which the first input data was generated and the time of the signal processing apparatus as the delay value. (Item 8) The signal processing device according to item 7, further comprising a synchronization communication means for performing time synchronization communication to synchronize the time with the aforementioned external device. (Item 9) The signal processing device according to any one of items 1 to 8, characterized in that the receiving means wirelessly receives the second input data. (Item 10) The first input data is captured image data, The signal processing apparatus according to any one of items 1 to 9, characterized in that the second input data is audio data. (Item 11) The signal processing device according to item 10, characterized in that the external device is a wireless microphone. (Item 12) A generation step that generates one or more first input data, A receiving step of receiving one or more second input data from an external device, A calculation step of calculating the delay value of the second input data relative to the first input data, A decision step in which, in accordance with the delay value calculated in the calculation step, a neural network model is selected from among a plurality of neural network models that take both or either the first input data and the second input data as inputs, and the use of one neural network model is determined. A method for processing a signal processing device, characterized by having the following features. (Item 13) A program to cause a computer to function as a signal processing device as described in any one of items 1 through 11. [Explanation of symbols]

[0113] 100 Signal Processing Device 101 Wireless Microphone 201 Control Unit 202 Imaging Department 203 Non-volatile memory 204 Working memory 205 Operation section 206 Display section 207 Mike 208 speakers 209 Power supply section 210 Recording media 211 Communications Department 212 Connection part 213 Neural Network Processing Unit

Claims

1. A generation means for generating one or more first input data, A receiving means for receiving one or more second input data from an external device, A calculation means for calculating the delay value of the second input data relative to the first input data, A determination means that determines the use of one of a plurality of neural network models that take both or either the first input data and the second input data as input, in accordance with the delay value calculated by the calculation means. A signal processing device characterized by having

2. The signal processing apparatus according to claim 1, wherein the determination means determines, in accordance with the delay value calculated by the calculation means, to use one of the following: a first neural network model that inputs both the first input data and the second input data, and a second neural network model that inputs the first input data but does not input the second input data.

3. The signal processing apparatus according to claim 2, characterized in that the determination means decides to use the first neural network when the delay value is less than or equal to a predetermined value, and decides to use the second neural network when the delay value is not less than or equal to a predetermined value.

4. The first neural network model described above is A first single-type data processing hierarchical unit that processes the first input data, A second single-type data processing hierarchical unit that processes the second input data, The signal processing apparatus according to claim 2, characterized by having a first single-type data processing layer and a multi-type processing layer for processing the output results of the second single-type data processing layer.

5. The aforementioned determination means is In the first mode, and when the delay value is less than or equal to a first predetermined value, it is decided to use a first neural network model that takes both the first input data and the second input data as inputs. If it is the first mode and the delay value is not less than or equal to the first predetermined value, it is decided to use a second neural network model that takes the first input data but does not take the second input data. In the second mode, and when the delay value is less than or equal to the first predetermined value, it is decided to use a third neural network model that takes both the first input data and the second input data as inputs. If it is the second mode and the delay value is not less than or equal to the first predetermined value, then it is decided to use the second neural network model. The first neural network model and the third neural network model are each, A first single-type data processing hierarchical unit that processes the first input data, A second single-type data processing hierarchical unit that processes the second input data, It comprises a first single-type data processing hierarchy and a multi-type processing hierarchy that processes the output results of the first single-type data processing hierarchy, The signal processing apparatus according to claim 1, characterized in that the second single-type data processing layer of the third neural network model has less computational complexity than the second single-type data processing layer of the first neural network model.

6. The signal processing apparatus according to claim 5, characterized in that the first neural network model is a neural network model that has been trained based on the input time difference between the first input data and the second input data.

7. The receiving means receives time information from the external device that generated the first input data, The signal processing apparatus according to claim 1, characterized in that the calculation means calculates the difference between the time information on which the first input data was generated and the time of the signal processing apparatus as the delay value.

8. The signal processing device according to claim 7, further comprising a synchronization communication means for performing time synchronization communication to synchronize the time with the external device.

9. The signal processing device according to claim 1, characterized in that the receiving means wirelessly receives the second input data.

10. The first input data is captured image data, The signal processing apparatus according to claim 1, characterized in that the second input data is audio data.

11. The signal processing device according to claim 10, characterized in that the external device is a wireless microphone.

12. A generation step that generates one or more first input data, A receiving step of receiving one or more second input data from an external device, A calculation step of calculating the delay value of the second input data relative to the first input data, A decision step in which, in accordance with the delay value calculated in the calculation step, a neural network model is selected from among a plurality of neural network models that take both or either the first input data and the second input data as inputs, and the use of one neural network model is determined. A method for processing a signal processing device, characterized by having the following features.

13. A program for causing a computer to function as a signal processing device according to any one of claims 1 to 11.

Images