Information processing device, information processing method, and information processing program

Abstract

This information processing device makes inferences (e.g., object detection) by decimating consecutive frames on a prescribed interval. Upon detecting an event by inference with respect to a frame, the information processing device makes inferences with respect to consecutive frames to the frame in which the event was detected. If the event is no longer detected as a result of the inferences with respect to the frames, the information processing device again makes inferences by decimating frames on the prescribed interval. The information processing device also makes inferences by going back to a frame preceding the frame in which the event was detected.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program for performing inference on frames of a video.

[0002] With the development of deep learning and the spread of IoT (Internet of Things) devices, applications have emerged that detect people from videos taken by surveillance cameras installed throughout the city and vehicles from videos taken by in-vehicle cameras.

[0003] To ensure real-time processing of the above video, it is desirable to perform the processing on the IoT device. However, due to the significant resource constraints on IoT devices, it is difficult to deploy deep learning models for highly accurate object detection.

[0004] Therefore, a technology has been proposed in which IoT devices and a cloud computer work together to perform object detection processing. However, this technology has a problem in that the inference cost of the entire system increases as the number of IoT devices increases. To solve this problem, for example, a technology has been proposed in which the IoT device (first layer) first performs inference processing such as object detection for video using a lightweight model, and the cloud computer (second layer) performs advanced inference processing only when it is determined necessary based on the output result of the inference processing (two-layer inference system, see Non-Patent Document 1).

[0005] Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, Matei Zaharia, "NoScope: Optimizing Neural Network Queries over Video at Scale", [online], Internet <URL: https: / / www.vldb.org / pvldb / vol10 / p1586-kang.pdf>

[0006] However, in the above-mentioned two-layer inference system, if the inference model used in the first layer is made lightweight in order to reduce the resources of the IoT device (first layer), the inference accuracy of the first layer will decrease, which will in turn affect the inference accuracy of the entire system.

[0007] For example, if the first layer infers that there is a low probability that the target object is included in the input data, even if the input data actually contains the target object, the second layer will not perform inference on that data, and as a result, the target object included in the input data will not be detected.

[0008] Therefore, the present invention aims to reduce the frequency of inference without reducing the accuracy of inference when performing inference processing on data, such as video, in which there is a high possibility of consecutive events occurring.

[0009] In order to solve the above-mentioned problems, the present invention is characterized by comprising a data input unit that accepts input of consecutive data, an inference unit that thins out the consecutive data and performs inference on the data, and performs inference on consecutive data starting from data at which an event is detected by the inference, and an output processing unit that outputs the inference results based on the inference on the data.

[0010] According to the present invention, when performing inference processing on data such as video data in which there is a high possibility of consecutive events occurring, it is possible to reduce the frequency of inference without reducing the accuracy of inference.

[0011] FIG. 1 is a diagram illustrating an overview of an information processing device. FIG. 2 is a diagram illustrating an example of frame inference. FIG. 3 is a diagram illustrating an example of processing executed by the information processing device. FIG. 4 is a diagram illustrating an example of the configuration of the information processing device. FIG. 5 is a flowchart illustrating an example of processing executed by the information processing device. FIG. 6 is a diagram illustrating an example of processing in which the information processing device performs inference by going back to past frames. FIG. 7 is a diagram illustrating an example of processing in which the information processing device performs inference on each frame of a video file. FIG. 8 is a diagram illustrating an example of the configuration of a computer that executes an information processing program.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0013] First, an overview of the information processing device of this embodiment will be described with reference to Fig. 1. Note that data to be processed by the information processing device is, for example, continuous data output from an IoT device. In the following, an example will be described in which the data is a video (continuous frames) captured by a surveillance camera or the like.

[0014] Among the frames that make up a video, a characteristic is that an event that occurs in a certain frame is likely to also occur in the adjacent consecutive frames. The inventors focused on this characteristic and configured an information processing device to perform inference for each frame as follows.

[0015] The information processing device first performs inference while thinning out input data (consecutive frames). The inference here is, for example, detection of an object captured in a frame (see FIG. 2). Then, if the information processing device determines that a predetermined event has occurred as a result of the inference for a frame (if it determines that an event has occurred), it performs inference for the frames from the next frame onward.

[0016] For example, if the information processing device determines that an object with a predetermined attribute (class) is captured in a predetermined area of ​​frame 201 based on the inference result (reference numeral 202) for frame 201 shown in Fig. 2, it determines that an event has occurred and executes inference for the frames consecutively from the next frame onwards. On the other hand, if the information processing device determines that a predetermined event has not occurred as a result of the inference for the frame (determines that no event has occurred), it continues to execute inference by thinning out frames in the same manner as before.

[0017] That is, the information processing device determines whether to thin out inferences for a frame based on whether a predetermined event occurs in the frame, thereby reducing the frequency of inferences for the frame and preventing missed detections.

[0018] A specific example of processing executed by an information processing device will be described with reference to Figure 3. As shown in Figure 3, the information processing device performs inference by thinning out consecutive frames (for example, performing inference every five frames), and if the inference result shows that no event is detected from the frames, the inference is skipped until the next inference timing. On the other hand, if the inference result shows that an event is detected from the frames, the information processing device performs inference on consecutive frames. Then, if the inference result shows that no event is detected, the information processing device thins out frames again and performs inference.

[0019] The information processing device may infer frames consecutively, and when an event is no longer detected in even one frame, it may resume the inference process by thinning out frames, or when an event is no longer detected in multiple consecutive frames (e.g., two frames), it may resume the inference process by thinning out frames.

[0020] [Configuration Example] Next, a configuration example of the information processing device 10 will be described with reference to Fig. 4. The information processing device 10 includes, for example, an input / output unit 11, a communication unit 12, a storage unit 13, and a control unit 14.

[0021] The input / output unit 11 is an interface that controls input and output of various data. The input / output unit 11 receives, for example, input of various settings to the information processing device 10.

[0022] The communication unit 12 is a communication interface for inputting and outputting various types of data via a network. The communication unit 12 receives, for example, data (such as consecutive frames) output from an IoT device via the network.

[0023] The storage unit 13 stores data, programs, etc. that are referenced when the control unit 14 executes various processes. The storage unit 13 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0024] The storage unit 13 stores various information such as parameters of a deep learning model used when the inference unit 142 (described later) performs inference on data. For example, the storage unit 13 stores setting information such as how many frames to thin out when the inference unit 142 thins out frames to perform inference (thinning amount), how many consecutive frames of no event detection are required before the inference unit 142 thins out frames again to perform inference (number of no-detection frames), and so on.

[0025] The storage unit 13 also stores definition information of events to be detected. The definition information of an event is, for example, information that defines the number of detected target objects in a frame, the detection size, and the degree of certainty required to determine that an event has occurred. The above setting information, definition information of events to be detected, etc. can be changed as appropriate by the user of the information processing device 10, etc.

[0026] The control unit 14 is responsible for overall control of the information processing device 10. The functions of the control unit 14 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the storage unit 13.

[0027] The control unit 14 includes, for example, a data input unit 141, an inference unit 142, and an output processing unit 143. The data input unit 141 receives input of continuous data. For example, the data input unit 141 receives input of data (continuous frames) captured by a surveillance camera via the communication unit 12.

[0028] The inference unit 142 thins out and performs inference on the continuous data received by the data input unit 141. When the inference unit 142 detects an event through inference on the data, it performs inference on the continuous data starting from the data where the event was detected.

[0029] For example, the inference unit 142 performs inference by thinning out consecutive frames received by the data input unit 141 at predetermined intervals. If the inference unit 142 detects an event indicated in the event definition information as a result of the inference for a frame, it continues to perform inference for subsequent frames. If the inference unit 142 determines that the event is no longer detected as a result of the inference for a frame, it stops inference for consecutive frames and performs inference again by thinning out the frames.

[0030] For example, the inference unit 142 performs an event determination (event presence / absence) for the previous frame using the inference result for the previous frame, and then assigns the determination result to the next frame as metadata (inference necessity flag).Then, if the inference necessity flag for the frame is "necessary," the inference unit 142 performs inference for the frame, and if the inference necessity flag is "no," the inference unit 142 does not perform inference for the frame.

[0031] The frame inference performed by the inference unit 142 includes, for example, identifying a BBOX (position information) of an object captured in the frame, identifying the class of the object in the BBOX (what the object is), calculating the confidence level of the BBOX, etc. For frame inference, for example, a deep learning model for detecting objects from an image is used.

[0032] The output processing unit 143 outputs the inference results for the frames by the inference unit 142. For example, the output processing unit 143 assigns meta-information indicating the inference results to each piece of data (frame) output from the inference unit 142 and outputs the data.

[0033] [Example of Processing Procedure] Next, an example of processing procedure executed by the information processing device 10 will be described with reference to FIG. 5. For example, when the data input unit 141 of the information processing device 10 receives input of continuous data (S1), the inference unit 142 performs inference while thinning out the data (S2). Then, if the inference unit 142 determines that an event has occurred based on the inference result of the data (Yes in S3), it performs inference continuously on the next data and thereafter (S4). Thereafter, it returns to S3. On the other hand, if the inference unit 142 determines that no event has occurred based on the inference result of the data (No in S3), it returns to S2 and performs inference while thinning out the next data and thereafter (S2).

[0034] By the information processing device 10 performing the above process, it is possible to reduce the frequency of inference for data (continuous data) that is the subject of inference, and to prevent missed detection.

[0035] When the information processing device 10 determines that an event has occurred as a result of inference on a frame, there is a possibility that the event has also occurred in a past frame for which inference was skipped. Therefore, when the information processing device 10 detects an event by inferring data, it may perform inference retroactively on the skipped past data (frames).

[0036] In this case, the inference unit 142 performs inference by tracing back to past frames starting from the frame in which the event is detected, as shown in Fig. 6. For example, the inference unit 142 performs inference on past frames for which inference has not yet been performed, starting from the frame immediately before the frame in which the event is detected.

[0037] In this way, the information processing device 10 can detect events without omission for frames to be processed.

[0038] When performing frame inference by going back to the past as described above, the inference unit 142 may perform batch processing on multiple past frames (e.g., four frames) for which inference has not yet been performed at once. In this way, the inference unit 142 can efficiently perform inference on past frames.

[0039] The inference unit 142 may perform 1. inference for frames beyond the frame in which the event was detected (inference for frames in the future direction) and 2. inference for frames beyond the frame in which the event was detected (inference for frames in the past direction) in the order of 1 → 2, or 2 → 1.

[0040] [Example of application to video file] The information processing device 10 may perform the above processing on frames sequentially output from an IoT device, or may perform the above processing on a set of frames for a predetermined period of time (for example, a video file). An example of the case where the information processing device 10 performs processing on a video file will be described with reference to FIG. 7 .

[0041] For example, when the information processing device 10 executes processing on a video file, it is possible to check from any frame in the video. Therefore, for example, as shown in Fig. 7, the inference unit 142 periodically checks future frames in the video file (detecting an event by thinning out frames at predetermined intervals starting from the first frame and performing inference) ((1)), and when an event is detected, it performs event checks continuously from the frame where the event was detected to future frames ((2)).

[0042] After that, when the inference unit 142 has finished checking up to the first frame on the future side, it then performs event checks on frames on the past side in succession ((3)).

[0043] In this way, the inference unit 142 can detect events in order, starting with the most recent event in a long video file, for example.

[0044] In the above (2) and (3), the inference unit 142 may determine non-detection of an event, for example, using the results of inference for multiple frames ((4) Non-detection determination based on results for multiple frames). For example, if the inference unit 142 does not detect an event in two consecutive frames, the event determination may be "no" and consecutive event checks may be stopped.

[0045] The amount of frames that the inference unit 142 will thin out to make an inference (thinning amount) and the number of consecutive frames after which an event is no longer detected to make an inference (number of undetected frames) may be fixed values ​​or may be dynamically changed values.

[0046] For example, if the input data is a video captured by an on-board camera, the inference unit 142 sequentially acquires information about the speed of the vehicle on which the on-board camera is mounted. If the speed of the vehicle increases, the inference unit 142 may reduce the amount of thinning in accordance with the speed.

[0047] By doing this, even if the speed of a passenger vehicle equipped with an on-board camera increases and the changes in each frame of the video become greater than before, the inference unit 142 infers frames at shorter intervals accordingly, thereby reducing missed detections.

[0048] Furthermore, when the speed of the passenger vehicle decreases, the inference unit 142 increases the amount of frame thinning in accordance with the speed. By doing so, when the change in each frame of the video becomes smaller than before due to a decrease in the speed of the passenger vehicle equipped with an on-board camera, the frequency of frame inference performed by the inference unit 142 can be reduced. As a result, the inference unit 142 can perform frame inference by effectively utilizing limited computational resources.

[0049] Furthermore, the amount of thinning and the number of non-detection frames may be dynamically changed based on the confidence level of the BBOX included in the inference result of the frame.

[0050] For example, if the result of frame inference by the inference unit 142 is that the certainty of the BBOX is just below the threshold for event determination (for example, the frame appears to show something that looks like a person, but the certainty is not very high), the amount of frame thinning may be reduced or the number of undetected frames may be increased.

[0051] According to the information processing device 10 described above, there is no degradation in inference accuracy of frames due to a low-precision model in the first layer, which occurs in the two-layer inference system described above. Furthermore, the information processing device 10 performs inference by thinning out frames when no event has occurred, thereby reducing the frequency of inference.

[0052] Furthermore, the information processing device 10 does not require a model equivalent to the first-layer model in a two-layer inference system. Therefore, the information processing device 10 does not require the implementation or tuning of a model equivalent to the first-layer model in a two-layer inference system. Furthermore, if events occur frequently in each frame, a two-layer inference system uses first-layer resources plus second-layer resources, but the information processing device 10 only needs resources equivalent to the second-layer resources. Therefore, the information processing device 10 can reduce the resources required for frame inference compared to conventional devices.

[0053] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0054] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0055] [Program] The information processing device 10 can be implemented by installing a program (information processing program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the information processing device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0056] 8 is a diagram showing an example of a computer that executes an information processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0057] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0058] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the information processing device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the information processing device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0059] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0060] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0061] REFERENCE SIGNS LIST 10 Information processing device 11 Input / output unit 12 Communication unit 13 Storage unit 14 Control unit 141 Data input unit 142 Inference unit 143 Output processing unit

Claims

1. An information processing device comprising: a data input unit that accepts input of consecutive data; an inference unit that thins out the consecutive data and performs inference on the data, and performs inference on consecutive data starting from data at which an event is detected by the inference; and an output processing unit that outputs inference results based on the inference on the data.

2. The information processing device according to claim 1, characterized in that the inference unit performs inference on the continuous data until the event is no longer detected.

3. The information processing device according to claim 1, characterized in that the inference unit takes data at which an event has been detected by the inference as a starting point and performs inference on consecutive data from the data and on data older than the data at which the inference has not been performed.

4. The information processing device according to claim 3, characterized in that the inference unit performs inference by batch processing on a plurality of data items that are older than the data and for which the inference has not been performed.

5. The information processing device according to claim 1, wherein the data is frames constituting a video.

6. An information processing method executed by an information processing device, comprising the steps of: accepting input of consecutive data; thinning out the consecutive data and performing inference on the data, and performing inference on consecutive data starting from data at which an event is detected by the inference; and outputting an inference result based on the inference on the data.

7. An information processing program for causing a computer to execute the steps of: accepting input of consecutive data; thinning out the consecutive data and performing inference on the data, and performing inference on the consecutive data starting from data at which an event is detected by the inference; and outputting an inference result based on the inference on the data.

Images