AI inference system for images

Abstract

This AI inference system comprises: an encoder circuit (101) which encodes an image captured by a camera (100); a decoder circuit (102) which decodes an image; a first inference processing unit (103) which executes first inference processing for the decoded image; a second inference processing unit (104) which executes second inference processing, which is more accurate than the first inference, only for a frame of an image of which a value of confidence for the predicted class of the first inference exceeds a threshold value; and a frame rate control unit (105) which reduces a frame rate of the encoder circuit (101) and the decoder circuit (102) from an initial value to a low rate value when the value of confidence for the predicted class is equal to or smaller than the threshold value.

Description

[Technical field]

[0001] The present invention relates to an event-driven AI inference system that performs inference on images captured by a camera only when an event occurs. [Background technology]

[0002] It is possible to apply AI (Artificial Intelligence) inference to the video captured by the camera. It is known that the amount of calculation and memory required for AI inference increases with the accuracy of the inference. For this reason, a method is generally used in which encoded video is sent from a camera or other device and decoded on a server before inference is performed.

[0003] However, there are few use cases where inference is required for all frames of video captured by a camera. The reason is that, for example, in the case of a security camera, inference is only required when a person is captured in the frame. In other words, inference when no person is captured in the frame is a waste of cost. Therefore, event-driven inference has been proposed, which performs inference only when an event occurs (Non-Patent Document 1).

[0004] Event-driven inference applies lightweight, low-cost inference to decoded frames and then decides whether more accurate inference is needed. It is common to use the predicted class (hereafter referred to as reliability) and the higher the reliability, the more accurate the inference can be. Reliability is a value that indicates the probability with which a Neural Network (NN) model judges its own output to be correct (Non-Patent Document 2).

[0005] For example, in the security camera example mentioned above, as shown in Fig. 6, image frames captured by the camera 100 and encoded by an encoder circuit 101 are decoded by, for example, a decoder circuit 102 on the server side. Then, a first inference processing unit 103 executes human detection inference on the decoded image frames, and only for frames with a certain level of reliability or higher, a second inference processing unit 104 executes face recognition inference to determine whether or not the person is suspicious. Such two-stage inference is called multi-layer inference.

[0006] Event-driven multi-layer inference is a technology that uses a low-cost model for the first-layer inference to reduce the time required to perform high-precision inference and reduce total costs. With event-driven multi-layer inference, the second-layer inference may not be performed as a result of the first-layer inference, but the streaming from the camera is always running, and there was an issue that the encoding and decoding costs could not be reduced. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Takeharu Eda et al., "Highly efficient event-driven inference to support AI services in the IOWN era," NTT Technical Journal, December 2020 issue [Non-Patent Document 2] “person-detection-retail-0002”, Intel, 2018,<https: / / docs.openvinotoolkit.org / 2018_R5 / person-detection-retail-0002.html> Summary of the Invention [Problem to be solved by the invention]

[0008] The present invention has been made to solve the above-mentioned problems, and aims to provide an AI inference system that can reduce power consumption compared to conventional systems. [Means for solving the problem]

[0009] The AI ​​inference system of the present invention is characterized in that it comprises an encoder circuit configured to encode an image captured by a camera, a decoder circuit configured to decode the image encoded by the encoder circuit, a first inference processing unit configured to execute a first inference process for each frame of the image decoded by the decoder circuit, a second inference processing unit configured to receive a reliability value of the first inference from the first inference processing unit, and to execute a second inference process having higher accuracy than the first inference only for frames of an image for which the reliability value exceeds a predetermined threshold, and a frame rate control unit configured to reduce the frame rates of the encoder circuit and the decoder circuit from their initial values ​​to a low-speed value when the reliability value is equal to or lower than the threshold.

[0010] In addition, in one configuration example of the AI ​​inference system of the present invention, the encoder circuit and the decoder circuit are provided for each camera, and the frame rate control unit reduces the frame rates of the encoder circuit and the decoder circuit corresponding to a camera among the multiple cameras that outputs an image whose reliability value is below the threshold value from an initial value to a low-speed value. In addition, in one configuration example of the AI ​​inference system of the present invention, the encoder circuit and the decoder circuit are provided for each camera, and the frame rate control unit is characterized in that when at least one first camera among the multiple cameras outputs an image whose reliability value exceeds the threshold, the frame rates of the encoder circuit and the decoder circuit corresponding to a second camera that is in a predetermined positional relationship with the first camera are kept at their initial values, and when a third camera that is not in a predetermined positional relationship with the first camera outputs an image whose reliability value is equal to or lower than the threshold, the frame rates of the encoder circuit and the decoder circuit corresponding to the third camera are reduced to the low-speed value. Effect of the Invention

[0011] According to the present invention, when the reliability value of the first inference is below a threshold value, the frame rates of the encoder circuit and the decoder circuit are reduced from their initial values ​​to a low-speed value, thereby reducing power consumption compared to conventional event-driven multi-layer inference. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing a configuration of an AI inference system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flowchart explaining the operation of the encoder circuit, the decoder circuit, the first inference processing unit, and the second inference processing unit of the AI ​​inference system according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart illustrating the operation of the frame rate control unit of the AI ​​inference system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram showing another configuration of an AI inference system according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a block diagram showing an example of the configuration of a computer that realizes an AI inference system according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram for explaining event-driven multi-layer reasoning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an AI inference system according to an embodiment of the present invention. The AI ​​inference system includes a surveillance camera 100, a decoder circuit 102 for decoding an image encoded by an encoder circuit 101 of the surveillance camera 100, a first inference processing unit 103 for executing an inference process for detecting a person on a frame of the decoded image, a second inference processing unit 104 for receiving a reliability value of the inference by the first inference processing unit 103 from the first inference processing unit 103 and executing a face recognition inference process for a person only on an image frame whose reliability value exceeds a predetermined threshold, and a frame rate control unit 105 for lowering the frame rates of the encoder circuit 101 and the decoder circuit 102 when the reliability value output by the first inference processing unit 103 is equal to or less than the threshold.

[0014] The decoder circuit 102, the first inference processing unit 103, the second inference processing unit 104, and the frame rate control unit 105 are provided in a location away from the surveillance camera 100, for example, inside a server device.

[0015] FIG. 2 is a flowchart for explaining the operations of the encoder circuit 101, the decoder circuit 102, the first inference processing unit 103, and the second inference processing unit 104, and FIG.

[0016] The encoder circuit 101 encodes an image captured by the surveillance camera 100 (Step S100 in FIG. 2). The decoder circuit 102 decodes the image encoded by the encoder circuit 101 (step S101 in FIG. 2).

[0017] The first inference processing unit 103 executes inference processing of human detection for each frame of the decoded image (Step S102 in FIG. 2). When the reliability value output from the first inference processing unit 103 exceeds a predetermined threshold (YES in step S103 in Figure 2), the second inference processing unit 104 performs face recognition inference processing of people present in the frames whose reliability values ​​exceed the threshold (step S104 in Figure 2).

[0018] The encoder circuit 101, the decoder circuit 102, the first inference processing unit 103, and the second inference processing unit 104 execute the processes of steps S100 to S104 for each frame.

[0019] On the other hand, when the reliability value output from the first inference processing unit 103 is equal to or lower than the above threshold value (NO in step S200 of FIG. 3), the frame rate control unit 105 temporarily stops the stream output from the encoder circuit 101 and the decoder circuit 102 (step S201 of FIG. 3).

[0020] Next, the frame rate control unit 105 accesses the clock generation circuits of the encoder circuit 101 and the decoder circuit 102, and rewrites the clock frequency values ​​set in the registers of the clock generation circuits, thereby reducing the frame rates of the encoder circuit 101 and the decoder circuit 102 from their initial values ​​to a predetermined low-speed value (step S202 in FIG. 3).

[0021] Then, the frame rate control unit 105 resets the encoder circuit 101 and the decoder circuit 102 to restart the stream output (step S203 in FIG. 3). In this way, encoding and decoding are performed at a low frame rate from the frame next to the frame whose reliability value is equal to or less than the threshold.

[0022] In addition, when the reliability value output from the first inference processing unit 103 exceeds a threshold value (YES in step S200), if the frame rates of the encoder circuit 101 and the decoder circuit 102 are set to low values ​​(YES in step S204 in Figure 3), the frame rate control unit 105 temporarily stops the stream output from the encoder circuit 101 and the decoder circuit 102 (step S205 in Figure 3).

[0023] Next, the frame rate control unit 105 accesses the clock generation circuits of the encoder circuit 101 and the decoder circuit 102, and rewrites the values ​​of the clock frequencies set in the registers of the clock generation circuits, thereby returning the frame rates of the encoder circuit 101 and the decoder circuit 102 from the low-speed values ​​to their initial values ​​(step S206 in FIG. 3).

[0024] Then, the frame rate control unit 105 resets the encoder circuit 101 and the decoder circuit 102 to restart the stream output (step S207 in FIG. 3). In this way, when the reliability value becomes equal to or less than the threshold and then exceeds the threshold, encoding and decoding are performed at the normal frame rate from the frame next to the frame whose reliability value became equal to or less than the threshold. The frame rate control unit 105 executes the process of steps S200 to S207 for each frame.

[0025] As described above, in this embodiment, when the reliability value of the inference by the first inference processing unit 103 is below a threshold value, high-precision inference by the second inference processing unit 104 is stopped and the frame rates of the encoder circuit 101 and the decoder circuit 102 are reduced, thereby making it possible to further reduce power consumption compared to conventional event-driven multi-layer inference.

[0026] The frame rate of the surveillance camera 100 is at most 60 FPS (frames per second). Therefore, there is an interval of about 16.6 milliseconds between frames. Therefore, if the processing of steps S200 to S207 is completed within this interval, no frames will be dropped. Lowering the frame rate reduces the cost required for data movement between the encoder circuit 101 and the decoder circuit 102 (e.g., DRAM (Dynamic Random Access Memory) access), and therefore reduces power consumption.

[0027] Also, as shown in the example of Fig. 4, in a use case where a plurality of surveillance cameras 100-1 to 100-4 are connected to an AI inference system, the above processes of Fig. 2 and Fig. 3 may be performed for each of the surveillance cameras 100-1 to 100-4, and the frame rates of the encoder circuits 101-1 to 101-4 and the decoder circuits 102-1 to 102-4 may be changed individually for each of the surveillance cameras 100-1 to 100-4. In the example of Fig. 4, the encoder circuits 101-1 to 101-4 and the decoder circuits 102-1 to 102-4 are provided for each of the surveillance cameras 100-1 to 100-4.

[0028] 4, when the reliability value of inference for an image of at least one of the surveillance cameras 100-1 to 100-4 exceeds a threshold, the frame rate control unit 105 not only keeps the frame rates of the encoder circuit 101 and the decoder circuit 102 corresponding to the camera that outputs an image whose reliability value exceeds the threshold at their initial values, but also keeps the frame rates of the encoder circuit 101 and the decoder circuit 102 corresponding to a camera that is in a predetermined positional relationship with this camera at their initial values. Furthermore, the frame rate control unit 105 sets the frame rates of the corresponding encoder circuit 101 and the decoder circuit 102 to a low value for a camera that is not in a predetermined positional relationship with the camera that outputs an image whose reliability value exceeds the threshold and that outputs an image whose reliability value is equal to or less than the threshold.

[0029] 4, for example, it is assumed that the reliability of inference for the image output from the surveillance camera 100-2 that is closest to the person 200 becomes the highest value. At this time, the frame rate control unit 105 keeps the frame rates of the encoder circuit 101-2 and the decoder circuit 102-2 corresponding to the surveillance camera 100-2 that output the image whose reliability value exceeds the threshold value at their initial values.

[0030] Furthermore, for surveillance cameras 100-1, 100-3 installed within a predetermined distance from surveillance camera 100-2, frame rate control unit 105 keeps the frame rates of the corresponding encoder circuits 101-1, 101-3 and decoder circuits 102-1, 102-3 at their initial values, regardless of the reliability of the inference.

[0031] The frame rate control unit 105 also reduces the frame rates of the encoder circuit 101-4 and the decoder circuit 102-4 corresponding to the surveillance camera 100-4 that is installed outside a predetermined distance range from the surveillance camera 100-2 and that outputs an image whose reliability value is equal to or less than a threshold value. The frame rate control unit 105 also keeps the frame rates of the corresponding encoder circuit 101 and decoder circuit 102 at their initial values ​​for a surveillance camera that outputs an image whose reliability value exceeds a threshold value, even if the surveillance camera is installed outside a predetermined distance range from the surveillance camera 100-2.

[0032] Information indicating the positional relationship between the surveillance cameras 100-1 to 100-4 is set in the frame rate control unit 105 in advance. In this way, by changing the frame rate according to the relative positions of the surveillance cameras 100-1 to 100-4, it becomes possible to optimize the power consumption of the entire AI inference system.

[0033] The first inference processing unit 103, the second inference processing unit 104, and the frame rate control unit 105 described in this embodiment are a central processing unit (CPU), a storage device, and an interface with the outside. This can be realized by a computer equipped with an interface and a program that controls these hardware resources. An example of the computer configuration is shown in Figure 5.

[0034] The computer includes a CPU 300, a storage device 301, and an interface device (I / F) 302. Encoder circuits 101-1 to 101-4, a decoder circuit 102, etc. are connected to the I / F 302. In such a computer, a program for implementing the method of the present invention is stored in the storage device 301. The CPU 300 executes the processes described in this embodiment in accordance with the program stored in the storage device 301. [Industrial Applicability]

[0035] The present invention can be applied to technology for improving the efficiency of AI inference. [Explanation of symbols]

[0036] 100, 100-1 to 100-4... surveillance cameras, 101, 101-1 to 101-4... encoder circuits, 102, 102-1 to 102-4... decoder circuits, 103... first inference processing unit, 104... second inference processing unit, 105... frame rate control unit.

Claims

1. an encoder circuit configured to encode an image captured by the camera; a decoder circuit configured to decode an image encoded by the encoder circuit; a first inference processing unit configured to execute a first inference process for each frame of an image decoded by the decoder circuit; a second inference processing unit configured to receive a reliability value of the first inference from the first inference processing unit, and to execute a second inference process having higher accuracy than the first inference only for image frames for which the reliability value exceeds a predetermined threshold; and a frame rate control unit configured to reduce the frame rates of the encoder circuit and the decoder circuit from an initial value to a lower value when the reliability value is equal to or less than the threshold.

2. 2. The AI ​​inference system according to claim 1, the encoder circuit and the decoder circuit are provided for each camera, The frame rate control unit reduces the frame rates of the encoder circuit and the decoder circuit corresponding to a camera among multiple cameras that outputs an image whose reliability value is below the threshold from an initial value to a low-speed value.

3. 2. The AI ​​inference system according to claim 1, the encoder circuit and the decoder circuit are provided for each camera, The frame rate control unit, when at least one first camera among the multiple cameras outputs an image whose reliability value exceeds the threshold, keeps the frame rates of the encoder circuit and the decoder circuit corresponding to a second camera in a predetermined positional relationship with the first camera at their initial values, and when a third camera that is not in a predetermined positional relationship with the first camera outputs an image whose reliability value is equal to or lower than the threshold, reduces the frame rates of the encoder circuit and the decoder circuit corresponding to the third camera to the low-speed value.

Images