Inference method, inference system, and inference program

Abstract

To track a subject by efficient interference by two layers of an edge and a cloud.SOLUTION: In an inference system 1, which has an edge and a server 10, an acquisition unit 15b acquires information pertaining to movement of a prescribed subject imaged using a first camera 20, among a plurality of cameras 20 of the edge, at least at a first time. An estimation unit 15c estimates, on the basis of the movement destination of the prescribed subject, a second camera 20 by which the prescribed subject is imaged at a second time that is after the first time.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an inference method, an inference system, and an inference program. [Background technology]

[0002] Recently, real-time applications using DNNs (Deep Neural Networks), such as video surveillance, voice assistants, and autonomous driving, have been emerging. These real-time applications require the ability to process a large number of queries in real time with limited resources while maintaining the accuracy of the DNN. To address this issue, a technology called model cascade has been proposed that uses a fast, low-accuracy lightweight model and a slow, high-accuracy high-accuracy model to speed up inference processing with little degradation in accuracy.

[0003] In model cascade, multiple models including lightweight models and high-precision models are used. When performing inference using model cascade, estimation is first performed using a lightweight model, and if the result is reliable, the result is adopted and the process is terminated. On the other hand, if the inference result of the lightweight model is unreliable, inference is subsequently performed using a high-precision model, and the result is adopted. For example, IDK Cascade (see, for example, Non-Patent Document 1) is known, which introduces an IDK (I Don't Know) classifier to determine whether the inference result of the lightweight model is reliable. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Wang, Xin, et al., “Idk cascades: Fast deep learning by learning not to overthink”, arXiv preprint arXiv:1706.00885 (2017). Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in Non-Patent Document 1 does not consider inference, and since the idea for learning is derived from fine tuning, it is premised on learning for the same purpose.In addition, since it is premised on processing all of the data from a large number of sensors, there are still issues to be addressed regarding the amount of transmission from the sensor to the edge and from the edge to the cloud, and the total amount of calculations at the edge and the cloud.

[0006] The present invention has been made in consideration of the above, and has an object to track a subject by efficiently performing inference at two layers, that is, an edge and a cloud. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objective, the inference method of the present invention is an inference method executed by an inference system having an edge and a server, and is characterized in that it includes an acquisition step of acquiring information on the movement of a specified subject captured by a first camera among the cameras of the edge at at least a first time, and an estimation step of estimating from the acquired image a second camera that will capture the specified subject at a second time that is in the future than the first time, based on the destination of the specified subject. Effect of the Invention

[0008] According to the present invention, it is possible to track a subject by efficiently performing inference at two layers, the edge and the cloud. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an overview of the inference system of this embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating a schematic configuration of the inference system of this embodiment. [Diagram 3] FIG. 3 is a diagram for explaining the process of the estimation unit. [Figure 4]FIG. 4 is a flowchart showing the procedure of the presentation process. [Diagram 5] FIG. 5 is a diagram for explaining the effect of the processing. [Figure 6] FIG. 6 is a diagram for explaining the effect of the processing. [Figure 7] FIG. 7 is a diagram illustrating an example of a computer that executes a presentation program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are indicated by the same reference numerals.

[0011] [Outline of the inference system] 1 is a diagram for explaining an overview of the inference system of this embodiment. The inference system 1 of this embodiment has a camera 20 as an example of an IoT device on the edge side, and a server 10 on the cloud side. In this inference system 1, the server 10 estimates the camera 20 that satisfies predetermined requirements, and performs a desired inference using images and information acquired from the estimated camera 20.

[0012] For example, as shown in Fig. 1, the inference system 1 is applied to tracking of a person. That is, the server 10 collates and identifies the person to be tracked. The server 10 also estimates the cameras 20 capable of capturing the identified person based on the positions of all the cameras, narrows down the number of cameras 20, and instructs tracking to the estimated cameras 20.

[0013] The camera 20, estimated by the server 10, detects the person specified from the image being captured and performs tracking to estimate the moving direction. Also, if the person deviates from the range that the camera 20 can capture, the camera 20 notifies the server 10 and leaves the tracking of the person to the server 10.

[0014] Specifically, as shown in FIG. 1, the server 10 collates and identifies the person to be tracked, estimates the camera 20 capable of capturing the identified person as a specified subject, and instructs only the estimated camera 20 (20A) to track the identified subject.

[0015] The camera 20 (20A) capturing the subject detects the subject, calculates its speed, and estimates its direction of travel. Furthermore, the camera 20 (20A) notifies the server 10 when the subject deviates from the range in which it can capture images. The server 10 estimates the next candidate camera 20 capable of capturing images based on the subject's speed and direction of travel notified from the camera 20 (20A), the analysis results of people flow such as movement patterns, and the position of the camera 20, and instructs the estimated camera 20 (20B) to detect and track the subject.

[0016] This enables the inference system 1 to efficiently track a desired person. In this way, the inference system reduces the amount of data transmission and the total amount of calculation in the inference process in two layers, the edge and the cloud, and makes it possible to efficiently perform the desired inference and track the subject.

[0017] It should be noted that the processing target of the inference system 1 is not limited to video. For example, it is also possible to estimate and track the position and direction of a sound source by processing an audio signal.

[0018] [Inference system configuration] 2 is a schematic diagram illustrating a schematic configuration of an inference system according to the present embodiment. As illustrated in FIG. 2, the inference system 1 according to the present embodiment includes a camera 20 as an example of an IoT device on the edge side, and a server 10 on the cloud side.

[0019] [Camera configuration] The camera 20 has an imaging section 22, a communication control section 23, a storage section 24, and a control section 25. The imaging section 22 constantly captures an image within the imaging range of the camera 20 itself to obtain an image.

[0020] The communication control unit 23 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 25 and an external device via an electric communication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 23 controls communication between the server 10 or the like and the control unit 25.

[0021] The storage unit 24 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 24 stores in advance a processing program for operating the camera 20, data used during execution of the processing program, and the like, or temporarily stores the data each time processing is performed. In this embodiment, the storage unit 24 stores, for example, a model for classifying images, which is used in processing by the detection unit 25a described later.

[0022] The control unit 25 is realized by a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), etc., and executes processing programs stored in memory to function as a detection unit 25a and a tracking unit 25b.

[0023] The detection unit 25a detects a person in the image captured by the imaging unit 22, and assigns a rectangle ID to a rectangle including the person. Furthermore, the detection unit 25a transmits an image obtained by cropping the rectangle to the server 10 when instructed by the server 10, which will be described later.

[0024] When instructed by the server 10 to track the identified person, the tracking unit 25b calculates the speed of movement of the identified person and estimates the moving direction. For example, when the server 10 notifies the tracking unit 25b of the rectangle ID of the identified person, the tracking unit 25b tracks the trajectory of the rectangle including the person, thereby calculating the speed of the person and estimating the moving direction of the person.

[0025] Then, the tracking unit 25b transmits the camera ID, the rectangle ID, the speed, and the estimated traveling direction to the server 10 via the communication control unit 23 at the timing when the person leaves the shooting range of the camera 20. Note that instead of the rectangle ID, a rectangle including the person may be identified using the coordinates of a BBOX or the like and transmitted to the server 10.

[0026] [Server configuration] The server 10 is realized by a general-purpose computer such as a personal computer, and includes a communication control unit 13, a storage unit 14, and a control unit 15.

[0027] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and an external device via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the camera 20 and the control unit 15.

[0028] The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance the processing program that operates the server 10 and data used during the execution of the processing program, or stores the data temporarily each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

[0029] In this embodiment, the storage unit 14 stores the positions of the cameras 20, models used in the processing by the estimation unit 15c described below, for classifying images and analyzing people flow, and the like.

[0030] The control unit 15 is realized using a CPU (Central Processing Unit) or the like, and executes a processing program stored in a memory. As a result, the control unit 15 functions as an identifying unit 15a, an acquiring unit 15b, and an estimating unit 15c, as exemplified in FIG. 2. Note that each of these functional units may be implemented in different hardware, or a part of each may be implemented in different hardware. The control unit 15 may also include other functional units.

[0031] The identification unit 15a identifies a person to be tracked. Specifically, the identification unit 15a identifies a person to be tracked who satisfies a predetermined condition from the image of the camera 20. For example, the identification unit 15a identifies the person to be tracked by matching the person in the image of the camera 20 with a query image of the person to be tracked.

[0032] The estimation unit 15c, which will be described later, estimates the camera 20 capturing the image of the identified person and instructs the camera 20 to start tracking processing. The estimation unit 15c also instructs the other cameras 20 to stop tracking processing.

[0033] The acquisition unit 15b acquires information on the movement of a specific subject captured by a first camera 20 among the cameras 20 of the edge at least at a first time. Specifically, when the acquisition unit 15b captures a person identified by the identification unit 15a, the acquisition unit 15b acquires the speed and estimated traveling direction of the person from the camera 20 estimated by the estimation unit 15c. The acquisition unit 15b also acquires an image of the person to be tracked. For example, the acquisition unit 15b acquires information on the movement when the subject to be tracked deviates from the capture range of the camera 20 that is tracking the subject.

[0034] The estimation unit 15c estimates the second camera 20 that will capture the predetermined subject at a second time that is future than the first time from the acquired movement information, based on the destination of the predetermined subject. Specifically, when the predetermined subject deviates from the capture range of the first camera 20, the estimation unit 15c estimates the second camera 20 based on the estimated destination of the predetermined subject. That is, when the estimation unit 15c is notified by the camera 20 that is tracking the predetermined subject that the subject to be tracked has deviated from the capture range of the camera, the estimation unit 15c estimates the destination of the subject and estimates the second camera 20 that can capture the subject at the destination.

[0035] Here, FIG. 3 is a diagram for explaining the processing of the estimation unit. For example, the estimation unit 15c estimates the second camera 20 using the direction and speed of movement of a predetermined subject as the movement information. Specifically, as shown in FIG. 3(a), the estimation unit 15c estimates the camera 20 that can next capture the subject based on the speed and moving direction of the subject acquired from the first camera 20. In the example shown in FIG. 3(a), the estimation unit 15c grasps that the person to be tracked is moving 80 m / min to the right from the capture range of camera A, and estimates cameras A, B, and C as candidates for the second camera 20 that can capture the subject in the next 5 minutes, for example.

[0036] Alternatively, the estimation unit 15c estimates the second camera using a probability distribution of a movement pattern of a predetermined subject, as shown in FIG. 3(b), in addition to or instead of the movement information acquired by the acquisition unit 15b. For example, the estimation unit 15c estimates the destination of the subject by analyzing the movement pattern of the subject between the accumulated cameras 20. That is, the estimation unit 15c estimates the camera C as the second camera 20 by analyzing a movement pattern in which, for example, 90% of people who pass through the shooting range of the camera A move toward the camera C in FIG. 3(b). At that time, the estimation unit 15c estimates the second camera 20 by estimating the destination of the subject using, for example, a model obtained as a result of the analysis.

[0037] When the tracking process starts, if it is not known which camera 20 the tracking target person is in the shooting range of, the acquisition unit 15b acquires the video of the camera 20, the identification unit 15a collates the tracking target person, and the estimation unit 15c estimates the camera 20 that can capture the estimated destination. Then, the estimation unit 15c instructs the estimated camera 20 to start the tracking process and instructs the other cameras 20 to stop the tracking process.

[0038] In this way, by narrowing down the necessary processing targets instead of always targeting all cameras 20, the inference system 1 can reduce the amount of data transmission and total amount of calculation for inference processing in two layers of edge and cloud, making it possible to efficiently perform desired inferences, such as tracking a desired person or other subject.

[0039] [Inference processing] Next, the inference process by the inference system 1 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the procedure of the inference process. The flowchart in Fig. 4 starts, for example, when the user performs an operation input to instruct the start of the process.

[0040] In the server 10, the acquisition unit 15b first acquires images from all cameras 20, the identification unit 15a collates the person to be tracked, and the estimation unit 15c estimates the camera 20 capable of capturing the estimated destination. Then, the estimation unit 15c instructs the estimated camera 20 to start tracking processing and instructs the other cameras 20 to stop tracking processing.

[0041] Then, the acquisition unit 15b acquires information on the movement of the person to be tracked (step S1). For example, when the subject to be tracked deviates from the shooting range of the camera 20 that is tracking the subject, the acquisition unit 15b acquires the speed and estimated traveling direction of the subject from the camera 20. At that time, the acquisition unit 15b acquires from the camera 20 a camera ID that identifies the camera and a rectangle ID of the subject.

[0042] Next, the estimation unit 15c estimates the destination of the subject (step S2). For example, the estimation unit 15c estimates the destination of the subject using the direction and speed of movement of the specified subject. Alternatively, the estimation unit 15c estimates the destination of the subject using a probability distribution of the movement pattern of the specified subject.

[0043] Then, the estimation unit 15c estimates the camera 20 capable of capturing the subject at the destination based on the information indicating the position of the camera 20 (step S3), and instructs the camera 20 to start the process of tracking the subject. The acquisition unit 15b also acquires data for verification to identify which camera 20 the person to be tracked is in, the identification unit 15a verifies the person to be tracked, and the estimation unit 15c estimates the camera 20 capturing the identified person and instructs the camera 20 to start the process of tracking. The estimation unit 15c also instructs the other cameras 20 to stop the process of tracking. After that, the estimation unit 15c returns the process to step S1. In this way, a series of inference processes is repeated until an instruction to end the tracking of the subject is given.

[0044] [effect] As described above, in the inference system 1 of this embodiment, the acquisition unit 15b acquires information on the movement of a predetermined subject captured by a first camera 20 among the edge cameras 20 at least at a first time. The estimation unit 15c estimates a second camera 20 that will capture the predetermined subject at a second time that is future than the first time, based on the movement destination of the predetermined subject.

[0045] Specifically, when a predetermined subject deviates from the shooting range of the first camera, the estimation unit 15c estimates the second camera.

[0046] As a result, in the inference system 1, within the shooting range of the first camera 20, only the first camera 20 tracks the specified subject. Furthermore, if the subject deviates from the shooting range of the first camera 20, the second camera 20 capable of shooting the subject is estimated, and only the second camera 20 tracks the subject. This allows the inference system 1 to efficiently track a desired person. In this way, the inference system reduces the amount of data transmission and total amount of calculation in the inference process using two layers, edge and cloud, making it possible to efficiently perform the desired inference and track the subject.

[0047] Here, Fig. 5 and Fig. 6 are diagrams for explaining the effect of the inference process. Conventionally, in person tracking using two-layer inference of edges and clouds, as shown in Fig. 5, images from cameras at all edges (or images of clipped areas including the subject) were transmitted to the cloud side.

[0048] Therefore, in the past, as shown by the bold frame in Fig. 6(a), even if the person to be tracked passes through the shooting range of some of the cameras, all cameras 20 (cameras A to C) continue to capture images at all times, and the person to be tracked is detected and matched for all people captured by the cameras in each frame. Therefore, as shown in Fig. 6(a), a certain amount of calculation resources continues to be consumed in all cameras A to C.

[0049] In contrast, in the inference system 1 of this embodiment, as shown in FIG. 6(b), the required amount of computational resources in each camera 20 increases or decreases according to the occurrence frequency of an event in which a person to be tracked appears in the shooting range. Therefore, it is possible to reduce the required amount of computational resources of the camera as long as the person to be tracked does not appear in the shooting range. Also, the camera 20 performs only light processing that requires a small amount of calculation to begin with. Therefore, in the inference system 1 of this embodiment, it is possible to reduce the required amount of computational resources and perform inference processing efficiently.

[0050] Specifically, the estimation unit 15c uses the direction and speed of movement of a predetermined subject as the movement information to estimate the second camera 20. As a result, the inference system 1 can perform inference processing efficiently by using the information on the direction and speed of movement of the subject acquired from the edge cameras 20 to track only the cameras 20 that can capture the subject.

[0051] Alternatively, the estimation unit 15c uses a probability distribution of a movement pattern of a predetermined subject in addition to or instead of the acquired movement information to estimate the second camera 20. This allows the inference system 1 to obtain highly accurate movement information and perform tracking processing when highly accurate movement information cannot be obtained from the edge camera 20.

[0052] [program] A program in which the processes executed by the server 10 or the camera 20 according to the above-mentioned embodiment are written in a language executable by a computer can also be created. As an embodiment, the server 10 or the camera 20 can be implemented by installing a presentation program that executes the above-mentioned presentation process as package software or online software on a desired computer. For example, the information processing device can be made to function as the server 10 or the camera 20 by executing the above-mentioned presentation program on the information processing device. The information processing device here includes a desktop or notebook personal computer. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and further slate terminals such as PDA (Personal Digital Assistant). The functions of the server 10 or the camera 20 may be implemented on a cloud server.

[0053] 7 is a diagram showing an example of a computer that executes an inference program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0054] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

[0055] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiments is stored in the hard disk drive 1031 or memory 1010, for example.

[0056] The inference program is stored in the hard disk drive 1031 as, for example, a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the hard disk drive 1031 stores the program module 1093 in which each process to be executed by the server 10 or the camera 20 described in the above embodiment is written.

[0057] Furthermore, data used for information processing by the inference program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-mentioned procedures.

[0058] The program module 1093 and program data 1094 related to the inference program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the inference program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0059] Although the embodiment to which the invention made by the present inventor is applied has been described above, the present invention is not limited by the description and drawings which form a part of the disclosure of the present invention according to the present embodiment. In other words, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on the present embodiment are all included in the scope of the present invention. [Explanation of symbols]

[0060] 10 Server 13,23 Communication control unit 14,24 Storage section 15, 25 Control section 15a Specific part 15b Acquisition part 15c Estimation part 20 Camera 22 Imaging unit 25a Detection unit 25b Tracking Section

Claims

1. 1. A method for tracking a desired object executed by a reasoning system, comprising: An acquisition step of acquiring information on the movement of a predetermined object photographed by a first camera at at least a first time by inference using the first camera, which is a lightweight inference; an estimation step of determining a second camera, which is selected from a plurality of cameras and is a camera that captures the specified subject at a second time that is future to the first time, by inference by a server, which is a non-lightweight inference, based on a destination of the specified subject; A step of controlling the server to perform light-weight inference on only the second camera, on information on the movement of the predetermined object; Including, The inference method, characterized in that the information on the movement of the specified subject includes a direction of travel of the subject estimated by the first camera.

2. 2. The inference method according to claim 1, wherein the inference step by the server determines the second camera by inference using information on the movement of the specified subject inferred by the first camera.

3. 2. The inference method according to claim 1, wherein the estimation step estimates the second camera when the predetermined subject deviates from a shooting range of the first camera.

4. 2. The method according to claim 1, wherein the first camera estimates a direction and a speed of movement of the predetermined object.

5. The inference method according to claim 1, characterized in that, when movement information with an accuracy equal to or higher than a predetermined value cannot be obtained from the first camera, the estimation step estimates the second camera using a probability distribution of the movement pattern of the specified subject.

6. An acquisition unit that acquires information on the movement of a predetermined subject photographed by a first camera at least at a first time by inference using the first camera, which is a lightweight inference; an estimation unit that obtains a second camera, which is selected from a plurality of cameras and is a camera that captures the specified subject at a second time that is in the future from the first time, by inference by a server, which is a non-lightweight inference, based on a destination of the specified subject; having The server controls the estimation unit so that the estimation unit performs light-weight inference on the second camera only, on information on the movement of the predetermined subject; An inference system characterized in that the information on the movement of the specified subject includes the direction of travel of the subject estimated by the first camera.

7. An acquisition step of acquiring information on the movement of a predetermined object photographed by a first camera at at least a first time by inference using the first camera, which is a lightweight inference; an estimation step of determining a second camera, which is selected from a plurality of cameras and is a camera that captures the specified subject at a second time that is future to the first time, by inference by a server, which is a non-lightweight inference, based on a destination of the specified subject; A step in which the server controls only the second camera to perform light-weight inference on information on the movement of the predetermined object; Run the following on your computer: The information on the movement of the specified subject includes a direction of travel of the subject estimated by the first camera.

Images