Learning device and program

Abstract

To provide a technique that enables high-precision object counting at low cost.SOLUTION: A learning device calculates a line neighborhood size, which is a size of a line neighborhood area, the line neighborhood size being used in an inference device that detects, from a video, a target object to be counted when passing through a line, and increases a count value when detecting the target object passing through the line and moving from a first area to a second area. The line neighborhood area is an area set in the vicinity of the line on the video. The learning device includes a buffer size calculation unit which calculates the line neighborhood size on the basis of sizes of target objects detected from the video, the target object not belonging to the first and second areas.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for counting objects that pass through a predetermined line among objects captured in video data. [Background technology]

[0002] There is an object counting technology that detects objects (such as humans) from video data and counts the number of objects passing through a line drawn on the boundary between the inside and outside of a facility.

[0003] Counting objects that pass through a predetermined line among those captured in video data allows us to grasp the number of objects present in a given space, which can be applied to determining the level of congestion within a facility and for marketing purposes. Considering the recent COVID-19 pandemic and the need to shift to new normal work styles and lifestyles, object counting technology has extremely high industrial applicability. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-242947 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when counting objects using conventional techniques such as optical flow calculations or general-purpose object detectors, there are issues such as high training costs and low counting accuracy.

[0006] The present invention has been made in view of the above points, and has an object to provide a technology that enables object counting to be performed at low cost and with high accuracy. [Means for solving the problem]

[0007] According to the disclosed technology, a learning device calculates a line neighborhood size, which is the size of a line neighborhood area used in an inference device that detects a target object that is a target of counting line crossings from an image, and increases a count value when it detects that the target object has crossed the line and moved from a first area to a second area, the line vicinity region is a region set in the vicinity of the line on the image, and does not belong to either the first region or the second region; a buffer size calculation unit that calculates the line neighborhood size based on the sizes of multiple target objects detected from the video; A learning device is provided. [Effects of the Invention]

[0008] The disclosed technology provides a technology that enables object counting to be performed with high accuracy at low cost. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 10 is a diagram illustrating an example of object counting. [Figure 2] FIG. 10 is a diagram illustrating a case where counting fails. [Figure 3] FIG. 1 is a configuration diagram of an inference device according to a first embodiment. [Figure 4] FIG. 1 is a configuration diagram of a learning device according to a first embodiment. [Figure 5] FIG. 2 is a diagram for explaining the operation of the inference device of the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining each region. [Figure 7] 10A and 10B are diagrams for explaining an example of the relationship between a line vicinity region and a line; [Figure 8] FIG. 10 is a diagram for explaining a counting method. [Figure 9] FIG. 2 is a diagram for explaining the operation of the learning device of the first embodiment. [Figure 10] FIG. 10 is a configuration diagram of an inference device according to a second embodiment. [Figure 11]FIG. 10 is a diagram for explaining the operation of the inference device of the second embodiment. [Figure 12] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] (Outline of this embodiment) In this embodiment, an inference device 100 (and an inference device 300) described below counts objects that pass through a predetermined line among one or more objects captured in video data. Figure 1 shows an example of object counting.

[0012] In the example in Figure 1, the object to be counted is a person, and a line is drawn on the boundary between the inside and outside of the facility. On the left side of Figure 1, an object passes through the line from the outside of the facility to the inside, so the number of objects entering the facility is incremented by 1 (represented as "In+1"). On the right side of Figure 1, an object passes through the line from the inside of the facility to the outside, so the number of objects leaving the facility is incremented by 1 (represented as "Out+1").

[0013] (About the assignment) To perform object counting, for example, a method (referred to as Method 1) can be used in which moving object regions are extracted from the input image based on the velocity field on a line obtained by optical flow calculation, which is a conventional technology, and the number of target objects contained in each obtained region is counted.

[0014] In this method, when counting the number of target objects in a moving object region, training data with ground truth data is used to train a regression model that inputs feature regions of appearance related to color, edges, etc. extracted from the moving object region and outputs the number of objects. However, the appearance of a moving object region changes significantly depending on the tilt angle at which the imaging device captures the target region and objects. Therefore, this method requires the acquisition and preparation of training data for each tilt angle of the camera, which poses a problem of high costs.

[0015] Another possible method for counting objects is to detect the target object from each image frame, associate the detection results between image frames to extract the trajectory of each object, and count by determining the position of each object at each time (method 2).In method 2, it is assumed that the passage of the line is determined by simply comparing the position of the object with the position of the line.

[0016] In Method 2, it is possible to use a general-purpose object detector, such as open source or commercial software. Method 2 solves the problem of Method 1, which is the high cost of acquiring and preparing training data. However, Method 2, which simply compares the position of an object with the position of a line, has the problem that the counting results are easily affected by a deterioration in the position detection accuracy of the object detection process.

[0017] An example is shown in Figure 2. In this example, three image frames (left, center, and right) are shown. Also shown are the correct In and Out values ​​for each of the three image frames, as well as the predictions made by Method 2.

[0018] As shown in Figure 2, the object indicated by A goes outside the line in the center image frame (Out:0 → Out:1) and then continues to move outside the line. However, the object detection result in the right image frame is inaccurate, and the estimated object position of A differs from the actual position, resulting in a false count of it entering the line (In:0 → In:1).

[0019] Reasons for the decrease in accuracy of object position detection include low image quality or resolution, and occlusion due to multiple objects being crowded together, which are very common. Therefore, Method 2 has the problem of low counting accuracy in many situations.

[0020] That is, methods 1 and 2 have problems such as high costs for preparing and acquiring training data or low counting accuracy. Below, a technique according to this embodiment, which is low cost and has high counting accuracy, will be described. Below, examples 1 and 2 will be described as examples of the technique according to this embodiment.

[0021] Example 1 <Configuration of inference device 100> Fig. 3 shows an example of the configuration of the inference device 100 in the first embodiment. As shown in Fig. 3, the inference device 100 has a detection unit 110, a tracking unit 120, a counting unit 130, a video data storage unit 140, a line information storage unit 150, and a line neighborhood buffer size storage unit 160. The functional unit integrating the detection unit 110 and the tracking unit 120 may be referred to as a "detection unit." The functions of each unit are outlined below.

[0022] Time-series image frames are stored as video data in video data storage unit 140. Real-time video data may be sequentially accumulated in video data storage unit 140, and object counting may be performed in real time by processing the video data, or past video data may be accumulated in video data storage unit 140, and object counting may be performed at a past point in time by processing the video data.

[0023] Information about lines that serve as a reference for object counting is stored in the line information storage unit 150. The line neighborhood buffer size storage unit 160 stores a line neighborhood buffer size obtained by the learning device 200, which will be described later. The line neighborhood buffer size to be stored in the line neighborhood buffer size storage unit 160 may be a line neighborhood buffer size determined by a system administrator or the like, without using the learning device 200.

[0024] 3 shows an example in which each storage unit is provided within inference device 100, but this is just one example. Any one or more or all of video data storage unit 140, line information storage unit 150, and line neighborhood buffer size storage unit 160 may be provided externally to inference device 100. When the storage unit is provided externally, for example, inference device 100 accesses the storage unit via a network to obtain information.

[0025] The detection unit 110 detects a target object from each image frame that constitutes the video data read from the video data storage unit 140. The "target object" is an object that is the target of line passage determination.

[0026] The tracking unit 120 receives the image frame at the current time, the detection result at the current time, and the past tracking result as input, and outputs the tracking result of the object at the current time. Note that the "current time" is the time of the latest image frame that is the target of the detection and tracking process.

[0027] The counting unit 130 receives the tracking results at each time, determines whether each tracking result is inside or outside the line, and performs object counting. More specifically, based on the object's position, line information, and line vicinity buffer size, the counting unit 130 classifies the area in which the object is located for each tracking result at each time into either an area separated by the line or unknown, and counts the object when the area in which the object is located changes from one of the areas except for unknown to one of the areas. In other words, when an object is located in the area near the line, the area in which the object is located is not explicitly identified. This allows for robust counting even if the position of an object detected near the line is inaccurate.

[0028] This configuration of the inference device 100 eliminates the need to prepare and acquire training data for each tilt angle of the camera, enabling low-cost object counting. Furthermore, the use of a line neighborhood buffer allows for highly accurate counting.

[0029] <Configuration of learning device 200> Fig. 4 shows an example configuration of the learning device 200. As shown in Fig. 4, the learning device 200 has a detection unit 210, a buffer size calculation unit 220, a video data storage unit 230, and a line information storage unit 240. The functions of each unit are outlined below.

[0030] Video data storage unit 230 stores a time series of image frames as video data. Line information storage unit 240 stores information on lines that serve as the basis for object counting. The images used by learning device 200 and inference device 100 may be the same or different. However, it is desirable that the locations and positions of the cameras that capture the images be consistent between learning device 200 and inference device 100.

[0031] 4 shows an example in which each storage unit is provided inside learning device 200, but this is just one example. One or more of video data storage unit 230 and line information storage unit 240 may be provided outside learning device 200. When the storage unit is provided outside, learning device 200 accesses the storage unit via a network to acquire information, for example.

[0032] The detection unit 210 detects an object from each of a predetermined number of image frames. The buffer size calculation unit 220 calculates and outputs the line neighborhood buffer size based on the object detection result by the detection unit 210.

[0033] Here, the line neighborhood buffer size is a parameter that determines the size of the line neighborhood area, and the larger the area, the more difficult it is to classify the area where the object is located near the line. The buffer size calculation unit 220 adaptively determines the line neighborhood buffer size in accordance with the position and size of the object detected by the detection unit 210, thereby making it possible to automatically determine the line neighborhood buffer size appropriate for the size of the object reflected in the video, thereby reducing the cost of tuning and enabling accurate counting.

[0034] <Details of the operation of the inference device 100> As described above, the video data storage unit 140 of the inference device 100 stores image frames in a time series, and the detection unit 110 (as well as the tracking unit 120 and counting unit 130) processes each image frame read out from the video data storage unit 140. An image of processing the image frames at each time is shown in Figure 5. As shown in Figure 5, processing is performed on the image frames at each time, starting with the image frame at time t = 0. The operation of the inference device 100 will be described in detail below with reference to Figures 5 to 8.

[0035] <About line information, line neighborhood buffer size, and area> First, the line information, line neighborhood buffer size, and area in this embodiment will be described.

[0036] In this embodiment, one piece of line information is defined by, for example, the (x, y) coordinates of both end points in the image. In this embodiment, only one line is defined in the image, but the technology according to the present invention can be similarly applied even when there are multiple lines.

[0037] In this embodiment, the area in the video is divided into an In area, an Out area, a line vicinity area, and a non-target area, as shown in FIG. 6, based on the line information and the line vicinity buffer size.

[0038] In the example of Figure 6, the In area is the lower area of ​​the area defined by the normal to the line that passes through both end points of the line. "Area defined by the normal to the line" means that an area exists in a direction perpendicular to the line. However, this is just one example of the definition of an area, and is not limited to this.

[0039] 6, the Out area is the upper area of ​​the area defined by the normal to the line passing through both end points of the line. Which side of the line is the In area / Out area is determined in advance.

[0040] The line neighborhood region is a region near a line defined by the normal of the line passing through the line endpoints, and the size of the region is defined by the line neighborhood buffer size d. In the example of Figure 6, the line neighborhood region is defined as a region whose width in the direction perpendicular to the line is d and whose length (length in the direction parallel to the line) is the length of the line.

[0041] The non-target area is an area that is not an In area, an Out area, or an area near a line. The In area and the Out area may be defined in other ways. For example, the area below the area defined by the line information that is not near the line may be defined as the In area, and the area above the area defined by the line information that is not near the line may be defined as the Out area.

[0042] As is clear from the above description of the line neighborhood buffer size, the line neighborhood buffer size is defined as one-dimensional data, and is used in combination with line information to define the line neighborhood area. In this embodiment, as shown in Figure 6, the line neighborhood area is defined as a rectangle with the line as its center line and defined by the line width and the line neighborhood buffer size, but the method of defining the line neighborhood area is not limited to this.

[0043] In the example of Fig. 6, in the region near the line, the length from the line to the Out region side is 0.5d, and the length from the line to the In region side is 0.5d, and the region near the line is symmetrical with respect to the line. However, this is just an example, and the region near the line may be asymmetric with respect to the line, for example, as shown in Cases A and B of Fig. 7.

[0044] 5, inference device 100 outputs object count results for each time in the video based on the video data, line information, and line neighborhood buffer size. Here, the count results for each time in the video consist of the number of target objects that have moved from the In area to the Out area and the number of target objects that have moved from the Out area to the In area up to that time, counting from the start of the video.

[0045] <Detection unit 110> Next, the detection unit 110 will be described. The detection unit 110 receives image frames at each time that make up the video, and outputs the position of the target object that appears in the image. In this embodiment, the position of the target object is output as position information of a rectangle that exactly surrounds the target object. One example of a method for representing a target object is to represent the target object as a rectangle that completely surrounds the object.In addition to a rectangle, the target object and its position may also be represented by mask information consisting of a set of pixel positions that capture the target object.

[0046] The method of detecting a target object from a video is itself a conventional technique, and the techniques disclosed in, for example, References 1 and 2 can be used. The names of the references are listed at the end of the specification.

[0047] <Tracking Unit 120> Next, a description will be given of the tracking unit 120. The tracking unit 120 receives as input an image frame at the current time, a target object detection result at the current time output from the detection unit 120, and a past tracking result (specifically, the tracking result at the immediately previous time), and outputs the tracking result at the current time.

[0048] Here, the tracking result is composed of an ID assigned to each target object appearing in the video and the position of the target object in the current image frame. The tracking process itself performed by the detection unit 120 is a conventional technique, and the techniques disclosed in, for example, References 3 and 4 can be used.

[0049] When using the technology disclosed in Reference 3, the tracking unit 120 matches objects between image frames based solely on the positional proximity of the detection results output from the detection unit 110, whereas when using the technology disclosed in Reference 4, the tracking unit 120 also takes into account appearance information extracted from the target object region in the image frame to match past tracking results with objects in the current image frame. The technology of Reference 4, which takes appearance information into account, generally provides higher tracking performance. On the other hand, the processing speed is faster when using the technology of Reference 3, which tracks based only on position, so there is a trade-off between the two.

[0050] <Counting unit 130> Next, the counting unit 130 will be described. The counting unit 130 receives the tracking results output by the tracking unit 120 as input, and outputs the counting results at each time in the video. The counting procedure executed by the counting unit 130 is as follows. The counting unit 130 includes a position storage unit that stores, for each target object, the area in which it is located (In area or Out area).

[0051] Based on the position of each tracked object at the current time, the counting unit 130 classifies the area in which each object is located into one of the In area, Out area, area near the line, and non-target area. This classification method can be arbitrarily used, and for example, it can be determined which of the above areas contains a representative point of a rectangle corresponding to the object (e.g., x, y coordinates of the center of the bottom side of the rectangle).

[0052] Next, for each target object, if the area stored in the position storage unit up to now is the In area and the position at the current time is determined to be the Out area, counting unit 130 increments the count value indicating movement from the In area to the Out area by 1 (adds 1 to the count value). Similarly, if the area stored in the position storage unit up to now is the Out area and the position at the current time is determined to be the In area, counting unit 130 increments the count value indicating movement from the Out area to the In area by 1.

[0053] Finally, if the current position of the target object is in the In area or the Out area, the counting unit 130 updates the area in which the target object is located, which is stored in the position storage unit, to the area in which the current target object is located.

[0054] More specifically, if the In area is stored in the position memory unit for a certain target object at time t=k, and the area determination result at time t=k+1 is either the In area, the area near the line, or the non-target area, the information stored in the position memory unit remains the In area, and if the area determination result at time t=k+1 is the Out area, the information stored in the position memory unit is updated to the Out area.

[0055] Similarly, for a certain target object, if the Out area is stored in the position memory unit at time t=k, and the area determination result at time t=k+1 is either the Out area, the area near the line, or the non-target area, the information stored in the position memory unit will remain as the Out area, and if the area determination result at time t=k+1 is the In area, the information stored in the position memory unit will be updated to the In area.

[0056] An example is shown in Figure 8. As shown by the trajectory of the target object (points connected by arrows), the target object is located in the In region at times t = 1 and t = 2, and at these times, the In region is stored in the position memory unit as the region where the target object exists. At times t = 3 and t = 4, the target object is located in the region near the line, so the region stored in the position memory unit is not updated and remains the In region.

[0057] At time t=5, the target object reaches the Out area. At this time, the area in which the target object exists changes from the In area to the Out area, so counting unit 130 increments the count value of the target object's movement from the In area to the Out area by 1, and updates the area stored in the position storage unit from the In area to the Out area.

[0058] The counting unit 130 performs the above processing on the tracking results of all target objects, and outputs a count value indicating the total number of movements from the In area to the Out area and a count value indicating the total number of movements from the Out area to the In area as the counting results at the current time.

[0059] <Details of the operation of the learning device 200> Next, a detailed description will be given of the learning device 200. As described above, the video data storage unit 230 of the learning device 200 stores image frames in time series, and the detection unit 210 performs processing on each image frame read from the video data storage unit 230. An image of processing the image frame at each time is shown in Fig. 9.

[0060] The operation of detection unit 210 is the same as the operation of detection unit 110 in inference device 100. The operation of buffer size calculation unit 220 in learning device 200 will now be described in detail.

[0061] <Buffer size calculation unit 220> The buffer size calculation unit 220 calculates the line neighborhood buffer size using as input the object detection results obtained by the detection unit 210 for a predetermined number of image frames. The buffer size calculation unit 220 adaptively determines the line neighborhood buffer size according to the size and position of the object detected by the detection unit 210.

[0062] The method for calculating the line neighborhood buffer size based on the object detection result is not limited to a specific method, but for example, the buffer size calculation unit 220 may calculate the line neighborhood buffer size so as to be proportional to the average size of the target objects detected by the detection unit 210. The size of the target object here can be defined by, for example, the height of a rectangle obtained as a detection result or the length of the diagonal of the rectangle.

[0063] For example, if the target objects passing through the line are mainly humans, the average size (height) of the target objects can be set to, for example, 165 cm, and the buffer size calculation unit 220 can set the height on the image of a 165 cm person appearing in the image at the line position as the line vicinity buffer size, which is h x a constant.

[0064] Alternatively, the buffer size calculation unit 220 may indirectly calculate the depth of the line representative point from the camera based on the object detection result, and determine the line neighborhood buffer size to be proportional to that value. An example of a specific determination method is as follows.

[0065] The set of detected target objects detected by the detection unit 210 and input to the buffer size calculation unit 220 is defined as {(x i ,y i ,w i ,h i )} N i=1 Here, (x i ,y i) is a representative point in a rectangle that represents the target object in the image. This point may be, for example, the center of gravity of the rectangle, a vertex of the rectangle, a point on a side of the rectangle (for example, the center of the side), or any other point. i is the width of the rectangle, and h i is the height of the rectangle. N is the number of objects. w i and h i are examples of the size of the target object.

[0066] Here, it is assumed that the line and the detected target objects are on the same plane and that the heights of the detected multiple target objects are approximately constant. For example, imagine a situation in which a line is drawn on the floor (or ground) at the boundary of the entrance and exit of a facility, and an object (here, a human) walks on the floor (or ground) to enter and exit the facility.

[0067] In this case, a, b, and c are each one-dimensional parameters, and for all object detection results, h i =ax i +by i +c (formula 1) The buffer size calculation unit 220 uses the object detection set to find the parameters a, b, and c in Equation 1 by using an optimization method such as the least squares method.

[0068] Here, the representative x and y coordinates of the line are (x l ,y l ), and a constant is d0, the buffer size calculation unit 220 calculates the line neighborhood buffer size d as follows: d=d0(ax l +by l +c) (Formula 2) It can be calculated as:

[0069] In the above example, the heights of the detected multiple target objects are generally constant. If there are multiple types of target objects, d may be calculated for each type.

[0070] For example, suppose there is a mixture of children and adults. The buffer size calculation unit 220 calculates the line neighborhood buffer size d1 from a group of adults as target objects using the above-mentioned method, calculates the line neighborhood buffer size d2 from a group of children as target objects using the above-mentioned method, and outputs d1 and d2.

[0071] In this case, detection unit 110 of inference apparatus 100 determines whether the detected object is an adult or a child, and counting unit 130 uses d1 as the line neighborhood buffer size for adult objects and d2 as the line neighborhood buffer size for child objects.

[0072] Example 2 Next, a description will be given of Example 2. In Example 1, the calculation of the line neighborhood buffer size by learning device 200 and the counting process by inference device 100 were independent. On the other hand, in Example 2, while the counting process is performed on input video data, the line neighborhood buffer size is successively updated based on the detection result of the target object, and the counting process is performed using the result.

[0073] Fig. 10 shows an example of the configuration of an inference device 300 in Example 2. As shown in Fig. 10, the inference device 300 includes a detection unit 110, a tracking unit 120, a counting unit 130, a video data storage unit 140, a line information storage unit 150, and a buffer size update unit 310. The buffer size update unit 310 may also be referred to as a buffer size calculation unit.

[0074] 10 shows an example in which each storage unit is provided within inference device 300, but this is just one example. One or more of video data storage unit 140 and line information storage unit 150 may be provided externally to inference device 300. When a storage unit is provided externally, for example, inference device 300 accesses the storage unit via a network to obtain information.

[0075] The detection unit 110, the tracking unit 120, and the counting unit 130 in the second embodiment perform the same processes as the detection unit 110, the tracking unit 120, and the counting unit 130 in the first embodiment, respectively.

[0076] The video data storage unit 140 stores image frames in time series, and the detection unit 110 (as well as the tracking unit 120, counting unit 130, and buffer size update unit 310) processes each image frame read from the video data storage unit 140. An image of processing the image frames at each time is shown in FIG. 11.

[0077] In the second embodiment, the buffer size update unit 310 receives the object detection results from a certain past time to the current time, and performs the same processing as the buffer size calculation unit 220 described in the first embodiment to calculate the line neighborhood buffer size at the current time, and inputs the line neighborhood buffer size to the counting unit 130.

[0078] Here, the object detection results used to update the line neighborhood buffer size from a certain time in the past to the current time may be all currently and previously stored detection results, or detection results obtained from up to a predetermined number of past image frames may be used.

[0079] (Example of hardware configuration) Inference device 100, learning device 200, and inference device 300 can all be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud. Note that, hereinafter, inference device 100, learning device 200, and inference device 300 will be collectively referred to as "devices."

[0080] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0081] Fig. 12 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 12 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS. Note that some of these devices may not be included. For example, if no display is required, the display device 1006 may not be included.

[0082] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0083] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions of the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, and functions as a transmitter and receiver. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0084] (Effects of the embodiment) As described above, according to this embodiment, a technique is provided that enables object counting to be performed with high accuracy at low cost.

[0085] (Summary of the embodiment) This specification discloses at least the inference device, learning device, object counting method, and program described in the following sections. (Section 1) An inference device that counts objects passing through a predetermined line from a video of the objects, a detection unit that detects, from the video, a target object that is to be counted as having passed through the line; a counting unit that increases a count value when it is detected that the target object has passed through the line and moved from the first area to the second area, A line vicinity area is set in the vicinity of the line on the image, and the line vicinity area does not belong to either the first area or the second area. Reasoning device. (Section 2) The counting unit storing the first area or the second area in a storage unit as an area in which the target object is located, and if the area in which the target object is located is the line vicinity area, not updating the area stored in the storage unit; When the area stored in the storage unit is the first area and the area in which the target object is located becomes the second area, the count value is increased. 2. The inference device of claim 1. (Section 3) When detecting that the target object has passed through the line and moved from the first area to the second area, the counting unit increases a count value corresponding to the movement from the first area to the second area, and when detecting that another target object has passed through the line and moved from the second area to the first area, the counting unit increases a count value corresponding to the movement from the second area to the first area. 2. The inference device according to claim 1 or 2. (Section 4) The image processing device further includes a buffer size update unit that calculates a line neighborhood size, which is the size of the line neighborhood area, based on the sizes of the plurality of target objects detected by the detection unit, and inputs the line neighborhood size to the count unit. An inference device according to any one of claims 1 to 3. (Section 5) A learning device that calculates a line neighborhood size, which is the size of the line neighborhood area used in the inference device according to any one of claims 1 to 3, comprising: a buffer size calculation unit that calculates the line neighborhood size based on the sizes of multiple target objects detected from the video; Learning device. (Section 6) The buffer size calculation unit generates a relational expression indicating a relationship between the positions and sizes of the plurality of target objects on the video image based on the positions and sizes of the plurality of target objects on the video image, and calculates the line neighborhood size by applying the position of the line to the relational expression. 6. The learning device according to claim 5. (Section 7) An object counting method executed by an inference device that counts objects passing through a predetermined line from a video of the objects, the method comprising: a detecting step of detecting, from the image, a target object whose crossing of the line is to be counted; a counting step of increasing a count value when it is detected that the target object has passed through the line and moved from the first area to the second area, A line vicinity area is set in the vicinity of the line on the image, and the line vicinity area does not belong to either the first area or the second area. Object counting method. (Section 8) A program for causing a computer to function as each part of the inference device described in any one of paragraphs 1 to 4. (Section 9) A program for causing a computer to function as a buffer size calculation unit in the learning device described in paragraph 5 or 6.

[0086] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0087] [References] Reference 1: X. Zhou, D. Wang, and P. Krahenbuhl. Objects as points. In arXiv preprint arXiv:1904.07850, 2019. Reference 2: E. Xie, P. Sun, X. Song, W. Wang, D. Liang, C. Shen, and P. Luo. Polarmask: Single shot instance segmentation with polar representation. In CVPR, 2020. Reference 3: A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft. 7Simple online and realtime tracking. In ICIP, 2016. Reference 4: Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu. Fairmot: On the fairness of detection and re-identification in multiple object tracking. In arXiv preprint arXiv:, 2020. [Explanation of symbols]

[0088] 100 Reasoning device 110 Detector 120 Tracking Department 130 Counting section 140 Video data storage unit 150 Line information storage section 160 Line neighborhood buffer size storage 200 Learning Device 210 Detector 220 Buffer size calculation unit 230 Video data storage unit 240 Line information storage section 300 Reasoning device 310 Buffer size update section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. A learning device that calculates a line neighborhood size, which is the size of a line neighborhood area used in an inference device that detects a target object that is a target of a line crossing count from an image, and increases a count value when it detects that the target object has crossed the line and moved from a first area to a second area, the line vicinity region is a region set in the vicinity of the line on the image, and does not belong to either the first region or the second region; a buffer size calculation unit that calculates the line neighborhood size based on the sizes of multiple target objects detected from the video; Learning device.

2. The buffer size calculation unit generates a relational expression indicating a relationship between the positions and sizes of the plurality of target objects on the video image based on the positions and sizes of the plurality of target objects on the video image, and calculates the line neighborhood size by applying the position of the line to the relational expression. The learning device according to claim 1 .

3. A program for causing a computer to function as the buffer size calculation unit in the learning device according to claim 1 or 2.

Images