Object quantity estimation device, object quantity estimation method and storage medium

By setting the local area size according to the image in the object number estimation device and using a reconstructed learning model, the problem of degradation of estimation accuracy caused by the inappropriate size is solved, and the high-precision estimation of the number of objects when the image shooting range is changed is realized.

CN112446275BActive Publication Date: 2024-12-03KK TOSHIBA
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202010122252.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-09-04
Filing Date
2020-02-27
Publication Date
2024-12-03
Estimated Expiration
2040-02-27

Smart Images

  • Figure CN112446275B_ABST
    Figure CN112446275B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an object quantity estimation device, an object quantity estimation method, and a storage medium, which estimate the number of objects with high accuracy. The object quantity estimation device includes an acquisition unit, a setting unit, and an estimation unit. The acquisition unit acquires an image. The setting unit sets a local area size as a unit for estimating the number of objects in the image according to the image. The estimation unit estimates the number of objects contained in the image according to each local area of ​​the set local area size.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is based on Japanese Patent Application No. 2019-161610 (filing date: September 4, 2019) and claims priority from that application. This application incorporates all the contents of that application by reference. Technical Field

[0002] Embodiments of the present invention relate to an object quantity estimation device, an object quantity estimation method, and a storage medium. Background Art

[0003] There is a known method for analyzing an image obtained by a surveillance camera or the like and estimating the number of people contained in the image. For example, a learning model is learned using the correspondence between an image and a correct label indicating the state of a crowd reflected in the image and the position of the crowd as teacher data. Furthermore, a technology for estimating the position of a crowd from an image using the learning model is disclosed. In addition, in the prior art, the number of objects contained in the image is estimated from the position of the crowd by estimating the objects contained in the image in each area of ​​a fixed size.

[0004] However, the size of a person included in an image varies depending on the image capturing range, etc. Therefore, if a fixed size as a unit of estimation does not fit the size of a person in an image, the estimation accuracy of the number of objects may decrease. Summary of the invention

[0005] The object quantity estimation device of the embodiment includes an acquisition unit, a setting unit, and an estimation unit. The acquisition unit acquires an image. The setting unit sets a local area size as a unit for estimating the number of objects in the image based on the image. The estimation unit estimates the number of objects contained in the image according to each local area of ​​the set local area size.

[0006] According to the above-mentioned object number estimating device, the number of objects can be estimated with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the object quantity estimation system according to the first embodiment.

[0008] Figure 2A It is a schematic diagram of an image of the first embodiment.

[0009] Figure 2B It is an explanatory diagram of a partial area of ​​the first embodiment.

[0010] Figure 3 Schematic diagram of the benchmark learning model of the first embodiment.

[0011] Figure 4 It is a schematic diagram showing the relationship between the local area size, the characteristic map, and the estimation result in the first embodiment.

[0012] Figure 5A It is a schematic diagram of an image of the first embodiment.

[0013] Figure 5B It is a schematic diagram of an image of the first embodiment.

[0014] Figure 6 FIG. 4 is a schematic diagram of an output image according to the first embodiment.

[0015] Figure 7 FIG. 4 is a schematic diagram of an output image according to the first embodiment.

[0016] Figure 8 Schematic diagram of the learning model of the first embodiment.

[0017] Fig. 9 : is a flowchart showing the flow of object quantity estimation processing according to the first embodiment.

[0018] Fig.10 : is a flowchart showing the flow of object quantity estimation processing according to the first embodiment.

[0019] Fig.11 It is a schematic diagram of the object quantity estimation system according to the second embodiment.

[0020] Fig.12 Schematic diagram of a teacher image according to the second embodiment.

[0021] Fig.13 This is a hardware structure diagram of the second embodiment.

[0022] Reference numerals

[0023] 10, 11: object number estimation system; 20, 21: object number estimation device; 20C: acquisition unit; 20D: setting unit; 20E, 21E: output control unit; 20F: acceptance unit; 20G, 21G: estimation unit; 50: reference learning model; 52: learning model. DETAILED DESCRIPTION

[0024] Hereinafter, an object quantity estimation device, an object quantity estimation method, and a storage medium will be described in detail with reference to the accompanying drawings.

[0025] (First Embodiment)

[0026] Figure 1 It is a schematic diagram showing an example of the object quantity estimation system 10 according to the present embodiment.

[0027] The object number estimation system 10 includes an object number estimation device 20 , a communication unit 10A, an imaging unit 10B, and a UI (user interface) unit 10C.

[0028] Object quantity estimation device 20 is connected to communication unit 10A, imaging unit 10B, and UI unit 10C so as to be able to exchange data or signals.

[0029] The object number estimation device 20 is, for example, a dedicated or general-purpose computer. The object number estimation device 20 estimates the number of objects included in an image (details will be described later).

[0030] The communication unit 10A communicates with an external device. For example, the communication unit 10A transmits various information to the external device or receives various information from the external device via a known communication line.

[0031] The imaging unit 10B is an imaging device that captures an image of an object. The imaging unit 10B obtains image data of a captured image by, for example, capturing an image. In the following description, the image data may be simply referred to as an image.

[0032] The imaging unit 10B acquires an image of a target area. The target area is a specific area in real space. The target area can be predetermined. For example, the target area is an area in a specific building, a specific region, a specific space, etc.

[0033] The imaging unit 10B acquires an image obtained by imaging an object included in the target area.

[0034] An object refers to an object whose number is estimated by the object quantity estimation device 20. The type of the object can be predetermined. For example, the object is a person, an animal, a cell, various organisms, a vehicle, an object that can fly (a manned aerial vehicle, an unmanned aerial vehicle (such as UAV (Unmanned Aerial Vehicle), an unmanned aircraft)), a robot, etc. In addition, the object is not limited to this. In this embodiment, the case where the object is a person is described as an example.

[0035] The UI unit 10C has a function of receiving an operation input by a user and a function of outputting various information. For example, the UI unit 10C includes a display 10D and an input unit 10E.

[0036] The display 10D displays various information and is, for example, a well-known organic EL (Electro-Luminescence) display, LCD (Liquid Crystal Display), or a projection device.

[0037] The input unit 10E receives various instructions from the user. The input unit 10E is, for example, a keyboard, a mouse, a touch panel, a microphone, and the like.

[0038] In addition, the UI unit 10C may be constituted by a touch panel having an input mechanism and an output mechanism. In addition, the UI unit 10C may also include a speaker that outputs sound.

[0039] Next, object quantity estimation device 20 will be described. Object quantity estimation device 20 includes processing unit 20A and storage unit 20B. Processing unit 20A, storage unit 20B, communication unit 10A, imaging unit 10B, display 10D, and input unit 10E are connected via bus 10F.

[0040] At least one of the storage unit 20B, the communication unit 10A, the imaging unit 10B, the display 10D, and the input unit 10E may be connected to the processing unit 20A via a network.

[0041] The storage unit 20B stores various data. In the present embodiment, the storage unit 20B stores the reference learning model 50 in advance. The reference learning model 50 is reconstructed into the learning model 52 by the processing of the processing unit 20A described later. The details of the reference learning model 50 and the learning model 52 will be described later.

[0042] The storage unit 20B is, for example, a RAM (Random Access Memory), a semiconductor storage element such as a flash memory, a hard disk, an optical disk, etc. In addition, the storage unit 20B may be a storage device provided outside the object quantity estimation device 20. In addition, the storage unit 20B may be a storage medium. Specifically, the storage medium may be a storage medium that is downloaded and stored or temporarily stores programs and various information via a LAN (Local Area Network), the Internet, etc. In addition, the storage unit 20B may be composed of a plurality of storage media.

[0043] The structure may be such that at least one of the storage unit 20B and the processing unit 20A is mounted on an external device such as a server device connected to a network. In addition, at least one of the functional units described later included in the processing unit 20A may be mounted on an external device such as a server device connected to the processing unit 20A via a network.

[0044] The processing unit 20A includes an acquisition unit 20C, a setting unit 20D, an output control unit 20E, an acceptance unit 20F, and an estimation unit 20G. At least one of the acquisition unit 20C, the setting unit 20D, the output control unit 20E, the acceptance unit 20F, and the estimation unit 20G is implemented, for example, by one or more processors. For example, the above-mentioned parts can be implemented by making a processor such as a CPU (Central Processing Unit) execute a program, that is, software. The above-mentioned parts can also be implemented by a dedicated processor such as an IC (Integrated Circuit), that is, hardware. The above-mentioned parts can be implemented by using software and hardware in combination. When using multiple processors, each processor can implement one of the parts, or can implement two or more of the parts.

[0045] The acquisition unit 20C acquires an image. The acquisition unit 20C acquires an image of a target for object quantity estimation.

[0046] Figure 2A 2 is a schematic diagram showing an example of an image 30 acquired by the acquisition unit 20C. The image 30 includes one or more persons 32. As described above, the person 32 is an example of an object.

[0047] return Figure 1 The description continues. The image 30 may be any image selected from a captured image obtained by the capturing unit 10B, an image acquired from an external device connected via the communication unit 10A, and an image pre-stored in the storage unit 20B. In addition, the external device connected via the communication unit 10A may be a capturing device connected via the communication unit 10A. The capturing device may be a device that obtains image data of a captured image by capturing. The image 30 is not limited to a captured image. For example, the image 30 may be any image selected from a captured image, an image read by a scanner, etc., and an image produced using a known image production application, etc.

[0048] The setting unit 20D sets the local area size in the image 30 based on the image 30 acquired by the acquisition unit 20C. The local area size is a unit for estimating the number of objects in the image 30.

[0049] Figure 2B FIG. 4 is an explanatory diagram of an example of a local area 40. Figure 2B, multiple local regions 40 (local regions 40A to 40D) of different local region sizes are shown as an example. Local region 40A is, for example, a local region 40 having a local region size of 8×8 pixels. Local region 40B is, for example, a local region 40 having a local region size of 16×16 pixels. Local region 40C is, for example, a local region 40 having a local region size of 32×32 pixels. Local region 40D is, for example, a local region 40 having a local region size of 64×64 pixels. In addition, the local region size is not limited to Figure 2B Example shown.

[0050] For example, the setting unit 20D uses a known detector to derive the size of the head of the person included in the image 30. Then, the setting unit 20D may (automatically) set the size of the local region based on the derived size of the head.

[0051] In the estimation unit 20G, which will be described in detail later, the number of people 32 is estimated for each local area 40 according to the local area size in the image 30. By performing the estimation process for each local area 40, the estimation unit 20G estimates the number of people 32 included in the image 30. In addition, the number of people 32 may be referred to as the number of objects in the description.

[0052] The reference learning model 50 is used for estimating the number of objects. In the present embodiment, it is assumed that the reference learning model 50 is stored in advance in the storage unit 20B.

[0053] Figure 3 : is a schematic diagram showing a reference learning model 50. The reference learning model 50 is a model for deriving an estimation result of the number of objects in a local region 40 of a local region size in the image 30 by repeating a convolution operation on the image 30. The reference learning model 50 is a neural network model.

[0054] In the reference learning model 50, a plurality of convolution layers 42 are generated by repeatedly performing convolution operations on the image 30 input as the input layer, thereby generating a plurality of feature maps 44 of different sizes (resolution and scale). Then, in the reference learning model 50, an estimation result 46 is output as a plurality of output maps (output layers) respectively corresponding to the plurality of feature maps 44. The estimation result 46 is an estimation result of the number of objects.

[0055] exist Figure 3 , as the characteristic graph 44 , characteristic graphs 44A to 44D are shown as an example.

[0056] The feature map 44A is the feature map 44 reduced to 1 / 8 the size. The estimation result 46A is the estimation result of the number of objects derived from the feature map 44A of this size. That is, the estimation result 46A is equivalent to the estimation result obtained by estimating the number of objects for the local area 40A of the local area size of 8×8 pixels in the image 30.

[0057] The feature map 44B is the feature map 44 reduced to 1 / 16 of the size. The estimation result 46B is the result of estimating the number of objects derived from the feature map 44B of this size. That is, the estimation result 46B is equivalent to the result of estimating the number of objects for the local area 40B of the local area size of 16×16 pixels.

[0058] The feature map 44C is the feature map 44 reduced to 1 / 32 of the size. The estimation result 46C is the result of estimating the number of objects derived from the feature map 44C of this size. That is, the estimation result 46C is equivalent to the result of estimating the number of objects for the local area 40C of the local area size of 32×32 pixels.

[0059] The feature map 44D is the feature map 44 reduced to 1 / 64 of the size. The estimation result 46D is the result of estimating the number of objects derived from the feature map 44D of this size. That is, the estimation result 46D is equivalent to the result of estimating the number of objects for the local area 40D of the local area size of 64×64 pixels.

[0060] Figure 4 4 is a schematic diagram showing an example of the relationship between the local area size of the local area 40B in the image 30, the feature map 44, and the estimation result 46. Figure 4 , a case where the local area size is 16×16 pixels is shown as an example.

[0061] like Figure 4 As shown, it is assumed that the number of objects in the local area 40B of the local area size of 16×16 pixels in the estimated image 30. In this case, the image 30 is reduced to 1 / 16 using the reference learning model 50, so as to obtain a feature map 44B corresponding to the local area 40B. Then, an estimation result 46B such as 6 persons is obtained from the feature map 44B.

[0062] Here, when the range of the target region reflected in the image 30 is changed, the size of the person 32 included in the image 30 changes.

[0063] The change in the scope of the target area means that when a captured image is used as the image 30 of the object quantity estimation object, the shooting range is changed. That is, the change in the scope of the target area means that at least one of the height from the ground, the shooting angle, the viewing angle, the shooting magnification, the resolution, and the shooting environment (brightness, etc.) of the installation position of the shooting unit 10B that shoots the image 30 or other shooting devices that shoot the image 30 is changed. In addition, when an image 30 other than a captured image is used, the size of the object contained in the image 30 is changed, etc.

[0064] Therefore, when the local region size of the local region 40 serving as the object number estimation unit is fixed regardless of the image 30 , the estimation accuracy of the object number may decrease.

[0065] Figure 5A as well as Figure 5B 1 is a schematic diagram showing an example of the image 30 in which the ranges of the target regions reflected in the image 30 are different. Figure 5A 1 is a schematic diagram showing an example of an image 30A having a narrow target region. Figure 5B 30A is a schematic diagram showing an example of an image 30B in which the range of the target region is wider than that of the image 30A. The image 30A and the image 30B are examples of the image 30 .

[0066] like Figure 5A as well as Figure 5B As shown in FIG. 1 , it is assumed that the local area size is fixed, and local areas 40 of the same local area size (e.g., local area 40B of a local area size of 16×16 pixels) are set in image 30A and image 30B. In this case, in image 30B where the range of the object area is wide, the local area 40B of the local area size includes the entire part (e.g., the head) that can identify the person 32 (see FIG. 1 ). Figure 5B On the other hand, in the image 30A having a narrow target area, the local area 40B having the local area size only includes a part of the part (e.g., the head) that can identify the person 32 ( Figure 5A ). Therefore, when the local area size is fixed regardless of the image 30 and the person 32 is estimated, the estimation accuracy may decrease.

[0067] return Figure 1 Therefore, in the present embodiment, the setting unit 20D sets the local region size based on the image 30 .

[0068] For example, the setting unit 20D sets the local area size selected by the user. In this case, for example, the user selects the local area size by selecting a desired estimation result 46 from the output image displayed by the output control unit 20E.

[0069] Specifically, the output control unit 20E outputs the output image. For example, the output control unit 20E displays the output image on the display 10D. In addition, the output control unit 20E may output the output image to an external device connected via the communication unit 10A. In this embodiment, the output control unit 20E displays the output image on the display 10D as an example for explanation.

[0070] The output image is an image including a list of estimation results 46 obtained by estimating the number of objects included in the image 30 for each of the local regions 40 having a plurality of local region sizes different from each other.

[0071] Figure 6 1 is a schematic diagram showing an example of an output image 54. The output image 54 includes the image 30 and a plurality of estimation results 46.

[0072] The estimation result 46 displayed in the output image 54 is information indicating the number of objects (that is, the number of people 32) estimated for each of the local regions 40 having a plurality of local region sizes that are different from each other. Figure 6 In FIG. 4 , estimation result images 48 ( 48A to 48D) of estimation result 46 ( 46A to 46D) are shown as an example. Figure 6 , as an example, a case where an estimated result image 48 includes the number of objects 47 (47A to 47D) included in the image 30 instead of the estimated result 46 for each local area 40. The number of objects 47 (47A to 47D) is a calculation result obtained by calculating the number of objects included in the entire image 30 by performing a process of estimating the number of objects (estimation result 46) for each local area 40 (40A to 40D) of the image 30 while changing the position of the local area 40.

[0073] Furthermore, the estimation result image 48 is preferably a superimposed image in which information indicating at least one of the estimation result 46 and the number of objects 47 is superimposed on the image 30 .

[0074] In addition, Figure 6, the estimated result image 48A is an image showing the number of objects 47A included in the image 30 estimated for each local area 40A having a local area size of 8×8 pixels. The estimated result image 48B is an image showing the number of objects 47B included in the image 30 estimated for each local area 40B having a local area size of 16×16 pixels. The estimated result image 48C is an image showing the number of objects 47C included in the image 30 estimated for each local area 40C having a local area size of 32×32 pixels. The estimated result image 48D is an image showing the number of objects 47D included in the image 30 estimated for each local area 40D having a local area size of 64×64 pixels.

[0075] For example, the output control unit 20E outputs the image 30 acquired by the acquisition unit 20C to the reference learning model 50, thereby generating a plurality of estimation result images 48 corresponding to a plurality of local area sizes different from each other. Then, the output control unit 20E may display an output image 54 including the image 30 and the plurality of estimation result images 48 on the display 10D.

[0076] The user selects any estimation result from the estimation results 46 (or the number of objects 47) by operating the input unit 10E while visually checking the displayed output image 54. That is, the user selects any estimation result from the estimation results 46 (or the number of objects 47) by selecting any estimation result image from the displayed plurality of estimation result images 48. Figure 6 , the case where the estimation result image 48A is selected is shown as an example. The user may select at least one estimation result image 48 from the displayed plurality of estimation result images 48, or may select two or more estimation result images 48. Here, the description will be continued assuming that one estimation result image 48 is selected.

[0077] Then, the user operates the input unit 10E to select the display position of the decision button D included in the output image 54. Then, the accepting unit 20F accepts the user's selection of the estimation result 46 included in the output image 54 from the input unit 10E.

[0078] The setting unit 20D may set the local region size corresponding to the received estimation result 46 (or the number of objects 47 ) as the local region size serving as the object number estimation unit of the image 30 .

[0079] In addition, the setting of the local area size by the setting unit 20D is not limited to the above-mentioned method.

[0080] For example, the setting unit 20D may set the local area size used to derive the estimation result 46 closest to the previously acquired correct number of objects contained in the image 30 among the plurality of estimation results 46 as the local area size serving as the object number estimation unit of the image 30 .

[0081] In this case, the setting unit 20D preferably uses the number of objects 47 instead of the estimated result 46. As described above, the number of objects 47 is a calculation result obtained by calculating the number of objects included in the entire image 30 by performing a process of estimating the estimated result 46 for each local area 40 of the image 30 while changing the position of the local area 40. Then, the setting unit 20D sets the local area size used to estimate the number of objects 47 closest to the correct number of objects obtained as the local area size as the object number estimation unit of the image 30.

[0082] In addition, the setting unit 20D may receive the correct number of objects contained in the image 30 from the input unit 10E. In this case, the user may input the correct number of objects contained in the image 30 by operating the input unit 10E. In addition, the setting unit 20D may use the number of objects contained in the image 30 estimated by an external device or the like as the correct number of objects contained in the image 30 obtained in advance.

[0083] In addition, the setting unit 20D may set one local area size for one image 30, or may set a plurality of local area sizes having different sizes. When setting a plurality of local area sizes, for example, the user may select two or more estimation result images 48 (estimation results 46) from among the plurality of estimation result images 48 (estimation results 46) included in the output image 54. Then, the setting unit 20D may set the local area size for estimating each estimation result 46 of the plurality of estimation result images 48 received from the input unit 10E via the receiving unit 20F.

[0084] Furthermore, the setting unit 20D may set different local region sizes for different regions within the image 30 .

[0085] Figure 7 Schematic diagram showing an example of the output image 54 .

[0086] For example, the user selects a plurality of estimation result images 48 (estimation results 46) of different local region sizes by operating the input unit 10E while visually checking the displayed output image 54. At this time, the user selects a specific region in the estimation result image 48 by operating the input unit 10E.

[0087] exist Figure 7As an example, the estimated result image 48A corresponding to the local area size of 8×8 pixels is shown. Figure 7 The upper area SA in the image 48B corresponding to the local area size of 16×16 pixels is selected. Figure 7 The state in which the lower area SB in is selected.

[0088] In this case, the setting unit 20D sets a local area size of 8×8 pixels for deriving the estimation result 46A in the estimation result image 48A for the area SA in the image 30. Alternatively, the setting unit 20D may set a local area size of 16×16 pixels for deriving the estimation result 46B in the estimation result image 48B for the area SB in the image 30.

[0089] return Figure 1 The description will be continued. The estimation unit 20G restructures the reference learning model 50 into a learning model 52 for deriving the estimation result 46 of the local region 40 having the local region size set by the setting unit 20D.

[0090] Figure 8 is a schematic diagram showing an example of the reconstructed learning model 52. For example, it is assumed that the setting unit 20D sets the local area size to 16×16 pixels. In this case, the estimation unit 20G Figure 3 The reference learning model 50 shown is reconstructed into a learning model 52 that reduces the image 30 to a feature map 44B of 1 / 16 size by repeated convolution operations, thereby outputting an estimation result 46B.

[0091] Specifically, the estimation unit 20G reconstructs the network of the learning model 52 by deleting the feature map 44 and the estimation result 46 formed by the convolution operation (lower order) in the learning model 52 that is later than the set local region size. Through this process, the estimation unit 20G reconstructs the learning model 52 based on the reference learning model 50.

[0092] Then, the estimation unit 20G stores the reconstructed learning model 52 in the storage unit 20B in association with identification information uniquely identifying the range of the target region of the image 30 used to reconstruct the learning model 52. For example, identification information of the imaging unit 10B that captured the image 30 may be used as the identification information.

[0093] When the acquisition unit 20C acquires a new image 30 , the estimation unit 20G estimates the number of objects included in the image 30 for each local region 40 having a local region size set based on the image 30 .

[0094] That is, the estimation unit 20G reads the learning model 52 corresponding to the identification information of the range of the object area of ​​the image 30 acquired by the acquisition unit 20C from the storage unit 20B. Then, the estimation unit 20G estimates the number of objects contained in the image 30 using the image 30 and the learning model 52. As described above, the learning model 52 is a learning model obtained by reconstructing the reference learning model 50 based on the local area size set for the image 30. Therefore, the estimation unit 20G obtains the estimation result 46 of the number of objects for each local area 40 as the local area size set for the image 30 by inputting the image 30 into the read learning model 52. Then, the estimation unit 20G estimates the number of objects contained in the image 30 by performing the estimation process on the entire image 30 while changing the position of the local area 40 in the image 30.

[0095] The estimation unit 20G may further perform any of the following processes: displaying the estimation result 46 on the display 10D, storing it in the storage unit 20B, and transmitting it to an external device via the communication unit 10A.

[0096] In addition, as described above, the setting unit 20D may set a plurality of local region sizes for one image 30. In this case, the estimation unit 20G stores the plurality of learning models 52 reconstructed for each of the plurality of local region sizes in the storage unit 20B in correspondence with the identification information that uniquely identifies the range of the object region of the image 30. Therefore, in this case, the estimation unit 20G reads the plurality of learning models 52 corresponding to the identification information of the range of the object region of the image 30 newly acquired by the acquisition unit 20C from the storage unit 20B.

[0097] In this case, the estimation unit 20G inputs the image 30 to the plurality of learning models 52 that have been read, thereby obtaining a plurality of estimation results 46 as outputs from the learning models 52. Then, the estimation unit 20G may use the weighted sum of the plurality of estimation results 46 as the estimation result 46 for each local region 40 of the image 30.

[0098] Specifically, the estimation unit 20G multiplies the plurality of estimation results 46 for the local area derived from the plurality of learning models 52 by a predetermined weight coefficient for each local area 40, and uses the weighted sum of the sum of the results obtained by these multiplications as the estimation result 46 for the local area 40. The weight coefficient may be pre-set according to the size of each local area and stored in the image 30B in advance. Then, the estimation unit 20G may estimate the number of objects contained in the image 30 by performing the estimation process on the entire image 30 while changing the position of the local area 40 in the image 30.

[0099] In addition, as described above, the setting unit 20D may set local area sizes of different sizes to different areas in the image 30 (see Figure 7 ). In this case, the setting unit 20D corresponds to the identification information that uniquely identifies the range of the object area of ​​the image 30, and the area information representing the position and range in the image 30 and the learning model 52 reconstructed according to the local area size set for the area of ​​the area information are stored in the storage unit 20B.

[0100] Then, the estimation unit 20G may obtain the estimation result 46 of the number of objects in each local region 40 using the learning model 52 corresponding to the local region size set for each region in the image 30. Then, the estimation unit 20G may estimate the number of objects included in the image 30 by performing the estimation process on the entire image 30 while changing the position of the local region 40 in the image 30.

[0101] Next, an example of the flow of object number estimation processing executed by object number estimation device 20 according to the present embodiment will be described. Fig. 9 as well as Fig.10 This is a flowchart showing an example of the flow of object quantity estimation processing.

[0102] Fig. 9 This is a flowchart showing an example of a flow of generating the learning model 52 in the object quantity estimation process.

[0103] The processing unit 20A determines whether the range of the target area has been changed (step S100 ).

[0104] The change in the scope of the target area means that when a captured image is used as the image 30 for estimating the number of objects, at least one of the new settings or changes of the imaging unit 10B that captures the captured image or other imaging devices that capture the captured image, the new settings or changes of the imaging unit 10B or the other imaging devices themselves, the shooting conditions, the height of the setting position from the ground, the shooting angle, the shooting environment (brightness, etc.), the viewing angle, and the resolution is changed. In addition, when an image 30 other than a captured image is used as the image 30 for estimating the number of objects, the size of the object contained in the image 30 is changed, the new image 30 is acquired, etc. can be listed. In addition, in step S100, the processing unit 20A can determine that the scope of the target area has been changed when change information indicating the change is input by the user's instruction to the input unit 10E, etc.

[0105] If a negative determination is made in step S100 (step S100: No), this routine ends. If an affirmative determination is made in step S100 (step S100: Yes), the routine proceeds to step S102.

[0106] In step S102 , the acquisition unit 20C acquires the image 30 of the range determined as the change target area in step S100 (step S102 ).

[0107] The setting unit 20D sets a local area size as a unit for estimating the number of objects in the image 30 based on the image 30 acquired in step S102 (step S104 ).

[0108] Next, the estimation unit 20G reconstructs the reference learning model 50 stored in the storage unit 20B into a learning model 52 for deriving the estimation result 46 of the local area 40 of the local area size set in step S104 (step S106). Then, the estimation unit 20G stores the learning model 52 reconstructed in step S106 in the storage unit 20B in association with the identification information that uniquely identifies the range of the object area of ​​the image 30 acquired in step S102 (step S108). Then, this routine ends.

[0109] Fig.10 This is a flowchart showing an example of the flow of object quantity estimation in the object quantity estimation process.

[0110] The acquisition unit 20C acquires the image 30 (step S200 ). The setting unit 20D reads the learning model 52 corresponding to the identification information that uniquely identifies the range of the target region of the image 30 acquired in step S200 from the storage unit 20B (step S202 ).

[0111] For example, the setting unit 20D may analyze the acquired image 30 using a known image analysis method to determine the range of the object area of ​​the image 30. In addition, as the identification information, it is sufficient to use identification information including information indicating the range of the determined object area. In addition, it is possible to form a structure in which the identification information or information indicating the range of the object area of ​​the image 30 is included in the image 30. In this case, it is sufficient to form a structure in which the information is included in the header part of the image 30, etc. Then, when the image 30 is sent to the object quantity estimation device 20, the sending side of the image 30 (for example, the imaging unit 10B) sends the image 30 with the information attached. In this case, the setting unit 20D may read the learning model 52 corresponding to the identification information contained in the image 30 from the storage unit 20B.

[0112] Next, the estimating unit 20G inputs the image 30 acquired in step S200 to the learning model 52 read in step S202 to estimate the number of objects included in the image 30 for each local region 40 of the local region size set for the image 30 (step S204 ). Then, this routine ends.

[0113] In the process of step S202, sometimes the learning model 52 corresponding to the identification information uniquely identifying the range of the object area of ​​the image 30 acquired in step S200 is not stored in the storage unit 20B. In this case, the processing unit 20A is set to Fig. 9 If an affirmative determination is made in step S100 (step S100: Yes), the processing of steps S104 to S108 may be performed on the image 30. Then, the processing of steps S202 to S204 may be performed next.

[0114] As described above, the object number estimation device 20 of the present embodiment includes an acquisition unit 20C, a setting unit 20D, and an estimation unit 20G. The acquisition unit 20C acquires the image 30. The setting unit 20D sets the local area size as the unit for estimating the number of objects in the image 30 based on the image 30. The estimation unit 20G estimates the number of objects included in the image 30 for each local area 40 of the set local area size.

[0115] As described above, in object number estimation device 20 according to the present embodiment, the local area size serving as the unit for estimating the number of objects in image 30 is set based on image 30 .

[0116] That is, in the object number estimation device 20 of the present embodiment, even when the range of the target area reflected in the image 30 is changed and the size of the person 32 included in the image 30 changes, the local area size corresponding to the image 30 is set. Then, the object number estimation device 20 can estimate the number of objects included in the image 30 according to each set local area size.

[0117] Therefore, the object number estimating device 20 according to the present embodiment can estimate the number of objects with high accuracy.

[0118] Furthermore, the reference learning model 50 and the learning model 52 may be models that output the estimation result 46 including at least one of the position and density of the persons 32 in the image 30 in addition to the number of persons 32 included. In this case, in addition to the above-mentioned effects, the object number estimation device 20 of the present embodiment can estimate at least one of the density of the persons 32 included in the image 30 and the position of the persons 32 in the image 30 with high accuracy.

[0119] (Second Embodiment)

[0120] In the second embodiment, a case where a learning model obtained by learning using a specific teacher image is used as the reference learning model 50 will be described.

[0121] In addition, the same reference numerals are given to the same functions and configurations as those in the first embodiment, and detailed description thereof will be omitted.

[0122] Fig.11 It is a schematic diagram showing an example of the object quantity estimation system 11 according to the present embodiment.

[0123] The object quantity estimation system 11 includes an object quantity estimation device 21, a communication unit 10A, an imaging unit 10B, and a UI unit 10C. The object quantity estimation device 21 is connected to the communication unit 10A, the imaging unit 10B, and the UI unit 10C so as to be able to exchange data or signals.

[0124] Object number estimation device 21 includes storage unit 20B and processing unit 21A. Object number estimation device 21 is the same as object number estimation device 20 of the above-described embodiment except that it includes processing unit 21A instead of processing unit 20A.

[0125] The processing unit 21A includes an acquisition unit 20C, a setting unit 20D, an output control unit 21E, a reception unit 20F, an estimation unit 21G, and a learning unit 21H.

[0126] The learning unit 21H learns the reference learning model 50. The learning unit 21H uses teacher data representing the correspondence between a teacher image and the number of objects, and learns the reference learning model 50 for deriving the estimation result 46 from the teacher image, wherein the number of objects is the number of objects estimated from each reduced image obtained by reducing the teacher image to each of a plurality of local area sizes different from each other.

[0127] The teacher image is an image including position information of an object and distribution information indicating the existence probability distribution of the object. Fig.12 is a schematic diagram showing an example of the teacher image 60. Fig.12 , a teacher image 60A and a teacher image 60B are shown as examples of the teacher image 60 .

[0128] like Fig.12 As shown, the teacher image 60 includes position information 62 and distribution information 64 of the person 32. The position information 62 is information indicating the position of the person 32 contained in the teacher image 60 in the image 30. The distribution information 64 is information indicating the existence probability distribution of the person 32. Fig.12 In FIG. 1 , a Gaussian distribution is shown as an example of representing the distribution information 64. Fig.12 , an example in which distribution information 64 indicating the existence probability of a head as an example of a part for specifying the person 32 is expressed by Gaussian distribution is shown.

[0129] For example, the learning unit 21H stores in advance in the storage unit 20B the teacher data indicating the correspondence between the teacher image 60 and the number of objects estimated from each of the reduced images obtained by reducing the teacher image 60 to each of a plurality of local area sizes different from each other. In addition, the reduced image of the distribution information 64 corresponding to the teacher image 60 is an image reduced in such a way that the sum of the pixels included before and after the reduction does not change.

[0130] Then, the learning unit 21H uses the teacher data to learn the reference learning model 50 (see Figure 3 ).

[0131] The output control unit 21E and the estimation unit 21G may perform the same processing as that of the first embodiment, except that the reference learning model 50 learned by the learning unit 21H is used as the reference learning model 50 .

[0132] That is, the estimating unit 21G may reconstruct the reference learning model 50 obtained by the learning of the learning unit 21H into the learning model 52. Then, the estimating unit 21G may estimate the number of objects included in the image 30 using the reconstructed learning model 52 and the image 30.

[0133] Here, if Fig.12 As shown, the teacher image 60 includes distribution information 64 indicating the probability distribution of the person 32. Specifically, the person 32 of the teacher image 60 is blurred due to the uniform Gaussian distribution. Therefore, a difference is generated between the distribution information 64 blurred due to the Gaussian distribution and the actual image of the teacher image 60. It is generally believed that this difference sometimes has an adverse effect on the reference learning model 50.

[0134] However, in the present embodiment, the object number estimation device 21 estimates the number of objects of the person 32 included in the image 30 for each local area 40 of the local area size set according to the image 30. Therefore, the object number estimation device 21 of the present embodiment performs learning by using the reduced distribution information 64, thereby becoming less susceptible to the influence of blur. Therefore, in addition to the effects of the above-mentioned embodiment, the object number estimation device 21 of the present embodiment can learn the high-precision learning model 50 and reconstruct the high-precision learning model 52.

[0135] Next, an example of the hardware configuration of the object number estimation device 20 and the object number estimation device 21 according to the above-described embodiment will be described.

[0136] Fig.13This is an example of a hardware configuration diagram of the object number estimation device 20 and the object number estimation device 21 according to the above-mentioned embodiment.

[0137] The object quantity estimation device 20 and the object quantity estimation device 21 of the above-mentioned embodiment include: a control device such as a CPU 86; storage devices such as a ROM (Read Only Memory) 88, a RAM (Random Access Memory) 90, and an HDD (Hard Disk Drive) 92; an interface unit 82 as an interface with various devices; an output unit 80 that outputs various information such as output information; an input unit 94 that accepts operations performed by a user; and a bus 96 that connects the various units, forming a hardware structure utilizing a conventional computer.

[0138] In the object number estimation device 20 and the object number estimation device 21 of the above-described embodiment, the CPU 86 reads the program from the ROM 88 to the RAM 90 and executes the program, whereby the above-described components are realized on the computer.

[0139] In addition, the program for executing the above-mentioned processing executed in the object number estimation device 20 and the object number estimation device 21 of the above-mentioned embodiment may be stored in the HDD 92. In addition, the program for executing the above-mentioned processing executed in the object number estimation device 20 and the object number estimation device 21 of the above-mentioned embodiment may be provided by being pre-embedded in the ROM 88.

[0140] In addition, the program for executing the above-mentioned processing executed in the object number estimation device 20 and the object number estimation device 21 of the above-mentioned embodiment can be stored in a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), floppy disk (FD) in an installable or executable form and provided as a computer program product. In addition, the program for executing the above-mentioned processing executed in the object number estimation device 20 and the object number estimation device 21 of the above-mentioned embodiment can be stored in a computer connected to a network such as the Internet and provided by downloading via the network. In addition, the program for executing the above-mentioned processing executed in the object number estimation device 20 and the object number estimation device 21 of the above-mentioned embodiment can be provided or distributed via a network such as the Internet.

[0141] In addition, the above describes the embodiments of the present invention, but the above embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. The embodiments and their variations are included in the scope and purpose of the invention, and are included in the invention recorded in the patent claims and the scope of their equivalents.

[0142] Furthermore, the above-mentioned embodiments can be summarized as the following technical solutions.

[0143] Technical Solution 1

[0144] An object quantity estimation device, comprising:

[0145] An acquisition unit, for acquiring an image;

[0146] a setting unit that sets a local area size as a unit for estimating the number of objects in the image based on the image; and

[0147] The estimating unit estimates the number of objects included in the image for each local area of ​​the set local area size.

[0148] Technical Solution 2

[0149] According to the object quantity estimation device described in technical solution 1,

[0150] The estimating unit estimates the number of objects included in the image using a learning model for deriving an estimation result of the number of objects in a region of the local region size in the image by repeating a convolution operation on the image, the image, and the set local region size.

[0151] Technical Solution 3

[0152] According to the object quantity estimation device described in technical solution 2,

[0153] The estimation unit reconstructs a baseline learning model into the learning model, and estimates the number of objects contained in the image using the reconstructed learning model and the image, wherein the baseline learning model is used to derive the estimation results of each of the local areas of a plurality of different local area sizes from the image by repeatedly performing a convolution operation on the image, and the learning model is used to derive the estimation results of the local area of ​​the set local area size.

[0154] Technical Solution 4

[0155] According to the object quantity estimation device described in technical solution 3,

[0156] A learning unit is provided, wherein the learning unit uses teacher data representing the correspondence between a teacher image and the number of objects to learn the reference learning model for deriving the estimation result from the teacher image, wherein the teacher image includes position information of the object and distribution information representing the probability distribution of the existence of the object, and the number of objects is the number of objects estimated from each reduced image obtained by reducing the teacher image to each of the plurality of local area sizes that are different from each other,

[0157] The estimating unit reconstructs the reference learning model into the learning model, and estimates the number of objects included in the image using the reconstructed learning model and the image, wherein the learning model is used to derive the estimation result estimated from the reduced image of the set local area size.

[0158] Technical Solution 5

[0159] The object quantity estimation device according to any one of technical solutions 2 to 4 comprises:

[0160] an output control unit that outputs an output image including a list of the estimation results obtained by estimating the number of objects included in the image for each of the plurality of local area sizes different from each other; and

[0161] a receiving unit for receiving a user's selection of the estimation result included in the output image,

[0162] The setting unit sets the local area size corresponding to the received estimation result.

[0163] Technical Solution 6

[0164] According to the object quantity estimation device described in any one of technical solutions 2 to 5,

[0165] The setting unit sets the local area size corresponding to the estimation result closest to the previously acquired correct number of objects contained in the image among the estimation results obtained by estimating the number of objects contained in the image for each local area of ​​a plurality of local area sizes different in size.

[0166] Technical Solution 7

[0167] According to the object quantity estimation device described in any one of technical solutions 2 to 6,

[0168] The setting unit sets a plurality of local area sizes having different sizes from each other,

[0169] The estimating unit estimates, as the estimation result, a weighted sum of results of estimating the number of objects included in the image for each of the plurality of local region sizes that are set.

[0170] Technical Solution 8

[0171] The object quantity estimation device according to any one of claims 1 to 7, wherein:

[0172] The setting unit sets different local area sizes for different areas in the image.

[0173] The estimating unit estimates, for each of the regions in the image, the number of objects in each local region of the local region size set for the region.

[0174] Technical Solution 9

[0175] A method for estimating the number of objects is a method for estimating the number of objects executed by a computer, and the method for estimating the number of objects comprises:

[0176] Steps to obtain the image;

[0177] A step of setting a local area size as a unit for estimating the number of objects in the image based on the image; and

[0178] The step of estimating the number of objects included in the image for each local area of ​​the set local area size.

[0179] Technical Solution 10

[0180] A storage medium storing an object quantity estimation program, the object quantity estimation program is used to cause a computer to execute the following steps:

[0181] Steps to obtain the image;

[0182] A step of setting a local area size as a unit for estimating the number of objects in the image based on the image; and

[0183] The step of estimating the number of objects included in the image for each local area of ​​the set local area size.

Claims

1. An object quantity estimation device comprising: An acquisition unit, for acquiring an image; a setting unit that sets a local area size as a unit for estimating the number of objects in the image based on the image; an estimating unit, for estimating the number of objects contained in the image according to each local area of ​​the set local area size; an output control unit that outputs an output image including a list of estimation results obtained by estimating the number of objects included in the image for each of the plurality of local area sizes different from each other; and a receiving unit for receiving a user's selection of the estimation result included in the output image, The estimating unit estimates the number of objects contained in the image using a learning model, the image, and the set local region size, wherein the learning model is a learning model for deriving the estimation result of the number of objects in a region of the local region size in the image by repeating a convolution operation on the image, The setting unit sets the local area size corresponding to the received estimation result.

2. The object quantity estimation device according to claim 1, wherein: The estimation unit reconstructs a baseline learning model into the learning model, and estimates the number of objects contained in the image using the reconstructed learning model and the image, wherein the baseline learning model is used to derive the estimation results of each of the local areas of a plurality of different local area sizes from the image by repeatedly performing a convolution operation on the image, and the learning model is used to derive the estimation results of the local area of ​​the set local area size.

3. The object quantity estimation device according to claim 2, wherein: A learning unit is provided, wherein the learning unit uses teacher data representing the correspondence between a teacher image and the number of objects to learn the reference learning model for deriving the estimation result from the teacher image, wherein the teacher image includes position information of the object and distribution information representing the probability distribution of the existence of the object, and the number of objects is the number of objects estimated from each reduced image obtained by reducing the teacher image to each of the plurality of local area sizes that are different from each other, The estimating unit reconstructs the reference learning model into the learning model, and estimates the number of objects included in the image using the reconstructed learning model and the image, wherein the learning model is used to derive the estimation result estimated from the reduced image of the set local area size.

4. An object quantity estimation device comprising: An acquisition unit, for acquiring an image; a setting unit that sets a local area size as a unit for estimating the number of objects in the image based on the image; an estimating unit, for estimating the number of objects contained in the image according to each local area of ​​the set local area size; an output control unit that outputs an output image including a list of estimation results obtained by estimating the number of objects included in the image for each of the plurality of local area sizes different from each other; and a receiving unit for receiving a user's selection of the estimation result included in the output image, The estimating unit estimates the number of objects contained in the image using a learning model, the image, and the set local region size, wherein the learning model is a learning model for deriving the estimation result of the number of objects in a region of the local region size in the image by repeating a convolution operation on the image, The setting unit sets the local area size corresponding to the estimation result closest to the correct number of objects contained in the image acquired in advance, among the estimation results obtained by estimating the number of objects contained in the image for each local area of ​​a plurality of local area sizes having different sizes.

5. The object quantity estimation device according to claim 1 or 4, wherein: The setting unit sets a plurality of local area sizes having different sizes from each other, The estimating unit estimates, as the estimation result, a weighted sum of results of estimating the number of objects included in the image for each of the plurality of local region sizes that are set.

6. The object quantity estimation device according to claim 1 or 4, wherein: The setting unit sets different local area sizes for different areas in the image. The estimating unit estimates, for each of the regions in the image, the number of objects in each local region of the local region size set for the region.

7. A method for estimating the number of objects, which is a method for estimating the number of objects executed by a computer, the method for estimating the number of objects comprising: an acquisition step of acquiring an image; a step of setting a local area size as a unit for estimating the number of objects in the image based on the image; A step of estimating the number of objects contained in the image according to each local area of ​​the set local area size; Outputting an output image including a list of estimation results obtained by estimating the number of objects included in the image for each of the plurality of local area sizes different from each other; as well as a step of accepting a user's selection of the estimation result included in the output image, In the estimating step, the number of objects contained in the image is estimated using a learning model, the image, and the set local area size, wherein the learning model is a learning model for deriving the estimation result of the number of objects in an area of ​​the local area size in the image by repeating a convolution operation on the image, In the setting step, the local area size is set corresponding to the received estimation result.

8. A program product for estimating the number of objects, configured to cause a computer to execute the following steps: an acquisition step of acquiring an image; a step of setting a local area size as a unit for estimating the number of objects in the image based on the image; A step of estimating the number of objects contained in the image according to each local area of ​​the set local area size; Outputting an output image including a list of estimation results obtained by estimating the number of objects included in the image for each of the plurality of local area sizes having different sizes; and The process of accepting a user's selection of the estimation result included in the output image comprises: In the estimating step, the number of objects contained in the image is estimated using a learning model, the image, and the set local area size, wherein the learning model is a learning model for deriving the estimation result of the number of objects in an area of ​​the local area size in the image by repeating a convolution operation on the image, In the setting step, the local area size is set corresponding to the received estimation result.

Citation Information

Patent Citations

  • Information processing system, information processing device, and program

    JP2019161610A

  • Image recognition method

    JP2017059207A

  • Information processing apparatus, information processing method, and computer-readable storage medium, for estimating state of objects

    US20180285656A1