Detection device, control method, detection model generation method, and recording medium

Abstract

The detection device, the control method, the model generation method for detection, and the recording medium of the present application can detect a person from an image captured using a fisheye lens at high speed and with high precision. A detection device (10) detects a person from a captured image (PI) using a learned model (140) constructed through multiclass learning.

Description

TECHNICAL FIELD

[0001] The present application relates to a detection device or the like that detects a person photographed from a photographed image photographed by a ceiling camera using a fisheye lens, and particularly to a detection device, a control method, a model generation method for detection, and a recording medium. BACKGROUND

[0002] Conventionally, various researches have been known that analyze an image photographed using a fisheye lens to detect a person. For example, an image sensor is disclosed in Patent Literature 1 described below, which detects a person from an image photographed over a wide range with high accuracy by using a plurality of dictionary information.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Laid-Open Patent Publication No. 2016-171526 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, the conventional technology described above has the following problem: since a plurality of dictionaries representing features of a person are used to detect a person from an image, the time required until a person is detected becomes longer than when the dictionary representing the features of the person is one.

[0008] An object of one aspect of the present application is to detect a person from an image photographed using a fisheye lens at high speed and with high accuracy.

[0009] TECHNICAL MEANS FOR SOLVING THE PROBLEMS

[0010] To solve the problem, a detection device of an embodiment of the present application detects a person photographed from a photographed image photographed by a ceiling camera using a fisheye lens, the detection device including: a division section that divides the photographed image into a plurality of regions; a region estimation section that calculates, for each of the plurality of regions, a probability that a position where a detection target estimated to be a person is located is included; and a determination section that determines whether the detection target is a person using the probability of each of the plurality of regions.

[0011] To solve the problem, a control method of an embodiment of the present application is a control method of a detection device that detects a person photographed from a photographed image photographed by a ceiling camera using a fisheye lens, the control method including: a division step of dividing the photographed image into a plurality of regions; a region estimation step of calculating, for each of the plurality of regions, a probability that a position where a detection target estimated to be a person is located is included; and a determination step of determining whether the detection target is a person using the probability of each of the plurality of regions.

[0012] To solve the problem, a model generation method of an embodiment of the present application is a model generation method performed by a model generation device that generates a learned model, the model generation method including: a step of acquiring teaching data, the teaching data being given, as a label, information indicating a position, shape, and size of a bounding box that encloses a person photographed in a photographed image, and information identifying a region that "contains a position at which the person photographed in the photographed image is located," for the photographed image that is divided into a plurality of regions; and a step of constructing a learned model by machine learning with respect to the teaching data, the learned model being a model that takes the photographed image as input and outputs information indicating a position, shape, and size of the bounding box and information indicating, for each of the plurality of regions, a probability that "contains a position at which the person photographed in the photographed image is located."

[0013] Effects of Invention

[0014] According to an embodiment of the present application, the following effects are achieved: a person can be detected at high speed and with high accuracy from an image photographed using a fisheye lens. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a block diagram showing the main part structure of the detection device of Embodiment 1 of the present application.

[0016] Figure 2 is a diagram showing the overall outline of a person detection system of the detection device including Figure 1

[0017] Figure 3 is a diagram showing an example in which a false detection occurs in a case where a person is to be detected from a photographed image.

[0018] Figure 4 is a flowchart illustrating an example of a generation process of a learned model used by the detection device of Figure 1

[0019] Figure 5 is a diagram showing an example of a photographed image that is divided into a plurality of regions.

[0020] Figure 6 is a diagram showing an example of teaching data learned in the generation process of Figure 4

[0021] Figure 7 is a flowchart illustrating an example of a person detection process in which a first determination process is performed on the output of the learned model generated in Figure 4 ​​​​

[0022] Figure 8 is a graph showing an example of a class in the output of the learned model generated in Figure 4

[0023] Figure 9 is a graph showing an example of a specific region determined by the detection device of Figure 1

[0024] Figure 10 is a graph showing an example in which the detection device of Figure 1 performs the first determination processing to determine the estimation result as correct.

[0025] Figure 11 is a graph showing an example in which the detection device of Figure 1 performs the first determination processing to determine the estimation result as incorrect.

[0026] Figure 12 is a graph showing an example in which a person is located near a boundary line of a plurality of regions.

[0027] Figure 13 is a graph showing an example in which the detection device of Figure 1 performs the first determination processing to determine the estimation result as correct even in a case where a person is located near a boundary line of a plurality of regions.

[0028] Figure 14 is a flowchart illustrating an example of a person detection processing in which the second determination processing is performed on the output of the learned model generated in Figure 4

[0029] is a graph showing an example in which the detection device of Figure 15 performs the second determination processing to determine the estimation result as correct. Figure 1

[0030] is a graph showing an example in which the detection device of Figure 16 performs the second determination processing to determine the estimation result as incorrect. Figure 1

[0031] is a graph showing an example in which the detection device of Figure 17 performs the second determination processing to determine the estimation result as correct even in a case where a position where a detection target is located is located near a boundary line of a plurality of regions. Figure 1

[0032] is a graph showing an example in which the detection device of Figure 18 sets a buffer region near a boundary line of a plurality of regions. Figure 1

[0033] Figure 19 ​​​is a flowchart illustrating an example of human detection processing that detects a human by executing the second determination processing while a buffer area is set.

[0034] Explanation of Symbols

[0035] 10: detection device

[0036] 20: ceiling camera

[0037] 120: division unit

[0038] 140: learned model (region prediction model)

[0039] 142: region prediction model

[0040] 150: estimation unit (region estimation unit)

[0041] 152: region estimation unit

[0042] 170: determination unit

[0043] 190: determination unit

[0044] 220: learning unit

[0045] 222: region learning unit (learning unit)

[0046] AR: region

[0047] BB: detection rectangle (bounding box)

[0048] CR: class reliability value

[0049] DT: teaching data

[0050] IA: specific region

[0051] OB: detection object

[0052] OR: object reliability

[0053] PI: photographed image

[0054] PR: probability

[0055] PS: position

[0056] S120: acquisition step

[0057] S130: learning step

[0058] S210: division step

[0059] S220: region estimation step

[0060] S250: determination step

[0061] S310: Segmentation Steps

[0062] S320: Regional Inference Steps

[0063] S350: Judgment Step

[0064] S410: Segmentation Steps

[0065] S420: Area Inference Steps

[0066] S470: Judgment Step

[0067] TH: Specified value Detailed Implementation

[0068] [Implementation Method 1]

[0069] The following is based on Figures 1 to 19 This description will focus on one aspect of an embodiment of the present invention (hereinafter also referred to as "this embodiment"). Furthermore, identical or equivalent parts will be labeled with the same symbols in the figures and their descriptions will not be repeated. In this embodiment, the detection device 10 will be described as a typical example of a detection device that detects a person captured from an image PI captured by a ceiling camera 20 using a fisheye lens.

[0070] In the following explanation, let “m”, “n”, “p”, “q”, “x”, and “y” each represent an integer greater than or equal to “1”. Furthermore, let “m” be an integer less than or equal to “n”, “p” and “q” be distinct integers, and “x” and “y” be distinct integers.

[0071] Furthermore, when it is necessary to distinguish between multiple regions AR, suffixes such as "(1)", "(2)", "(3)", ..., "(n)" are added to the symbols for differentiation. For example, they are recorded as "Region AR(1)", "Region AR(2)", "Region AR(3)", ..., "Region AR(n)" for differentiation. When it is not necessary to distinguish between the individual regions AR, they are simply referred to as "Region AR". The same applies to the detection rectangle BB, the detection object OB, the position PS, etc.

[0072] To facilitate understanding of one aspect of the detection device 10 of the present invention, firstly using Figure 2 To illustrate the outline of the human detection system 1 including the detection device 10.

[0073] §1. Applicable Examples

[0074] (Overall overview of the human detection system)

[0075] Figure 2 This is a diagram showing an overall overview of the human detection system 1, including the detection device 10. (See diagram for example.) Figure 2As shown, the person detection system 1 includes a ceiling camera 20 that generates a captured image PI, and a detection device 10 that performs image analysis on the captured image PI generated by the ceiling camera 20 to detect a person captured in the captured image PI. The ceiling camera 20 and the detection device 10 are communicably connected to each other, for example, via a communication cable that is a Universal Serial Bus (USB) cable.

[0076] The ceiling camera 20 is a capturing device capable of capturing a wide range of a capturing subject space using a fisheye lens (ultra-wide-angle lens). The ceiling camera 20 is provided, for example, on the ceiling of the factory Fa, and generates a captured image PI that captures a plurality of work sites Ar in the factory Fa in a form of overlooking from above (obliquely above). The ceiling camera 20 outputs the generated captured image PI to the detection device 10.

[0077] The detection device 10 acquires the captured image PI generated by the ceiling camera 20 from the ceiling camera 20. The detection device 10 performs image analysis on the acquired captured image PI to detect a person captured in the captured image PI.

[0078] (Necessity of detecting a person with high accuracy)

[0079] In order to perform process improvement of a work process performed in the factory Fa, it is required to detect a worker (person) located in the factory Fa with high accuracy from the captured image PI that captures the factory Fa. However, it is not easy to detect a person (human body) captured in the captured image PI with high accuracy by image analysis of the captured image PI captured by the ceiling camera 20 or the like.

[0080] (Occurrence of false detection)

[0081] Figure 3 is a diagram that shows an example of occurrence of false detection in a case where a person is to be detected from the captured image PI captured by the ceiling camera 20. In image analysis on the captured image PI using a learned model, for example, a detection rectangle BB (also referred to as a "Bounding Box") is arranged in a manner of surrounding a detection object OB that is presumed to be a "person (human body)" around the detection object OB.

[0082] In Figure 3 In the illustrated captured image PI, two detection rectangles BB that surround detection objects OB presumed to be persons are set, specifically, a detection rectangle BB(1) and a detection rectangle BB(2) are set.

[0083] In Figure 3In the illustrated captured image PI, the detection object OB surrounded by the detection rectangle BB (1) in the detection rectangle BB that surrounds the detection object OB presumed to be a person is actually also a "person". However, the detection object OB surrounded by the detection rectangle BB (2) is not actually a "person".

[0084] As Figure 3 As illustrated, it is not easy to accurately detect (presume) a "person photographed in the captured image PI" from the captured image PI even when using the learned model.

[0085] (Suppression of false detection)

[0086] Therefore, the detection device 10 uses the learned model 140 constructed through multiclass learning when detecting a person (human body) through image analysis of the captured image PI, thereby reducing false detection of a person. That is, the detection device 10 divides the captured image PI into a plurality of regions AR, and presumes a "region AR that contains a position PS (for example, a position under the feet of the "person", a position under the feet) of the detection object OB presumed to be a person" as the class of the detection object OB. Further, the detection device 10 uses the class of the detection object OB presumed to verify whether the presumption that the detection object OB is a person is correct or not.

[0087] That is, the detection device 10 learns, in addition to the rectangle information given as a label to the captured image PI in the teaching data DT, the "kind (class) of the person photographed in the captured image PI" given as a label to the captured image PI, to construct the learned model 140.

[0088] Specifically, in the teaching data DT learned by the detection device 10, the captured image PI is given rectangle information as a label, which determines the position / shape / size of the detection rectangle BB that surrounds the person photographed in the captured image PI.

[0089] Further, the detection device 10 divides the captured image PI into a plurality of regions AR. Further, in the teaching data DT learned by the detection device 10, the captured image PI is given information (class) that determines the region AR to which the position PS (for example, a position under the feet of the "person", a position under the feet) of the person photographed in the captured image PI belongs, as a label. The detection device 10 learns the recognition information of the region AR to which the position PS of the person photographed in the captured image PI belongs, in other words, the recognition information of the region AR that contains the position PS of the person photographed in the captured image PI, as the class of the person.

[0090] The detection device 10 performs learning (machine learning) on a set, that is, a data set DS, of "teaching data DT to which rectangular information and a category are assigned as labels for a captured image PI" and constructs a learned model 140.

[0091] Details will be described later, but for example, the captured image PI is divided into a plurality of regions AR by at least one of a plurality of straight lines (a part of the plurality of straight lines) intersecting each other at a prescribed angle at a substantially center of the captured image PI and a distance from the substantially center.

[0092] The detection device 10 performs inference using the learned model 140 constructed by learning the category of the detected person in the captured image PI, and thereby infers the detection rectangle BB that encloses the person and the region AR (category) to which the position PS of the person belongs from the captured image PI.

[0093] The multi-category learning performed by the detection device 10 refers to learning in which, in addition to learning of "where to place the detection rectangle BB" (rectangular information), the category of the detection object OB (object), that is, the region AR "containing the position PS of the detection object OB" is learned. The detection device 10 uses the learned model 140 constructed by the multi-category learning, and thereby, in addition to inference of "where to place the detection rectangle BB", infers the category of the detection object OB, that is, infers the region AR "to which the position PS of the detection object OB belongs".

[0094] The detection device 10 uses the inferred category of the detection object OB to verify whether or not the inference that "the detection object OB is a person" is correct, and thereby detects (infers) a person in the captured image PI with high accuracy from the captured image PI.

[0095] In particular, the detection device 10 determines the region AR "in which the detection rectangle BB placed by the learned model 140 for the captured image PI" as a specific region IA. Also, the detection device 10 uses the inferred category of the detection object OB and the determined specific region IA to verify whether or not the inference that "the detection object OB enclosed by the detection rectangle BB is a person" is correct.

[0096] For example, the detection device 10 compares the consistency of the category output from the learned model 140 and the position of the detection rectangle BB (more accurately, the region AR in which the detection rectangle BB is placed), and if they are different, the detection rectangle BB is regarded as a false detection and is removed.

[0097] Also, for example, the detection device 10 uses the category output from the learned model 140, the region AR in which the detection rectangle BB is placed, and the object reliability OR of the detection rectangle BB to determine whether or not the detection rectangle BB is correct.

[0098] The detection device 10 described so far is summarized as follows. That is, the detection device 10 is a detection device that detects a person photographed in a photographed image PI from the photographed image PI photographed by the ceiling camera 20 using a fisheye lens. The detection device 10 includes a division section 120, a region estimation section 152 (estimation section 150), and a determination section 190. Figure 2 Figure 3 The division section 120 divides the photographed image PI into a plurality of regions AR. The estimation section 150 (for example, the region estimation section 152) calculates a probability PR that a position PS in which a detection object OB estimated to be a "person" is present is included with respect to each of the plurality of regions AR. The determination section 190 determines whether the detection object OB is a person using the probability PR of each of the plurality of regions AR.

[0099] According to the above-described structure, the detection device 10 divides the photographed image PI into a plurality of regions AR, and calculates a probability PR that a position PS in which a detection object OB estimated to be a "person" is present is included with respect to each of the plurality of regions AR. Furthermore, the detection device 10 verifies the estimation result that the detection object OB is a person using the probability PR of each of the plurality of regions AR. The probability PR is a probability that the region AR includes the position PS in which the detection object OB estimated to be a "person" is present.

[0100] According to the above-described structure, the detection device 10 divides the photographed image PI into a plurality of regions AR, and calculates a probability PR that a position PS in which a detection object OB estimated to be a "person" is present is included with respect to each of the plurality of regions AR. Furthermore, the detection device 10 verifies the estimation result that the detection object OB is a person using the probability PR of each of the plurality of regions AR. The probability PR is a probability that the region AR includes the position PS in which the detection object OB estimated to be a "person" is present.

[0101] Here, generally, in a case where a person is to be detected from an image using a dictionary representing a feature of a person, analysis of the image is required for each dictionary. Therefore, in a case where a plurality of dictionaries representing a feature of a person are to be used to improve the detection accuracy of a person, analysis of the image is also required a plurality of times, and the time required to detect a person becomes long.

[0102] In contrast to this, the detection device 10 determines whether a detection object OB is a person using the probability PR that a position PS in which the detection object OB estimated to be a person is present is included, which is calculated with respect to each of a plurality of regions AR into which a photographed image PI is divided, thereby improving the detection accuracy of a person. That is, the detection device 10 does not detect a person from a photographed image PI using a plurality of dictionaries representing a feature of a person, but determines whether a detection object OB estimated to be a person is actually a person using the probability PR of each of a plurality of regions AR, thereby improving the detection accuracy of a person.

[0103] A method of using a plurality of dictionaries representing a feature of a person is a method of improving the accuracy of estimation itself, and in contrast to this, the detection device 10 improves the detection accuracy by verifying the estimated result (estimation that the detection object OB is a person) (that is, removing an erroneous estimation result).

[0104] ​Therefore, the detection device 10 does not need to use a plurality of dictionaries representing features of a person when detecting a person from the captured image PI, and can shorten the time required to detect a person from the captured image PI compared to a case where a plurality of dictionaries are used to detect a person from the captured image PI.

[0105] Further, the detection device 10 uses the probability PR of each of the plurality of regions AR to determine (verify) whether the detection object OB presumed to be a person is actually a person, and thus can detect a person from the captured image PI with high accuracy.

[0106] Therefore, the detection device 10 has an effect that a person can be detected from an image (captured image PI) captured using a fisheye lens at high speed and with high accuracy.

[0107] Further, as described above, the detection device 10 does not need to use a plurality of dictionaries representing features of a person when detecting a person from the captured image PI.

[0108] Therefore, the detection device 10 has an effect that the effort required to prepare a dictionary (e.g., a learned model) required to detect a person from an image captured using a fisheye lens can be reduced, and the capacity of a memory required to store the dictionary can also be reduced.

[0109] The detection device 10 further includes a determination section 170 that determines a region AR in which the detection rectangle BB that surrounds the detection object OB is located, among the plurality of regions AR, as a specific region IA. Further, the determination section 190 uses the probability PR of each of the plurality of regions AR and the specific region IA to determine "whether the detection object OB is a person".

[0110] According to the configuration, the detection device 10 determines the specific region IA that is a region AR in which the detection rectangle BB is located, among the plurality of regions AR, and uses the probability PR of each of the plurality of regions AR and the specific region IA to determine "whether the detection object OB is a person".

[0111] For example, it can be considered that, in a case where the detection object OB presumed to be a person is actually a person, the consistency of the probability PR of each of the plurality of regions AR and the region AR in which the detection rectangle BB that surrounds the detection object OB is located (i.e., the specific region IA) is also high.

[0112] Therefore, the detection device 10 uses the probability PR of each of the plurality of regions AR and the specific region IA to determine "whether the detection object OB is a person", i.e., to verify the presumption that "the detection object OB is a person".

[0113] Therefore, the detection device 10 has an effect that a person can be detected from an image (captured image PI) captured using a fisheye lens with high accuracy.

[0114] The determination unit 170 can also determine, as the specific region IA, a region AR that includes the position PS of the detection object OB presumed to be "a person" (for example, a position equivalent to the position of the feet of the detection object OB presumed to be "a person") surrounded by the detected rectangle BB or the position PS of the detection object OB. The determination unit 170 can also calculate the position PS (feet position) of the detection object OB from the detection rectangle BB, for example, can calculate the position PS (feet position) of the detection object OB from the position, shape, and size of the detection rectangle BB. Furthermore, the determination unit 170 can also determine the center position (or the center of gravity position) of the detection rectangle BB as the position PS (feet position) of the detection object OB presumed to be "a person" surrounded by the detected rectangle BB.

[0115] In the detection device 10, the region presumption unit 152 (presumption unit 150) calculates the probability PR of each of the plurality of regions AR from the captured image PI using the region prediction model 142 (learned model 140). The region prediction model 142 (learned model 140) is a learned model that takes the captured image PI as input and outputs the probability PR of each of the plurality of regions AR including "a position PS at which a detection object OB surrounded by a detected rectangle BB is present".

[0116] According to the structure, the detection device 10 calculates the probability PR of each of the plurality of regions AR from the captured image PI using the region prediction model 142 (learned model 140). Thus, the detection device 10 achieves the effect of being able to calculate the probability PR of each of the plurality of regions AR from the captured image PI with high accuracy using the region prediction model 142 (learned model 140).

[0117] The detection device 10 can include a learning unit 220 that constructs the region prediction model 142 (learned model 140) by machine learning with respect to teaching data DT. That is, the learning unit 220 performs machine learning with respect to the captured image PI for teaching data DT to which "information indicating "a region AR including a position PS at which a person photographed in the captured image PI is present" (that is, a class)" is given as a label.

[0118] According to the structure, the detection device 10 constructs the region prediction model 142 (learned model 140) by machine learning with respect to teaching data DT to which "information indicating a region AR including a position PS at which a person is present" is given as a label for the captured image PI.

[0119] Thus, the detection device 10 achieves the effect of being able to construct the region prediction model 142 (learned model 140) that can calculate the probability PR of each of the plurality of regions AR from the captured image PI with high accuracy by machine learning with respect to the teaching data DT. Thus, the detection device 10 achieves the effect of being able to construct the region prediction model 142 (learned model 140) that can calculate the probability PR of each of the plurality of regions AR from the captured image PI with high accuracy by machine learning with respect to the teaching data DT.

[0120] §2. Structural Examples

[0121] Up to this point, using Figure 2 and Figure 3 This section provides an overview of the human detection system 1. Next, while referring to... Figure 1 The detailed structure of the detection device 10 is explained at the same time.

[0122] (Detailed structure of the detection device)

[0123] Figure 1 This is a block diagram showing the main structural components of the detection device 10. For example... Figure 1 As shown, the detection device 10 includes a storage unit 100, an image acquisition unit 110, a segmentation unit 120, an inference unit 150, a determination unit 170, a judgment unit 190, a teaching data generation unit 210, and a learning unit 220 as functional blocks.

[0124] In addition to the functional blocks described above, the detection device 10 may also include a display unit and a communication unit. The display unit displays the results of human detection processing, captured images (PI), etc., and the communication unit outputs these as data to an external device. However, for the sake of brevity, structures not directly related to this embodiment are omitted from the description and block diagrams. However, depending on the actual implementation, the detection device 10 may also include the omitted structures.

[0125] The functional blocks included in the detection device 10 can be implemented, for example, by having the arithmetic unit read a program stored in a storage device (storage unit 100) implemented using read-only memory (ROM), non-volatile random access memory (NVRAM), etc., and execute it in a random access memory (RAM) (not shown). Examples of components that can be used as arithmetic units include, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MPU), a floating-point number processing unit (FPU), a physical processing unit (PPU), a microcontroller, or combinations thereof.

[0126] Hereinafter, details will be provided for each of the image acquisition unit 110, segmentation unit 120, estimation unit 150, determination unit 170, judgment unit 190, teaching data generation unit 210, and learning unit 220.

[0127] (Details of functional blocks other than the storage department)

[0128] The image acquisition unit 110 acquires the captured image PI from the ceiling camera 20. The image acquisition unit 110 outputs the captured image PI from the ceiling camera 20 to the segmentation unit 120.

[0129] The segmentation unit 120 acquires a captured image PI from the image acquisition unit 110 and segments the acquired captured image PI into multiple regions AR using a predetermined segmentation method. The predetermined segmentation method can be preset by the user, and the user can also update (change) the preset method. For example, the user-preset method is stored in the storage unit 100. The segmentation unit 120 saves the captured image PI segmented into multiple regions AR as segmentation image information 130 in the storage unit 100.

[0130] The inference unit 150 performs image analysis using the learned model 140 on the captured image PI (i.e., the captured image PI stored in the segmentation image information 130) that has been segmented into multiple regions AR by the segmentation unit 120. The inference unit 150 can also output rectangle information and category from captured images PI that have not been segmented into multiple regions AR by the segmentation unit 120, using the learned model 140. In other words, the captured image PI input to the learned model 140 by the inference unit 150 may or may not have been segmented into multiple regions AR by the segmentation unit 120.

[0131] exist Figure 1 In the example shown, the inference unit 150 includes a human inference unit 151 and a region inference unit 152. The human inference unit 151 uses the learned model 140 (especially the human prediction model 141) to output rectangle information from the captured image PI. The region inference unit 152 uses the learned model 140 (especially the region prediction model 142) to output the category of the detected object OB surrounded by the detected rectangle BB from the captured image PI (specifically, the probability PR of multiple regions AR calculated for each detected rectangle BB).

[0132] Figure 1In the present embodiment, the person estimation unit 151 and the region estimation unit 152 are described separately for the sake of convenience in understanding the detection device 10, but the estimation unit 150 including the person estimation unit 151 and the region estimation unit 152 can also be implemented by one neural network. In other words, the functions of the person estimation unit 151 and the region estimation unit 152 can also be implemented by one neural network. Furthermore, the neural network implementing the function of the person estimation unit 151 and the neural network implementing the function of the region estimation unit 152 can also be separated.

[0133] In the following description, an example in which the estimation unit 150 is implemented as one neural network will be described. The estimation unit 150 can also be implemented as, for example, a convolutional neural network (CNN), a fast region-based convolutional network (Fast R-CNN), or a deep neural network (DNN).

[0134] The estimation unit 150 implemented as a neural network outputs the rectangular information and the class from the captured image PI using the learned model 140. Furthermore, the estimation unit 150 outputs the object reliability OR for each detection rectangle BB, the object reliability OR being information indicating the magnitude of the likelihood that some object is enclosed by the detection rectangle BB.

[0135] The object reliability OR is information indicating the likelihood that the detection rectangle BB contains some object (detection target OB) or that the detection rectangle BB contains only a background, and is information indicating the magnitude of the likelihood that the detection rectangle BB contains some object by the value thereof. That is, the object reliability OR is information indicating the magnitude of the likelihood that the detection rectangle BB encloses some detection target OB (the likelihood that some object is enclosed by the detection rectangle BB).

[0136] First, the estimation unit 150 acquires the learned model 140 with reference to the storage unit 100. Next, the estimation unit 150 acquires the captured image PI divided into a plurality of regions AR with reference to the divided image information 130 of the storage unit 100. Furthermore, the estimation unit 150 outputs the rectangular information, the class, and the object reliability OR from the captured image PI using the learned model 140. That is, the estimation unit 150 configures one or more detection rectangles BB in the captured image PI, and calculates, for each detection rectangle BB configured, the probability PR of each of the plurality of regions AR, "containing the position PS of the detection target OB enclosed by the detection rectangle BB". Furthermore, the estimation unit 150 calculates the object reliability OR for each detection rectangle BB.

[0137] The estimation unit 150 saves the estimation result estimated from the captured image PI using the learned model 140 to the estimation result information table 160. Specifically, the estimation unit 150 saves the rectangular information, the category, and the object reliability OR to each of the rectangular information table 161, the probability table 162, and the object reliability table 163 of the storage unit 100, respectively.

[0138] The determination unit 170 determines, for each of one or more detection rectangles BB arranged in the captured image PI by the estimation unit 150, a region AR in which the detection rectangle BB is present, that is, a region AR containing the detection rectangle BB as a specific region IA.

[0139] Specifically, the determination unit 170 refers to the rectangular information table 161 of the storage unit 100 to acquire the position / shape / size of each of one or more detection rectangles BB arranged in the captured image PI. Also, the determination unit 170 refers to the segmented image information 130 of the storage unit 100 to acquire the position / shape / size of each of a plurality of regions AR in which the captured image PI is segmented. The determination unit 170 determines, for each of one or more detection rectangles BB arranged in the captured image PI, in which region AR the detection rectangle BB is contained, based on the “position / shape / size of the detection rectangle BB” and the “position / shape / size of each of a plurality of regions AR”.

[0140] The determination unit 170 can also determine the position PS of the detection object OB surrounded by the detection rectangle BB and estimated as “a person”, for example, can also determine the position (underfoot position) of the detection object OB estimated as “a person” corresponding to the underfoot of the “person”. The determination unit 170 can also calculate the position PS (underfoot position) of the detection object OB from the detection rectangle BB, for example, can also calculate the position PS (underfoot position) of the detection object OB from the position, shape, and size of the detection rectangle BB. Also, the determination unit 170 can also determine the center position (or the center of gravity position) of the detection rectangle BB as the position PS (underfoot position) of the detection object OB surrounded by the detection rectangle BB and estimated as “a person”. The determination unit 170 can also determine the region AR in which the position PS (underfoot position) of the detection object OB is present, that is, the region AR containing the position PS as a specific region IA.

[0141] The determination unit 170 saves the specific region IA determined for each detection rectangle BB to the specific region table 180 of the storage unit 100. The determination unit 170 can also determine the position PS (underfoot position) of the detection object OB surrounded by the detection rectangle BB and estimated as “a person” as the specific region IA for each detection rectangle BB, and can also save the determined position PS for each detection rectangle BB to the specific region table 180.

[0142] The determination unit 190 determines whether the detection object OB surrounded by the detection rectangle BB is actually a person using the class of the detection object OB. That is, the determination unit 190 determines whether the rectangular information output from the photographing image PI by the estimation unit 150 using the learned model 140 is correct using the class of the detection object OB output from the photographing image PI by the estimation unit 150 using the learned model 140. In other words, the determination unit 190 determines whether the detection object OB surrounded by the detection rectangle BB is actually a person using the class of the detection object OB.

[0143] The determination unit 190 adopts the detection rectangle BB (i.e., the detection object OB) as a correct estimation when it is determined that the detection object OB surrounded by the detection rectangle BB is actually a person. The determination unit 190 removes the detection rectangle BB (i.e., the detection object OB) as a false detection (false estimation) when it is determined that the detection object OB surrounded by the detection rectangle BB is not actually a person.

[0144] In particular, the determination unit 190 determines whether the detection object OB surrounded by the detection rectangle BB is actually a person using the class of the detection object OB and the area AR, i.e., the specific area IA, in which the detection rectangle BB is located.

[0145] Specifically, the determination unit 190 acquires the "PRs of the respective areas AR" for each detection rectangle BB with reference to the estimation result information table 160, and in particular, with reference to the probability table 162. Also, the determination unit 190 acquires the object reliability OR for each detection rectangle BB with reference to the estimation result information table 160, and in particular, with reference to the object reliability table 163. Furthermore, the determination unit 190 acquires the specific area IA for each detection rectangle BB with reference to the specific area table 180. The determination unit 190 determines whether the detection object OB surrounded by the detection rectangle BB is a person using at least one of the "PRs of the respective areas AR" for each detection rectangle BB and the object reliability OR for each detection rectangle BB, and the specific area IA for each detection rectangle BB.

[0146] For example, in the first verification method, the determination unit 190 determines whether the area AR in which the "position PS of the detection object OB surrounded by the detection rectangle BB" has the highest probability PR coincides with the specific area IA for each detection rectangle BB. The determination unit 190 determines that the detection object OB surrounded by the detection rectangle BB is a person when it is confirmed that the area AR in which the "position PS of the detection object OB surrounded by the detection rectangle BB" has the highest probability PR coincides with the specific area IA of the detection rectangle BB. The determination unit 190 determines that the detection object OB surrounded by the detection rectangle BB is not a person (i.e., the detection rectangle BB is a false detection) when it is confirmed that the area AR in which the "position PS of the detection object OB surrounded by the detection rectangle BB" has the highest probability PR does not coincide with the specific area IA of the detection rectangle BB.

[0147] For example, in the second verification method, the determination section 190 determines, for each detection rectangle BB, whether or not "a class reliability value CR of the detection rectangle BB calculated by multiplying a probability PR of a region AR corresponding to the specific region IA by an object reliability OR of the detection rectangle BB" is greater than a prescribed value TH. The determination section 190 determines that the detection object OB surrounded by the detection rectangle BB is a person when it is confirmed that the class reliability value CR of the detection rectangle BB is greater than the prescribed value TH. The determination section 190 determines that the detection object OB surrounded by the detection rectangle BB is not a person (i.e., the detection rectangle BB is a false detection) when it is confirmed that the class reliability value CR of the detection rectangle BB is the prescribed value TH or less.

[0148] The teaching data generation section 210 generates teaching data DT by assigning, as a label, "rectangle information and a class" received from a user to the captured image PI (i.e., the divided image information 130) divided into a plurality of regions AR by the division section 120.

[0149] The learning section 220 constructs the learned model 140 by supervised learning with respect to a data set DS, which is a set of the teaching data DT generated by the teaching data generation section 210.

[0150] In the example shown in FIG. 1, the learning section 220 includes a person learning section 221 that learns rectangle information and a region learning section 222 that learns a class of each detection object OB surrounded by a detected rectangle BB. Figure 1 In the example shown in FIG. 1, the learning section 220 includes a person learning section 221 that learns rectangle information and a region learning section 222 that learns a class of each detection object OB surrounded by a detected rectangle BB. Figure 1 In the example shown in FIG. 1, the learning section 220 includes a person learning section 221 that learns rectangle information and a region learning section 222 that learns a class of each detection object OB surrounded by a detected rectangle BB.

[0151] In the following description, an example in which the learning section 220 is implemented as one neural network is described. The learning section 220 may, for example, be implemented as a convolutional neural network (CNN), a fast region-based convolutional network (Fast R-CNN), or a deep neural network (DNN).

[0152] (Details of the storage section)

[0153] The storage section 100 is a storage device that stores various data used by the detection device 10. In addition, the storage section 100 can also non-temporarily store (1) a control program, (2) an Operating System (OS) program, (3) an application program for executing various functions possessed by the detection device 10, and (4) various data read out when the application program is executed, which are executed by the detection device 10. The data of (1) to (4) is stored, for example, in a non-volatile storage device such as a read only memory (ROM), a flash memory, an Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM (registered trademark)), a Hard Disc Drive (HDD), or the like. The detection device 10 can also include a non-illustrated temporary storage section. The temporary storage section is a so-called work memory that temporarily stores data used in calculation and calculation results and the like in the course of various processes executed by the detection device 10, and includes a volatile storage device such as a Random Access Memory (RAM). Which data is stored in which storage device is appropriately decided in accordance with the purpose of use, convenience, cost, or physical limitations of the detection device 10, or the like. The storage section 100 further holds the divided image information 130, the learned model 140, the estimation result information table 160, and the specific region table 180.

[0154] The divided image information 130 is information indicating the captured image PI divided into the plurality of regions AR by the division section 120, and is information held by the division section 120.

[0155] The learned model 140 is a learned model that takes the captured image PI as input, and takes as output the rectangular information and the probability PR (i.e., the class) of each of the plurality of regions AR, of "the position PS where the detection target OB surrounded by the detection rectangle BB is present". The learned model 140 is constructed by the learning section 220, and is held by the learning section 220 to the storage section 100.

[0156] In Figure 1In the illustrated example, the learned model 140 includes a person prediction model 141 and a region prediction model 142. The person prediction model 141 is a learned model that takes the captured image PI as input and outputs rectangular information indicating the position / shape / size of the detection rectangle BB that encloses a detection object OB that is estimated to be a "person". The region prediction model 142 is a learned model that takes the captured image PI as input and outputs the class of each detection object OB enclosed by the detection rectangle BB, i.e., the probability PR of each of the plurality of regions AR.

[0157] Figure 1 In the illustrated example, the learned model 140 includes a person prediction model 141 and a region prediction model 142. The person prediction model 141 is a learned model that takes the captured image PI as input and outputs rectangular information indicating the position / shape / size of the detection rectangle BB that encloses a detection object OB that is estimated to be a "person". The region prediction model 142 is a learned model that takes the captured image PI as input and outputs the class of each detection object OB enclosed by the detection rectangle BB, i.e., the probability PR of each of the plurality of regions AR.

[0158] In the illustrated example, the learned model 140 includes a person prediction model 141 and a region prediction model 142. The person prediction model 141 is a learned model that takes the captured image PI as input and outputs rectangular information indicating the position / shape / size of the detection rectangle BB that encloses a detection object OB that is estimated to be a "person". The region prediction model 142 is a learned model that takes the captured image PI as input and outputs the class of each detection object OB enclosed by the detection rectangle BB, i.e., the probability PR of each of the plurality of regions AR.

[0159] In the illustrated example, the learned model 140 includes a person prediction model 141 and a region prediction model 142. The person prediction model 141 is a learned model that takes the captured image PI as input and outputs rectangular information indicating the position / shape / size of the detection rectangle BB that encloses a detection object OB that is estimated to be a "person". The region prediction model 142 is a learned model that takes the captured image PI as input and outputs the class of each detection object OB enclosed by the detection rectangle BB, i.e., the probability PR of each of the plurality of regions AR.

[0160] In the illustrated example, the learned model 140 includes a person prediction model 141 and a region prediction model 142. The person prediction model 141 is a learned model that takes the captured image PI as input and outputs rectangular information indicating the position / shape / size of the detection rectangle BB that encloses a detection object OB that is estimated to be a "person". The region prediction model 142 is a learned model that takes the captured image PI as input and outputs the class of each detection object OB enclosed by the detection rectangle BB, i.e., the probability PR of each of the plurality of regions AR.

[0161] For example, with respect to the detection rectangle BB(x), the probabilities PR of each of the regions AR(0) through AR(n), i.e., the probability PR(x-0), the probability PR(x-1), the probability PR(x-2),..., and the probability PR(x-n), are saved to the probability table 162. Similarly, for example, with respect to the detection rectangle BB(y), the probabilities PR of each of the regions AR(0) through AR(n), i.e., the probability PR(y-0), the probability PR(y-1), the probability PR(y-2),..., and the probability PR(y-n), are saved to the probability table 162.

[0162] In the object reliability table 163, for each of the detection rectangles BB configured in the captured image PI by the prediction unit 150 using the learned model 140, an object reliability OR is stored. The object reliability OR is information representing the probability that certain objects are surrounded by the detection rectangles BB. In other words, the object reliability OR of each detection rectangle BB is stored in the object reliability table 163.

[0163] In the specific region table 180, information on the identification of a specific region IA for each detection rectangle BB is stored. In other words, in the specific region table 180, information on the identification of the region AR, i.e., the specific region IA, that contains the detection rectangle BB is stored for each detection rectangle BB.

[0164] In the specific region table 180, the "position PS (e.g., the foot position) of the detected object OB surrounded by the detection rectangle BB and presumed to be a 'person'" can also be stored as a specific region IA for each detection rectangle BB. The position PS (foot position) of the detected object OB is calculated based on the detection rectangle BB, for example, based on the position, shape, and size of the detection rectangle BB. Moreover, the specific region IA of each detection rectangle BB can also be a region AR that includes the "position PS of the detected object OB surrounded by the detection rectangle BB and presumed to be a 'person'". As mentioned above, the "position PS (foot position) of the detected object OB surrounded by the detection rectangle BB and presumed to be a 'person'" can also be the center position or centroid position of the detection rectangle BB.

[0165] §3. Examples of Actions

[0166] The following is for reference Figures 4 to 19 This will illustrate the various processes performed by the detection device 10. Specifically, using... Figures 4 to 6 To illustrate the learning process performed by the detection device 10, using Figures 7 to 19 This explains the human detection process performed by the detection device 10.

[0167] (Summary of learning processing)

[0168] The detection device 10 performs a learning process, for example (i.e., learning the generation process of model 140). In the learning process, the detection device 10 (especially the teaching data generation unit 210) first assigns rectangular information and category to the captured image PI, which is divided into multiple regions AR, as labels, and generates teaching data DT.

[0169] The rectangular information is information indicating "the position / shape / size of the detection rectangle BB that encloses the person photographed in the photographed image PI". For example, the rectangular information is "the X-axis coordinate and the Y-axis coordinate of the right upper point of the detection rectangle BB" and "the X-axis coordinate and the Y-axis coordinate of the left lower point of the detection rectangle BB".

[0170] The category is the identification information of "the region AR that includes the position PS (for example, a position equivalent to the position under the feet of the person, a position under the feet) where the person photographed in the photographed image PI is positioned". The category can also be said to be information that determines the region AR that includes "the detection rectangle BB that encloses the person photographed in the photographed image PI".

[0171] Also, the detection device 10 (particularly, the learning unit 220) performs learning (supervised learning) on the set of teaching data DT, that is, the data set DS, and constructs the learned model 140. The learning unit 220 saves the constructed learned model 140 to the storage unit 100.

[0172] As described above, in the detection device 10, the learning unit 220 constructs the learned model 140 by learning (for example, deep learning) on the set of teaching data generated by the teaching data generation unit 210, that is, the data set DS. In this way, the detection device 10 also has a function as a model generation device that constructs the learned model 140.

[0173] (Example of learning processing)

[0174] Figure 4 is a flowchart that illustrates an example of the generation processing (learning processing) of the learned model 140 used by the detection device 10. The learned model 140 is, for example, generated by the detection device 10 performing learning (machine learning) on the set of teaching data DT, that is, the data set DS. However, the subject that generates the learned model 140 by learning on the data set DS is not necessarily the detection device 10. It can also be a device different from the detection device 10 that includes the teaching data generation unit 210 and the learning unit 220 and generates the learned model 140 by learning on the data set DS.

[0175] Hereinafter, an example will be described in which the learned model 140 is generated (constructed) by the detection device 10 performing learning (for example, supervised learning) on the data set DS.

[0176] As described above, Figure 4As illustrated, firstly, the teaching data generation unit 210 generates teaching data DT (S110) that classifies the people photographed, based on the position PS (more precisely, the region AR containing the position PS) of each person photographed. That is, in the teaching data DT generated by the teaching data generation unit 210, the photographed image PI is assigned the category of the people photographed in the photographed image PI (identification information including the region AR containing the position PS (foot position) of the person).

[0177] Specifically, the instructional data generation unit 210 first refers to the segmented image information 130 in the storage unit 100 to obtain a "piloted image PI segmented into multiple regions of AR". The instructional data generation unit 210 then presents the obtained "piloted image PI segmented into multiple regions of AR" to the user, urging the user to annotate the "piloted image PI segmented into multiple regions of AR". Furthermore, the instructional data generation unit 210 obtains rectangle information and category from the user's annotations to assign labels (label information) to the "piloted image PI segmented into multiple regions of AR".

[0178] Learning unit 220 obtains the teaching data DT generated by teaching data generation unit 210 (S120). Learning unit 220 performs multi-class learning (machine learning) on ​​the set of teaching data DT obtained from teaching data generation unit 210, i.e., dataset DS (S130), and generates (builds) a learned model 140. For example, learning unit 220 generates learned model 140 through instructional learning on dataset DS.

[0179] The learning unit 220 saves the generated learned model 140 to the storage unit 100 (S140).

[0180] If this is the end of the usage... Figure 4 As explained, the model generation method performed by the detection device 10 is a "model generation method performed by a model generation device that generates a learned model", which includes the following two processes. That is, the model generation method includes: an acquisition step (S120) to acquire teaching data DT; and a learning step (S130) to construct a learned model 140 by learning from the set of teaching data DT, i.e., dataset DS.

[0181] In the teaching data DT acquired in the acquisition step, the captured image PI, which is segmented into multiple regions AR, is assigned the following information as a label: (A) information representing the position / shape / size of the detection rectangle BB surrounding the person captured in the captured image PI (rectangle information), and (B) information on the AR identification of the region PS containing the location of the person captured in the captured image PI (category) as labels.

[0182] The learned model 140 constructed in the learning step is a learned model that outputs (C) rectangular information and (D) a probability PR of each of the plurality of regions AR "containing a position PS of a person photographed in the photographed image PI" as output, with the photographed image PI as input.

[0183] According to the configuration, the model generation method constructs the learned model 140 by machine learning with respect to the teaching data DT (more accurately, a collection of the teaching data DT, that is, the data set DS). The learned model 140 outputs the following two kinds of information when the photographed image PI is input. That is, (C) information indicating a position / shape / size of the detection rectangle BB (rectangular information) and (D) a probability PR of each of the plurality of regions AR "containing a position PS of a person photographed in the photographed image PI".

[0184] The rectangular information is information indicating a position / shape / size of the detection rectangle BB that encloses the detection object OB presumed to be a "person", and is information containing the presumption that the detection object OB enclosed by the detection rectangle BB is a "person".

[0185] Accordingly, the model generation method has the effect of being able to construct the learned model 140 that outputs the rectangular information containing the presumption that the detection object OB is a "person" and the probability PR (that is, the category) of each of the plurality of regions AR when the photographed image PI is input.

[0186] (Segmentation example of photographed image)

[0187] Figure 5 is a diagram indicating an example of the photographed image PI segmented into the plurality of regions AR by the detection device 10. For example, the detection device 10 (particularly, the segmentation section 120) can also segment the photographed image PI into the plurality of regions AR according to a distance from the center of the photographed image PI. In other words, the segmentation section 120 can also segment the photographed image PI into the plurality of regions AR with a circumference of one or more concentric circles (or ellipses) centered on the "center of the photographed image PI" as a boundary line.

[0188] Further, for example, the segmentation section 120 can also segment the photographed image PI into a plurality of sector-shaped regions AR with a central angle set to a prescribed angle. In other words, the segmentation section 120 can also segment into a plurality of regions AR with a portion of a plurality of straight lines (a plurality of straight lines extending radially from the approximate center of the photographed image PI, intersecting each other at a prescribed angle) intersecting each other at a prescribed angle in the approximate center of the photographed image PI as a boundary line.

[0189] Further, for example, the division section 120 can also divide a plurality of regions AR with at least one of a circumference of one or more concentric circles (ellipses) centered on "the approximate center of the captured image PI" and a portion of a plurality of straight lines intersecting each other at a prescribed angle at the approximate center of the captured image PI as a boundary line.

[0190] In Figure 5 In the example illustrated in (A) of the above, the division section 120 sets the inside of one circle (ellipse) centered on "the center of the captured image PI" in the captured image PI as the region AR(0). Also, the division section 120 divides the region outside of the circle (ellipse) centered on "the center of the captured image PI" in the captured image PI into six regions AR of regions AR(1) to AR(6) by a portion of a plurality of straight lines intersecting each other at 60 degrees at the center of the captured image PI. Thus, Figure 5 The captured image PI illustrated in (A) of the above is divided by the division section 120 into seven regions AR of the region AR(0), the region AR(1), the region AR(2),..., and the region AR(6).

[0191] In Figure 5 In (A) of the above, the region AR(0) can also be further divided by the division section 120 by a portion of a plurality of straight lines intersecting each other at a prescribed angle at the center of the captured image PI. For example, the division section 120 can also divide the region AR(0) of (A) into regions AR(0-0), AR(0-1), AR(0-2),..., and AR(0-n). Figure 5 The region AR(0) of (A) can also be further divided by the division section 120 into regions AR(0-0), AR(0-1), AR(0-2),..., and AR(0-n).

[0192] In Figure 5 In the example illustrated in (B) of the above, the division section 120 divides the captured image PI into six regions AR of regions AR(0) to AR(5) by a portion of a plurality of straight lines (a portion of straight lines) intersecting each other at 60 degrees at the center of the captured image PI. Thus, Figure 5 The captured image PI illustrated in (B) of the above is divided by the division section 120 into six regions AR of the region AR(0), the region AR(1), the region AR(2),..., and the region AR(5).

[0193] In Figure 5 In the example illustrated in (C) of the above, the division section 120 divides the captured image PI into sixteen regions AR of regions AR(0) to AR(15) by a portion of a plurality of straight lines intersecting each other at 22.5 degrees at the center of the captured image PI. Thus, Figure 5 The captured image PI illustrated in (C) of the above is divided by the division section 120 into sixteen regions AR of the region AR(0), the region AR(1), the region AR(2),..., and the region AR(15).

[0194] It is not necessary for the division section 120 to divide the captured image PI by the part of the plurality of straight lines intersecting each other at 22.5 degrees or 60 degrees at the center of the captured image PI. The division section 120 can also divide the captured image PI by the plurality of straight lines (part of straight lines) intersecting each other at 1 degree at the approximate center of the captured image PI, and can also divide the captured image PI by the plurality of straight lines (part of straight lines) intersecting each other at 45 degrees at the approximate center of the captured image PI.

[0195] In Figure 5 In the example illustrated in (D) of FIG. 10, the division section 120 sets the inside of the smaller one of the two concentric circles (ellipses) in the captured image PI, which is centered on the "center of the captured image PI", as the region AR(0). Also, the detection device 10 sets the region of the inside of the larger one of the two concentric circles (ellipses) in the captured image PI, which is centered on the "center of the captured image PI", and the region of the outside of the smaller one of the two concentric circles (ellipses) as the region AR(1). Further, the detection device 10 divides the region of the outside of the larger one of the two concentric circles (ellipses) in the captured image PI into six regions AR by the part of the plurality of straight lines intersecting each other at 60 degrees at the center of the captured image PI, as the regions AR(2) to AR(7). Thus, Figure 5 The captured image PI illustrated in (D) of FIG. 10 is divided by the division section 120 into the eight regions AR of the region AR(0), the region AR(1), the region AR(2),..., and the region AR(7).

[0196] As explained using Figure 5 , the captured image PI is divided by the division section 120 into a plurality of regions AR by a prescribed division method. The "prescribed division method" is, for example, set or changed by the user.

[0197] (Example of learning processing)

[0198] Figure 6 is a diagram that shows an example of the teaching data DT learned in the generation processing (learning processing) of Figure 4 , which is generated by the teaching data generation section 210 and learned by the learning section 220. In Figure 6 The captured image PI illustrated in (D) of FIG. 10 is divided by the division section 120 into the eight regions AR of the region AR(0), the region AR(1), the region AR(2),..., and the region AR(7). Figure 6The photographing image PI is divided into six regions AR (1) to AR (6) by a plurality of straight lines intersecting with each other at an angle of 60 degrees in the illustrated example.

[0199] In Figure 6 In the illustrated teaching data DT, a detection rectangle BB (1) (i.e., rectangle information indicating the position / shape / size of the detection rectangle BB (1)) that surrounds the person photographed in the photographing image PI is given as a label for the photographing image PI.

[0200] Also, in Figure 6 In the illustrated teaching data DT, "1" is given as a label for the photographing image PI as a class of the person photographed in the photographing image PI. The class of the person photographed in the photographing image PI refers to information that identifies a region AR that contains the position PS at which the person photographed in the photographing image PI is present from other regions AR.

[0201] In Figure 6 In the illustrated teaching data DT, the position PS at which the person photographed in the photographing image PI is present is contained in the region AR (1) among the regions AR (0) to AR (6) obtained by dividing the photographing image PI. As described above, the "position PS at which the person photographed in the photographing image PI is present" refers to, for example, a "position under the feet (under-feet position) of the person photographed in the photographing image PI". In Figure 6 In the illustrated example, the position under the feet (under-feet position) of the person (the person surrounded by the detection rectangle BB (1)) photographed in the photographing image PI is contained in the region AR (1). Therefore, in Figure 6 In the illustrated teaching data DT, the class of the person photographed in the photographing image PI is set to "1".

[0202] The detection device 10 (particularly, the learning unit 220) learns Figure 6 The illustrated set of teaching data DT, i.e., the data set DS, is used to construct the learned model 140. That is, the learning unit 220 learns the teaching data DT in which the photographing image PI is given the detection rectangle BB that surrounds the person photographed in the photographing image PI and the identification information of the region AR that contains the position PS at which the person photographed in the photographing image PI is present as labels. The learning unit 220 learns the teaching data DT in which the photographing image PI is given the detection rectangle BB that surrounds the person photographed in the photographing image PI and the identification information of the region AR that contains the position PS at which the person photographed in the photographing image PI is present as labels.

[0203] (Outline of person detection processing)

[0204] The detection device 10 executes a person detection process of detecting "a person imaged in the imaged image PI" from the imaged image PI. The person detection process includes an inference process of performing inference using the learned model 140 constructed through the learning process, and a determination process of determining whether or not the result of the inference process (inference result) is correct.

[0205] In the inference process, the detection device 10 (particularly, the inference unit 150 (learned model 140)) inputs the imaged image PI to the learned model 140, thereby outputting the rectangular information and the category. Specifically, the learned model 140 outputs information indicating the position / shape / size of the detection rectangle BB that encloses the detection object OB inferred as "a person", as the rectangular information. Also, the learned model 140 outputs the recognition information of the region AR including the position PS at which the detection object OB inferred as "a person" is enclosed by the detection rectangle BB, that is, outputs "the probability PR of including the position PS" with respect to each of the plurality of regions AR, as the category. The detection device 10 executes the inference process using the learned model 140 constructed through deep learning, for example, thereby being able to improve the detection speed (inference speed) and the detection accuracy (inference accuracy).

[0206] In the determination process, the detection device 10 (particularly, the determination unit 190) determines whether or not the inference result is correct, specifically, determines whether or not the rectangular information output from the learned model 140 is correct using the category output from the learned model 140. Specifically, the determination unit 190 executes the first determination process or the second determination process in the determination process.

[0207] In the first determination process, the determination unit 190 determines whether or not the category output from the learned model 140 is correct, and if it is determined that the category is correct, it is also determined that the detection information output from the learned model 140 is correct. That is, if the determination unit 190 determines that the category is correct, it is also determined that the inference that "the detection object OB enclosed by the detection rectangle BB is a person" is correct. Also, if the determination unit 190 determines that the category output from the learned model 140 is incorrect, it is also determined that the detection information output from the learned model 140 is incorrect. That is, if the determination unit 190 determines that the category is incorrect, it is also determined that the inference that "the detection object OB enclosed by the detection rectangle BB is a person" is incorrect.

[0208] In the second determination processing, the determination section 190 determines whether or not a class reliability value CR calculated by multiplying the probability PR of the region AR in which the detection rectangle BB is present (i.e., the region AR corresponding to the specific region IA) by the object reliability OR of the detection rectangle BB is greater than a prescribed value TH. If the determination section 190 determines that the class reliability value CR is greater than the prescribed value TH, it determines that the detection information output by the learned model 140 is correct, i.e., determines that the presumption that "the detection object OB surrounded by the detection rectangle BB is a person" is correct. If the determination section 190 determines that the class reliability value CR is equal to or less than the prescribed value TH, it determines that the detection information output by the learned model 140 is incorrect, i.e., determines that the presumption that "the detection object OB surrounded by the detection rectangle BB is a person" is incorrect.

[0209] The first determination processing will be described below using Figures 7 to 13 , and the second determination processing will be described using Figures 14 to 19 .

[0210] (Example of Person Detection Processing Including First Determination Processing)

[0211] Figure 7 is a flowchart illustrating an example of person detection processing that performs the first determination processing on the output of the learned model 140 to detect a person. As shown in Figure 7 , first, the image acquisition section 110 acquires a captured image PI, and the division section 120 divides the captured image PI acquired by the image acquisition section 110 into a plurality of regions AR (S210). That is, first, the image acquisition section 110 acquires a captured image PI captured by the ceiling camera 20 from the ceiling camera 20. The image acquisition section 110 outputs the acquired captured image PI to the division section 120. The division section 120 divides the captured image PI received from the image acquisition section 110 into a plurality of regions AR by a prescribed division method. The division section 120 saves the captured image PI divided into a plurality of regions AR as the divided image information 130 to the storage section 100.

[0212] The presumption section 150 acquires the learned model 140 with reference to the storage section 100. Also, the presumption section 150 acquires a captured image PI (a captured image PI divided into a plurality of regions AR) for which the presumption processing should be performed with reference to the divided image information 130 of the storage section 100. The presumption section 150 performs person detection and class output using the learned model 140 for the captured image PI (a captured image PI divided into a plurality of regions AR) acquired with reference to the divided image information 130 (S220). The presumption section 150 can also perform person detection and class output using the learned model 140 for a captured image PI that is not divided into a plurality of regions AR by the division section 120.

[0213] The estimation unit 150 saves the result of the estimation processing for the captured image PI using the learned model 140 to the estimation result information table 160 of the storage unit 100. Specifically, the estimation unit 150 saves the rectangular information, the class, and the object reliability OR output by the learned model 140 to which the captured image PI is input to the storage unit 100.

[0214] The rectangular information is information indicating the position / shape / size of each of one or more detection rectangles BB arranged in the captured image PI by the estimation unit 150 (learned model 140), that is, information indicating the position / shape / size of each detection rectangle BB. The rectangular information is saved to the rectangular information table 161 of the storage unit 100.

[0215] The class is the probability PR of each of the plurality of regions AR, "containing the position PS of the detection object OB surrounded by the detection rectangle BB," calculated for each detection object OB (that is, each detection rectangle BB) surrounded by the detection rectangle BB. The class is saved to the probability table 162 of the storage unit 100.

[0216] The object reliability OR is information indicating the size of the "possibility of some object being surrounded by each detection rectangle BB." The object reliability OR is saved to the object reliability table 163 of the storage unit 100.

[0217] The detection device 10 executes the processing of S230 to S270 for each detection rectangle BB (that is, each detection object OB) with the number of persons detected from the captured image PI by the estimation unit 150, in other words, the number of detection rectangles BB arranged by the estimation unit 150.

[0218] The determination unit 190 acquires the class (output class) for each detection object OB (that is, each detection rectangle BB) surrounded by the detection rectangle BB by referring to the probability table 162 of the storage unit 100 (S230), that is, acquires the probability PR of each of the plurality of regions AR. Specifically, the determination unit 190 acquires the probability PR of each of the plurality of regions AR, "containing the position PS of the detection object OB surrounded by the detection rectangle BB and estimated as a "person," for each detection rectangle BB.

[0219] For example, the determination unit 190 acquires the probability PR (0-1), PR (1-1), PR (2-1),..., PR (n-1) of each of the plurality of regions AR for the detection rectangle BB (1). Likewise, the determination unit 190 acquires the probability PR (0-2), PR (1-2), PR (2-2),..., PR (n-2) of each of the plurality of regions AR for the detection rectangle BB (2).

[0220] The determination section 170 acquires the position PS (for example, a position equivalent to a position under the feet of the "person", a position under the feet) of the detection object OB surrounded by the detection rectangle BB and presumed to be a "person" for each detection rectangle BB (S240). For example, the determination section 170 refers to the storage section 100 to acquire the rectangle information table 161, and calculates the position PS (position under the feet) of the detection object OB from the detection rectangle BB (position, shape, size of the detection rectangle BB) for each detection rectangle BB. The determination section 170 can also determine the center position (or the barycentric position) of the detection rectangle BB as the position PS (position under the feet) of the detection object OB. The determination section 170 sets the position PS (position under the feet) of the detection object OB calculated from the detection rectangle BB as the position PS of the detection object OB surrounded by the detection rectangle BB and presumed to be a "person" for each detection rectangle BB.

[0221] The determination section 190 determines whether or not the position under the feet (position PS) of the detection object OB is included in the output category (that is, the region AR in which the probability PR is the largest) for each detection object OB (that is, each detection rectangle BB surrounding the detection object OB) (S250).

[0222] Specifically, the determination section 190 first refers to the probability table 162 of the storage section 100 to acquire the probability PR of each region AR for each detection object OB (that is, the detection rectangle BB). Next, the determination section 190 selects the region AR in which the probability PR is the highest for each detection object OB (that is, the detection rectangle BB). Furthermore, the determination section 190 determines whether or not the position PS (position under the feet) of each detection object OB is included in the region AR in which the probability PR is the highest (that is, the output category) selected for each detection object OB (that is, the detection rectangle BB).

[0223] If the position PS (position under the feet) of the detection object OB is included in the output category (Yes in S250), the determination section 190 adopts the detection rectangle BB as a correct detection result (S260). That is, the determination section 190 determines that the position PS (position under the feet) of the detection object OB is included in the output category and that the presumption that the detection object OB surrounded by the detection rectangle BB is a person is correct.

[0224] If the position PS (position under the feet) of the detection object OB is not included in the output category (No in S250), the determination section 190 removes the detection rectangle BB as a false detection (S270). That is, the determination section 190 determines that the position PS (position under the feet) of the detection object OB is not included in the output category and that the presumption that the detection object OB surrounded by the detection rectangle BB is a person is incorrect.

[0225] In Figure 7 In the illustrated first determination processing, the detection device 10 (particularly, the determination section 190) determines, for each detection object OB (i.e., the detection rectangle BB), whether the position PS of the detection object OB is included in the "region AR with the highest probability PR". However, it is not necessary for the determination section 190 to determine, in the first determination processing, whether the position PS of the detection object OB is included in the "region AR with the highest probability PR".

[0226] It is only necessary for the determination section 190 to determine, in the first determination processing, whether the class of the detection object OB (i.e., the region AR with the highest probability PR) output from the learned model 140 is correct or not for each detection object OB (i.e., the detection rectangle BB). It is only necessary for the determination section 190 to determine, in the first determination processing, whether the position PS of the detection object OB (i.e., the (actual) position PS of the detection object OB surrounded by the detection rectangle BB and estimated as "a person") estimated by the learned model 140 coincides with the position PS at the level of the region AR. The (actual) position PS of the detection object OB estimated as "a person" is calculated from the detection rectangle BB (the position, shape, and size of the detection rectangle BB). Further, the position PS can be, for example, the center position (or the center of gravity position) of the detection rectangle BB.

[0227] That is, it is only necessary for the determination section 190 to determine, in the first determination processing, whether the class (i.e., the region AR with the highest probability PR) output from the learned model 140 coincides with the region AR (i.e., the specific region IA) in which the detection rectangle BB actually exists. Details will be described later, but the determination section 190 can also determine, in the first determination processing, whether the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR" coincides with the region AR (i.e., the specific region IA) in which the detection rectangle BB actually exists.

[0228] For example, in S240, the determination section 170 determines, for each detection rectangle BB, the region AR in which the detection rectangle BB exists as the specific region IA. The determination section 170 can also determine, as the specific region IA, the region AR including the position PS (the position under the feet) of the detection object OB calculated from the detection rectangle BB (the position, shape, and size of the detection rectangle BB). Further, the determination section 170 can also determine, as the specific region IA, the region AR including the center position (or the center of gravity position) of the detection rectangle BB.

[0229] Specifically, the determination section 170 acquires the rectangular information table 161 and the divided image information 130 with reference to the storage section 100. The determination section 170 determines the "region AR in which the detection rectangle BB is present", that is, the specific region IA, for each detection rectangle BB, based on the acquired rectangular information table 161 and the divided image information 130. The determination section 170 saves information identifying the specific region IA determined for each detection rectangle BB to the specific region table 180 of the storage section 100.

[0230] For example, in S250, the determination section 190 can determine whether the specific region IA of each detection rectangle BB (that is, each detection object OB) coincides with the "region AR in which the probability PR is the greatest" for each detection rectangle BB (that is, each detection object OB).

[0231] Specifically, the determination section 190 acquires the specific region IA of each detection rectangle BB with reference to the specific region table 180 of the storage section 100. Also, the determination section 190 acquires the "probability PR of each of the regions AR" of each detection rectangle BB (that is, each detection object OB) with reference to the probability table 162 of the storage section 100, and selects the region AR in which the probability PR is the greatest for each detection rectangle BB. Also, the determination section 190 determines whether the specific region IA of each detection rectangle BB (that is, each detection object OB) coincides with the "region AR in which the probability PR is the greatest" for each detection rectangle BB (that is, each detection object OB).

[0232] If the "specific region IA of the detection rectangle BB coincides with the "region AR in which the probability PR is the greatest", the determination section 190 adopts the detection rectangle BB as a correct detection result, that is, determines that the presumption that "the detection object OB surrounded by the detection rectangle BB is a person" is correct.

[0233] If the "specific region IA of the detection rectangle BB does not coincide with the "region AR in which the probability PR is the greatest", the determination section 190 removes the detection rectangle BB as a false detection, that is, determines that the presumption that "the detection object OB surrounded by the detection rectangle BB is a person" is incorrect.

[0234] Details will be described later, but the determination section 190 can also determine whether the specific region IA coincides with the "region AR in which the probability PR is the greatest" or the "region AR adjacent to the region AR in which the probability PR is the greatest".

[0235] Specifically, the determination unit 190 may also select the region AR with the highest probability PR and the region AR adjacent to the region AR with the highest probability PR for each detection rectangle BB (i.e., each detection object OB) based on the "probabilities PR of each of the multiple regions AR". Furthermore, the determination unit 190 may also determine for each detection rectangle BB (i.e., each detection object OB) whether a specific region IA of the detection rectangle BB matches the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR".

[0236] In the first determination process, the determination unit 190 uses a specific region IA of the detection rectangle BB to determine whether the "category of the detection object OB" output by the inference unit 150 (learned model 140) from the captured image PI is correct for each detection object OB (i.e., detection rectangle BB).

[0237] (Example of model output after learning)

[0238] Figure 8 This is a graph representing examples of categories in the output of the learned model 140. Specifically, Figure 8 This is a diagram illustrating an example of the categories (i.e., the probabilities PR of "containing the location PS of the detected object OB") output by the detection device 10 from the captured image PI using the learned model 140. Figure 8 In the example shown, the detection device 10 (especially the segmentation unit 120) segments the captured image PI into seven regions AR: region AR(0), region AR(1), region AR(2), ..., region AR(6).

[0239] exist Figure 8 In the example shown, two detection rectangles BB are configured in the captured image PI, specifically, detection rectangle BB(1) and detection rectangle BB(2). Detection rectangle BB(1) and detection rectangle BB(2) are configured to enclose the detection object OB that is presumed to be "human (human body)".

[0240] Moreover, in Figure 8In the example shown, the class of each of the two detection objects OB presumed to be "a person" is output (calculated) by the learned model 140. Specifically, the learned model 140 calculates the probability PR including "the position PS(1) at which the detection object OB(1) surrounded by the detection rectangle BB(1) is present" with respect to each of the seven regions AR from the region AR(0) to the region AR(6). Likewise, the learned model 140 calculates the probability PR including "the position PS(2) at which the detection object OB(2) surrounded by the detection rectangle BB(2) is present" with respect to each of the seven regions AR from the region AR(0) to the region AR(6).

[0241] In Figure 8 In the example shown, "0.596" is calculated as the probability PR that the region AR(0) includes the position PS(1), that is, the probability PR(0-1). "0.357" is calculated as the probability PR that the region AR(1) includes the position PS(1), that is, the probability PR(1-1). "0.032" is calculated as the probability PR that the region AR(2) includes the position PS(1), that is, the probability PR(2-1). "0.004" is calculated as the probability PR that the region AR(3) includes the position PS(1), that is, the probability PR(3-1). "0.006" is calculated as the probability PR that the region AR(4) includes the position PS(1), that is, the probability PR(4-1). "0.002" is calculated as the probability PR that the region AR(5) includes the position PS(1), that is, the probability PR(5-1). "0.006" is calculated as the probability PR that the region AR(6) includes the position PS(1), that is, the probability PR(6-1).

[0242] In Figure 8 In the example shown, "0.642" is calculated as the probability PR that the region AR(0) includes the position PS(2), that is, the probability PR(0-2). "0.1" is calculated as the probability PR that the region AR(1) includes the position PS(2), that is, the probability PR(1-2). "0.179" is calculated as the probability PR that the region AR(2) includes the position PS(2), that is, the probability PR(2-2). "0.014" is calculated as the probability PR that the region AR(3) includes the position PS(2), that is, the probability PR(3-2). "0.013" is calculated as the probability PR that the region AR(4) includes the position PS(2), that is, the probability PR(4-2). "0.035" is calculated as the probability PR that the region AR(5) includes the position PS(2), that is, the probability PR(5-2). "0.017" is calculated as the probability PR that the region AR(6) includes the position PS(2), that is, the probability PR(6-2).

[0243] (Determination with respect to a specific region)

[0244] Figure 9 is a diagram indicating an example of a specific region IA determined by the detection device 10 (particularly, the determination section 170). The detection device 10 (particularly, the segmentation section 120) segments the photographed image PI illustrated Figure 9 into seven regions AR from the region AR(0) to the region AR(6). Also, the detection device 10 (particularly, the determination section 170) determines, for each detection rectangle BB arranged in the photographed image PI, a region AR in which the detection rectangle BB is present as a specific region IA.

[0245] The determination section 170 can also determine a position PS (underfoot position) of the detection object OB surrounded by the detection rectangle BB, and determine a region AR containing the position PS as the specific region IA of the detection rectangle BB. Specifically, the determination section 170 calculates the position PS (underfoot position) of the detection object OB from the detection rectangle BB (position, shape, and size of the detection rectangle BB), and determines a region AR containing the calculated position PS as the specific region IA. The determination section 170 can also determine a region AR containing a center position (or a center of gravity position) of the detection rectangle BB as the specific region IA of the detection rectangle BB.

[0246] In the photographed image PI illustrated Figure 9 in the photographed image PI illustrated, two detection rectangles BB are arranged. Specifically, the detection device 10 (particularly, the estimation section 150 (learned model 140)) estimates, for each of the detection rectangles BB, a position PS (underfoot position) of the detection object OB surrounded by the detection rectangle BB. Figure 9 The photographed image PI illustrated is arranged with the detection rectangle BB(1) and the detection rectangle BB(2).

[0247] The determination section 170 determines a region AR in which the detection rectangle BB(1) is present as a specific region IA(1), and a region AR in which the detection rectangle BB(2) is present as a specific region IA(2). For example, the determination section 170 determines a position PS(1) (underfoot position) of the detection object OB(1) surrounded by the detection rectangle BB(1), and a position PS(2) (underfoot position) of the detection object OB(2) surrounded by the detection rectangle BB(2). Specifically, the determination section 170 calculates the position PS(1) from the detection rectangle BB(1), and calculates the position PS(2) from the detection rectangle BB(2). For example, the determination section 170 determines a center position (or a center of gravity position) of the detection rectangle BB(1) as the position PS(1), and a center position (or a center of gravity position) of the detection rectangle BB(2) as the position PS(2). Also, the determination section 170 determines a region AR containing the position PS(1) (center position of the detection rectangle BB(1)) as the specific region IA(1), and a region AR containing the position PS(2) (center position of the detection rectangle BB(2)) as the specific region IA(2).

[0248] In Figure 9 the example, the determining section 170 determines that the region AR(1) among the seven regions AR from the region AR(0) to the region AR(6) contains the position PS(1) (the center position of the detection rectangle BB(1)). Thus, the determining section 170 determines the region AR(1) as the specific region IA(1) of the detection rectangle BB(1).

[0249] Also, the determining section 170 determines that the region AR(6) among the seven regions AR from the region AR(0) to the region AR(6) contains the position PS(2) (the center position of the detection rectangle BB(2)). Thus, the determining section 170 determines the region AR(6) as the specific region IA(2) of the detection rectangle BB(2).

[0250] (Example in which the Speculation Result is Determined to be Correct by Performing the First Determination Process)

[0251] Figure 10 is a view that shows an example in which the detection device 10 (particularly, the determining section 190) determines the speculation result to be correct by performing the first determination process. In Figure 10 , the determining section 190 performs the first determination process on the detection rectangle BB(1) among the two detection rectangles BB exemplified in Figure 9 , determines that the speculation that "the detection object OB(1) surrounded by the detection rectangle BB(1) is a person (human body)" is correct.

[0252] As explained using Figure 9 , the detection device 10 (particularly, the determining section 170) determines that the specific region IA(1) of the detection rectangle BB(1) arranged in the captured image PI exemplified in Figure 10 is the region AR(1).

[0253] In Figure 10 , the detection device 10 (particularly, the speculation section 150 (the learned model 140)) outputs the class of the "detection object OB(1) surrounded by the detection rectangle BB(1)". That is, the learned model 140 calculates (outputs) the "probability PR that the position PS(1) where the detection object OB(1) is located is contained" with respect to each of the seven regions AR from the region AR(0) to the region AR(6).

[0254] In Figure 10In this case, the learned model 140 calculates the probability PR(0-1) that the region AR(0) contains the "position PS(1) where the detection target OB(1) is present" as "0.3". The learned model 140 calculates the probability PR(1-1) that the region AR(1) contains the "position PS(1) where the detection target OB(1) is present" as "0.7". The learned model 140 calculates the probability PR(2-1) that the region AR(2) contains the "position PS(1) where the detection target OB(1) is present" as "0.05". The learned model 140 calculates the probability PR(3-1) that the region AR(3) contains the "position PS(1) where the detection target OB(1) is present" as "0.05". The learned model 140 calculates the probability PR(4-1) that the region AR(4) contains the "position PS(1) where the detection target OB(1) is present" as "0.02". The learned model 140 calculates the probability PR(5-1) that the region AR(5) contains the "position PS(1) where the detection target OB(1) is present" as "0.075". The learned model 140 calculates the probability PR(6-1) that the region AR(6) contains the "position PS(1) where the detection target OB(1) is present" as "0.075".

[0255] The determination section 190 selects the region AR(1) as the region AR having the highest probability PR of containing the "position PS(1) where the detection target OB(1) is present" among the seven regions AR from the region AR(0) to the region AR(6). The determination section 190 confirms whether the "region AR(1) having the highest probability PR of containing the "position PS(1) where the detection target OB(1) is present" coincides with the "region AR(1) that is the specified region IA(1)". Then, when the determination section 190 confirms that the region AR(1) having the highest probability PR coincides with the region AR(1) that is the specified region IA(1), the determination section 190 determines that the estimation that the "detection target OB(1) surrounded by the detection rectangle BB(1) is a person (human body)" is correct.

[0256] (Example in which the determination section 190 determines that the estimation result is incorrect by performing the first determination processing)

[0257] Figure 11 is a diagram illustrating an example in which the detection device 10 (particularly, the determination section 190) determines that the estimation result is incorrect by performing the first determination processing. In this case, the determination section 190 performs the first determination processing on the detection rectangle BB(2) among the two detection rectangles BB illustrated in Figure 11 Figure 9 The determination section 190 determines that the estimation that the "detection target OB(2) surrounded by the detection rectangle BB(2) is a person (human body)" is incorrect.

[0258] As used herein, the term "determination" means that the determination section 190 determines that the estimation result is correct or incorrect. Figure 9 ​As explained, the detection device 10 (especially the determination section 170) determines Figure 11 The specific region IA(2) of the detection rectangle BB(2) arranged in the example photographed image PI is the region AR(6).

[0259] In Figure 11 , the detection device 10 (especially the presumption section 150 (learned model 140)) outputs the class of the "detection object OB(2) surrounded by the detected rectangle BB(2)". That is, the learned model 140 calculates (outputs) the "probability PR that the region AR(0) to the region AR(6) contains the position PS(2) where the detection object OB(2) is located" for each of the seven regions AR.

[0260] In Figure 11 , the learned model 140 calculates the probability PR(0-2) that the region AR(0) contains the "position PS(2) where the detection object OB(2) is located" as "0.1". The learned model 140 calculates the probability PR(1-2) that the region AR(1) contains the "position PS(2) where the detection object OB(2) is located" as "0.2". The learned model 140 calculates the probability PR(2-2) that the region AR(2) contains the "position PS(2) where the detection object OB(2) is located" as "0.3". The learned model 140 calculates the probability PR(3-2) that the region AR(3) contains the "position PS(2) where the detection object OB(2) is located" as "0.1". The learned model 140 calculates the probability PR(4-2) that the region AR(4) contains the "position PS(2) where the detection object OB(2) is located" as "0.05". The learned model 140 calculates the probability PR(5-2) that the region AR(5) contains the "position PS(2) where the detection object OB(2) is located" as "0.05". The learned model 140 calculates the probability PR(6-2) that the region AR(6) contains the "position PS(2) where the detection object OB(2) is located" as "0.2".

[0261] The determination section 190 selects the region AR(2) as the region AR with the highest probability PR that contains the "position PS(2) where the detection object OB(2) is located" among the seven regions AR from the region AR(0) to the region AR(6). The determination section 190 confirms whether the "region AR(2) with the highest probability PR that contains the "position PS(2) where the detection object OB(2) is located" coincides with the "region AR(6) as the specific region IA(2)". Also, the determination section 190 determines that the presumption that the "detection object OB(2) surrounded by the detected rectangle BB(2) is a person (human being)" is wrong when the region AR(2) with the highest probability PR is confirmed not to coincide with the region AR(6) as the specific region IA(2).

[0262] As explained using Figure 10 and Figure 11 , in the detection device 10, the determination section 190 determines that "the detection object OB is a person" when the specific region IA coincides with "the region AR having the highest probability PR among the plurality of regions AR".

[0263] According to the configuration, the detection device 10 determines that "the detection object OB is a person" when "the region AR in which the detection rectangle BB that surrounds the detection object OB presumed to be a person is located (i.e., the specific region IA)" coincides with "the region AR having the highest probability PR among the plurality of regions AR". That is, the detection device 10 determines that "the detection object OB is a person" when the specific region IA coincides with "the region AR having the highest probability PR including the position PS in which the detection object OB presumed to be a person is located".

[0264] It can be considered that, in a case where the detection object OB presumed to be a person is actually a person, the region AR in which the detection rectangle BB that surrounds the detection object OB is located, i.e., the specific region IA, coincides with "the region AR having the highest probability PR including the position PS in which the detection object OB is located" with high likelihood.

[0265] Therefore, the detection device 10 determines that "the detection object OB is a person" when the specific region IA coincides with "the region AR having the highest probability PR among the plurality of regions AR".

[0266] Therefore, the detection device 10 has the effect that it can detect a person with high accuracy from an image (captured image PI) captured using a fisheye lens.

[0267] In a case where the determination section 170 determines the position PS (underfoot position) of "the detection object OB surrounded by the detection rectangle BB and presumed to be a person" as the specific region IA, the determination section 190 can also determine "whether or not the underfoot position is included in the region AR having the highest probability PR". The determination section 190 can also determine that "the detection object OB is a person" when the underfoot position is included in the region having the highest probability PR.

[0268] (Regarding a modification example of the first determination processing)

[0269] In the example explained using Figure 10 and Figure 11 , the determination section 190 verifies the correctness of the presumption that "the detection object OB surrounded by the detection rectangle BB is a person (human body)" by confirming the coincidence of the following two. That is, the determination section 190 verifies the correctness of the presumption according to the coincidence / non-coincidence of "the region AR having the highest probability PR including the position PS in which the detection object OB surrounded by the detection rectangle BB is located" and the region AR in which the detection rectangle BB is located, i.e., the specific region IA.

[0270] However, as Figure 12 As illustrated, when the detected object OB, which is presumed to be a "person", is located near the boundary of multiple regions AR, the region AR with the highest probability PR sometimes does not match the specific region IA, even though the presumption is correct.

[0271] Figure 12 This is an example of an image PI taken near the boundary of multiple regions AR, representing a detected object OB presumed to be a "person". The detection device 10 (especially the segmentation unit 120) will... Figure 12 The illustrated captured image PI is divided into seven regions AR up to region AR(0) and region AR(6).

[0272] exist Figure 12 In this process, the prediction unit 150 (learned model 140) configures a detection rectangle BB that surrounds the detection object OB, which is predicted to be "human (human body)," for the captured image PI, which has been segmented into seven regions AR (0) to AR (6). Specifically, the learned model 140 configures a detection rectangle BB (3) for the captured image PI.

[0273] exist Figure 12 In this process, the detection device 10 (especially the determination unit 170) determines the specific region IA (3) of the seven regions AR from region AR (0) to region AR (6), which is the region AR in which the detection rectangle BB (3) is located, as region AR (2). For example, the determination unit 170 calculates the position PS (3) (foot position) of the detection object OB (3) based on the detection rectangle BB (3), and determines the region AR (2) containing the calculated position PS (3) as the specific region IA (3). The determination unit 170 may also determine the region AR (2) containing the center position (or centroid position) of the detection rectangle BB (3) as the specific region IA (3).

[0274] Figure 13 The diagram illustrates the following example, where, despite the discrepancy between the region AR with the highest probability PR and the specific region IA, the detection device 10 performs the first determination process and still determines that the conjecture "the detected object OB surrounded by the detection rectangle BB is a person (human body)" is correct.

[0275] If used Figure 12 As explained, the determination part 170 determines Figure 13 The specific region IA(3) of the detection rectangle BB(3) configured in the illustrated captured image PI is region AR(2).

[0276] exist Figure 13In this process, the detection device 10 (especially the inference unit 150 (learned model 140)) outputs the category of "the detected object OB (3) surrounded by the detection rectangle BB (3)". That is, the learned model 140 calculates (outputs) the probability PR of the position PS (3) where the detected object OB (3) is located for each of the seven regions AR from region AR (0) to region AR (6).

[0277] exist Figure 13 In the training, after model 140 completed the training, the probability PR(0-3) of region AR(0) containing the position PS(3) of the detected object OB(3) surrounded by the detected rectangle BB(3) was calculated to be "0.2". After model 140 completed the training, the probability PR(1-3) of region AR(1) containing the position PS(3) of the detected object OB(3) was calculated to be "0.45". After model 140 completed the training, the probability PR(2-3) of region AR(2) containing the position PS(3) of the detected object OB(3) was calculated to be "0.4". After model 140 completed the training, the probability PR(3-3) of region AR(3) containing the position PS(3) of the detected object OB(3) was calculated to be "0.05". After model 140 completed the training, the probability PR(4-3) of region AR(4) containing the position PS(3) of the detected object OB(3) was calculated to be "0.02". After learning, Model 140 calculated the probability PR(5-3) that region AR(5) contains the location PS(3) of the detected object OB(3) as "0.075". After learning, Model 140 calculated the probability PR(6-3) that region AR(6) contains the location PS(3) of the detected object OB(3) as "0.075".

[0278] exist Figure 13 In the above, the region AR with the highest probability PR containing "the location PS(3) of the detected object OB(3)" is region AR(1). The region AR with the second highest probability PR containing "the location PS(3) of the detected object OB(3)" is region AR(2), which is adjacent to region AR(1). The probability PR(1-3) of region AR(1) is "0.45" and below "0.5", and the probability PR(2-3) of region AR(2) is "0.4" and below "0.5".

[0279] exist Figure 13 In the process, the determination unit 190 selects region AR (1) from the seven regions AR (0) to region AR (6) as the region AR with the highest probability PR containing "the location PS (3) of the detected object OB (3)".

[0280] Further, the determination section 190 confirms whether the region AR (1) having the highest probability PR is inconsistent with the region AR (2) that is the specific region IA (3).

[0281] Therefore, in Figure 13 , the determination section 190 further confirms that the region AR adjacent to the region AR (1) having the highest probability PR is the region AR (2).

[0282] Further, the determination section 190 determines whether the "region AR (2) adjacent to the region AR (1) having the highest probability PR" is consistent with the "region AR (2) that is the specific region IA (3)". The determination section 190 determines that the detection rectangle BB (3) is correct when it is confirmed that the "region AR (2) adjacent to the region AR (1) having the highest probability PR" is consistent with the "region AR (2) that is the specific region IA (3)". That is, the determination section 190 determines that the presumption that the "detection object OB (3) surrounded by the detection rectangle BB (3) is a person (human body)" is correct.

[0283] As described using Figure 13 , the determination section 190 can also determine that the "detection object OB surrounded by the detection rectangle BB" is a person when the specific region IA is consistent with the "region AR having the highest probability PR" or the "region AR adjacent to the region AR having the highest probability PR".

[0284] Figure 13 In the example shown in FIG. 17, the region AR (1) is adjacent (adjacent) to the region AR (2). Further, the region AR having the highest probability PR among the seven regions AR from the region AR (0) to the region AR (6) including the position PS (3) where the detection object OB (3) is present is the region AR (1). Moreover, the position PS (3) where the detection object OB (3) surrounded by the detection rectangle BB (3) and presumed to be a "person (human body)" is present is located near the boundary line between the region AR (1) and the region AR (2).

[0285] In this case, the determination section 190 determines whether the specific region IA (3) of the detection rectangle BB (3) is consistent with the "region AR (1) having the highest probability PR" or the "region AR (2) adjacent to the region AR (1)". Further, when the specific region IA (3) of the detection rectangle BB (3) is consistent with the "region AR (1) having the highest probability PR" or the "region AR (2) adjacent to the region AR (1)", the determination section 190 determines that the presumption that the detection object OB (3) is a person is correct.

[0286] For example, the determination section 190 determines whether the detection rectangle BB (3) includes the region AR (1) having the highest probability PR in the example shown in FIG. 17. Figure 13 In the example shown in FIG. 17, the determination section 190 determines whether the detection rectangle BB (3) includes the region AR (1) having the highest probability PR. Figure 13The part indicated by a chain line at a point in the middle. Furthermore, the determination unit 190 confirms that "the detection rectangle BB(3) is contained within..." Figure 13 When the part indicated by the chain line is "in the middle", the inference that "the object being tested is a person" is correct.

[0287] The determination unit 190 can also determine whether the position PS(3) of the detection object OB(3) which is surrounded by the detection rectangle BB(3) and is presumed to be a "human (human body)" is included. Figure 13 The part indicated by the chain line at a point. Specifically, the determination unit 190 can also determine whether the position PS(3) calculated by the determination unit 170 based on the detection rectangle BB(3) is included. Figure 13 The determination unit 190 may also set the center position (or centroid position) of the detection rectangle BB(3) as position PS(3) and determine whether "position PS(3) is included in the middle". Figure 13 "In the part indicated by a chain line at a point in the middle." Furthermore, the determination unit 190 can also confirm that "position PS(3) is included." Figure 13 When the part indicated by the chain line is "in the middle", the inference that "the object being tested is a person" is correct.

[0288] exist Figure 7 In S250, the determination unit 190 determines whether the "position PS (foot position) of the detected object OB surrounded by the detection rectangle BB" is included in the output category (i.e., the "region AR with the highest probability PR"). However, in S250, the determination unit 190 may also determine whether the "position PS (foot position) of the detected object OB surrounded by the detection rectangle BB" is included in either of the following two regions AR. That is, the determination unit 190 may also determine whether the "position PS of the detected object OB" is included in the "region AR with the highest probability PR" or the "position adjacent to the region AR with the highest probability PR". In other words, the determination unit 190 may also determine whether a specific region IA of the detection rectangle BB is consistent with the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR".

[0289] exist Figure 13 The part indicated by the chain at a point, in addition to region AR(1), also includes the following two regions. That is, it includes: "the region adjacent to region AR(1) when region AR(2) adjacent to region AR(1) is divided into multiple regions"; and "the region adjacent to region AR(1) when region AR(6) adjacent to region AR(1) is divided into multiple regions".

[0290] As explained so far, for example in a case where the detection rectangle BB is arranged so as to straddle the region AR(x) and the region AR(y) that are adjacent to each other, the determination section 190 uses the specific region IA of the detection rectangle BB to determine whether the category is correct or not as follows. That is, the determination section 190 determines whether the specific region IA of the detection rectangle BB (that is, the region AR(x) or the region AR(y)) coincides with the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR".

[0291] In other words, in a case where the position PS of the detection target OB calculated from the detection rectangle BB is in the vicinity of the boundary line of the region AR(x) and the region AR(y) that are adjacent to each other, the determination section 190 uses the specific region IA of the detection rectangle BB to determine whether the category is correct or not as follows. That is, the determination section 190 determines whether the specific region IA of the detection rectangle BB (that is, the region AR(x) or the region AR(y)) coincides with the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR". The "vicinity of the boundary line" can also be the buffer region BA described later. The position PS is calculated from the detection rectangle BB (the position, shape, and size of the detection rectangle BB), and can be, for example, the center position (or the barycentric position) of the detection rectangle BB. Figure 18

[0292] In a case where the detection rectangle BB is arranged so as to straddle the region AR(x) and the region AR(y) that are adjacent to each other, it can be considered that the specific region IA of the detection rectangle BB is the region AR(x) or the region AR(y). Also, it can be considered that the "region AR with the highest probability PR" is the region AR(x) or the region AR(y), and the "region AR adjacent to the region AR with the highest probability PR" is the region AR(y) or the region AR(x).

[0293] The processing of the determination section 190 explained above can be summarized as follows. That is, the determination section 190 determines that the detection target OB is a person when the specific region IA coincides with the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR" among the plurality of regions AR. Figure 13 According to the structure, the detection device 10 determines that the "detection target OB is a person" when the specific region IA coincides with the "region AR with the highest probability PR" or the "region AR adjacent to the region AR with the highest probability PR" among the plurality of regions AR.

[0294]

[0295] ​​Here, it is assumed that a person is imaged in a manner that straddles two of the plurality of regions AR, or that the position of the person is located near the boundary of two regions AR, and the like in the captured image PI.

[0296] In this case, it can be considered that the region AR in which the detection rectangle BB that surrounds the detection object OB is located, that is, the specific region IA, coincides with any one of the regions AR below with high probability in the case where the detection object OB that is presumed to be a person is actually a person. That is, it can be considered that the specific region IA coincides with the "region AR with the highest probability PR among the plurality of regions AR" or the "region AR among the plurality of regions AR that is adjacent to the region AR with the highest probability PR" with high probability.

[0297] Therefore, the detection device 10 determines that the detection object OB is a person when the specific region IA coincides with the "region AR with the highest probability PR among the plurality of regions AR", and determines that the detection object OB is a person when the specific region IA coincides with the "region AR among the plurality of regions AR that is adjacent to the region AR with the highest probability PR".

[0298] Therefore, the detection device 10 has the effect of being able to detect a person with high precision from the captured image PI even in the case where a person is imaged near the boundary of two of the plurality of regions AR in the captured image PI.

[0299] In the case where the determination section 170 determines the underfoot position as the specific region IA, the determination section 190 can also determine whether or not the underfoot position is included in the "region AR with the highest probability PR" or the "region AR that is adjacent to the region AR with the highest probability PR". The determination section 190 can also determine that the detection object OB is a person when the underfoot position is included in the "region AR with the highest probability PR" or the "region AR that is adjacent to the region AR with the highest probability PR".

[0300] (Example of Person Detection Process Including Second Determination Process)

[0301] Figure 14 is a flowchart that illustrates an example of a person detection process that performs the second determination process on the output of the learned model 140 to detect a person. Figure 14 The processes of S310 and S320 of the illustrated second determination process are the same as Figure 7 The processes of S210 and S220 of the illustrated first determination process are the same as well.

[0302] That is, the image acquisition unit 110 acquires the photographed image PI, and the division unit 120 divides the photographed image PI acquired by the image acquisition unit 110 into a plurality of regions AR (S310). The estimation unit 150 performs person detection and class output using the learned model 140 with respect to the photographed image PI acquired with reference to the divided image information 130 (the photographed image PI divided into a plurality of regions AR) (S320).

[0303] The detection device 10 executes the processes of S330 to S370 for each detection rectangle BB (that is, each detection object OB) with the number of persons detected from the photographed image PI by the estimation unit 150, in other words, the number of detection rectangles BB arranged by the estimation unit 150.

[0304] The determination unit 190 acquires the object reliability OR and the probability PR of the class (that is, the region AR corresponding to the specific region IA) to which the detection rectangle BB belongs with respect to each detection rectangle BB (that is, the detection object OB) with reference to the object reliability table 163 and the probability table 162 (S330).

[0305] Specifically, the determination unit 190 acquires the object reliability OR of each detection rectangle BB with reference to the object reliability table 163. Also, the determination unit 190 acquires the specific region IA of each detection rectangle BB with reference to the specific region table 180. Also, the determination unit 190 acquires the probability PR of the region AR corresponding to the acquired specific region IA with reference to the probability table 162.

[0306] The determination unit 190 multiplies the object reliability OR of each detection rectangle BB by the probability PR of the region AR corresponding to the specific region IA of the detection rectangle BB to calculate the class reliability value CR of the detection rectangle BB (S340). Also, the determination unit 190 determines whether the class reliability value CR of the detection rectangle BB is greater than the predetermined value TH (S350).

[0307] If the class reliability value CR of the detection rectangle BB is greater than the predetermined value TH (Yes in S350), the determination unit 190 adopts the detection rectangle BB as a correct detection result (S360). That is, the determination unit 190 determines that the class reliability value CR of the detection rectangle BB is greater than the predetermined value TH and that the estimation that the detection object OB surrounded by the detection rectangle BB is a person is correct.

[0308] If the category reliability value CR of the detection rectangle BB is equal to or lower than the prescribed value TH (NO in S350), the determination section 190 removes the detection rectangle BB as a false detection (S370). That is, the determination section 190 determines that the category reliability value CR of the detection rectangle BB is equal to or lower than the prescribed value TH and the presumption that the detection object OB surrounded by the detection rectangle BB is a person is false.

[0309] (Example of determining the presumption result as correct by performing the second determination processing)

[0310] Figure 15 is a diagram illustrating an example in which the detection device 10 (particularly, the determination section 190) determines the presumption result as correct by performing the second determination processing. In Figure 15 , the determination section 190 performs the second determination processing on the detection rectangle BB (1) disposed in the captured image PI illustrated in Figure 15 , determines that the presumption that the detection object OB (1) surrounded by the detection rectangle BB (1) is a person (human body) is correct.

[0311] As explained using Figure 9 , the detection device 10 (particularly, the determination section 170) determines that the specific region IA (1) of the detection rectangle BB (1) disposed in the captured image PI illustrated in Figure 15 is the region AR (1).

[0312] In Figure 15 , the detection device 10 (particularly, the presumption section 150 (learned model 140)) outputs the category of the detection object OB (1) surrounded by the detection rectangle BB (1). That is, the learned model 140 calculates (outputs) the probability PR that the position PS (1) where the detection object OB (1) is present is included for each of the seven regions AR from the region AR (0) to the region AR (6).

[0313] In Figure 15In this case, the learned model 140 calculates the probability PR(0-1) that the region AR(0) contains the "position PS(1) where the detection object OB(1) is present" as "0". The learned model 140 calculates the probability PR(1-1) that the region AR(1) contains the "position PS(1) where the detection object OB(1) is present" as "0.4". The learned model 140 calculates the probability PR(2-1) that the region AR(2) contains the "position PS(1) where the detection object OB(1) is present" as "0.005". The learned model 140 calculates the probability PR(3-1) that the region AR(3) contains the "position PS(1) where the detection object OB(1) is present" as "0.45". The learned model 140 calculates the probability PR(4-1) that the region AR(4) contains the "position PS(1) where the detection object OB(1) is present" as "0.005". The learned model 140 calculates the probability PR(5-1) that the region AR(5) contains the "position PS(1) where the detection object OB(1) is present" as "0.14". The learned model 140 calculates the probability PR(6-1) that the region AR(6) contains the "position PS(1) where the detection object OB(1) is present" as "0".

[0314] Further, in the case where the learned model 140 determines that the detection object OB(1) is a person, the learned model 140 calculates the probability PR(1-1) that the region AR(1) contains the "position PS(1) where the detection object OB(1) is present" as "0.4". Figure 15 In this case, the learned model 140 calculates the object reliability OR(1) of the detection rectangle BB(1) as "920".

[0315] The determination unit 190 multiplies the probability PR(1-1) of the region AR(1) corresponding to the "specific region IA(1) determined with respect to the detection rectangle BB(1)" by the object reliability OR(1) of the detection rectangle BB(1) to calculate the class reliability value CR(1) of the detection rectangle BB(1). That is, the determination unit 190 multiplies the "probability PR(1-1) of the region AR(1): 0.4" by the "object reliability OR(1) of the detection rectangle BB(1): 920" to calculate the "class reliability value CR(1) of the detection rectangle BB(1): 368".

[0316] The determination unit 190 determines that the detection object OB surrounded by the detection rectangle BB is a person when the class reliability value CR of the detection rectangle BB is greater than the predetermined value TH. For example, when the determination unit 190 determines that the "class reliability value CR(1) of the detection rectangle BB(1): 368" is greater than the "predetermined value TH: 350", the determination unit 190 determines that the detection object OB(1) surrounded by the detection rectangle BB(1) is a person.

[0317] (Example of determining that the result of the second determination process is incorrect)

[0318] Figure 16 is a diagram indicating an example in which the detection device 10 (particularly, the determination section 190) executes the second determination processing and determines that the presumption is wrong. In Figure 16 , the determination section 190 executes the second determination processing on the detection rectangle BB(2) disposed in the photographed image PI illustrated in Figure 16 , and determines that the presumption that the detection object OB(2) surrounded by the detection rectangle BB(2) is a person (human body) is wrong.

[0319] As explained using Figure 9 , the detection device 10 (particularly, the determination section 170) determines that the specific region IA(2) of the detection rectangle BB(2) disposed in the photographed image PI illustrated in Figure 16 is the region AR(6).

[0320] In Figure 16 , the detection device 10 (particularly, the presumption section 150 (learned model 140)) outputs the class of the detection object OB(2) surrounded by the detection rectangle BB(2). That is, the learned model 140 calculates (outputs) the probability PR that the region AR(0) to the region AR(6) contains the position PS(2) at which the detection object OB(2) is located for each of the seven regions AR.

[0321] In Figure 16 , the learned model 140 calculates the probability PR(0-2) that the region AR(0) contains the position PS(2) at which the detection object OB(2) is located as "0.1". The learned model 140 calculates the probability PR(1-2) that the region AR(1) contains the position PS(2) at which the detection object OB(2) is located as "0.2". The learned model 140 calculates the probability PR(2-2) that the region AR(2) contains the position PS(2) at which the detection object OB(2) is located as "0.3". The learned model 140 calculates the probability PR(3-2) that the region AR(3) contains the position PS(2) at which the detection object OB(2) is located as "0.1". The learned model 140 calculates the probability PR(4-2) that the region AR(4) contains the position PS(2) at which the detection object OB(2) is located as "0.05". The learned model 140 calculates the probability PR(5-2) that the region AR(5) contains the position PS(2) at which the detection object OB(2) is located as "0.05". The learned model 140 calculates the probability PR(6-2) that the region AR(6) contains the position PS(2) at which the detection object OB(2) is located as "0.2".

[0322] Further, in Figure 16In the process, after learning the model 140, the object reliability OR(2) of the detected rectangle BB(2) is calculated to be "625".

[0323] The determination unit 190 multiplies the probability PR (6-2) of the region AR (6) corresponding to the specific region IA (2) determined about the detection rectangle BB (2) by the object reliability OR (2) of the detection rectangle BB (2) to calculate the category reliability value CR (2) of the detection rectangle BB (2). That is, the determination unit 190 multiplies the probability PR (6-2) of the region AR (6): 0.2 by the object reliability OR (2) of the detection rectangle BB (2): 625 to calculate the category reliability value CR (2): 125 of the detection rectangle BB (2).

[0324] The determination unit 190 checks whether the category reliability value CR of the detection rectangle BB is greater than the specified value TH. If it checks that the category reliability value CR of the detection rectangle BB is less than or equal to the specified value TH, it determines that the assumption that "the detection object OB surrounded by the detection rectangle BB is a person" is incorrect. For example, when it checks that "the category reliability value CR(2) of the detection rectangle BB(2): 125" is less than or equal to "the specified value TH: 350", the determination unit 190 determines that the assumption that "the detection object OB(2) surrounded by the detection rectangle BB(2) is a person" is incorrect.

[0325] use Figure 15 and Figure 16 The processing of the determination unit 190 described herein can be summarized as follows. For example, if the determination unit 190 determines that "the detected object OB is a person" when the category reliability value CR, calculated by multiplying the probability PR of "the region AR corresponding to the specific region IA among multiple regions AR" by the object reliability OR of the detection rectangle BB, is greater than a predetermined value TH, then the determination unit 190 determines that "the detected object OB is a person". The object reliability OR of the detection rectangle BB is a value representing the probability that certain objects are surrounded by the detection rectangle BB.

[0326] According to the structure, when the detection device 10 determines that "the detected object OB is a person" when the probability PR of "the region AR corresponding to the specific region IA in multiple regions AR" is multiplied by the object reliability OR of the detection rectangle BB and the result is greater than the specified value TH.

[0327] It can be assumed that, if the detected object OB, which is presumed to be a human, is actually a human, the probability PR that "the region AR, i.e., the specific region IA, where the detection rectangle BB surrounding the detected object OB is located" contains "the location PS of the detected object OB" is sufficiently high. Moreover, it can be assumed that, if the detected object OB, which is presumed to be a human, is actually a human, the value representing the probability that certain objects are surrounded by "the detection rectangle BB surrounding the detected object OB," i.e., the object reliability OR of the detection rectangle BB, is also sufficiently high.

[0328] Therefore, the detection device 10 determines that the detection object OB is a person when the category reliability value CR calculated by multiplying the probability PR of the region AR corresponding to the specific region IA among the plurality of regions AR by the object reliability OR of the detection rectangle BB is greater than the prescribed value TH.

[0329] Therefore, the detection device 10 functions to enable high-precision detection of a person from the image (captured image PI) captured using the fisheye lens.

[0330] In a case where the determination section 170 determines the underfoot position as the specific region IA, the determination section 190 can also determine that the detection object OB is a person when the category reliability value CR calculated by multiplying the probability PR of the region AR including the underfoot position by the object reliability OR is greater than the prescribed value TH.

[0331] (Second Determination Process)

[0332] In the example described using Figure 15 and Figure 16 , the determination section 190 calculates the category reliability value CR of the detection rectangle BB by multiplying the probability PR of the region AR corresponding to the specific region IA determined with respect to the detection rectangle BB by the object reliability OR of the detection rectangle BB. Also, the determination section 190 determines whether the calculated category reliability value CR is greater than the prescribed value TH. If the calculated category reliability value CR is greater than the prescribed value TH, the determination section 190 determines that the presumption that the detection object OB surrounded by the detection rectangle BB is a person is correct. If the calculated category reliability value CR is the prescribed value TH or less, the determination section 190 determines that the presumption that the detection object OB surrounded by the detection rectangle BB is a person is incorrect.

[0333] However, as exemplified in Figure 17 , in a case where the detection object OB presumed to be a person is located near the boundary line of the plurality of regions AR, the category reliability value CR of the detection rectangle BB is sometimes the prescribed value TH or less even though the presumption is correct due to the small probability PR. That is, in a case where the detection rectangle BB straddles two regions AR among the plurality of regions AR, the category reliability value CR of the detection rectangle BB is sometimes the prescribed value TH or less even though the detection object OB surrounded by the detection rectangle BB is a person due to the small probability PR.

[0334] Figure 17 is a diagram indicating an example in which the detection device 10 (particularly, the determination section 190) determines the presumption result to be correct even in a case where the detection rectangle BB (in other words, the position PS where the detection object OB is located) is located near the boundary line of the plurality of regions AR by performing the second determination process.Figure 17 In the example shown, the detection rectangle BB(3) is configured across the region AR(1) and the region AR(2) that are adjacent to each other. In other words, the position PS(3) of the detection object OB(3) surrounded by the detection rectangle BB(3) is located near the boundary line between the region AR(1) and the region AR(2). The position PS(3) is calculated from the detection rectangle BB(3), and can be, for example, the center position or the barycentric position of the detection rectangle BB(3).

[0335] In Figure 17 , the detection device 10 (particularly the division section 120) divides the captured image PI into seven regions AR of the regions AR(0) to AR(6). Also, in Figure 17 , the estimation section 150 (the learned model 140) configures one detection rectangle BB that surrounds the detection object OB estimated as "person (human body)" for the captured image PI divided into seven regions AR of the regions AR(0) to AR(6). Specifically, the learned model 140 configures the detection rectangle BB(3) for the captured image PI. Further, in Figure 17 , the detection device 10 (particularly the determination section 170) determines that the region AR in which the detection rectangle BB(3) is located among the seven regions AR of the regions AR(0) to AR(6), that is, the specific region IA(3) is the region AR(2). The determination section 170 calculates the position PS(3) (underfoot position) of the detection object OB(3) surrounded by the detection rectangle BB(3) from the detection rectangle BB(3). Also, the determination section 170 determines the region AR including the calculated position PS(3), that is, the region AR(2) as the specific region IA(3). For example, the determination section 170 can determine the region AR including the center position (or the barycentric position) of the detection rectangle BB(3), that is, the region AR(2) as the specific region IA(3).

[0336] In Figure 17 , the detection device 10 (particularly the estimation section 150 (the learned model 140)) outputs the class of the detection object OB(3) surrounded by the detection rectangle BB(3). That is, the learned model 140 calculates (outputs) the probability PR of including the position PS(3) in which the detection object OB(3) is located for each of the seven regions AR of the regions AR(0) to AR(6).

[0337] In Figure 17In this case, the learned model 140 calculates the probability PR(0-3) that the region AR(0) contains the "position PS(3) where the detection target OB(3) is present" as "0.3". The learned model 140 calculates the probability PR(1-3) that the region AR(1) contains the "position PS(3) where the detection target OB(3) is present" as "0.3". The learned model 140 calculates the probability PR(2-3) that the region AR(2) contains the "position PS(3) where the detection target OB(3) is present" as "0.45". The learned model 140 calculates the probability PR(3-3) that the region AR(3) contains the "position PS(3) where the detection target OB(3) is present" as "0.05". The learned model 140 calculates the probability PR(4-3) that the region AR(4) contains the "position PS(3) where the detection target OB(3) is present" as "0.02". The learned model 140 calculates the probability PR(5-3) that the region AR(5) contains the "position PS(3) where the detection target OB(3) is present" as "0.075". The learned model 140 calculates the probability PR(6-3) that the region AR(6) contains the "position PS(3) where the detection target OB(3) is present" as "0.075".

[0338] Further, in the case where the learned model 140 calculates the probability PR(0-3) that the region AR(0) contains the "position PS(3) where the detection target OB(3) is present" as "0.3", the learned model 140 calculates the probability PR(1-3) that the region AR(1) contains the "position PS(3) where the detection target OB(3) is present" as "0.3". The learned model 140 calculates the probability PR(2-3) that the region AR(2) contains the "position PS(3) where the detection target OB(3) is present" as "0.45". The learned model 140 calculates the probability PR(3-3) that the region AR(3) contains the "position PS(3) where the detection target OB(3) is present" as "0.05". The learned model 140 calculates the probability PR(4-3) that the region AR(4) contains the "position PS(3) where the detection target OB(3) is present" as "0.02". The learned model 140 calculates the probability PR(5-3) that the region AR(5) contains the "position PS(3) where the detection target OB(3) is present" as "0.075". The learned model 140 calculates the probability PR(6-3) that the region AR(6) contains the "position PS(3) where the detection target OB(3) is present" as "0.075". Figure 17 In this case, the learned model 140 calculates the object reliability OR(3) of the detection rectangle BB(3) as "800".

[0339] The determination section 190 calculates the average of the probability PR(2-3) of the determined specific region IA(3), that is, the region AR(2), and the probability PR(1-3) of the region AR(1) adjacent to the region AR(2). That is, the determination section 190 calculates the average of "the probability PR(2-3) of the region AR(2): 0.45" and "the probability PR(1-3) of the region AR(1): 0.3", that is, "0.375".

[0340] The determination section 190 multiplies the calculated average by the object reliability OR(3) of the detection rectangle BB(3) to calculate the category reliability value CR(3) of the detection rectangle BB(3). That is, the determination section 190 multiplies "the average: 0.375" by "the object reliability OR(3): 800" to calculate "the category reliability value CR(3): 300".

[0341] The determination section 190 determines whether the category reliability value CR of the detected rectangle BB is greater than a predetermined value TH, and if it is determined that the category reliability value CR of the detected rectangle BB is greater than the predetermined value TH, it is determined that the presumption that "the detection object OB surrounded by the detected rectangle BB is a person" is correct. For example, when it is determined that "the category reliability value CR (3) of the detected rectangle BB (3): 300" is "a predetermined value TH: 250" or less, the determination section 190 determines that the presumption that "the detection object OB (3) surrounded by the detected rectangle BB (3) is a person" is correct.

[0342] In Figure 14 In the example shown in FIG. 17, the detected rectangle BB (3) exists across the region AR (2) that is the specific region IA (3) and the region AR (1) that is adjacent to the region AR (2), that is, the detection object OB (3) is located near the boundary line between the region AR (2) and the region AR (1). In other words, the position PS (3) of the detection object OB (3) calculated from the detected rectangle BB (3) is located near the boundary line between the region AR (2) that is the specific region IA (3) and the region AR (1) that is adjacent to the region AR (2). For example, the position PS (3) can also be the center position or the barycentric position of the detected rectangle BB (3).

[0343] Therefore, the determination section 190 multiplies the average value of the probability PR (2-3) of the region AR (2) and the probability PR (1-3) of the region AR (1) by the object reliability OR (3) of the detected rectangle BB (3) to calculate the category reliability value CR (3) of the detected rectangle BB (3). Also, the determination section 190 determines whether the presumption that "the detection object OB (3) surrounded by the detected rectangle BB (3) is a person" is correct or not based on whether the category reliability value CR (3) of the detected rectangle BB (3) is greater than the predetermined value TH.

[0344] In the case where the detected rectangle BB (that is, the position PS of the detection object OB surrounded by the detected rectangle BB) is located near the boundary line between the specific region IA (that is, the region AR (x)) of the detected rectangle BB and the region AR (y) that is adjacent to the region AR (x), the determination section 190 performs the following processing.

[0345] That is, the determination section 190 calculates the probability PR(x) that the region AR(x) contains the position PS of the detection object OB surrounded by the detection rectangle BB and the probability PR(y) that the region AR(y) contains the position PS of the detection object OB surrounded by the detection rectangle BB. The determination section 190 multiplies the average of the probability PR(x) and the probability PR(y) by the object reliability OR of the detection rectangle BB to calculate the class reliability value CR of the detection rectangle BB. Also, the determination section 190 determines whether the presumption that the detection object OB surrounded by the detection rectangle BB is a person is correct or not, depending on whether the class reliability value CR of the detection rectangle BB is greater than a predetermined value TH.

[0346] In other words, the determination section 190 calculates the average of the probabilities PR of the two regions AR in place of the probability PR of the class to which the detection rectangle BB belongs (i.e., the probability PR of the region AR that is the specific region IA) in S330. Figure 14 In the illustrated S330, the average of the probabilities PR of the two regions AR is calculated in place of the probability PR of the class to which the detection rectangle BB belongs (i.e., the probability PR of the region AR that is the specific region IA). That is, the determination section 190 calculates the average of the probability PR(x) of the region AR(x) that is the specific region IA and the probability PR(y) of the region AR(y) that is adjacent to the region AR(x). Also, the determination section 190 calculates the class reliability value CR in S340 in place of the object reliability OR of the detection rectangle BB. Figure 15 In the illustrated S340, the class reliability value CR is calculated by multiplying the average of the probability PR(x) and the probability PR(y) by the object reliability OR of the detection rectangle BB in place of the probability PR of the region AR that is the specific region IA.

[0347] In the illustrated S340, the class reliability value CR is calculated by multiplying the average of the probability PR(x) and the probability PR(y) by the object reliability OR of the detection rectangle BB in place of the probability PR of the region AR that is the specific region IA. Figure 16 In the illustrated S340, the class reliability value CR is calculated by multiplying the average of the probability PR(x) and the probability PR(y) by the object reliability OR of the detection rectangle BB in place of the probability PR of the region AR that is the specific region IA. Figure 17 In the illustrated example, the detection rectangle BB (i.e., the position PS of the detection object OB surrounded by the detection rectangle BB) is located near the boundary line of the plurality of regions AR, and the predetermined value TH is "250". Figure 18 In the illustrated example, the detection rectangle BB (i.e., the position PS of the detection object OB surrounded by the detection rectangle BB) is located near the boundary line of the plurality of regions AR, and the predetermined value TH is "250".

[0348] Thus, the determination section 190 can change the value of the predetermined value TH depending on whether the detection rectangle BB (i.e., the position PS of the detection object OB surrounded by the detection rectangle BB) is located near the boundary line of the plurality of regions AR. Also, the determination section 190 can set the value of the predetermined value TH to be fixed regardless of whether the detection rectangle BB (i.e., the position PS of the detection object OB surrounded by the detection rectangle BB) is located near the boundary line of the plurality of regions AR.

[0349] (Example of Setting a Buffer Region)

[0350] Figure 18is a diagram illustrating an example in which the detection device 10 (for example, the division section 120) sets a buffer region BA in the vicinity of the boundary line of the plurality of regions AR with respect to the captured image PI. As illustrated in Figure 18 The division section 120 can also set a buffer region BA in the vicinity of the boundary line of the plurality of regions AR. Specifically, the division section 120 can set a buffer region BA(x-y) in the vicinity of the boundary line of the region AR(x) and the region AR(y).

[0351] Figure 19 In the example illustrated in the drawing, the division section 120 sets a buffer region BA(1-2) in the vicinity of the boundary line of the region AR(1) and the region AR(2). The division section 120 sets a buffer region BA(2-3) in the vicinity of the boundary line of the region AR(2) and the region AR(3). The division section 120 sets a buffer region BA(3-4) in the vicinity of the boundary line of the region AR(3) and the region AR(4). The division section 120 sets a buffer region BA(4-5) in the vicinity of the boundary line of the region AR(4) and the region AR(5). The division section 120 sets a buffer region BA(5-6) in the vicinity of the boundary line of the region AR(5) and the region AR(6). The division section 120 sets a buffer region BA(6-1) in the vicinity of the boundary line of the region AR(6) and the region AR(1).

[0352] The detection device 10 executes the processing exemplified above to determine whether the presumption that "the detection object OB is a person" is correct or not in a case where the position PS of the detection object OB surrounded by the detection rectangle BB is included in the buffer region BA. Figure 19

[0353] Figure 19 is a flowchart illustrating an example of a person detection process in which the second determination processing is executed in a case where the buffer region BA is set. Figure 14 The processing up to S410, S420, and S470 to S490 of the second determination processing exemplified above is the same as the processing up to Figures 17 to 19 The processing up to S310, S320, and S350 to S370 of the second determination processing exemplified above is the same as the processing up to

[0354] That is, the image acquisition section 110 acquires the captured image PI, and the division section 120 divides the captured image PI acquired by the image acquisition section 110 into the plurality of regions AR (S410). The presumption section 150 executes person detection and class output using the learned model 140 with respect to the captured image PI acquired with reference to the divided image information 130 (the captured image PI divided into the plurality of regions AR) (S420).

[0355] ​The detection device 10 executes the processes of S430 to S490 for each detection rectangle BB (i.e., for each detection object OB) up to the number of persons detected from the captured image PI by the presumption section 150, in other words, up to the number of detection rectangles BB arranged by the presumption section 150.

[0356] The determination section 190 determines, for each detection rectangle BB, whether or not the detection rectangle BB belongs to within the buffer area BA (inside the buffer area BA) (S430), i.e., determines whether or not the detection rectangle BB is located inside the buffer area BA. The determination section 190 can also determine, for each detection rectangle BB, whether or not the position PS of the detection object OB surrounded by the detection rectangle BB belongs to within the buffer area BA. The position PS of the detection object OB is calculated from the detection rectangle BB (position, shape, and size of the detection rectangle BB), and can be, for example, the center position (or the barycentric position) of the detection rectangle BB.

[0357] If the detection rectangle BB belongs to within the buffer area BA (Yes in S430), the determination section 190 calculates the average of the probabilities PR of the following two areas AR that are adjacent to each other in the buffer area BA, as the probability PR (category probability) of the detection rectangle BB. That is, the determination section 190 calculates the average of the probability PR of the area AR corresponding to the specific area IA, and the probability PR of the area AR adjacent to the area AR corresponding to the specific area IA.

[0358] For example, in the case where the detection rectangle BB is located within the buffer area BA(x-y) of the "specific area IA of the detection rectangle BB, i.e., the area AR(x)", and the "area AR(y) adjacent to the area AR(x)", the determination section 190 sets the probability PR (category probability) of the detection rectangle BB to the following value. That is, the determination section 190 sets the probability PR of the detection rectangle BB to the average of the probability PR(x) of the area AR(x) and the probability PR(y) of the area AR(y).

[0359] If the detection rectangle BB does not belong to within the buffer area BA (No in S430), the determination section 190 sets the probability PR (category probability) of the detection rectangle BB to the probability PR of the area AR to which the detection rectangle BB belongs, i.e., to the probability PR of the "specific area IA of the detection rectangle BB, i.e., the area AR".

[0360] For example, in the case where the detection rectangle BB belongs to only the "specific area IA of the detection rectangle BB, i.e., the area AR", the determination section 190 sets the probability PR (category probability) of the detection rectangle BB to the probability PR of the "specific area IA of the detection rectangle BB, i.e., the area AR".

[0361] The determination section 190 can also determine whether the detection rectangle BB belongs inside the buffer area BA according to whether the buffer area BA contains the position PS of the detection object OB surrounded by the detection rectangle BB (the position PS calculated from the detection rectangle BB). For example, the determination section 190 can also determine whether the detection rectangle BB belongs inside the buffer area BA according to whether the buffer area BA contains the center position (or the center of gravity position) of the detection rectangle BB.

[0362] The determination section 190 multiplies the object reliability OR of each detection rectangle BB by the class probability of the detection rectangle BB to calculate the class reliability value CR of the detection rectangle BB (S460). The determination section 190 then determines whether the class reliability value CR of the detection rectangle BB is greater than the predetermined value TH (S470).

[0363] If the class reliability value CR of the detection rectangle BB is greater than the predetermined value TH (Yes in S470), the determination section 190 adopts the detection rectangle BB as a correct detection result (S480). That is, the determination section 190 determines that the class reliability value CR of the detection rectangle BB being greater than the predetermined value TH and the speculation that the detection object OB surrounded by the detection rectangle BB is a person is correct.

[0364] If the class reliability value CR of the detection rectangle BB is equal to or less than the predetermined value TH (No in S470), the determination section 190 removes the detection rectangle BB as a false detection (S490). That is, the determination section 190 determines that the class reliability value CR of the detection rectangle BB being equal to or less than the predetermined value TH and the speculation that the detection object OB surrounded by the detection rectangle BB is a person is incorrect.

[0365] As described using Figures 7 to 19 the determination section 190 can determine that the detection object OB is a person if the class reliability value CR calculated by multiplying the average of the probabilities PR of the two areas AR by the object reliability OR of the detection rectangle BB is greater than the predetermined value TH. That is, the determination section 190 can calculate the class reliability value CR from the average of the probabilities PR of the area AR corresponding to the specific area IA and the area AR adjacent to the area AR corresponding to the specific area IA among the plurality of areas AR.

[0366] According to the configuration, the detection device 10 calculates an average of the probability PR of the region AR corresponding to the specific region IA and the probability PR of the region AR adjacent to the region AR corresponding to the specific region IA, among the plurality of regions AR. Also, the detection device 10 determines that the detection target OB is a person when a category reliability value CR calculated by multiplying the calculated average by the object reliability OR of the detection rectangle BB is greater than a predetermined value TH.

[0367] Here, a case is assumed in which a person is captured in the captured image PI in a manner that spans two regions AR among the plurality of regions AR, or a position at which the person is captured is located near a boundary of the two regions AR, and the like.

[0368] In this case, it can be considered that, when the detection target OB presumed to be a person is actually a person, both the probability PR of the region AR corresponding to the specific region IA and the probability PR of the region AR adjacent to the region AR corresponding to the specific region IA are sufficiently high values. Also, it can be considered that, when the detection target OB presumed to be a person is actually a person, the value indicating the size of the possibility that the certain object is surrounded by the detection rectangle BB, that is, the object reliability OR of the detection rectangle BB is also a sufficiently high value.

[0369] Accordingly, the detection device 10 calculates an average of the probability PR of the region AR corresponding to the specific region IA and the probability PR of the region AR adjacent to the region AR corresponding to the specific region IA. Also, the detection device 10 determines that the detection target OB is a person when a category reliability value CR of the detection rectangle BB calculated by multiplying the average by the object reliability OR of the detection rectangle BB is greater than a predetermined value TH.

[0370] Accordingly, the detection device 10 has an effect that, even in a case in which a person is captured in the captured image PI near a boundary of two regions AR among the plurality of regions AR, the person can be detected from the captured image PI with high accuracy.

[0371] In a case in which the determination section 170 determines the underfoot position as the specific region IA, the determination section 190 can also calculate an average of the probability PR of the region AR including the underfoot position and the probability PR of the region AR adjacent to the region AR including the underfoot position. Also, the determination section 190 can determine that the detection target OB is a person when a category reliability value CR of the detection rectangle BB calculated by multiplying the calculated average by the object reliability OR of the detection rectangle BB is greater than a predetermined value TH.

[0372] (Outline of the Control Method Performed by the Detection Device 10)

[0373] Up to this point Figure 7 The control method performed by the detection device 10 described above can be summarized as follows. That is, the control method performed by the detection device 10 is a control method of a detection device that detects a person from a captured image PI captured by a ceiling camera using a fisheye lens. The control method includes a division step, a region estimation step (estimation step), and a determination step.

[0374] In the division step, the captured image PI is divided into a plurality of regions AR. In the region estimation step (estimation step), the probability PR that a position PS in which a detection object OB estimated to be a person is present is included is calculated for each of the plurality of regions AR. In the determination step, whether the detection object OB is a person is determined using the probability PR of each of the plurality of regions AR.

[0375] For example, the division step corresponds to S210 of Figure 14 , S310 of Figure 19 , and S410 of Figure 7 . The region estimation step (estimation step) corresponds to S220 of Figure 14 , S320 of Figure 19 , and S420 of Figure 7 . The determination step corresponds to S250 of Figure 14 , S350 of Figure 19 , and S470 of ​ .

[0376] According to the configuration, the control method divides the captured image PI into a plurality of regions AR, and calculates the probability PR that a position PS in which a detection object OB estimated to be a person is present is included for each of the plurality of regions AR. Furthermore, the control method verifies the estimation result that the detection object OB is a person using the probability PR of each of the plurality of regions AR.

[0377] Here, generally, in a case where a person is to be detected from an image using a dictionary representing a feature of a person, analysis of the image is required for each dictionary. Therefore, in a case where a plurality of dictionaries representing a feature of a person are to be used to improve the accuracy of detection of a person, analysis of the image is also required a plurality of times, and the time required to detect a person becomes longer.

[0378] In contrast, the control method calculates a probability PR that "a position PS in which the detection object OB is presumed to be present" is included for each of a plurality of regions AR into which the captured image PI is divided. Then, the control method determines "whether the detection object OB is a person" using the calculated probability PR for each of the plurality of regions AR, thereby improving the detection accuracy of persons. That is, the control method does not detect a person from the captured image PI using a plurality of dictionaries that represent characteristics of a person, but determines whether the detection object OB presumed to be a person is actually a person using the probability PR for each of the plurality of regions AR, thereby improving the detection accuracy of persons.

[0379] The method using a plurality of dictionaries that represent characteristics of a person is a method of improving the accuracy of presumption of itself, and in contrast, the control method improves the detection accuracy by verifying (i.e., removing false presumption results) the presumed result (the presumption that the detection object OB is a person).

[0380] Therefore, the control method does not need to use a plurality of dictionaries that represent characteristics of a person when detecting a person from the captured image PI, and can shorten the time required to detect a person from the captured image PI compared to a case in which a plurality of dictionaries are used to detect a person from the captured image PI.

[0381] Further, the control method determines (verifies) whether the detection object OB presumed to be a person is a person using the probability PR for each of the plurality of regions AR, and thus can detect a person from the captured image PI with high accuracy.

[0382] Therefore, the control method has the effect of being able to detect a person from an image (captured image PI) captured using a fisheye lens at high speed and with high accuracy.

[0383] Further, as described above, the control method does not need to use a plurality of dictionaries that represent characteristics of a person when detecting a person from the captured image PI.

[0384] Therefore, the control method has the effect of being able to reduce the effort required to prepare a dictionary (e.g., a learned model) required to detect a person from an image captured using a fisheye lens, and also being able to reduce the capacity of a memory required to store the dictionary.

[0385] §4. Modified Example

[0386] It is not necessary for the detection device 10 to perform the learning process, i.e., the subject that generates the learned model 140 is not necessarily the detection device 10. For the detection device 10 that detects "a person photographed in the photographed image PI" from the photographed image PI, it is not necessary to generate the learned model 140 as long as it is possible to detect "a person photographed in the photographed image PI" from the photographed image PI using the learned model 140. In other words, the detection device 10 does not necessarily include the teaching data generation section 210 and the learning section 220.

[0387] The learned model 140 can also be generated by a learned model generation device that includes the teaching data generation section 210 and the learning section 220, which is different from the detection device 10. Also, the detection device 10 can detect "a person photographed in the photographed image PI" from the photographed image PI using the learned model 140 generated by the learned model generation device.

[0388] The determination section 170 can also calculate the position PS of the detection object OB based on the detection rectangle BB, for example, can calculate the position PS of the detection object OB based on the position, shape, and size of the detection rectangle BB.

[0389] 〔Example of Implementation by Software〕

[0390] The functional blocks of the detection device 10 (specifically, the image acquisition section 110, the segmentation section 120, the estimation section 150, the determination section 170, the determination section 190, the teaching data generation section 210, and the learning section 220) can also be implemented by logic circuits (hardware) formed on integrated circuits (IC chips) and the like. Also, these functional blocks can be implemented by software using CPUs, GPUs, DSPs, and the like.

[0391] In the latter case, the detection device 10 includes a CPU, GPU, DSP, or the like that executes commands of a program that implements each function, a read only memory (ROM) or storage device (referred to as a "recording medium") in which the program and various data are recorded so as to be readable by a computer (or CPU, GPU, DSP, or the like), and a random access memory (RAM) or the like that expands the program. Also, the purpose of the present application is achieved by the computer (or CPU, GPU, DSP, or the like) reading and executing the program from the recording medium. As the recording medium, a "non-transitory tangible medium" can be used, such as a tape, disk, card, semiconductor memory, programmable logic circuit, or the like. Also, the program can be provided to the computer via any transmission medium (communication network or broadcast wave, or the like) that can transmit the program. In addition, the present application can also be implemented in the form of a data signal embedded in a carrier wave that realizes the program by electronic transmission.

[0392] [SUMMARY]

[0393] As described above, a detection device of one aspect of the present application detects a person from a captured image captured by a ceiling camera using a fisheye lens, the detection device including: a division section that divides the captured image into a plurality of regions; a region estimation section that calculates, for each of the plurality of regions, a probability that a position where a detection object estimated to be a person is located is included; and a determination section that determines whether the detection object is a person using the probability for each of the plurality of regions.

[0394] According to the configuration, the detection device divides the captured image into a plurality of regions, calculates, for each of the plurality of regions, a probability that a position where a detection object estimated to be a person is located is included, and uses the probability for each of the plurality of regions to verify the estimation result that the detection object is a person.

[0395] Here, generally, in a case where a person is to be detected from an image using a dictionary that represents a feature of a person, analysis of the image is required for each dictionary. Therefore, in a case where a plurality of dictionaries that represent a feature of a person are to be used to improve the accuracy of person detection, analysis of the image is also required a plurality of times, and the time required to detect a person becomes longer.

[0396] In contrast, the detection device uses the probability that each of the plurality of regions into which the captured image is divided contains a position at which a detection target that is presumed to be a person is located to determine whether the detection target is a person, thereby improving the accuracy of person detection. That is, the detection device does not use a plurality of dictionaries that represent characteristics of a person to detect a person from the captured image, but uses the probability of each of the plurality of regions to determine whether a detection target that is presumed to be a person is a person, thereby improving the accuracy of person detection.

[0397] The method of using a plurality of dictionaries that represent characteristics of a person is a method of improving the accuracy of presumption of itself, and in contrast, the detection device improves the accuracy of detection by verifying (i.e., removing false presumption results) the presumed result (the presumption that the detection target is a person).

[0398] Therefore, the detection device does not need to use a plurality of dictionaries that represent characteristics of a person when detecting a person from the captured image, and can shorten the time required to detect a person from the captured image compared to a case in which a plurality of dictionaries are used to detect a person from the captured image.

[0399] Furthermore, the detection device uses the probability of each of the plurality of regions to determine (verify) whether a detection target that is presumed to be a person is a person, and thus can accurately detect a person from the captured image.

[0400] Therefore, the detection device has the effect of being able to accurately and quickly detect a person from an image (captured image) captured using a fisheye lens.

[0401] Furthermore, as described above, the detection device does not need to use a plurality of dictionaries that represent characteristics of a person when detecting a person from the captured image.

[0402] Therefore, the detection device has the effect of being able to reduce the effort required to prepare a dictionary (e.g., a learned model) required to detect a person from an image captured using a fisheye lens, and also being able to reduce the capacity of a memory required to store the dictionary.

[0403] The detection device of one aspect of the present application can further include a determination section that determines a region in which a bounding box (Bounding Box) that encloses the detection target is located among the plurality of regions as a specific region, and the determination section can use the probability of each of the plurality of regions and the specific region to determine whether the detection target is a person.

[0404] According to the configuration, the detection device determines the specific region in which the bounding box is located among the plurality of regions, and uses the probability of each of the plurality of regions and the specific region to determine whether the detection target is a person.

[0405] For example, it can be considered that, in a case where the detection object presumed to be a person is actually a person, the probability of each of the plurality of regions is also high in consistency with the region (i.e., the specific region) in which the bounding box that surrounds the detection object is located.

[0406] Therefore, the detection device determines whether the detection object is a person using the probability of each of the plurality of regions and the specific region, i.e., verifies the presumption that the detection object is a person.

[0407] Therefore, the detection device has an effect of being able to detect a person with high accuracy from an image (captured image) captured using a fisheye lens.

[0408] The determination section can also determine, as the specific region, a region that surrounds the detection object presumed to be a person and that includes a position of the detection object (e.g., a position under the feet of the detection object presumed to be a person) or a position of the detection object. The determination section can also calculate the position of the detection object (position under the feet) from the bounding box, for example, can calculate the position of the detection object (position under the feet) from the position, shape, and size of the bounding box. Furthermore, the determination section can also determine the center position (or the center of gravity position) of the bounding box as the position of the detection object (position under the feet) that is surrounded by the bounding box and that is presumed to be a person.

[0409] In the detection device of one aspect of the present disclosure, the determination section can determine that the detection object is a person when the specific region coincides with the region of the plurality of regions in which the probability is highest.

[0410] According to the configuration, the detection device determines that the detection object is a person when the specific region coincides with the region of the plurality of regions in which the probability is highest. That is, the detection device determines that the detection object is a person when the region (the specific region) in which the bounding box that surrounds the detection object presumed to be a person is located coincides with the region in which the probability of the position in which the detection object presumed to be a person is located is highest.

[0411] It can be considered that, in a case where the detection object presumed to be a person is actually a person, the probability that the specific region in which the bounding box that surrounds the detection object is located coincides with the region in which the probability of the position in which the detection object is located is highest is high.

[0412] Therefore, the detection device determines that the detection object is a person when the specific region coincides with the region of the plurality of regions in which the probability is highest.

[0413] Therefore, the detection device has an effect of being able to detect a person with high accuracy from an image (captured image) captured using a fisheye lens.

[0414] In a case where the determination section determines the position (underfoot position) of the detection object surrounded by the bounding box and presumed to be a "person" as the specific region, the determination section can also determine whether the underfoot position is included in the region with the highest probability. The determination section can also determine that the detection object is a person when the underfoot position is included in the region with the highest probability.

[0415] In a detection device of an aspect of the present application, the determination section can also determine that the detection object is a person when the specific region coincides with the region with the highest probability among the plurality of regions or a region adjacent to the region with the highest probability among the plurality of regions.

[0416] According to the configuration, the detection device determines that the detection object is a person when the specific region coincides with the region with the highest probability among the plurality of regions or a region adjacent to the region with the highest probability among the plurality of regions.

[0417] Here, a case is assumed in which a person is photographed in such a manner as to straddle two regions among the plurality of regions or a position of a person is photographed near a boundary between two regions, and the like in the captured image.

[0418] In such a case, it can be considered that, in a case where the detection object presumed to be a person is actually a person, the specific region in which the bounding box surrounding the detection object is located coincides with a high probability with any one of the following regions. That is, it can be considered that the specific region coincides with a high probability with the region with the highest probability among the plurality of regions or a region adjacent to the region with the highest probability among the plurality of regions.

[0419] Therefore, the detection device determines that the detection object is a person when the specific region coincides with the region with the highest probability among the plurality of regions. Furthermore, the detection device determines that the detection object is a person when the specific region coincides with a region adjacent to the region with the highest probability among the plurality of regions.

[0420] Therefore, the detection device has an effect of being able to accurately detect a person from a captured image even in a case where a person is photographed near a boundary between two regions among the plurality of regions in the captured image.

[0421] In a case where the determination section determines the underfoot position as the specific region, the determination section can also determine whether the underfoot position is included in the region with the highest probability or a region adjacent to the region with the highest probability. The determination section can also determine that the detection object is a person when the underfoot position is included in the region with the highest probability or a region adjacent to the region with the highest probability.

[0422] In the detection device of one aspect of the present application, the determination section can determine that the detection target is a person when a class reliability value calculated by multiplying the probability of the region corresponding to the specific region among the plurality of regions by an object reliability value is greater than a predetermined value, the object reliability value being a value indicating the degree of likelihood that the certain object is enclosed by the bounding box.

[0423] According to the configuration, the detection device determines that the detection target is a person when a class reliability value calculated by multiplying the probability of the region corresponding to the specific region among the plurality of regions by the object reliability value is greater than a predetermined value.

[0424] It can be considered that the probability that the specific region in which the bounding box enclosing the detection target is located contains the position in which the detection target is located is sufficiently high when the detection target presumed to be a person is actually a person. Further, it can be considered that the object reliability value indicating the degree of likelihood that the certain object is enclosed by the bounding box enclosing the detection target is also a sufficiently high value when the detection target presumed to be a person is actually a person.

[0425] Therefore, the detection device determines that the detection target is a person when a class reliability value calculated by multiplying the probability of the region corresponding to the specific region among the plurality of regions by the object reliability value is greater than a predetermined value.

[0426] Therefore, the detection device has an effect that a person can be detected with high accuracy from an image (captured image) captured using a fisheye lens.

[0427] In the case where the determination section determines the underfoot position as the specific region, the determination section can determine that the detection target is a person when a class reliability value calculated by multiplying the probability of the region containing the underfoot position by the object reliability value is greater than a predetermined value.

[0428] In the detection device of one aspect of the present application, the determination section can determine that the detection target is a person when a class reliability value calculated by multiplying the probability of the region corresponding to the specific region among the plurality of regions by an object reliability value is greater than a predetermined value, the object reliability value being a value indicating the degree of likelihood that the certain object is enclosed by the bounding box.

[0429] According to the configuration, the detection device calculates an average of the probability of the region corresponding to the specific region among the plurality of regions and the probability of the region adjacent to the region corresponding to the specific region among the plurality of regions. Also, the detection device determines that the detection target is a person when a class reliability value calculated by multiplying the average by a value indicating the degree of possibility that the certain object is enclosed by the bounding box, i.e., an object reliability, is greater than a predetermined value.

[0430] Here, a case is assumed in which a person is photographed in such a manner as to straddle two regions among the plurality of regions, or a case in which a position at which a person is photographed is near a boundary between two regions, or the like.

[0431] In such a case, it can be considered that, in a case in which the detection target presumed to be a person is actually a person, both the probability of the region corresponding to the specific region and the probability of the region adjacent to the region corresponding to the specific region are sufficiently high values. Also, it can be considered that, in a case in which the detection target presumed to be a person is actually a person, the value indicating the degree of possibility that the certain object is enclosed by the bounding box enclosing the detection target, i.e., the object reliability, is also a sufficiently high value.

[0432] Accordingly, the detection device calculates an average of the probability of the region corresponding to the specific region and the probability of the region adjacent to the region corresponding to the specific region. Also, the detection device determines that the detection target is a person when a class reliability value calculated by multiplying the average by the object reliability is greater than a predetermined value.

[0433] Accordingly, the detection device has an effect of being able to accurately detect a person from the photographed image even in a case in which a person is photographed near a boundary between two regions among the plurality of regions in the photographed image.

[0434] In a case in which the determination section determines the underfoot position as the specific region, the determination section can also calculate an average of the probability of the region including the underfoot position and the probability of the region adjacent to the region including the underfoot position. Also, the determination section can determine that the detection target is a person when a class reliability value calculated by multiplying the calculated average by the object reliability is greater than a predetermined value.

[0435] In the detection device of one aspect of the present application, the region presumption section can also calculate the probability of each of the plurality of regions from the photographed image using a region prediction model that is a learned model that takes the photographed image as input and outputs the probability of each of the plurality of regions including a position at which the detection target is located.

[0436] According to the configuration, the detection device calculates the probability of each of the plurality of regions from the captured image using the region prediction model as a learned model that takes the captured image as input and outputs the probability of each of the plurality of regions including the position where the detection target is present.

[0437] Accordingly, the detection device has an effect that the probability of each of the plurality of regions can be calculated from the captured image with high accuracy using the region prediction model.

[0438] The detection device according to one aspect of the present disclosure can further include a learning unit that constructs a region prediction model as a learned model by machine learning with respect to teaching data that is labeled with information indicating a region including a position where a person is present with respect to the captured image, the learned model taking the captured image as input and outputting the probability of each of the plurality of regions including the position where the detection target is present.

[0439] According to the configuration, the detection device constructs the region prediction model by machine learning with respect to teaching data that is labeled with information indicating a region including a position where a person is present with respect to the captured image.

[0440] Accordingly, the detection device has an effect that the region prediction model that can calculate the probability of each of the plurality of regions from the captured image with high accuracy can be constructed by machine learning with respect to the teaching data.

[0441] To address the problem, a control method according to one aspect of the present disclosure is a control method of a detection device that detects a person from a captured image captured by a ceiling camera using a fisheye lens, the control method including: a division step of dividing the captured image into a plurality of regions; a region estimation step of calculating, with respect to each of the plurality of regions, a probability of including a detection target that is estimated to be a person; and a determination step of determining whether the detection target is a person using the probability of each of the plurality of regions.

[0442] According to the configuration, the control method divides the captured image into a plurality of regions, calculates, with respect to each of the plurality of regions, a probability of including a detection target that is estimated to be a person, and verifies the estimation result that the detection target is a person using the probability of each of the plurality of regions.

[0443] Here, in general, in a case where a dictionary representing a feature of a person is used to detect a person from an image, analysis of the image is required for each dictionary. Therefore, in a case where a plurality of dictionaries representing a feature of a person are used to improve the accuracy of detection of a person, analysis of the image is required a plurality of times, and the time required to detect a person becomes longer.

[0444] In contrast, the control method uses the probability that each of the plurality of regions into which the captured image is divided contains a position in which a detection target that is presumed to be a person is present, to determine whether the detection target is a person, thereby improving the accuracy of detection of a person. That is, the control method does not use a plurality of dictionaries representing a feature of a person to detect a person from the captured image, but uses the probability of each of the plurality of regions to determine whether a detection target that is presumed to be a person is a person, thereby improving the accuracy of detection of a person.

[0445] The method of using a plurality of dictionaries representing a feature of a person is a method of improving the accuracy of presumption itself, and in contrast, the control method improves the accuracy of detection by verifying (i.e., removing false presumption results) the presumed result (the presumption that the detection target is a person).

[0446] Therefore, the control method does not need to use a plurality of dictionaries representing a feature of a person when detecting a person from the captured image, and can shorten the time required to detect a person from the captured image, as compared to a case where a plurality of dictionaries are used to detect a person from the captured image.

[0447] Furthermore, the control method uses the probability of each of the plurality of regions to determine (verify) whether a detection target that is presumed to be a person is a person, and thus can detect a person from the captured image with high accuracy.

[0448] Therefore, the control method has the effect of being able to detect a person from a captured image captured using a fisheye lens at high speed and with high accuracy.

[0449] Furthermore, as described above, the control method does not need to use a plurality of dictionaries representing a feature of a person when detecting a person from the captured image.

[0450] Therefore, the control method has the effect of being able to reduce the effort required to prepare a dictionary (e.g., a learned model) required to detect a person from an image captured using a fisheye lens, and also being able to reduce the capacity of a memory required to store the dictionary.

[0451] To solve the problem, a model generation method according to one aspect of the present application is a model generation method performed by a model generation device that generates a learned model, the model generation method including: a step of acquiring teaching data, the teaching data being teaching data in which, for a captured image that is divided into a plurality of regions, (A) information indicating a position, a shape, and a size of a bounding box that encloses a person photographed in the captured image, and (B) information identifying a region that "contains a position at which the person photographed in the captured image is present" are given as a label; and a step of constructing a learned model by machine learning with respect to the teaching data, the learned model being a model that, when the captured image is input, outputs (C) information indicating a position, a shape, and a size of the bounding box and (D) a probability that each of the plurality of regions "contains a position at which the person photographed in the captured image is present".

[0452] According to the structure, the model generation method constructs the learned model by machine learning with respect to the teaching data. The learned model, when the captured image is input, outputs the following two kinds of information. That is, (C) information indicating a position, a shape, and a size of the bounding box (rectangular information), and (D) a probability that each of the plurality of regions "contains a position at which the person photographed in the captured image is present".

[0453] The rectangular information is information indicating a position, a shape, and a size of the bounding box that encloses a detection target that is presumed to be a person, and is information that includes a presumption that the detection target enclosed by the bounding box is a person.

[0454] Therefore, the model generation method has an effect of being able to construct the learned model that, when the captured image is input, outputs the rectangular information that includes a presumption that the detection target is a person and the probability that each of the plurality of regions.

[0455] The present application is not limited to the embodiments described above, and various changes can be made within the scope of the claims, and embodiments obtained by appropriately combining the technical features of the different embodiments disclosed in the respective embodiments are also encompassed within the technical scope of the present application.

Claims

1. A detection apparatus that detects a photographed person from a photographed image photographed by a ceiling camera using a fisheye lens, the detection apparatus comprising: a division section that divides the photographed image into a plurality of regions; a region estimation section that calculates, for each of the plurality of regions, a probability that a position where a detection target estimated to be a person is located is included; a determination section that determines whether the detection target is a person using the probability of each of the plurality of regions; and a specification section that specifies a region where a bounding box that encloses the detection target is located among the plurality of regions as a specified region, the determination section determines whether the detection target is a person using the probability of each of the plurality of regions and the specified region, the determination section determines that the detection target is a person when the specified region coincides with a region having the highest probability among the plurality of regions.

2. The detection apparatus according to claim 1, wherein the determination section determines that the detection target is a person when the specified region coincides with a region having the highest probability among the plurality of regions, or a region adjacent to the region having the highest probability among the plurality of regions.

3. A detection apparatus that detects a photographed person from a photographed image photographed by a ceiling camera using a fisheye lens, the detection apparatus comprising: a division section that divides the photographed image into a plurality of regions; a region estimation section that calculates, for each of the plurality of regions, a probability that a position where a detection target estimated to be a person is located is included; a determination section that determines whether the detection target is a person using the probability of each of the plurality of regions; and a specification section that specifies a region where a bounding box that encloses the detection target is located among the plurality of regions as a specified region, the determination section determines whether the detection target is a person using the probability of each of the plurality of regions and the specified region, the determination section determines that the detection target is a person when a class reliability value calculated by multiplying the probability of a region corresponding to the specified region among the plurality of regions by an object reliability degree is greater than a predetermined value, the object reliability degree being a value indicating a degree of possibility that an object is enclosed by the bounding box.

4. The detection apparatus according to claim 3, wherein the determination section determines that the detection target is a person when a class reliability value calculated by multiplying an average value of the probability of a region corresponding to the specified region among the plurality of regions and the probability of a region adjacent to the region corresponding to the specified region among the plurality of regions by an object reliability degree is greater than a predetermined value, the object reliability degree being a value indicating a degree of possibility that an object is enclosed by the bounding box.

5. The detection apparatus according to claim 3, wherein the region estimation section calculates the probability of each of the plurality of regions from the photographed image using a region prediction model that is a learned model that inputs the photographed image and outputs, as an output, the probability of each of the plurality of regions that includes a position where the detection target is located.

6. The detection apparatus according to claim 3, further comprising: The learning unit constructs a region prediction model as a learned model by machine learning with respect to teaching data that is labeled with information indicating a region including a position of a person photographed in the photographed image, the learned model being a model that takes the photographed image as input and outputs a probability of each of the plurality of regions including a position of a detection target.

7. A control method of a detection device that detects a person photographed from a photographed image photographed by a ceiling camera using a fisheye lens, the control method comprising: a division step of dividing the photographed image into a plurality of regions; a region estimation step of calculating, for each of the plurality of regions, a probability of including a position of a detection target estimated to be a person; a determination step of determining whether the detection target is a person using the probability of each of the plurality of regions; a determination step of determining whether the detection target is a person using the probability of each of the plurality of regions and the specific region, in the determination step, when the specific region coincides with the region having the highest probability among the plurality of regions, the detection target is determined to be a person.

8. A model generation method of a detection model, the model generation method being a method of generating a learned model used by the detection device according to any one of claims 1 to 7, the model generation method comprising: an acquisition step of acquiring teaching data that is labeled with (A) information indicating a position, shape, and size of a bounding box surrounding a person photographed in a photographed image divided into a plurality of regions and (B) information identifying a region including a position of the person photographed in the photographed image; a learning step of constructing the learned model by machine learning with respect to the teaching data, the learned model being a model that takes the photographed image as input and outputs (C) information indicating the position, shape, and size of the bounding box and (D) a probability of each of the plurality of regions including a position of the person photographed in the photographed image. The information processing program is used to cause a computer to function as the division unit, the region estimation unit, and the determination unit. ​ ​ 9. A recording medium, which records an information processing program for causing a computer to function as the detection device according to claim 1, and is readable by the computer, wherein, ​

Images