Program generation device, object sensing system, anchor setting method, and computer-readable recording medium recording an anchor setting program

Abstract

A program generation device, an object sensing system, an anchor setting method, and a computer-readable recording medium recording an anchor setting program. The purpose is to suppress the amount of processing and to sense an object with high precision. The system comprises: teacher data, the teacher data including a plurality of image data containing information about the region of an object; a setting unit, the setting unit setting an anchor as information about a frame, the frame information defining the region of each cell in which the presence or absence of an object is detected from an image; and a learning unit, the learning unit performing machine learning on the teacher data based on the information from the setting unit to generate a learning program for extracting an object from an image. The setting unit acquires information about the aspect ratio of the object region and the anchor in the teacher data, calculates the degree of consistency between the anchor and the object region at each aspect ratio while varying the size of the anchor, calculates the adoption rate of the object region as the ratio of the degree of consistency above a threshold, and determines the size of the anchor used in the learning program based on the calculated result.

Description

Technical Field

[0001] The present disclosure relates to a program generation device, an object sensing system, an anchor setting method, and a computer-readable recording medium having an anchor setting program recorded thereon. Background Art

[0002] As a system for detecting objects from acquired images, there is a system that uses a learning program completed through deep learning (machine learning) of many images to detect objects. In general object detection using deep learning, convolution processing using specific filter coefficients is first performed on the input image to extract feature quantities. Then, in the feature quantity space with different resolutions obtained during the convolution process, rectangular areas called anchors (Bounding Boxes) are configured, and a score representing the similarity of the object is calculated for each anchor based on the feature quantities in the area. Using the calculated score, the anchors with scores above the threshold are regressed, thereby adjusting their size and outputting them as detection results.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2018-22484

[0006] Patent Document 2: Japanese Patent No. 5172749 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] In deep learning, multiple anchor shapes are set and used to detect objects. This improves object detection accuracy. However, increasing the number of anchors also increases the computational processing load. Therefore, the number of usable anchors is limited by processing power and the time required to obtain computational results. Therefore, there is a need to improve object detection accuracy while suppressing the increase in the number of set anchors.

[0009] In order to solve the above problems, an object of at least one embodiment of the present disclosure is to provide a program generation device, an object sensing system, an anchor setting method, and a computer-readable recording medium recording an anchor setting program that can appropriately set the shape of an anchor and sense an object with high precision.

[0010] Technical Solution

[0011] The present disclosure provides a program generation device for generating an object sensing program for detecting whether an image contains an object, wherein:

[0012] The program generation device includes: teacher data including a plurality of image data including region information of an object; a setting unit that sets an anchor as information of a frame that determines the region of each cell for detecting the presence or absence of an object from an image; and a learning unit that performs machine learning on the teacher data based on the information from the setting unit to generate a learned program for extracting the object from the image.

[0013] The setting unit obtains information on the aspect ratio of the object area and the anchor of the teacher data, calculates the consistency between the anchor and the object area at each aspect ratio while changing the size of the anchor, calculates the adoption rate of the object area as the proportion of the consistency above the threshold, and determines the size of the anchor used in the learning completion program based on the calculated result.

[0014] Furthermore, the present disclosure provides an object sensing system comprising: the program generation device described above; and

[0015] The object sensing device includes a computing unit for executing the learning program generated by the program generating device, a camera unit for acquiring an image, and a notification unit for notifying an operator.

[0016] The computing unit analyzes the image acquired by the camera unit using the learning program, and when detecting that an object is included in the image, the notification unit notifies the user of the image.

[0017] In addition, the present disclosure provides an anchor setting method for setting an anchor used in an object sensing program for detecting whether an image contains an object, the anchor setting method comprising the following steps: obtaining teacher data including a plurality of image data including area information containing an object; obtaining anchor information as frame information, the frame information determining the area of each cell for detecting the presence or absence of an object from the image; obtaining information on the aspect ratio of the object area and the anchor of the teacher data, calculating the consistency between the anchor and the object area at each aspect ratio while changing the size of the anchor, calculating the adoption rate of the object area as the proportion of the consistency above a threshold; and determining the size of the anchor used in the learning completion program based on the calculated result.

[0018] In addition, the present disclosure provides an anchor setting program, which performs processing for setting an anchor used in an object sensing program for detecting whether an object is included in an image, and the anchor setting program performs the following steps: obtaining teacher data including multiple image data including area information including an object; obtaining anchor information as frame information, the frame information determining the area of each cell for detecting the presence or absence of an object from the image; obtaining information on the aspect ratio of the object area and the anchor of the teacher data, calculating the consistency between the anchor and the object area at each aspect ratio while changing the size of the anchor, calculating the adoption rate of the object area as the proportion of the consistency above a threshold; and determining the size of the anchor used in the learning completion program based on the calculated result.

[0019] Beneficial effects

[0020] With the above configuration, the shape of the anchor can be appropriately set, thereby achieving an effect of being able to sense an object with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a block diagram showing an example of an object sensing system.

[0022] Figure 2 This is an explanatory diagram for explaining an example of image processing in an object sensing system.

[0023] Figure 3 This is an explanatory diagram for explaining an example of image processing.

[0024] Figure 4 This is an explanatory diagram for explaining an example of image processing.

[0025] Figure 5 This is an explanatory diagram for explaining an example of image processing.

[0026] Figure 6 This is an explanatory diagram for explaining the anchor.

[0027] Figure 7 This is an explanatory diagram for explaining the anchor.

[0028] Figure 8 This is a flowchart showing an example of processing of the anchor setting unit.

[0029] Figure 9 This is an explanatory diagram for explaining an example of processing by the anchor setting unit.

[0030] Figure 10 This is an explanatory diagram for explaining an example of processing by the anchor setting unit.

[0031] Figure 11This is a graph showing an example of the relationship between the size of the anchor and the adoption rate.

[0032] Figure 12 This is a graph showing an example of the relationship between the detection rate and the false detection rate for each anchor size.

[0033] Figure 13 This is a flowchart showing an example of the operation of the learning unit.

[0034] Figure 14 This is a flowchart showing an example of the operation of the object sensing device.

[0035] Figure 15 This is a flowchart showing another example of the processing of the anchor setting unit.

[0036] Figure 16 This is an explanatory diagram for explaining another example of the processing of the anchor setting unit.

[0037] Figure 17 This is a flowchart showing another example of the processing of the anchor setting unit.

[0038] Figure 18 This is an explanatory diagram for explaining another example of the processing of the anchor setting unit. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments of the present disclosure will be described in detail based on the accompanying drawings. It should be noted that the present invention is not limited to these embodiments. In addition, the constituent elements of the following embodiments include elements that can be substituted and easily substituted by those skilled in the art, or substantially the same elements. Moreover, the constituent elements described below can be appropriately combined. In addition, when there are multiple embodiments, each embodiment can also be combined.

[0040] <Object Sensing System>

[0041] Figure 1 This is a block diagram illustrating an example of an object sensing system. Object sensing system 100 of this embodiment includes a program generation device 10 and an object sensing device 102. In object sensing system 100, program generation device 10 uses machine learning, such as deep learning, to generate a learned program capable of performing image determination processing for detecting objects from images. Object sensing device 102 executes the learned program to detect objects. Object sensing device 102 is installed in, for example, a mobile object such as a vehicle or an aircraft, or a building.

[0042] The program generation device 10 includes an input unit 12, an output unit 14, a computing unit 16, and a storage unit 18. The input unit 12 includes input devices such as a keyboard and a mouse, a touch panel, or a microphone for collecting voice from an operator, and outputs a signal corresponding to the operation performed by the operator on the input device to the computing unit 16. The output unit 14 includes a display device such as a monitor, and displays a screen containing various information such as processing results and images of processing objects based on the display signal output from the computing unit 16. In addition, the output unit 14 may include a recording device that outputs data through a recording medium. In addition, the program generation device 10 may include a communication unit that uses a communication interface to transmit data as the input unit 12 and the output unit 14. The communication unit transmits various data and programs obtained by communicating with an external device and saves them to the storage unit 18. The communication unit can be connected to the external device via a wired communication line or a wireless communication line.

[0043] The operation unit 16 includes integrated circuits (processors) such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a memory as a work area, and performs various processes by executing various programs using these hardware resources. Specifically, the operation unit 16 reads out the program stored in the storage unit 18 and expands it in the memory, and performs various processes by causing the processor to execute the commands contained in the program expanded in the memory. The operation unit 16 includes a teacher data generation unit (an example of a data generation unit) 30, an anchor setting unit (an example of a setting unit) 32, a learning unit 34, and an object sensing processing unit (an example of a processing unit) 36. Before explaining the various parts of the operation unit 16, the storage unit 18 is explained.

[0044] The storage unit 18 is composed of a nonvolatile storage device such as a magnetic storage device or a semiconductor storage device, and stores various programs and data. The storage unit 18 includes image data 40, setting data 42, a learning execution program 44, an anchor setting program 46, an object detection program 48, and a learning completion program 50.

[0045] In addition, the data stored in the storage unit 18 includes image data 40 and setting data 42. The image data 40 includes teacher data for learning. The teacher data is data that establishes a correspondence between the image data and the area (bounding box) in which the object is displayed when the object is included in the image. The image of the teacher data can be divided into data for learning and data for evaluating the accuracy of the program after learning. In addition, the image data may also include image data for detecting the object. The setting data 42 includes the setting information of the anchor described later, information on the conditions for executing the learning completion program, etc.

[0046] Programs stored in the storage unit 18 include a learning execution program 44 , an anchor setting program 46 , an object sensing program 48 , and a learning completion program 50 .

[0047] The learning execution program 44 performs deep learning processing on the teacher data included in the image data 40 based on the settings of the setting data 42, and generates a learning completion program 50. As a deep learning model, the following deep learning model can be used: by setting bounding boxes called so-called anchors such as R-CNN (Regions with Convolutional Neural Networks), YOLO (You Only Look Once), and SSD (Single Shot multibox Detector) on the image and processing the feature quantities within the set anchors, it is possible to detect whether the image contains an object.

[0048] The anchor setting program 46 sets the anchors used when performing image processing using the deep learning model in the learning execution program 44 and the learning completion program 50. The anchor setting program 46 also sets the size of the anchors. Furthermore, the anchor setting program 46 preferably sets the aspect ratio of the anchors and the number of anchors used. The information set by the anchor setting program 46 is stored in the setting data 42.

[0049] The object sensing program 48 is a program that uses the learning program 50 to perform object detection. The object sensing program 48 also includes settings for image acquisition and output of determination results. The object sensing program 48 can also configure settings for processing image data. The learning program 50 is generated by executing the learning execution program 44. By executing the learning program 50 through the computing unit that performs image processing, the object sensing program 48 can calculate the feature value (score) used as the judgment criterion for learning and perform object detection based on the feature value.

[0050] The storage unit 18 can install the learning execution program 44, the anchor setting program 46 and the object sensing program 48 by reading the learning execution program 44, the anchor setting program 46 and the object sensing program 48 recorded in the recording medium, or can install the learning execution program 44, the anchor setting program 46 and the object sensing program 48 by reading the learning execution program 44, the anchor setting program 46 and the object sensing program 48 provided on the network.

[0051] The functions of each part of the operation unit 16 are explained. Each part of the operation unit 16 can be executed by executing the program stored in the storage unit 18. When there is an object in the image, the teacher data generation unit 30 establishes a correspondence between the frame information (bounding box) of the area representing the object and the image data. The set frame is a rectangle. For example, the teacher data generation unit 30 sets the frame information according to the operation input to the input unit 12 while the output unit 14 displays the image. The operation input to the input unit 12 is an operation in which the operator inputs the information of the position of the enclosing frame (the object) while looking at the image. In addition, the teacher data generation unit 30 can obtain the result of the image extraction processing performed by the object sensing processing unit 36. In this case, the operation of the operator who determines whether the position of the extracted frame is the correct answer of the teacher data can be detected, and the data determined by the operator that the position of the frame is correct can be obtained as the teacher data.

[0052] The anchor setting unit 32 executes the processing of the anchor setting program 46 to set anchor information used in image processing of the deep learning model executed by the learning unit 34 and the object detection processing unit 36. The processing of the anchor setting unit 32 will be described later.

[0053] The learning unit 34 executes the processing of the learning execution program 44 using the anchor settings set by the anchor setting unit 32, performs deep learning as teacher data of the image data 40, and generates a learning completion program. The processing of the learning unit 34 will be described later.

[0054] The object detection processing unit 36 uses the object detection program 48 and the learning program 50 to perform object detection processing, which determines whether an object is included in a captured image. The processing of the object detection processing unit 36 will be described later.

[0055] It should be noted that in this embodiment, the program generation device 10 includes the training data generation unit 30 and the object detection processing unit 36. However, the program generation device 10 may not be included. In other words, the training data may be generated by another device. Furthermore, the object detection processing unit 36 that performs the process of detecting objects from images may only include the object detection device 102.

[0056] As described above, the object sensing device 102 is installed on a mobile object or a building. The object sensing device 102 can be configured to communicate with the program generation device 10, but it may not have such a function. Object sensing devices 102 without such a function have various pre-set processing conditions and perform object sensing based on these conditions. The object sensing device 102 can output detection results to a control device that controls the installed mechanism. This allows, for example, a mobile object to stop or avoid an object upon detecting it.

[0057] The object sensing device 102 includes a camera unit 112, a computing unit 114, a storage unit 116, and a notification unit 118. The camera unit 112 captures an image of the object's field of view. The camera unit 112 can capture images continuously at a predetermined frame rate or in response to a predetermined operation.

[0058] The computing unit 114 includes integrated circuits (processors) such as a CPU and GPU, as well as memory as a workspace. It uses these hardware resources to execute various programs and perform various processes. Specifically, the computing unit 114 reads programs stored in the storage unit 18 and expands them into the memory. The computing unit 114 executes the commands contained in the expanded programs to perform various processes. The computing unit 114 executes the programs stored in the storage unit 116 to detect objects from images.

[0059] The storage unit 116 is composed of a nonvolatile storage device such as a magnetic storage device or a semiconductor storage device, and stores various programs and data. The storage unit 116 stores an object sensing program 120 and a learning completion program 122 .

[0060] Notification unit 118 notifies the operator. Notification unit 118 can be a speaker, light-emitting device, display, or the like. When processing by computing unit 114 detects an object contained in an image, notification unit 118 notifies the operator of the presence of the object. If the object is a person, notification can be directed to the person being sensed.

[0061] Figure 2 This is an explanatory diagram for explaining an example of image processing in an object sensing system. Figures 3 to 5 Each of them is an explanatory diagram for explaining an example of image processing.

[0062] In this embodiment, the object sensing processing unit 36 determines whether an image contains an object by performing image processing based on a set of deep learning. The learning unit 34 performs machine learning, such as deep learning, using the training data generated by the training data generating unit 30 to generate a learning program executed by the object sensing processing unit 36.

[0063] Here, the deep learning of this embodiment performs convolution processing on the image of the object and sets the feature quantity obtained from multiple pixels as a cell information. Then, as Figure 2 As shown, the process of repeatedly converting feature values obtained from multiple cells into information for a single cell is repeated. This allows multiple feature maps with different cell sizes to be acquired for each image. Deep learning uses the anchors assigned to each cell when acquiring the feature map to detect objects.

[0064] That is, for an image, such as Figure 2 As shown, feature map (segmentation map) 202 is processed to generate feature map 202A, which has fewer segments than image data 202. One cell 210A in feature map 202A has a larger proportion of the entire image than cell 210. The same process is repeated multiple times on feature map 202A to generate feature map 202B, which has only one cell (region) 210B defined. The process for transitioning to a different number of segments uses calculations performed by deep learning to set various parameters.

[0065] Here, in the deep learning setting, anchor 212 is set as the frame information for obtaining information in order to calculate the evaluation of a cell. In this embodiment, anchor 212 is set so that its center coincides with the center of the cell being evaluated. Furthermore, anchor 212 is set to a size corresponding to the cell; the larger the cell, the larger the anchor 212. Furthermore, multiple anchors 212 are set for processing using deep learning.

[0066] Next, teacher data will be described. Figure 3 The image 220 includes a dog 230 and a cat 232. In the case of sensing the object, when the object is a dog, a frame 224 is set in the area where the dog 230 is displayed. The frame 224 is area information, which is information indicating that the object is displayed, that is, a boundary frame. The frame 224 is position information on the image 220. In addition, when the object is a cat, a frame 226 is set in the area where the cat 232 is displayed. In addition, when the object is an animal, both the frame 224 and the frame 226 may be set for one image 220. Figure 3 As shown, the information of the frames 224 and 226 surrounding the object and the image 220 are associated with each other, and are image data of the correct answer data and the teacher data.

[0067] The object sensing system 100 performs a learning process on data of a plurality of images including the image 220 containing information of frames 224 and 226 of image data while adapting anchors set for the teacher data to each cell, thereby generating a learned model capable of extracting objects.

[0068] Figure 4 and Figure 5 Schematic diagram of the analysis Figure 3 The status of the image. Figure 4 The feature quantity map (segmentation map) 240 shown divides the image into 8 rows and 8 columns. Figure 5 The feature map 240a shown is divided into four rows and four columns. In feature map 240, as shown by anchor unit 242 corresponding to cell 252, multiple anchors 250a, 250b, 250c, and 250d with different aspect ratios are applied to each cell. For each anchor, feature values of the image within the anchor region are compared. The same applies to feature map 240a.

[0069] exist Figure 3 In the case of the image 220 shown, Figure 4 In the feature map 240 shown, the detected anchors correspond to the frame 226 of the cat 232 and the anchor unit 244 that divides the image region into the size of the cell 242. The anchor corresponding to the frame 224 of the dog 230 is not detected because it has a different size than the anchors in the feature map 240. The anchor corresponding to the frame 224 of the dog 230 is detected among the anchors included in the anchor unit 246 set in the feature map 240a with the smaller number of divisions.

[0070] As described above, the object sensing system 100 detects whether an object is included in image data by applying an anchor to each cell of the feature map and processing the image within the anchor through deep learning.

[0071] Here, when the consistency of the ratio of overlap between the bounding box and the anchor in the teacher data, which is an area where the object is displayed, can be maintained at a high state, the accuracy of learning also becomes higher, and the object can be detected with high accuracy. Here, the consistency is evaluated by IoU (Intersection over Union). Specifically, it is the percentage of (the consistent part of the bounding box and the anchor) / (the union of the bounding box and the anchor). On the other hand, when the area where the object is displayed is at the boundary of the anchor in the teacher data, the consistency of any anchor during deep learning becomes low, the amount of learning in deep learning does not increase, and a situation occurs in which the object cannot be detected by the learning completion program. In addition, during actual detection, an anchor with a high degree of consistency with the area containing the object is not generated, resulting in a situation in which detection cannot be performed. In contrast, the object sensing system 100 performs the following processing.

[0072] <Anchor Setting Method>

[0073] Next, use Figures 6 to 12 , the method of setting the anchor is explained. Figure 6 and Figure 7 Each of them is an explanatory diagram for explaining the anchor. Figure 8This is a flowchart showing an example of processing of the anchor setting unit. Figure 9 and Figure 10 Each of them is an explanatory diagram for explaining an example of processing of the anchor setting unit. Figure 11 This is a graph showing an example of the relationship between the size of the anchor and the adoption rate. Figure 12 This is a graph showing an example of the relationship between the detection rate and the false detection rate for each anchor size.

[0074] exist Figure 6 and Figure 7 In the example shown, anchors 212 are configured with four shapes: 212a, 212b, 212c, and 212d. These anchors 212a, 212b, 212c, and 212d have the same area but different aspect ratios. These anchors 212a, 212b, 212c, and 212d represent the configurations for cell 282. The vertical dimensions of anchors 212a, 212b, 212c, and 212d decrease in this order. Anchor 212a is a vertically elongated rectangle. Anchor 212d is a horizontally elongated rectangle.

[0075] In deep learning, by setting the shape of the anchor 212 to a plurality of shapes, it is possible to set an anchor that is close to the shape of the bounding box 280 or 280a, which is the area set to contain the object. Figure 6 In the case of , the consistency between bounding box 280 and anchor 212a is 45%, the consistency between bounding box 280 and anchor 212b is 80%, the consistency between bounding box 280 and anchor 212c is 60%, and the consistency between bounding box 280 and anchor 212d is 30%. In addition, Figure 7 In this case, the consistency between bounding box 280a and anchor 212a is 30%, the consistency between bounding box 280a and anchor 212b is 30%, the consistency between bounding box 280a and anchor 212c is 30%, and the consistency between bounding box 280a and anchor 212d is 5%. Here, the consistency is evaluated using IoU.

[0076] The anchor setting unit 32 of this embodiment can improve the accuracy of object sensing by setting anchors based on the bounding box of the teacher data. Figure 8 , the anchor setting process is explained. Figure 8 The illustrated processing can be realized by processing executed by the anchor setting unit 32 .

[0077] The anchor setting unit 32 obtains training data containing object region information, that is, bounding box information (step S12). The anchor setting unit 32 reads the anchor settings (step S14). The anchor settings are information about the aspect ratio of all anchors used in learning and the size of the base cell.

[0078] Here, Figure 9 and Figure 10 A case where the bounding box 284 is arranged across the cell 285a and the cell 285b is shown. Figure 9 and Figure 10 This shows a case where the sizes of anchors associated with the same cell 285a and cell 285b are different. Figure 9 Anchor 286a used in cell 285a and anchor 286b used in cell 285b adjacent to cell 285a are shown. Figure 10 Anchor 288a used in cell 285a and anchor 288b used in cell 285b adjacent to cell 285a are shown. Figure 10 Anchors 288a and 288b are Figure 9 The anchors 286a and 286b are large in size. Figure 10 The aspect ratio of anchors 288a, 288b is Figure 9 The length-to-width ratios of the anchors 286a, 286b are the same. Figure 9 and Figure 10 A bounding box 284 of the object is set in the mid-span cells 285a and 285b.

[0079] like Figure 9 As shown in FIG. 1 , when there is a gap between the anchor 286a and the adjacent anchor 286b, the degree of coincidence with the area of the bounding box of the object tends to be difficult to increase. Figure 10 As shown, when the size of the anchor 288a is such that there is no gap between the adjacent anchor 288b, the degree of coincidence with the area of the bounding box of the object decreases depending on the position, and the detection rate tends to increase. Therefore, it is preferable that the size of the anchor be as Figure 10 Thus, evaluation is performed on the case where the size increases and the case where the size decreases, centering on the size at which there is no gap between the anchor 288a and the adjacent anchor 288b.

[0080] The anchor setting unit 32 calculates the IoU at each position based on the object region information and the anchor information (step S16 ).

[0081] The anchor setting unit 32 determines whether the evaluation of all anchor sizes has been completed (step S18). If the anchor setting unit 32 determines that the evaluation of all anchor sizes has not been completed ("No" in step S18), the anchor sizes are changed (step S20) and the process returns to step S16. In other words, the anchors are changed to the unevaluated sizes and the IoU is calculated for the changed sizes.

[0082] When the anchor setting unit 32 determines that the evaluation of the sizes of all anchors has been completed (YES in step S18 ), the size (dimension) of the anchor is determined based on the evaluation result (step S22 ).

[0083] The anchor setting unit 32 compares the anchors with the object region information when the anchor sizes are set to different sizes and evaluates the adoption rate, thereby detecting the size of the anchor that can extract the object region information included in the training data.

[0084] like Figure 11 As shown, when the threshold value of IoU (IoU threshold) is set to various values, the adoption rate of the object area of the teacher data is calculated. The adoption rate is (the number of objects with IoU above the threshold) / (the number of objects included in all image data of the teacher data). The anchor setting unit 32 compares each of the multiple anchors generated with the set size with the area information of the object. When the IoU is above the threshold in the case of any anchor, it is included in the number of objects with IoU above the threshold. For example, when the threshold value of IoU is different, the anchor setting unit 32 calculates the size of the anchor with the highest adoption rate for that value.

[0085] Figure 12 When the IoU threshold is set to IoU threshold condition D, IoU threshold condition E, and IoU threshold condition F, an anchor of a size with a high adoption rate is set, and the evaluation of the image data for evaluation is performed to calculate the detection rate and the false detection rate. Here, the evaluation of the detection rate and the false detection rate can be performed by Figure 13 The learning program generation method described in the above is used to generate a learning program, and the generated learning program is used to extract the object from the result of the object extraction process of the unit of the image data for evaluation. Figure 12 As shown, when comparing with the same false positive rate, the detection rate can be higher than the benchmark in any case. Here, the IoU threshold becomes a learning parameter for deep learning. The IoU threshold is based on Figure 12 The result of the curve graph shown in FIG is used to determine the IoU threshold value that achieves the desired sensing rate and false sensing rate. The determined IoU threshold value is set as a parameter during learning. It should be noted that the anchor setting unit 32 may not calculate Figure 12 The detection rate and false detection rate of Figure 11 The information is used to determine the size of the anchor.

[0086] <Learn how to complete the program generation method>

[0087] Figure 13 This is a flowchart showing an example of the operation of the learning unit. Figure 13 The processing shown is executed by the learning unit 34 performing computational processing on the learning execution program. The learning unit 34 generates the learning completion program using the information on the anchor size determined by the anchor setting unit 32. Furthermore, the learning unit 34 may also generate the learning completion program using the anchor size determination process performed by the anchor setting unit 32, that is, the information on the size of the anchors that served as candidates when determining the anchors in step S22 described above.

[0088] The learning unit 34 obtains the teacher data containing the regional information of the object (step S30). The learning unit 34 reads the anchor settings (step S32). That is, the learning unit 34 reads the size information and aspect ratio information of the anchor set by the anchor setting unit 32. The learning unit 34 performs deep learning based on the teacher data and the anchor information (step S34). The learning unit 34 sets a deep learning model for the image of the teacher data based on the anchor information, and uses the set model to learn the image of the teacher data. As a result, the learning unit 34 generates a learning completion program that performs learning using the teacher data.

[0089] The learning unit 34 evaluates the learning results using the evaluation images (step S36). Here, the evaluation images are a data set that includes both images containing the object and images not containing the object. The evaluation images are associated with information indicating whether the object is contained or not. The learning unit 34 detects the object in the evaluation images using the learning completion program at the evaluation time point to evaluate whether the object can be detected in the evaluation images containing the object, and whether the object is falsely detected in the evaluation images not containing the object. The learning unit 34 calculates the detection rate, false detection rate, and other evaluation criteria as evaluation data.

[0090] After the learning unit 34 calculates the evaluation, it determines whether the learning is complete (step S38). The evaluation criteria for the completion of learning can be set arbitrarily. For example, the number of learning times or the amount of calculation can be used as a benchmark, or the processing can be terminated when the detection rate or false detection rate meets the set performance.

[0091] If the learning unit 34 determines that the learning is not completed ("No" in step S38), the conditions for deep learning are adjusted (step S40) and the process returns to step S34. Thus, the learning process is executed again. Here, there are no particular restrictions on the conditions for deep learning, but as the learning program at the start of learning in step S34, there are a learning program that sets the current time point, a partial replacement of the image of the teacher data, etc. If the learning unit 34 determines that the learning is completed ("Yes" in step S38), the program of the learning result is set to the learning completion program (step S42) and the process ends.

[0092] As described above, the learning unit 34 performs deep learning processing using the anchors set by the anchor setting unit 32 and generates a learned program.

[0093] <Object Sensing Method>

[0094] Next, use Figure 14 This section explains how to detect objects using a learned program. Figure 14This is a flowchart showing an example of the operation of the object sensing device. Figure 14 The processing is described as being performed by the object sensing device 102 , but the image data may be supplied to the program generation device 10 and the object sensing processing unit 36 may perform the same processing.

[0095] The object sensing device 102 reads the learned program (step S50 ). The object sensing device 102 acquires the learned program generated by the program generating device 10 . The object sensing device 102 acquires image data (step S52 ). Specifically, the object sensing device 102 acquires images using the camera unit 112 .

[0096] The object sensing device 102 analyzes the image data based on the learned program (step S54). The object sensing device 102 detects whether the image data contains an object using the learned program generated by deep learning using the anchors set by the anchor setting unit 32 through the computing unit 114.

[0097] The object sensing device 102 determines whether an object is present based on the analysis result of step S54 (step S56). If the object sensing device 102 determines that an object is present ("Yes" in step S56), the notification unit 118 notifies that the object has been detected (step S58). If the object sensing device 102 determines that there is no object ("No" in step S56), or after executing the process of step S58, it determines whether the process is complete (step S60). If the object sensing device 102 determines that the process is not complete ("No" in step S60), it returns to step S52, acquires the next image data, and performs object sensing processing. If the object sensing device 102 determines that the process is complete ("Yes" in step S60), it ends this process.

[0098] As described above, this embodiment uses the regional information of the object in the teacher data to compare regions of various situations in which the size (scale) of the anchor is changed, calculates the degree of consistency, calculates the size of the anchor in which the number of anchors that are consistent at a ratio above a threshold value increases, and performs deep learning using the calculated anchor size. This can further improve the accuracy of object detection. In addition, by determining the size of the anchor by performing regional comparison processing, deep learning is performed on the anchors and various combinations, and compared to finding the optimal conditions, the accuracy of object detection can be increased with significantly less calculations, and the size of the anchor can be determined. In addition, by using the information of the teacher data that sets the regional information, processing can be performed without generating new data.

[0099] <Other Examples of Anchor Setting Methods>

[0100] Here, the anchor setting unit 32 may determine the aspect ratio of the anchor based on information on the frame of the target region of the teacher data. Figure 15 This is a flowchart showing another example of the processing of the anchor setting unit. Figure 16 This is an explanatory diagram for explaining another example of the processing of the anchor setting unit.

[0101] The anchor setting unit 32 obtains the training data (step S70). The anchor setting unit 32 extracts the distribution of the aspect ratio of the object region (step S72). The anchor setting unit 32 detects the aspect ratio of the set region for all images in the training data. If the aspect ratio information is pre-set, the anchor setting unit 32 can read the set aspect ratio information.

[0102] The anchor setting unit 32 calculates the distribution of aspect ratios (step S74). Figure 16 As shown, the distribution of the aspect ratios of the bounding boxes set in the teacher data as the learning data is calculated.

[0103] The anchor setting unit 32 determines multiple aspect ratios of anchors based on the aspect ratio distribution (step S76). Specifically, based on the aspect ratio distribution of the training data, for example, the 2% and 98% positions of the distribution, and the positions of the aspect ratios divided equally between these two positions are used as the aspect ratios of the anchors. Alternatively, the peak position of the distribution may be used as the aspect ratio of the anchors.

[0104] As described above, by setting the aspect ratio of the anchor based on the aspect ratio distribution of the teacher data, the aspect ratio of the anchor can be determined based on the aspect ratio of the detected object in the image, thereby improving the accuracy of object detection using the anchor.

[0105] <Other Examples of Anchor Setting Methods>

[0106] Furthermore, after determining the anchor size, the anchor setting unit 32 may evaluate the anchors to be used and reduce the number of anchors. Specifically, the anchor setting unit 32 may evaluate the aspect ratio combinations of the anchors to be used and not use anchors with aspect ratios that have a small impact on the detection rate. Figure 17 This is a flowchart showing another example of the processing of the anchor setting unit. Figure 18 This is an explanatory diagram for explaining another example of the processing of the anchor setting unit.

[0107] The anchor setting unit 32 determines Figure 8 This process is then performed after determining the size (dimensions) of the anchors shown. The anchor setting unit 32 acquires the anchor size information (step S80). The anchor setting unit 32 reads the teacher data (learning data) (step S82). The anchor setting unit 32 calculates the adoption rate of the learning data for each combination of the aspect ratios of the anchors (step S84).

[0108] For example, in Figure 18 In the example shown, in the initial setting, as shown in pattern P1, five anchors with different aspect ratios of 3, 2, 1, 1 / 2, and 1 / 3 are used. In contrast, the adoption rate is calculated for the case where four anchors are used. Pattern P2 is a combination of 2, 1, 1 / 2, and 1 / 3, other than the anchor with an aspect ratio of 3. Pattern P3 is a combination of 3, 1, 1 / 2, and 1 / 3, other than the anchor with an aspect ratio of 2. Pattern P4 is a combination of 3, 2, 1 / 2, and 1 / 3, other than the anchor with an aspect ratio of 1. Pattern P5 is a combination of 3, 2, 1, and 1 / 3, other than the anchor with an aspect ratio of 1 / 2. Pattern P6 is a combination of 3, 2, 1, and 1 / 2, other than the anchor with an aspect ratio of 1 / 3. The anchor setting unit 32 calculates the adoption rate for all patterns.

[0109] The anchor setting unit 32 compares the adoption rates of the respective anchor combinations (step S86). Figure 18 As shown, the anchor setting unit 32 compares the adoption rate of the learning data in each mode. The anchor setting unit 32 determines the combination of aspect ratios to be used (step S88). The anchor setting unit 32 sets the combination in which the reduction in the adoption rate of the learning data is within the threshold and the number of anchors used is smaller as the combination of aspect ratios to be used. When the number of anchors is the same, the combination with the highest adoption rate of the learning data is set as the combination of anchors with the aspect ratios to be used. Figure 18 In the example shown, the anchor combination of P6 is used.

[0110] As described above, the anchor setting unit 32 evaluates anchor combinations, and by reducing the number of anchor combinations used while minimizing the reduction in adoption rate, it is possible to suppress a decrease in object detection accuracy and reduce the amount of computation. Furthermore, by using teacher data for evaluation, it is possible to select an anchor combination that is suitable for object detection.

[0111] Description of Reference Numerals

[0112] 10 Program Generation Device

[0113] 12 Input section

[0114] 14 Output

[0115] 16 Operation unit

[0116] 18 Storage

[0117] 30 Teacher Data Generation Department

[0118] 32 Anchor Setting Unit

[0119] 34 Learning Department

[0120] 36 Object sensing processing unit

[0121] 40 Image Data

[0122] 42 Setting data

[0123] 44 Learning execution program

[0124] 46 Anchor Setting Procedure

[0125] 48, 120 Object Sensing Program

[0126] 50, 122 Learning completion procedure

[0127] 100 Object Sensing System

[0128] 102 Object Sensing Device

[0129] 112 Camera Department

[0130] 114 Operation Unit

[0131] 116 Storage Department

[0132] 118 Notification Department

[0133] 212 Anchor

[0134] 230, 232 Bounding box.

Claims

1. A program generation device for generating an object sensing program for detecting whether an object is included in an image, wherein: The program generation device includes: teacher data, the teacher data including a plurality of image data including area information of an object; a setting unit that sets an anchor as frame information that determines an area of each cell for detecting the presence or absence of an object from an image; and a learning unit that performs machine learning on teacher data based on information from the setting unit to generate a learning-completed program for extracting an object from an image, The setting unit obtains information on the aspect ratio of the object area and the anchor of the teacher data, calculates the consistency between the anchor and the object area at each aspect ratio while changing the size of the anchor, and calculates the ratio of the consistency above the threshold as the adoption rate of the object area. The ratio is the ratio of the number of objects with a consistency above the threshold to the number of objects contained in all image data of the teacher data. Based on the threshold, the setting unit determines the size of the anchor with the highest adoption rate calculated as a result of the calculation as the size of the anchor used in the learning completion program.

2. The program generation device according to claim 1, wherein The setting unit calculates the adoption rate of each of the plurality of coincidence degrees by setting the threshold value to a plurality of coincidence values, and determines the sizes of the plurality of anchors based on the calculated results.

3. The program generation device according to claim 1, wherein The setting unit calculates the aspect ratio of the target region of the teacher data, and determines the aspect ratio of the anchor based on the distribution of the aspect ratio of the target region.

4. The program generation device according to claim 2, wherein: The setting unit calculates the aspect ratio of the target region of the teacher data, and determines the aspect ratio of the anchor based on the distribution of the aspect ratio of the target region.

5. The program generation device according to any one of claims 1 to 4, wherein: The setting unit calculates the detection rate for each aspect ratio set based on the determined anchor size for the teacher data, and determines a combination of the aspect ratios of the anchors used in the learning completion program based on the calculated result. The program generation device according to claim 5 , wherein: The setting unit sets some of the anchors for which the aspect ratio of the detection rate is calculated as anchors to be used in the learning completion procedure.

7. An object sensing system comprising: The program generation device according to any one of claims 1 to 6; and An object sensing device includes a computing unit that executes a learning completion program generated in the program generation device, a camera unit that acquires an image, and a notification unit that notifies an operator. The computing unit analyzes the image acquired by the camera unit using the learning completion program, and when it senses that an object is included in the image, the notification unit notifies the operator.

8. An anchor setting method for setting an anchor used in an object sensing process for detecting whether an object is included in an image, wherein: The anchor setting method comprises the following steps: acquiring teacher data including a plurality of image data including region information of an object; Acquire anchor information as frame information that specifies an area of each cell for detecting the presence or absence of an object from an image; obtaining information on aspect ratios of the object region and the anchor of the teacher data, calculating the degree of consistency between the anchor and the object region at each aspect ratio while varying the size of the anchor, and calculating a ratio of objects having a degree of consistency above a threshold as an adoption rate of the object region, the ratio being the ratio of the number of objects having a degree of consistency above the threshold to the number of objects included in all image data of the teacher data; as well as Based on the threshold value, the size of the anchor having the highest adoption rate calculated as a result of the calculation is determined as the size of the anchor used in the learning completion procedure.

9. A computer-readable recording medium having an anchor setting program recorded thereon, the computer-readable recording medium recording the anchor setting program, the anchor setting program causing a computer to execute processing for setting an anchor used in an object sensing program for detecting whether an object is included in an image, the anchor setting program causing the computer to execute the following steps: acquiring teacher data including a plurality of image data including region information of an object; Acquire anchor information as frame information that specifies an area of each cell for detecting the presence or absence of an object from an image; obtaining information on aspect ratios of the object region and the anchor of the teacher data, calculating the degree of consistency between the anchor and the object region at each aspect ratio while varying the size of the anchor, and calculating a ratio of objects having a degree of consistency above a threshold as an adoption rate of the object region, the ratio being the ratio of the number of objects having a degree of consistency above the threshold to the number of objects included in all image data of the teacher data; as well as Based on the threshold value, the size of the anchor having the highest adoption rate calculated as a result of the calculation is determined as the size of the anchor used in the learning completion procedure.

Images