Methods, devices, and storage media for setting anchor frames based on image regions

Abstract

This invention discloses a method, device, and medium for setting anchor boxes based on image regions, relating to the field of image detection technology. The method includes: statistically analyzing the bounding box sizes of targets to be detected within each image region, and clustering these bounding box sizes to obtain anchor box sizes for each image region; acquiring a feature map obtained from feature extraction of the image to be detected; dividing the feature map into multiple sub-feature maps corresponding to different image regions; performing convolution processing on each sub-feature map to obtain a direct prediction result for each image region; and obtaining the predicted bounding box size for each image region based on the anchor box size and scaling ratio. This embodiment sets anchor boxes of different sizes for different image regions, enhancing edge detection performance and reducing the number of invalid anchor boxes.

Description

Technical Field

[0001] The embodiments of the present invention relate to image detection technology, and more particularly to a method, device and storage medium for setting anchor frames based on image regions. Background Technology

[0002] In object detection tasks, identifying targets of different sizes and regions within an image is a common challenge. This is typically addressed using the anchor box method. Specifically, based on the actual size of the target, several typical bounding boxes are obtained through manual selection or clustering methods, and these boxes are then set at several fixed points in the image. During detection, the object detection model makes minor adjustments to these pre-defined bounding boxes to obtain more accurate detection results.

[0003] In real-world applications, many targets appear at the edges of images. If existing anchor box setting methods are still used, a large number of anchor boxes will extend beyond the image boundaries, increasing the number of invalid anchor boxes. Moreover, the target recognition effect at the image edges is relatively poor. Summary of the Invention

[0004] This invention provides a method, device, and storage medium for setting anchor frames based on image regions, so as to set anchor frames of different sizes for different image regions, enhance edge recognition effect, and reduce the number of invalid anchor frames.

[0005] In a first aspect, embodiments of the present invention provide a method for setting anchor frames based on image regions, including:

[0006] The statistical data focuses on the bounding box size of the target to be detected within each image region, and the bounding box size of the target to be detected within each image region is clustered to obtain the anchor box size within each image region; the dataset includes multiple images;

[0007] A feature map is obtained from the image to be detected after feature extraction; the image to be detected includes multiple non-overlapping image regions.

[0008] The feature map is divided into multiple sub-feature maps corresponding to different image regions. The multiple sub-feature maps are convolved to obtain the direct prediction result for each image region. The direct prediction result includes at least the size scaling ratio of the prediction box offset within each image cell relative to the anchor box.

[0009] The size of the predicted bounding box for each image region is obtained based on the anchor box size and the size scaling ratio of each image region. The size of the predicted bounding box is used to predict the true size of the target to be detected.

[0010] Secondly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0011] One or more processors;

[0012] Memory, used to store one or more programs.

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the anchor frame setting method based on the image region as described in any embodiment.

[0014] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the anchor frame setting method based on image region as described in any embodiment.

[0015] This invention considers that when a target to be detected appears in different image regions, it will exhibit different shape characteristics due to incomplete display. Therefore, it clusters anchor box sizes within different image regions. Because the anchor box sizes conform to the shape characteristics of the target to be detected within the region, very few anchor boxes will exceed the image boundaries. By convolving the sub-feature maps corresponding to different image regions and further processing them using the corresponding region's anchor box sizes, predicted box sizes that conform to the presentation characteristics of the target to be detected in different regions are obtained. Because the predicted box sizes conform to the shape characteristics of the target to be detected within the region, the target positions located within each image region can be accurately predicted. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an anchor frame setting method based on an image region provided by an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the bounding box of a target to be detected according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of another bounding box of a target to be detected provided in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the feature map provided in an embodiment of the present invention;

[0021] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0024] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0025] This invention provides a method for setting anchor frames based on image regions, the flowchart of which is shown below. Figure 1 As shown, this is applicable to situations where anchor boxes and prediction boxes are set in an image. This embodiment is executed by an electronic device. Combined with... Figure 1 The method provided in this embodiment specifically includes:

[0026] S110. Statistical data are concentrated on the bounding box size of the target to be detected in each image region, and the bounding box size of the target to be detected in each image region is clustered separately to obtain the anchor box size in each image region.

[0027] The dataset includes multiple images of the same size, identical to the image to be detected, and all images display the target to be detected. This embodiment does not limit the type of target to be detected; it can be a person, vehicle, or object, etc.

[0028] Optionally, on each image in the dataset, the bounding box of the target to be detected can be drawn using manual annotation or target recognition methods. Figure 2 This is a schematic diagram of a bounding box of a target to be detected according to an embodiment of the present invention. The bounding box is rectangular, and a bounding box is uniquely represented by its center position (indicated by a dark dot), width, and height.

[0029] Multiple images in the dataset and the image to be detected are pre-divided into multiple image regions using the same method. In other words, the image regions in the dataset and the image to be detected are identical. These multiple image regions do not overlap. It should be noted that the image regions are only divided in terms of size and location; the image is not cropped. See also... Figure 2 The image contains multiple regions, including edge regions and central regions. The central region is outlined with a dashed line. Clearly, targets appearing in the central region are displayed completely, with their bounding boxes having similar width and height; targets appearing in the edge regions are not displayed completely, with their bounding boxes having larger differences in width and height. Therefore, it is necessary to calculate the bounding box dimensions for each image region separately.

[0030] In one optional implementation, the first step is to obtain the center positions of the bounding boxes of the targets to be detected in the multiple images, and to summarize the targets to be detected appearing in each image region based on the center positions of the bounding boxes of the targets to be detected. Optionally, see [link to relevant documentation]. Figure 2 The target object (cat) whose center position appears in the edge region is considered as the target object appearing in the edge region, and the target object (cat and dog) whose center position appears in the image is considered as the target object appearing in the middle region. Preferably, in Figure 2 On this basis, Figure 3 This is a schematic diagram of another bounding box of a target to be detected provided in an embodiment of the present invention. The edge regions include left and right edge regions, top and bottom edge regions, and corner regions. Any image in the dataset or the image to be detected is uniformly divided into a 9-grid, numbered sequentially from 1 to 9. Numbers 1, 3, 7, and 9 are corner regions, numbers 4 and 6 are left and right edge regions, numbers 2 and 8 are top and bottom edge regions, and number 5 is the middle region.

[0031] The statistical data sets determine which grid the center of the bounding box of each target to be detected falls on, and the targets to be detected are divided into the following 4 groups according to the grid number.

[0032] Group 1: Targets whose center position appears in the left and right edge regions are considered to be targets appearing in the left and right edge regions. Targets in this group appearing at the left and right edges of the image are considered incompletely displayed. Characteristics include: narrow and tall bounding boxes with small length and width values.

[0033] The second group: Targets whose center position appears in the upper and lower edge regions are considered to be targets appearing in the upper and lower edge regions. Targets in this group appearing at the upper and lower edges of the image are considered incompletely displayed. Characteristics include: tall and narrow bounding boxes with small length and width values.

[0034] The third group: Targets whose center position appears in the corner region are considered to be targets appearing in the corner region. Targets in this group appear in the four corners of the image and are considered incompletely displayed. The characteristics are: the bounding box's length and width are similar and the length and width values ​​are relatively small.

[0035] The fourth group: Targets appearing at the center of the image are considered to appear within the central region. Targets appearing in the center of the image are considered to be fully displayed. Since this embodiment does not limit the type of target, to accommodate anchor box settings for various targets, targets of any shape can appear in the central region. Therefore, targets in all regions of the image are considered to appear within the central region. A key feature is that the bounding box can be large or small, high or low, wide or narrow.

[0036] The second step is to calculate the bounding box size of the target object appearing in each image region, including the width and height of the bounding box at the pixel level.

[0037] Step 3: Cluster the bounding box dimensions of the target to be detected in each image region to obtain the anchor box dimensions of each image region.

[0038] Optionally, the K-means clustering algorithm can be used to cluster the bounding box dimensions within each of the above groups to obtain several clusters, and the center point of each cluster can be taken as the anchor box size.

[0039] It should be noted that clustering each group actually involves clustering the bounding box dimensions within each image region. When clustering the bounding box dimensions within the central region, the clustering is performed on the bounding box dimensions of the target objects that appear at the center of the image.

[0040] For ease of description and differentiation, anchor frame dimensions include both width and height, with each anchor frame dimension consisting of a set of width and height. The number of anchor frame dimensions can be the same or different across different image regions. For example, the left and right edge regions and the top and bottom edge regions each have 3 anchor frame dimensions, the corner regions have 2 anchor frame dimensions, and the center region has 3 anchor frame dimensions.

[0041] S120. Obtain the feature map of the image to be detected after feature extraction.

[0042] This embodiment does not limit the method of feature extraction; feature extraction can be performed using at least one convolutional layer.

[0043] S130. Divide the feature map into multiple sub-feature maps corresponding to different image regions, and perform convolution processing on the multiple sub-feature maps to obtain the direct prediction result for each image region.

[0044] Figure 4 This is a schematic diagram of a feature map provided in an embodiment of the present invention. The feature values ​​in the feature map have a one-to-one correspondence with pixel blocks in the image, and the feature map can be divided in the same way as the image. For example, dividing the feature map into a nine-square grid yields nine sub-feature maps, each corresponding to a different pixel. Figure 3 The image region at the corresponding position in the middle.

[0045] See also Figure 4 A corresponding convolutional layer is set for each sub-feature map. Each sub-feature map is passed through the corresponding convolutional layer in parallel, thus undergoing convolution operations to obtain the direct prediction result for each image region. The direct prediction result includes at least the scaling ratio of the prediction box offset within each image cell relative to the anchor box size (t). h ,t w It also includes the offset of the prediction box relative to the anchor box within the cell (t). x ,t y The calculation methods for offset and confidence level are not improved in this embodiment; only the dimensions of the anchor frame and prediction frame are considered.

[0046] S140. The size of the predicted bounding box for each image region is obtained based on the anchor box size and the size scaling ratio of each image region. The size of the predicted bounding box is used to predict the true size of the target to be detected.

[0047] This step applies the anchor box size to the size scaling factor in the direct prediction result within the same image area. See the following formula, which reflects the size characteristics of the anchor box in the prediction box size.

[0048]

[0049]

[0050] Among them, t h t is the scaling ratio of the height of the predicted bounding box relative to the height of the anchor box. w p is the scaling factor of the prediction box width relative to the anchor box width. w and p h These are the width and height of the anchor frame, respectively, b w and b h The width and height of the prediction box are respectively.

[0051] Optionally, after obtaining the predicted bounding boxes within each image region, the predicted bounding boxes are filtered using bounding box regression and IOU algorithms to predict the true size of the target to be detected.

[0052] Preferably, after obtaining the predicted bounding box size of each image region based on the anchor box size and size scaling ratio of each image region, the method further includes: merging the predicted bounding box sizes of multiple image regions according to the position of the image regions; the merged predicted bounding box sizes are used to predict the true size of the target to be detected.

[0053] by Figure 3 Taking the image region shown as an example, four sets of output results corresponding to the four image regions will be obtained, namely the size of the predicted box. The sizes of the four sets of predicted boxes are merged according to the position of the image region to obtain the size of all predicted boxes in the entire image to be detected. Thus, the prediction of the true size of the target to be detected can be performed at once without the need to predict separately for each image region.

[0054] Based on the above embodiments, the number of anchor box sizes can vary across different image regions. To improve clustering performance, different numbers of clusters (2, 3, 4, ...) are traversed, and the inter-cluster distances are tested for each cluster number. The cluster with the largest inter-cluster distance is selected, which represents the number of anchor box sizes. The number of anchor box sizes obtained through this method fully considers the distribution of bounding box sizes and accurately represents the size characteristics of the bounding boxes. The number of anchor box sizes within each image region clustered using this method is generally different.

[0055] Within the same image region, there is a one-to-one correspondence between the anchor box size and the predicted box offset within each image cell in the direct prediction result, so that they can be multiplied one-to-one to obtain the size of the predicted box. Therefore, it is necessary to control the number of size scaling ratios in the direct prediction result corresponding to each image region. Optionally, the number of size scaling ratios can be controlled by controlling the number of convolution kernels. Specifically, the multiple sub-feature maps are convolved to obtain the direct prediction result for each image region, including: determining the number of convolution kernels based on the number of anchor box sizes within the image region corresponding to each sub-feature map; passing the multiple sub-feature maps through the corresponding number of convolution kernels to obtain the direct prediction result, wherein the direct prediction result includes at least the size scaling ratio of the predicted box offset within each image cell relative to the anchor box; the number of predicted boxes offset within each image cell is consistent with the number of anchor box sizes.

[0056] For example, the middle region has 3 anchor box sizes, and the corresponding sub-feature map of the middle region is 3×3 in size with 80 categories. The sub-feature map of the middle region is processed by 3×(80+5) = 255 convolutional kernels, where 3 represents the number of anchor box sizes. The resulting direct prediction is a feature map containing 3×3 cells. Each image cell includes 3 predicted bounding boxes scaled relative to the anchor box size. Within the same image region (e.g., the middle region), the scaled bounding box size relative to the anchor box within each image cell is multiplied by the corresponding anchor box size to obtain the predicted bounding box size for each image region. Finally, the sizes of the 3×3×3 = 27 predicted bounding boxes within the middle region are obtained.

[0057] For example, the upper and lower edge regions have three anchor box sizes, with 80 categories. The sub-feature maps of the upper and lower edges are both 3×3 in size. These sub-feature maps are merged to obtain a 3×6 feature map. The corresponding sub-feature maps of the upper and lower edge regions are then processed by 3×(80+5) = 255 convolutional kernels, where 3 represents the number of anchor box sizes. The resulting direct prediction is a feature map containing 3×6 cells. Each image cell includes three predicted bounding boxes scaled relative to the anchor box size. Within the same image region (e.g., the upper and lower edge regions), the scaled bounding box size relative to the anchor box within each image cell is multiplied by the corresponding anchor box size to obtain the predicted bounding box size for each image region. Finally, the sizes of 3×6×3 = 54 predicted bounding boxes are obtained within the upper and lower edge regions.

[0058] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the device includes a processor 40, a memory 41, an input device 42, and an output device 43; the number of processors 40 in the device can be one or more. Figure 5 Taking a processor 40 as an example; the processor 40, memory 41, input device 42, and output device 43 in the device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0059] The memory 41, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the image region-based anchor frame setting method in this embodiment of the invention. The processor 40 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 41, thereby implementing the aforementioned image region-based anchor frame setting method.

[0060] The memory 41 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 41 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 41 may further include memory remotely located relative to the processor 40, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0061] Input device 42 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 43 may include display devices such as a display screen.

[0062] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the anchor frame setting method based on an image region according to any embodiment.

[0063] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0064] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0065] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0066] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for setting anchor frames based on image regions, characterized in that, include: The statistical data focuses on the bounding box size of the target to be detected within each image region, and the bounding box size of the target to be detected within each image region is clustered to obtain the anchor box size within each image region; the dataset includes multiple images; Multiple image regions include edge regions and central regions; The statistical data focuses on the bounding box size of the target to be detected within each image region, including: obtaining the center position of the bounding box of the target to be detected in the multiple images; identifying the target to be detected whose center position appears in the edge region as the target to be detected within the edge region; identifying the target to be detected whose center position appears in the image as the target to be detected within the middle region; and calculating the bounding box size of the target to be detected within each image region. A feature map is obtained from the image to be detected after feature extraction; the image to be detected includes multiple non-overlapping image regions. The feature map is divided into multiple sub-feature maps corresponding to different image regions. The multiple sub-feature maps are convolved to obtain the direct prediction result for each image region. The direct prediction result includes at least the size scaling ratio of the prediction box offset within each image cell relative to the anchor box. The size of the predicted bounding box for each image region is obtained based on the anchor box size and the size scaling ratio of each image region. The size of the predicted bounding box is used to predict the true size of the target to be detected. The process of obtaining a direct prediction result for each image region by performing convolution processing on the multiple sub-feature maps includes: determining the number of convolution kernels based on the number of anchor box sizes within the image region corresponding to each sub-feature map; passing the multiple sub-feature maps through the corresponding number of convolution kernels to obtain a direct prediction result, wherein the direct prediction result includes at least the scaling ratio of the predicted box offset within each image cell relative to the anchor box size; and the number of predicted boxes offset within each image cell is consistent with the number of anchor box sizes.

2. The method according to claim 1, characterized in that, The edge region includes left and right edge regions, top and bottom edge regions, and corner regions; The step of defining a target whose center location appears in the edge region as a target appearing in the edge region includes: The target to be detected whose center position appears in the left and right edge regions is defined as the target to be detected that appears in the left and right edge regions. The target to be detected whose center position appears in the upper and lower edge regions is defined as the target to be detected that appears in the upper and lower edge regions. The target to be detected whose center position appears in the corner area is considered as the target to be detected within the corner area.

3. The method according to claim 1, characterized in that, Within the same image region, the offset prediction box and the anchor box size within each image cell unit have a one-to-one correspondence. The step of obtaining the predicted bounding box size for each image region based on the anchor box size and size scaling ratio for each image region includes: Within the same image region, the size of the predicted bounding box for each image region is obtained by multiplying the size scaling ratio of each predicted bounding box relative to the anchor box within each image cell by the corresponding anchor box size.

4. The method according to claim 1, characterized in that, After obtaining the size of the predicted bounding box for each image region based on the anchor box size and the size scaling ratio for each image region, the method further includes: The sizes of the prediction boxes of multiple image regions are merged according to their locations. The dimensions of the merged prediction boxes are used to predict the true size of the target to be detected.

5. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image region-based anchor frame setting method as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the anchor frame setting method based on the image region as described in any one of claims 1-4.

Images