Training sample set determination, model training, object detection methods and devices

Abstract

This application relates to the field of deep learning technology, specifically to a method and apparatus for determining a training sample set, training a model, and detecting objects, solving the problem of low detection accuracy of deep learning models for small objects in images to be detected. The method for determining the training sample set includes: performing a movement operation on the target region in the sample image to determine multiple enhanced target regions; determining a training sample set corresponding to the sample image based on multiple candidate regions and multiple enhanced target regions; and training a network model based on the training sample set to obtain an object detection model. By performing a movement operation on the target region in the sample image, the probability of small target regions being selected as training sample sets is increased without significantly affecting the selection of large target regions, thereby improving the detection accuracy of the object detection model obtained by training the network model using this training sample set for small targets.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, specifically to a training sample set determination method and apparatus, a model training method and apparatus, an object detection method and apparatus, as well as a computer-readable storage medium and electronic device. Background Technology

[0002] Object detection is a crucial application in the field of deep learning, and with the continuous development of deep learning technology, its application scenarios are expanding. Object detection is widely used in intelligent transportation, video surveillance, vehicle-to-infrastructure (V2I) communication, and other scenarios. Small object detection is an important branch of object detection and also one of its most challenging aspects. Specifically, small objects in the image to be detected often exhibit motion blur and are easily occluded, leading to lower detection accuracy of deep learning models for small objects in the image. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method and apparatus for determining a training sample set, a method and apparatus for model training, a method and apparatus for object detection, as well as a computer-readable storage medium and an electronic device, which solve the problem of low detection accuracy of deep learning models for small targets in images to be detected.

[0004] In a first aspect, one embodiment of this application provides a method for determining a training sample set, comprising: performing a movement operation on a target region in a sample image to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; determining a training sample set corresponding to the sample image based on multiple candidate regions and multiple enhanced target regions corresponding to the sample image; training a network model based on the training sample set corresponding to the sample image to obtain a target detection model, wherein the training sample set corresponding to the sample image includes at least one candidate region among the multiple candidate regions corresponding to the sample image.

[0005] In conjunction with the first aspect of this application, in some embodiments, a target region in a sample image is moved to determine multiple enhanced target regions, including: moving the target region in the sample image based on a preset path to determine multiple enhanced target regions.

[0006] In conjunction with the first aspect of this application, in some embodiments, a target region in a sample image is moved based on a preset path to determine multiple enhanced target regions, including: a translation operation is performed on the target region in the sample image based on the preset path to determine multiple enhanced target regions; and / or a rotation operation is performed on the target region in the sample image based on the preset path to determine multiple enhanced target regions.

[0007] In conjunction with the first aspect of this application, in some embodiments, determining a training sample set corresponding to a sample image based on multiple candidate regions and multiple enhancement target regions corresponding to a sample image includes: for each current enhancement target region among the multiple enhancement target regions, calculating the intersection-union ratio (IUU) between the multiple candidate regions and the current enhancement target region, and determining the overlap degree corresponding to each of the multiple candidate regions; determining the training sample set corresponding to the current enhancement target region based on the overlap degree and overlap threshold corresponding to each of the multiple candidate regions; and determining the training sample set corresponding to the sample image based on the training sample sets corresponding to each of the multiple enhancement target regions.

[0008] In conjunction with the first aspect of this application, in some embodiments, the overlap threshold includes a fixed overlap threshold and a dynamic overlap threshold. Determining a training sample set corresponding to the current enhancement target region based on the overlap corresponding to each of multiple candidate regions and the overlap threshold includes: determining a first training sample set corresponding to the current enhancement target region based on the overlap corresponding to each of the multiple candidate regions and the fixed overlap threshold, the first training sample set including at least one candidate region among the multiple candidate regions; determining the overlap corresponding to the current enhancement target region based on the overlap corresponding to each of the multiple candidate regions, and determining the mean and standard deviation of the overlap based on the overlap corresponding to each of the multiple enhancement target regions; determining a stability coefficient of the current enhancement target region based on the first training sample set corresponding to each of the multiple enhancement target regions; determining a dynamic overlap threshold of the current enhancement target region based on the mean, standard deviation, and stability coefficient of the current enhancement target region; determining a second training sample set corresponding to the current enhancement target region based on the overlap corresponding to each of the multiple candidate regions and the dynamic overlap threshold of the current enhancement target region; and determining a training sample set corresponding to the current enhancement target region based on the first and second training sample sets.

[0009] In conjunction with the first aspect of this application, in some embodiments, before performing a movement operation on a target region in a sample image and determining multiple enhanced target regions, the method further includes: for each current target region in the multiple target regions of the sample image, pre-moving the current target region to determine multiple pre-enhanced target regions corresponding to the current target region, wherein the multiple pre-enhanced target regions include the current target region and the pre-moved target region corresponding to the current target region; determining the degree of overlap between the current target region and the multiple pre-enhanced target regions corresponding to the current target region based on the multiple candidate regions corresponding to the sample image and the multiple pre-enhanced target regions corresponding to the current target region; and determining the movement order of the multiple target regions based on the degree of overlap of each of the multiple target regions, so as to perform a movement operation on the multiple target regions in the sample image according to the movement order.

[0010] In conjunction with the first aspect of this application, in some embodiments, determining the degree of overlap of the current target region based on multiple candidate regions corresponding to a sample image and multiple pre-enhanced target regions corresponding to the current target region includes: for each current pre-enhanced target region corresponding to the current target region, calculating the sum of the intersection-union ratios (IUR) of the current pre-enhanced target region with the multiple candidate regions, and calculating the sum of the center distances of the current pre-enhanced target region with the multiple candidate regions; determining the sum of the IUR of the current target region based on the sum of the IUR of the multiple pre-enhanced target regions, and determining the sum of the center distances of the current target region based on the sum of the center distances of the multiple pre-enhanced target regions; and determining the degree of overlap of the current target region based on the ratio of the sum of the IUR of the current target region to the sum of the center distances of the current target region.

[0011] Secondly, one embodiment of this application provides a model training method, comprising: performing a movement operation on a target region in a sample image to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; determining a training sample set corresponding to the sample image based on N candidate regions and the multiple enhanced target regions, wherein the training sample set corresponding to the sample image includes at least one candidate region among the N candidate regions, where N is a positive integer; and training an initial network model based on the training sample set corresponding to the sample image to generate a target detection model, wherein the target detection model is used to detect targets in the image to be detected.

[0012] In conjunction with the second aspect of this application, in some embodiments, an initial network model is trained based on a training sample set corresponding to a sample image to generate an object detection model, including: training the initial network model based on the training sample set corresponding to the sample image to obtain a preliminary object detection model; determining a fine-tuning training sample set corresponding to the sample image based on M candidate regions corresponding to the sample image and the target region in the sample image, wherein M is a positive integer, M is less than N, and the fine-tuning training sample set is at least one candidate region among the M candidate regions; adjusting the number of output channels of the preliminary object detection model based on the M candidate regions corresponding to the sample image to obtain an intermediate object detection model; and training the intermediate object detection model based on the fine-tuning training sample set corresponding to the sample image to generate an object detection model.

[0013] In conjunction with the second aspect of this application, in some embodiments, before determining the fine-tuning training sample set corresponding to the sample image based on the M candidate regions corresponding to the sample image and the target region in the sample image, the method further includes: determining the distribution heatmap corresponding to the sample image based on the distribution information of the target region in the sample image; and determining the M candidate regions corresponding to the sample image based on the distribution heatmap corresponding to the sample image and the N candidate regions corresponding to the sample image.

[0014] Thirdly, one embodiment of this application provides a target detection method, including: determining an image to be detected; using a target detection model to detect the image to be detected and determining a target region in the image to be detected, wherein the target detection model is trained based on the model training method mentioned in the second aspect.

[0015] Fourthly, one embodiment of this application provides a training sample set determination apparatus, comprising: a moving module configured to perform a moving operation on a target region in a sample image to determine a plurality of enhanced target regions, wherein the plurality of enhanced target regions include the moved target region; and a sample determination module configured to determine a training sample set corresponding to the sample image based on a plurality of candidate regions and a plurality of enhanced target regions corresponding to the sample image, and to train a network model based on the training sample set corresponding to the sample image to obtain a target detection model, wherein the training sample set corresponding to the sample image includes at least one candidate region among the plurality of candidate regions corresponding to the sample image.

[0016] Fifthly, one embodiment of this application provides a model training apparatus, comprising: a moving module configured to perform a moving operation on a target region in a sample image to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; a sample determination module configured to determine a training sample set corresponding to the sample image based on N candidate regions and the multiple enhanced target regions, wherein the training sample set corresponding to the sample image includes at least one candidate region among the N candidate regions, where N is a positive integer; and a training module configured to train an initial network model based on the training sample set corresponding to the sample image to generate a target detection model, wherein the target detection model is used to detect targets in the image to be detected.

[0017] In a sixth aspect, one embodiment of this application provides a target detection apparatus, comprising: an image determination module configured to determine an image to be detected; and a detection module configured to use a target detection model to detect the image to be detected and determine a target region in the image to be detected, wherein the target detection model is trained based on the model training method mentioned in the second aspect.

[0018] In a seventh aspect, one embodiment of this application provides a computer-readable storage medium storing instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the methods mentioned in the first to third aspects.

[0019] Eighthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing computer-executable instructions; and a processor for executing the computer-executable instructions to implement the methods mentioned in the first to third aspects above.

[0020] The training sample set determination method provided in this application first performs a movement operation on the target region in the sample image to determine multiple enhanced target regions, including the moved target region. Then, based on the multiple candidate regions and multiple enhanced target regions corresponding to the sample image, a training sample set corresponding to the sample image is determined. A network model is trained based on the training sample set corresponding to the sample image to obtain a target detection model. Since the side length of a small target region (i.e., the region of a small target) is generally tens of pixels, moving the region of a small target by a few pixels can effectively increase the probability that the region of a small target is selected as a positive sample. However, the side length of a large target region (i.e., the region of a large target) is generally hundreds of pixels (or even thousands of pixels), so moving the region of a large target by a few pixels has little impact on the probability that the region of a large target is selected as a positive sample. Therefore, by performing a movement operation on the target region in the sample image, the probability that the region of a small target is selected as a training sample set is increased without affecting the selection of large target regions as much as possible, thereby improving the detection accuracy of the target detection model trained using the training sample set for small targets. Attached Figure Description

[0021] Figure 1 The diagram shown is an application scenario diagram of the training sample set determination method provided in an embodiment of this application.

[0022] Figure 2 The diagram shown is an application scenario illustration of the target detection method provided in an embodiment of this application.

[0023] Figure 3 The diagram shown is a flowchart illustrating a method for determining a training sample set according to an embodiment of this application.

[0024] Figure 4a The image shown is a sample image provided in one embodiment of this application.

[0025] Figure 4b The image shown is a sample image provided in another embodiment of this application.

[0026] Figure 4c The image shown is a sample image provided in another embodiment of this application.

[0027] Figure 5 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application.

[0028] Figure 6a The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application.

[0029] Figure 6bThe diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application.

[0030] Figure 7 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application.

[0031] Figure 8 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application.

[0032] Figures 9a to 9d The image shown is a sample image provided in another embodiment of this application.

[0033] Figures 9e to 9h The image shown is a sample image provided in another embodiment of this application.

[0034] Figure 10 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application.

[0035] Figure 11 The diagram shows the relationship between the center distance between the target region and the candidate region and the classification confidence of the target.

[0036] Figure 12 The diagram shown is a flowchart of a model training method provided in an embodiment of this application.

[0037] Figure 13 The diagram shown is a flowchart of a model training method provided in another embodiment of this application.

[0038] Figure 14 The diagram shown is a schematic diagram of the principle of adjusting the number of output channels of a model provided in an embodiment of this application.

[0039] Figure 15 The diagram shown is a schematic diagram of the pruning principle of a model provided in an embodiment of this application.

[0040] Figure 16 The diagram shown is a flowchart of a model training method provided in another embodiment of this application.

[0041] Figure 17 The image shown is a heat map of the distribution corresponding to a sample image provided in an embodiment of this application.

[0042] Figure 18 The diagram shows the generation principle of M candidate regions.

[0043] Figure 19 The diagram shown is a flowchart of a target detection method provided in an embodiment of this application.

[0044] Figure 20The diagram shown is a schematic diagram of the training sample set determination device provided in an embodiment of this application.

[0045] Figure 21 The diagram shown is a schematic diagram of the training sample set determination device provided in another embodiment of this application.

[0046] Figure 22 The diagram shown is a schematic diagram of the structure of a model training device provided in an embodiment of this application.

[0047] Figure 23 The diagram shown is a schematic diagram of the structure of a model training device provided in another embodiment of this application.

[0048] Figure 24 The diagram shown is a schematic diagram of the target detection device provided in an embodiment of this application.

[0049] Figure 25 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] Object detection is widely used in intelligent transportation, video surveillance, vehicle-to-infrastructure (V2I) communication, and other scenarios. Small object detection is an important branch of object detection and also one of its challenges. Specifically, small objects in the image to be detected are characterized by motion blur and susceptibility to occlusion. Detecting target regions in the image using a deep learning model requires first training the original deep learning model with a training sample set. Then, the trained deep learning model is used to detect target regions in the image, thereby identifying the target regions (including target regions of various sizes).

[0052] However, in the process of selecting the training sample set in the sample images, small targets in the sample images also have characteristics such as motion blur and susceptibility to occlusion. As a result, the number of positive samples including small targets in the training sample set selected from the sample images is relatively small. Consequently, when the original deep learning model is trained using this training sample set, the original deep learning model cannot fully learn the features of small targets, resulting in low detection accuracy of the trained deep learning model in the small target region of the image to be detected.

[0053] Specifically, in the process of selecting a training sample set from sample images, due to the small area occupied by small targets in the sample images, their susceptibility to occlusion, and motion blur, the proportion of small targets in the selected training sample set is inconsistent with the proportion of small targets in the images to be detected in actual applications. Therefore, using a training sample set obtained by directly selecting positive samples from the sample images (e.g., the Adaptive Training Sample Selection (ATSS) method) to train a deep learning model results in a low accuracy of small target detection in the trained deep learning model.

[0054] The training sample set determination method provided in this application first performs a movement operation on the target region in the sample image to determine the moved target region. Then, based on multiple candidate regions corresponding to the sample image and the moved target region, the training sample set corresponding to the sample image is determined. The network model is trained based on the training sample set corresponding to the sample image to obtain the target detection model.

[0055] Specifically, this application embodiment performs a movement operation on the target region in the sample image to simulate the movement of small targets in real-world application scenarios. Compared to methods that increase the number of positive samples for small targets by lowering the selection threshold for positive samples, this application's movement operation on the target region in the sample image can effectively increase the number of positive samples for small targets, while having little or no impact on the number of positive samples for large targets. Specifically, since a small target region (i.e., the region of a small target) generally refers to a region with a side length of tens of pixels, moving the region of a small target by a few pixels can effectively increase the probability that the region of a small target will be selected as a positive sample. However, a large target region (i.e., the region of a large target) generally refers to a region with a side length of hundreds of pixels (or even thousands of pixels), so moving the region of a large target by a few pixels has little impact on the probability that the region of a large target will be selected as a positive sample. Therefore, by performing a movement operation on the target region in the sample image, the probability that the region of a small target will be selected as part of the training sample set is increased without significantly affecting the selection of samples from the region of a large target, thereby improving the detection accuracy of the target detection model trained using this training sample set for small targets.

[0056] Exemplary scenario

[0057] Figure 1 The diagram shown is an application scenario diagram of the training sample set determination method provided in an embodiment of this application. Figure 1The illustrated scenario includes a processor 110 and an image acquisition device 120 communicatively connected to the processor 110. Specifically, the processor 110 performs a movement operation on a target region in a sample image to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; based on multiple candidate regions and multiple enhanced target regions corresponding to the sample image, it determines a training sample set corresponding to the sample image, and trains a network model based on the training sample set corresponding to the sample image to obtain a target detection model, wherein the training sample set corresponding to the sample image is at least one candidate region among the multiple candidate regions corresponding to the sample image. The image acquisition device 120 generates a sample image and sends the generated sample image to the processor 110 so that the processor 110 can perform the above operations.

[0058] In one embodiment of this application, the processor 110 can also be used to perform a movement operation on the target region in the sample image, determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; determine a training sample set corresponding to the sample image based on N candidate regions and multiple enhanced target regions corresponding to the sample image, wherein the training sample set corresponding to the sample image is at least one candidate region among the N candidate regions, and N is a positive integer; and train an initial network model based on the training sample set corresponding to the sample image to generate a target detection model, wherein the target detection model is used to detect targets in the image to be detected.

[0059] Figure 2 The diagram shown is an application scenario illustration of the target detection method provided in an embodiment of this application. Figure 2 The scenario shown is an intelligent transportation scenario. Specifically, the object detection method can be applied to a vehicle-to-infrastructure (V2I) system within an intelligent transportation scenario. The V2I system includes a server 210 and an image acquisition device 220 communicatively connected to the server 210. Specifically, the image acquisition device 220 is used to acquire images of roads and vehicles, pedestrians, etc., on the roads, and sends the acquired images to the server 210. After receiving the image to be detected acquired by the image acquisition device 220, the server 210 uses an object detection model to detect the image and determine the target region in the image. The object detection model is obtained based on the model training method of this application.

[0060] Exemplary methods

[0061] Figure 3 The diagram shown is a flowchart illustrating a method for determining a training sample set according to an embodiment of this application. Figure 3 As shown, the method for determining the training sample set includes the following steps.

[0062] Step 310: Perform a movement operation on the target region in the sample image to determine multiple enhanced target regions.

[0063] Specifically, the sample image is the original image used to train the network model. A portion of the image containing the target region is extracted from the sample image and input into the network model to train it, resulting in the object detection model. The target region can be a large target region, a small target region, or a medium-sized target region (i.e., a medium-sized target region). The terms "large target region" and "small target region" are relative; generally, the area of ​​a large target region is several times larger than that of a small target region. Multiple enhanced target regions include the moved target regions. Specifically, multiple enhanced target regions can be multiple moved target regions obtained through multiple movement operations. Multiple enhanced target regions can also include target regions that have not undergone movement operations.

[0064] For example, the area of ​​a large target may include 960*480 pixels, and the area of ​​a small target may include 60*30 pixels, meaning the area of ​​the large target is 16 times the area of ​​the small target. The area between the areas of the large and small targets is the area of ​​the medium target.

[0065] Specifically, the movement operation can be a translation operation, a rotation operation, or other types of movement operations; this application does not impose any specific limitations.

[0066] Step 320: Based on multiple candidate regions and multiple enhanced target regions corresponding to the sample image, determine the training sample set corresponding to the sample image, and train the network model based on the training sample set corresponding to the sample image to obtain the target detection model.

[0067] Specifically, the training sample set corresponding to the sample image includes at least one candidate region from a plurality of candidate regions corresponding to the sample image. The training sample set is a collection of training samples. That is, if a candidate region is selected as a positive sample, then that candidate region is a training sample in the training sample set. The network model can be a deep learning network model. The object detection model is used to detect object regions in the image to be detected, in order to determine the object regions in the image to be detected.

[0068] In practical applications, multiple candidate regions can be determined using candidate bounding boxes. That is, multiple candidate regions are obtained by placing multiple candidate bounding boxes on the sample image. Each candidate bounding box corresponds to one candidate region.

[0069] For example, determining the training sample set corresponding to a sample image based on multiple candidate regions and multiple enhancement target regions can be performed as follows: for each enhancement target region, calculate the intersection area between the multiple candidate regions and the enhancement target region, and determine the candidate regions whose intersection areas satisfy a preset intersection threshold as training samples. Determining the training samples corresponding to a sample image based on multiple candidate regions and enhancement target regions can also be performed as follows: calculate the Intersection over Union (IOU) ratio between the multiple candidate regions and the enhancement target regions, and determine the candidate regions whose IOUs satisfy a preset IOU threshold as training samples.

[0070] For example, Figure 4a The image shown is a sample image provided in one embodiment of this application. Figure 4a The regions enclosed by multiple solid lines that are evenly distributed in the middle are the multiple candidate regions. Figure 4a The area enclosed by the dashed line is the target area. The dashed rectangle on the left can be a small target area, that is, an area containing an image of a small target. The dashed rectangle on the upper right can be a medium-sized target area, that is, an area containing an image of a medium-sized target. The dashed rectangle on the lower right can be a large target area, that is, an area containing an image of a large target.

[0071] For example, Figure 4b The image shown is a sample image provided in another embodiment of this application. Specifically, Figure 4b The image shows the pair. Figure 4a The sample image obtained after moving the target region in the image. For example... Figure 4a and Figure 4b As shown, Figure 4a After all three target regions in the image were shifted to the right, the result was... Figure 4b The image shown.

[0072] For example, Figure 4c The image shown is a sample image provided in another embodiment of this application. Specifically, Figure 4c The image shows the pair. Figure 4b The sample image obtained after moving the target region in the image. For example... Figure 4b and Figure 4c As shown, Figure 4b After all three target regions were shifted downwards, the following was obtained: Figure 4c The image shown.

[0073] By moving the target region in the sample image, the probability of selecting the region of small target as a training sample is increased without affecting the selection of the regions of large and medium targets as much as possible. This improves the detection accuracy of the target detection model trained using this training sample set for small targets.

[0074] Figure 5 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application. Figure 3 Extending from the illustrated embodiment Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0075] like Figure 5 As shown in the embodiments of this application, the step of moving the target region in the sample image to determine multiple enhanced target regions includes the following steps.

[0076] Step 510: Based on a preset path, perform a movement operation on the target region in the sample image to determine multiple enhanced target regions.

[0077] Specifically, the preset path can be a pre-set movement path for moving the target area.

[0078] In practical applications, the preset path can be the movement path of a small target obtained through empirical statistics. For example, in intelligent transportation scenarios, vehicles travel from left to right, so the preset path can be set to a left-to-right path. That is, the target area is moved along a left-to-right path. Exemplarily, based on the preset path, a movement operation is performed on the target area in the sample image. This can be a single movement operation or multiple movement operations; this application does not specifically limit the number of movements. Each movement operation yields a moved target area. For example, moving the target area 3 pixels from left to right yields a moved target area, and then moving it 5 pixels from left to right yields another moved target area, and so on.

[0079] Based on a preset path, the target region in the sample image is moved to determine multiple enhanced target regions, which further simulates the motion of small targets in real-world applications. This makes the training sample set closer to the image to be detected, and further improves the detection accuracy of the target detection model for small targets obtained by training the network model using this training sample set.

[0080] Figure 6a The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application. Figure 6bThe diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application. Figure 5 Extending from the illustrated embodiment Figure 6a and Figure 6b The illustrated embodiment will be described in detail below. Figure 6a and Figure 6b The illustrated embodiments and Figure 5 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0081] like Figure 6a and Figure 6b As shown in the embodiments of this application, the steps of moving the target region in the sample image based on a preset path to determine multiple enhanced target regions include the following steps.

[0082] Step 610: Based on a preset path, perform a translation operation on the target region in the sample image to determine multiple enhanced target regions.

[0083] Step 620: Based on a preset path, rotate the target region in the sample image to determine multiple enhanced target regions.

[0084] In practical applications, moving the target region in a sample image to determine multiple enhanced target regions can be performed as step 610, which involves translating the target region in the sample image based on a preset path to determine multiple enhanced target regions; or as step 620, which involves rotating the target region in the sample image based on a preset path to determine multiple enhanced target regions; or as steps 610 and 620, where first translating the target region in the sample image based on a preset path to determine multiple enhanced target regions, and then rotating the target region in the sample image based on a preset path to determine multiple enhanced target regions; or first rotating the target region in the sample image based on a preset path to determine multiple enhanced target regions, and then translating the target region in the sample image based on a preset path to determine multiple enhanced target regions; or performing steps 610 or 620 multiple times to obtain multiple enhanced target regions.

[0085] Moving the target region in the sample image can be done by translating and / or rotating the target region, which enriches the types of moving operations and further increases the number of candidate regions containing small targets in the training sample set. This further improves the target detection accuracy of the target detection model for small targets obtained by training the network model using this training sample set.

[0086] Figure 7 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application. Figure 3Extending from the illustrated embodiment Figure 7 The illustrated embodiment will be described in detail below. Figure 7 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0087] like Figure 7 As shown in the embodiments of this application, the step of determining the training sample set corresponding to the sample image based on multiple candidate regions and multiple enhancement target regions corresponding to the sample image includes the following steps.

[0088] Step 710: For each current enhanced target region among the multiple enhanced target regions, calculate the intersection-union ratio between the multiple candidate regions and the current enhanced target region, and determine the overlap degree corresponding to each of the multiple candidate regions.

[0089] Specifically, the overlap of each of the multiple candidate regions can be obtained by calculating the intersection-union ratio of the multiple candidate regions with the current enhanced target region. That is, the overlap of the candidate regions can be the intersection-union ratio of the candidate regions with the current enhanced target region.

[0090] Step 720: Based on the overlap and overlap threshold of each of the multiple candidate regions, determine the training sample set corresponding to the current enhancement target region.

[0091] Specifically, the overlap threshold can be a pre-set threshold. By comparing the overlap of multiple candidate regions with the overlap threshold, the training sample set corresponding to the current enhancement target region can be obtained. For example, if the overlap of a candidate region is greater than or equal to the overlap threshold, then the candidate region is determined as a training sample corresponding to the current enhancement target region; if the overlap of a candidate region is less than the overlap threshold, then the candidate region is not determined as a training sample corresponding to the current enhancement target region. Since the intersection-union ratio (IUR) of multiple candidate regions with the current enhancement target region can be calculated, i.e., the overlap of each candidate region can be obtained, the training sample set corresponding to the current enhancement target region may include at least one training sample, or it may not include any training samples. Specifically, if the overlap of multiple candidate regions is all less than the overlap threshold, then the training sample set corresponding to the current enhancement target region does not include any training samples, i.e., it is an empty set.

[0092] Step 730: Determine the training sample set corresponding to the sample image based on the training sample sets corresponding to each of the multiple enhanced target regions.

[0093] Specifically, each current enhanced target region corresponds to a training sample set, and the training sample set corresponding to the sample image includes the training sample sets corresponding to each of the multiple enhanced target regions.

[0094] The training sample set corresponding to the sample image is determined based on the overlap degree and overlap threshold of multiple candidate regions. The method is simple and efficient.

[0095] Figure 8 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application. Figure 7 Extending from the illustrated embodiment Figure 8 The illustrated embodiment will be described in detail below. Figure 8 The illustrated embodiments and Figure 7 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0096] like Figure 8 As shown in the embodiments of this application, the step of determining the training sample set corresponding to the current enhancement target region based on the overlap degree and overlap threshold of each of the multiple candidate regions includes the following steps.

[0097] Step 810: Based on the overlap degree and fixed overlap degree threshold of each of the multiple candidate regions, determine the first training sample set corresponding to the current enhanced target region.

[0098] Specifically, the first training sample set includes at least one candidate region from a plurality of candidate regions. The first training sample set is a collection of first training samples. That is, if a candidate region is selected as a positive sample, then that candidate region is a first training sample in the first training sample set. The fixed overlap threshold can be a pre-set threshold. By comparing the overlap corresponding to each of the plurality of candidate regions with the fixed overlap threshold, the first training sample set corresponding to the current augmentation target region can be obtained. For example, if the overlap corresponding to a candidate region is greater than or equal to the fixed overlap threshold, then the candidate region is determined as the first training sample corresponding to the current augmentation target region; if the overlap corresponding to a candidate region is less than the fixed overlap threshold, then the candidate region is not determined as the first training sample corresponding to the current augmentation target region.

[0099] Step 820: Based on the overlap of each of the multiple candidate regions, determine the overlap of the current enhancement target region, and based on the overlap of each of the multiple enhancement target regions, determine the mean and standard deviation of the overlap.

[0100] Specifically, the overlap of each of the multiple candidate regions is the intersection-union ratio (IUGR) between the multiple candidate regions and the current augmented target region. The overlap of the current augmented target region can be the sum of the overlaps of the multiple candidate regions, or the average of the overlaps of the multiple candidate regions. The mean of the overlap is the average of the overlaps of the multiple augmented target regions. The standard deviation of the overlap is the standard deviation of the overlaps of the multiple augmented target regions.

[0101] Step 830: Determine the stability coefficient of the current augmented target region based on the first training sample set corresponding to each of the multiple augmented target regions.

[0102] Specifically, the first training sample set corresponding to the current augmented target region includes at least one candidate region from multiple candidate regions. This first training sample set is obtained by comparing the overlap of each candidate region with a fixed overlap threshold. Therefore, the first training sample set consists of at least one candidate region among the multiple candidate regions that satisfies the fixed overlap threshold. Based on the first training sample sets corresponding to each of the multiple augmented target regions, the stability coefficient of the current augmented target region is determined as follows: if there is an intersection between the first training sample sets corresponding to each of the multiple augmented target regions, the stability coefficient can be set to a first value; if there is no intersection between the first training sample sets corresponding to each of the multiple augmented target regions, the stability coefficient can be set to a second value, where the first value is greater than the second value.

[0103] Step 840: Determine the dynamic overlap threshold of the current augmented target region based on the mean of the overlap, the standard deviation of the overlap, and the stability coefficient of the current augmented target region.

[0104] For example, the dynamic overlap threshold of the current enhanced target region can be calculated using the following formula (1).

[0105] IOU thre =IOU mean +a×IOU Sstd (1)

[0106] Among them, IOU thre Indicates the dynamic overlap threshold of the current augmented target region, IOU. mean IOU represents the mean of the overlap. Sstd The standard deviation of the overlap is represented by 'a', and 'a' represents the stability coefficient of the current augmented target region. If there is an intersection between the first training sample sets corresponding to the multiple augmented target regions, the stability coefficient 'a' of the multiple augmented target regions can be set to 1. If there is no intersection between the first training sample sets corresponding to the multiple augmented target regions, the stability coefficient 'a' of the multiple augmented target regions can be set to 0.1.

[0107] Figures 9a to 9d The image shown is a sample image provided in another embodiment of this application. Figures 9a to 9d As shown, uniformly distributed hollow square regions represent multiple candidate regions, rectangular regions represent the target region, solid square regions represent the first training sample set, and arrows indicate the movement direction of the target region. Figure 9a After shifting the target area to the right, we get Figure 9b The target area in Figure 9b After shifting the target region upwards, we obtain Figure 9c The target area in Figure 9c After shifting the target region to the left, we get Figure 9d The target area in the text. For example... Figures 9a to 9d As shown, it can be seen that there is no overlap between the first training sample sets corresponding to the current enhanced target region obtained after each move. Therefore, the stability coefficient 'a' of multiple enhanced target regions can be set to 0.1.

[0108] Figures 9e to 9h The image shown is a sample image provided in another embodiment of this application. Figures 9e to 9h As shown, uniformly distributed hollow square regions represent multiple candidate regions, rectangular regions represent the target region, solid square regions represent the first training sample set, and arrows indicate the movement direction of the target region. Figure 9e After shifting the target area to the right, we get Figure 9f The target area in Figure 9f After shifting the target region upwards, we obtain Figure 9g The target area in Figure 9g After shifting the target region to the left, we get Figure 9h The target area in the text. For example... Figures 9e to 9h As shown, it can be seen that there is an intersection between the first training sample sets corresponding to the current enhanced target region obtained after each move. Therefore, the stability coefficient 'a' of multiple enhanced target regions can be set to 1.

[0109] Step 850: Based on the overlap of each of the multiple candidate regions and the dynamic overlap threshold of the current enhanced target region, determine the second training sample set corresponding to the current enhanced target region.

[0110] Specifically, the second training sample set includes multiple second samples. If the overlap of a candidate region is greater than or equal to the dynamic overlap threshold of the current augmented target region, the candidate region is determined as a second training sample; if the overlap of a candidate region is less than the dynamic overlap threshold of the current augmented target region, the candidate region is not determined as a second training sample.

[0111] Step 860: Based on the first training sample set and the second training sample set, determine the corresponding training sample set for the current enhanced target region.

[0112] For example, the training sample set corresponding to the current augmented target region includes a first training sample set and a second training sample set. That is, the first training sample set and the second training sample set are combined into a training sample set corresponding to the current augmented target region.

[0113] By calculating the dynamic overlap threshold and using it to determine the training sample set, a more suitable threshold can be determined for each current enhanced target region, further improving the target detection accuracy of the target detection model for small targets obtained by training the network model using this training sample set.

[0114] Figure 10 The diagram shown is a flowchart illustrating a method for determining a training sample set according to another embodiment of this application. Figure 3 Extending from the illustrated embodiment Figure 10 The illustrated embodiment will be described in detail below. Figure 10 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0115] like Figure 10 As shown in the embodiments of this application, before the step of moving the target region in the sample image and determining multiple enhanced target regions, the following steps are also included.

[0116] Step 1010: Perform a pre-movement on the current target area to determine multiple pre-enhanced target areas corresponding to the current target area.

[0117] Specifically, the multiple pre-enhanced target regions include the current target region and the pre-moved target region corresponding to the current target region.

[0118] For example, for each current target region in multiple target regions in a sample image, the current target region is pre-moved to determine multiple pre-enhanced target regions corresponding to the current target region.

[0119] Step 1020: Based on the multiple candidate regions corresponding to the sample image and the multiple pre-enhanced target regions corresponding to the current target region, determine the degree of overlap between the current target region and the sample image.

[0120] Specifically, the degree of overlap corresponding to the current target region can be calculated as follows: For each of the multiple pre-enhanced target regions corresponding to the current target region, calculate the intersection-union ratio and center distance between the current pre-enhanced target region and the multiple candidate regions to obtain the intersection-union ratio and center distance corresponding to the current pre-enhanced target region. Then, calculate the sum of the intersection-union ratios and the sum of the center distances corresponding to the multiple pre-enhanced target regions corresponding to the current target region, and use the ratio of the sum of the intersection-union ratios to the sum of the center distances as the degree of overlap corresponding to the current target region.

[0121] This application obtains the result by statistically analyzing the center distance between the target region and the candidate region of the small target. Figure 11 The diagram shows the relationship between the center distance between the target region and the candidate region of a small target and the target's classification confidence. Figure 11The horizontal axis represents the center distance between the target region and the candidate region of the small target, and the vertical axis represents the classification confidence of the target. From Figure 11 It can be seen that the smaller the center distance between the target region and the candidate region of a small target, the higher the classification confidence of the target; conversely, the larger the center distance between the target region and the candidate region, the lower the classification confidence of the target. Therefore, by using the ratio of the sum of intersection-union ratios to the sum of center distances as the degree of overlap corresponding to the current target region, the classification confidence of the target can be improved.

[0122] Step 1030: Based on the degree of overlap of the multiple target regions, determine the movement order of the multiple target regions, so as to perform movement operations on the multiple target regions in the sample image according to the movement order.

[0123] Specifically, based on the degree of overlap of each of the multiple target areas, the movement order of the multiple target areas is determined. This can be achieved by arranging the multiple target areas in ascending order of overlap, thereby obtaining the movement order of the multiple target areas.

[0124] For example, the movement order of multiple target areas can be obtained using the following formula (2).

[0125]

[0126] Where SortRefList represents the move order, and IOU sum This represents the sum of the intersection-union ratios (IoU) of each of the multiple pre-enhanced target regions corresponding to the current target region. CTD sum This represents the sum of the center distances of the multiple pre-enhanced target regions corresponding to the current target region.

[0127] By determining the movement order of multiple target regions and performing movement operations on multiple target regions in the sample image according to the movement order, the movement order of multiple target regions can be determined offline, thereby reducing the time for determining the training sample set and improving the extraction efficiency of the subsequent training sample set.

[0128] Figure 12 The diagram shown is a schematic flowchart of a model training method provided in an embodiment of this application. Figure 12 As shown, the model training method of this application embodiment includes the following steps.

[0129] Step 1210: Perform a movement operation on the target region in the sample image to determine multiple enhanced target regions.

[0130] Specifically, multiple enhanced target areas include the target area after it has been moved.

[0131] Step 1220: Based on the N candidate regions and multiple enhancement target regions corresponding to the sample image, determine the training sample set corresponding to the sample image.

[0132] Specifically, the training sample set corresponding to the sample image includes at least one candidate region among N candidate regions, where N is a positive integer.

[0133] For example, the specific implementation of steps 1210 and 1220 can be found in the above-described embodiment of determining the training sample set, and will not be repeated here.

[0134] Step 1230: Based on the training sample set corresponding to the sample images, train the initial network model to generate the object detection model.

[0135] Specifically, the object detection model is used to detect objects in the image to be detected. The initial network model can be a deep learning model.

[0136] By moving the target region in the sample image, the probability of selecting the region of a small target as a training sample set is increased without affecting the selection of large target regions as much as possible. This improves the detection accuracy of the target detection model trained using the training sample set for small targets.

[0137] Figure 13 The diagram shown is a flowchart illustrating a model training method provided in another embodiment of this application. Figure 12 Extending from the illustrated embodiment Figure 13 The illustrated embodiment will be described in detail below. Figure 13 The illustrated embodiments and Figure 12 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0138] like Figure 13 As shown in the embodiments of this application, the step of training an initial network model based on the training sample set corresponding to the sample image to generate an object detection model includes the following steps.

[0139] Step 1310: Based on the training sample set corresponding to the sample images, train the initial network model to obtain the preliminary target detection model.

[0140] Specifically, an initial network model is trained using the training sample set until the initial network model converges, thus obtaining a preliminary object detection model.

[0141] Step 1320: Based on the M candidate regions, N candidate regions, and target region in the sample image, determine the fine-tuning training sample set corresponding to the sample image.

[0142] Specifically, M is a positive integer, less than N, and the fine-tuning training sample set includes at least one of the M and N candidate regions. The M candidate regions can be non-uniformly distributed, while the N candidate regions can be uniformly distributed.

[0143] In practical applications, multiple candidate regions can be determined by laying candidate boxes. For example, laying M candidate boxes yields M candidate regions. Laying N candidate boxes yields N candidate regions. The laying density of N candidate boxes is greater than the laying density of M candidate boxes.

[0144] For example, new candidate boxes can be generated based on N candidate boxes, M candidate boxes, and candidate box parameters. Specifically, M candidate boxes can be determined from the N candidate boxes, and then the M candidate boxes can be expanded, with each candidate box in the M candidate boxes being expandable into multiple candidate boxes. In practical applications, expanded candidate boxes can be generated by calculating the index values ​​of the N candidate boxes.

[0145] Specifically, for N uniformly distributed candidate boxes, the parameters of the candidate boxes can be set as: anchor_scale = [4], anchor_offset = [0,0], that is, one anchor point generates one candidate box. Since the purpose of this application is to improve the accuracy of small targets, more small candidate boxes can be laid for an anchor point, that is, the candidate box with parameters as: anchor_scale = [1|2], anchor_offset = [[0,0]|[0.5,0.5]] can be used as the parameters of M candidate boxes. That is, the current anchor point will generate 5 candidate boxes, of which the first candidate box is named the regular candidate box (that is, one of the N candidate boxes), which is used to maintain the detection accuracy of targets of various sizes, and the remaining 4 candidate boxes are named the extended candidate boxes (that is, one of the M candidate boxes), which are used to improve the detection accuracy of small targets.

[0146] Step 1330: Based on the M candidate regions corresponding to the sample images, adjust the number of output channels of the preliminary target detection model to obtain the intermediate target detection model.

[0147] For example, by adjusting the number of output channels of the initial target detection model (i.e., pruning the initial target detection model), the model with the adjusted number of output channels is determined as the intermediate target detection model. The pruning ratio of the initial target detection model can be selected according to actual needs.

[0148] Figure 14 The diagram shown illustrates the principle of adjusting the number of output channels in a model provided in one embodiment of this application. Figure 14As shown, the training sample size is 1×480×480×3. After passing through the backbone network of the initial object detection model, the resulting image size is 1×60×60×256. Then, the image with a size of 1×60×60×256, after processing by the regression branch, yields an output channel count of 1×3600×4. The image with a size of 1×60×60×256, after processing by the classification branch, yields an output channel count of 1×3600×15. Adjusting the output channel count of the initial object detection model, the intermediate object detection model's regression branch outputs 1×1000×4 channels, and the classification branch outputs 1×1000×15 channels. Specifically, the regression branch can include convolutional layers and fully connected layers, where convolutional layers are denoted by Conv and fully connected layers by FC. The parameters of the convolutional layer Conv in the regression branch of the initial object detection model can be 256×3×3×256, and the parameters of the fully connected layer FC in the regression branch can be 256×(60×60×4). The parameters of the convolutional layer Conv in the classification branch of the initial object detection model can be 256×3×3×256, and the parameters of the fully connected layer FC in the classification branch can be 256×(60×60×15). The parameters of the convolutional layer Conv in the regression branch of the intermediate object detection model can be 256×3×3×256, and the parameters of the fully connected layer FC in the regression branch can be 256×(1000×4). The parameters of the convolutional layer Conv in the classification branch of the intermediate object detection model can be 256×3×3×256, and the parameters of the fully connected layer FC in the classification branch can be 256×(1000×15).

[0149] Figure 15 The diagram shown is a schematic diagram illustrating the pruning principle of a model provided in an embodiment of this application. Figure 15 As shown, the area above the dashed line represents the standard candidate boxes, which do not require pruning of the initial object detection model. The area below the dashed line represents the expanded candidate boxes, which are used to prune the initial object detection model. Figure 15 As can be seen, some of the expanded candidate boxes (solid black boxes) were cropped.

[0150] Step 1340: Based on the fine-tuned training sample set corresponding to the sample images, train the intermediate target detection model and generate the target detection model.

[0151] In practical applications, directly using M non-uniform candidate regions and target regions to determine training samples leads to high training difficulty and affects training efficiency. Using only N uniformly distributed candidate regions and target regions results in low detection accuracy for small targets. Increasing the number of N uniformly distributed candidate regions (i.e., reducing the area of ​​each candidate region) to improve detection accuracy for small targets also leads to low training efficiency. Therefore, this application first trains an initial network model using a training sample set corresponding to the sample images to obtain a preliminary object detection model. Then, using the idea of ​​model pruning, the number of output channels of the preliminary object detection model is adjusted based on the M candidate regions corresponding to the sample images to obtain an intermediate object detection model. The intermediate object detection model is then trained using a fine-tuned training sample set to generate the final object detection model. This approach improves both the detection accuracy for small targets and the training efficiency of the model.

[0152] Figure 16 The diagram shown is a flowchart illustrating a model training method provided in another embodiment of this application. Figure 13 Extending from the illustrated embodiment Figure 16 The illustrated embodiment will be described in detail below. Figure 16 The illustrated embodiments and Figure 13 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0153] like Figure 16 As shown in the embodiments of this application, before the step of determining the fine-tuning training sample set corresponding to the sample image based on the M candidate regions corresponding to the sample image and the target region in the sample image, the following steps are also included.

[0154] Step 1610: Based on the distribution information of the target region in the sample image, determine the distribution heatmap corresponding to the sample image.

[0155] Specifically, the distribution information of target regions in the sample image is generally that the target regions of small targets are distributed in the edge regions of the sample image, while the target regions of large or medium targets are distributed in the central regions of the sample image.

[0156] Figure 17 The image shown is a heatmap of the distribution corresponding to a sample image provided in an embodiment of this application. The heatmap of the distribution corresponding to the sample image can be obtained by processing the sample image using a heatmap extraction model, or it can be extracted using methods such as grayscale thresholding. Figure 17 As shown, the area enclosed by the white rectangle (i.e., the high-frequency area) can be the target area of ​​a small target.

[0157] Step 1620: Based on the distribution heatmap corresponding to the sample image and the N candidate regions corresponding to the sample image, determine the M candidate regions corresponding to the sample image.

[0158] Specifically, high-frequency regions can be obtained from the distribution heatmap corresponding to the sample image. By laying out N candidate boxes, the candidate boxes at the positions corresponding to the high-frequency regions of the distribution heatmap are determined as M candidate boxes.

[0159] Figure 18 The diagram shows the generation principle of M candidate regions. Figure 18 , Figure 18 The image on the left contains 30 candidate boxes (used to identify 30 candidate regions). Figure 18 The image in the middle is a heatmap of the distribution corresponding to the sample image. Figure 18 The image on the right contains 11 candidate boxes (used to identify 11 candidate regions).

[0160] Based on the distribution information of target regions in the sample images, a distribution heatmap corresponding to the sample images is determined. Then, based on the distribution heatmap corresponding to the sample images and the N candidate regions corresponding to the sample images, M candidate regions corresponding to the sample images are determined. More candidate regions can be expanded from the M candidate regions (i.e., regions where small targets appear more frequently), thereby increasing the probability that regions of small targets are selected as training samples, and thus improving the detection accuracy of the target detection model trained using this training sample set for small targets.

[0161] Figure 19 The diagram shown is a schematic flowchart of a target detection method provided in an embodiment of this application. Figure 19 As shown, the target detection method of this application embodiment includes the following steps.

[0162] Step 1910: Determine the image to be detected.

[0163] Specifically, the image to be detected can be an image containing target regions of their respective sizes.

[0164] Step 1920: Use the object detection model to detect the image to be detected and determine the target region in the image to be detected.

[0165] Specifically, the object detection model is trained based on the model training method described in the above embodiments.

[0166] In the above embodiments, by moving the target region in the sample image, the probability of selecting the region of a small target as a training sample set is increased without significantly affecting the selection of large target regions. This improves the detection accuracy of the target detection model trained using this training sample set for small targets. Therefore, the target detection model trained using the model training method in the above embodiments detects the image to be detected, identifies the target region in the image, and improves the detection accuracy of the target region of small targets.

[0167] The above text combined Figures 3 to 19 The method embodiments of this application are described in detail below, in conjunction with... Figures 20 to 24 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0168] Exemplary device

[0169] Figure 20 The diagram shown is a structural schematic of a training sample set determination device provided in an embodiment of this application. Figure 20 As shown, the training sample set determination device 2000 of this application embodiment includes: a moving module 2010 and a sample determination module 2020.

[0170] Specifically, the moving module 2010 is configured to perform a moving operation on the target region in the sample image to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region. The sample determination module 2020 is configured to determine a training sample set corresponding to the sample image based on multiple candidate regions and multiple enhanced target regions corresponding to the sample image, and train a network model based on the training sample set corresponding to the sample image to obtain a target detection model, wherein the training sample set corresponding to the sample image includes at least one candidate region among the multiple candidate regions corresponding to the sample image.

[0171] In one embodiment of this application, the moving module 2010 is further configured to perform a moving operation on the target region in the sample image based on a preset path to determine multiple enhanced target regions.

[0172] In one embodiment of this application, the moving module 2010 is further configured to perform a translation operation on the target region in the sample image based on a preset path to determine multiple enhanced target regions; and / or perform a rotation operation on the target region in the sample image based on a preset path to determine multiple enhanced target regions.

[0173] In one embodiment of this application, the sample determination module 2020 is further configured to: calculate the intersection-union ratio (IUU) of multiple candidate regions with the current enhanced target region for each of the multiple enhanced target regions, and determine the overlap degree corresponding to each of the multiple candidate regions; determine the training sample set corresponding to the current enhanced target region based on the overlap degree and overlap threshold corresponding to each of the multiple candidate regions; and determine the training sample set corresponding to the sample image based on the training sample set corresponding to each of the multiple enhanced target regions.

[0174] In one embodiment of this application, the overlap threshold includes a fixed overlap threshold and a dynamic overlap threshold. The sample determination module 2020 is further configured to: determine a first training sample set corresponding to the current enhancement target region based on the overlap corresponding to each of the multiple candidate regions and the fixed overlap threshold; the first training sample set includes at least one candidate region among the multiple candidate regions; determine the overlap corresponding to the current enhancement target region based on the overlap corresponding to each of the multiple candidate regions, and determine the mean and standard deviation of the overlap based on the overlap corresponding to each of the multiple enhancement target regions; determine the stability coefficient of the current enhancement target region based on the first training sample set corresponding to each of the multiple enhancement target regions; determine the dynamic overlap threshold of the current enhancement target region based on the mean, standard deviation, and stability coefficient of the current enhancement target region; determine a second training sample set corresponding to the current enhancement target region based on the overlap corresponding to each of the multiple candidate regions and the dynamic overlap threshold of the current enhancement target region; and determine the training sample set corresponding to the current enhancement target region based on the first and second training sample sets.

[0175] Figure 21 The diagram shown is a structural schematic of a training sample set determination device provided in another embodiment of this application. Figure 20 Extending from the illustrated embodiment Figure 21 The illustrated embodiment will be described in detail below. Figure 21 The illustrated embodiments and Figure 20 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0176] like Figure 21 As shown, the training sample set determination device 2000 of this application embodiment further includes: a pre-movement module 2030, an overlap degree calculation module 2040, and a movement order determination module 2050.

[0177] Specifically, the pre-movement module 2030 is configured to pre-move each current target region among multiple target regions in the sample image, determining multiple pre-enhanced target regions corresponding to the current target region. These multiple pre-enhanced target regions include the current target region and its corresponding pre-moved target region. The overlap calculation module 2040 is configured to determine the overlap degree of the current target region based on multiple candidate regions corresponding to the sample image and the multiple pre-enhanced target regions corresponding to the current target region. The movement order determination module 2050 is configured to determine the movement order of the multiple target regions based on their respective overlap degrees, so as to perform movement operations on the multiple target regions in the sample image according to the movement order.

[0178] In one embodiment of this application, the overlap calculation module 2040 is further configured to: for each of the multiple pre-enhanced target regions corresponding to the current target region, calculate the sum of the intersection-union ratios (IUR) of the current pre-enhanced target region with the multiple candidate regions, and calculate the sum of the center distances of the current pre-enhanced target region with the multiple candidate regions; determine the sum of the IUR of the current target region based on the sum of the IUR of the multiple pre-enhanced target regions, and determine the sum of the center distances of the current target region based on the sum of the center distances of the multiple pre-enhanced target regions; and determine the overlap degree of the current target region based on the ratio of the sum of the IUR of the current target region to the sum of the center distances of the current target region.

[0179] Figure 22 The diagram shown is a structural schematic of a model training device provided in an embodiment of this application. Figure 22 As shown, the model training device 2200 of this application embodiment includes: a moving module 2210, a sample determination module 2220 and a training module 2230.

[0180] Specifically, the moving module 2210 is configured to perform a moving operation on the target region in the sample image to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region. The sample determination module 2220 is configured to determine a training sample set corresponding to the sample image based on N candidate regions and multiple enhanced target regions corresponding to the sample image, wherein the training sample set corresponding to the sample image includes at least one candidate region from the N candidate regions, where N is a positive integer. The training module 2230 is configured to train an initial network model based on the training sample set corresponding to the sample image to generate a target detection model, wherein the target detection model is used to detect targets in the image to be detected.

[0181] In one embodiment of this application, the training module 2230 is further configured to: train an initial network model based on the training sample set corresponding to the sample image to obtain a preliminary object detection model; determine a fine-tuning training sample set corresponding to the sample image based on M candidate regions, N candidate regions, and the target region in the sample image, wherein M is a positive integer, M is less than N, and the fine-tuning training sample set includes at least one candidate region among the M candidate regions and the N candidate regions; adjust the number of output channels of the preliminary object detection model based on the M candidate regions corresponding to the sample image to obtain an intermediate object detection model; and train the intermediate object detection model based on the fine-tuning training sample set corresponding to the sample image to generate an object detection model.

[0182] Figure 23 The diagram shown is a structural schematic of a model training device provided in another embodiment of this application. Figure 22 Extending from the illustrated embodiment Figure 23 The illustrated embodiment will be described in detail below. Figure 23 The illustrated embodiments and Figure 22 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0183] like Figure 23 As shown, the model training device 2200 in this embodiment of the application further includes a heatmap determination module 2240 and a candidate region determination module 2250.

[0184] Specifically, the heatmap determination module 2240 is configured to determine the distribution heatmap corresponding to the sample image based on the distribution information of the target region in the sample image. The candidate region determination module 2250 is configured to determine M candidate regions corresponding to the sample image based on the distribution heatmap corresponding to the sample image and N candidate regions corresponding to the sample image.

[0185] Figure 24 The diagram shown is a structural schematic of a target detection device provided in an embodiment of this application. Figure 24 As shown, the target detection device 2400 of this application embodiment includes: an image determination module 2410 and a detection module 2420.

[0186] Specifically, the image determination module 2410 is configured to determine the image to be detected. The detection module 2420 is configured to use a target detection model to detect the image to be detected and determine the target region in the image to be detected, wherein the target detection model is trained based on the model training method in the above embodiments.

[0187] Figures 20 to 24The operation and functions of the movement module 2010, sample determination module 2020, pre-movement module 2030, overlap degree calculation module 2040, and movement order determination module 2050 in the provided training sample set determination device 2000, as well as the movement module 2210, sample determination module 2220, training module 2230, heatmap determination module 2240, and candidate region determination module 2250 in the model training device 2200, and the image determination module 2410 and detection module 2420 in the target detection device 2400, can be referred to the above. Figures 3 to 19 The methods provided will not be repeated here to avoid duplication.

[0188] Exemplary electronic devices

[0189] Figure 25 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 25 As shown, the electronic device 2500 includes: one or more processors 2501 and a memory 2502; and computer program instructions stored in the memory 2502, which, when executed by the processor 2501, cause the processor 2501 to perform the method as described in any of the above embodiments.

[0190] The processor 2501 may be a central processing unit (CPU) or other form of processing unit with data transfer capability and / or instruction execution capability, and may control other components in the electronic device to perform the desired function.

[0191] The memory 2502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 2501 may execute the program instructions to implement the steps in the methods of the various embodiments of this application described above, and / or other desired functions.

[0192] In one example, electronic device 2500 may further include: input device 2503 and output device 2504, these components being connected via a bus system and / or other forms of connection mechanism. Figure 25 Interconnection (not shown in the image).

[0193] In addition, the input device 2503 may also include, for example, a keyboard, a mouse, a microphone, etc.

[0194] The output device 2504 can output various information to the outside. The output device 2504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0195] Of course, for the sake of simplicity, Figure 25 Only some of the components of the electronic device 2500 relevant to this application are shown in this illustration, omitting components such as buses, input devices / output interfaces, etc. In addition, the electronic device 2500 may include any other suitable components depending on the specific application.

[0196] Exemplary computer-readable storage media

[0197] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods of any of the above embodiments.

[0198] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0199] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0200] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0201] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0202] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0203] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0204] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0205] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0206] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a training sample set, characterized in that, include: A target region in a sample image is moved to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; Based on the multiple candidate regions and the multiple enhanced target regions corresponding to the sample image, a training sample set corresponding to the sample image is determined, and a network model is trained based on the training sample set corresponding to the sample image to obtain a target detection model. The training sample set corresponding to the sample image includes at least one candidate region among the multiple candidate regions corresponding to the sample image. The step of determining the training sample set corresponding to the sample image based on multiple candidate regions and multiple enhancement target regions corresponding to the sample image includes: For each of the plurality of enhanced target regions, Calculate the intersection-union ratio (IUU) of the multiple candidate regions with the current enhanced target region, and determine the overlap degree of each of the multiple candidate regions. Based on the overlap degree and overlap threshold of each of the multiple candidate regions, the training sample set corresponding to the current enhancement target region is determined; Based on the training sample sets corresponding to each of the multiple enhanced target regions, the training sample set corresponding to the sample image is determined; The overlap threshold includes a fixed overlap threshold and a dynamic overlap threshold. Determining the training sample set corresponding to the current enhancement target region based on the overlap and overlap threshold corresponding to each of the multiple candidate regions includes: Based on the overlap degree and fixed overlap threshold corresponding to each of the multiple candidate regions, a first training sample set corresponding to the current enhanced target region is determined, wherein the first training sample set includes at least one candidate region among the multiple candidate regions; Based on the overlap of each of the multiple candidate regions, the overlap of the current enhancement target region is determined, and based on the overlap of each of the multiple enhancement target regions, the mean and standard deviation of the overlap are determined. Based on the first training sample set corresponding to each of the plurality of augmented target regions, the stability coefficient of the current augmented target region is determined; Based on the mean of the overlap, the standard deviation of the overlap, and the stability coefficient of the current enhanced target region, a dynamic overlap threshold for the current enhanced target region is determined. Based on the overlap degree of each of the multiple candidate regions and the dynamic overlap degree threshold of the current enhanced target region, a second training sample set corresponding to the current enhanced target region is determined. Based on the first training sample set and the second training sample set, the training sample set corresponding to the current enhanced target region is determined.

2. The method for determining the training sample set according to claim 1, characterized in that, The process of moving the target region in the sample image to determine multiple enhanced target regions includes: Based on a preset path, a movement operation is performed on the target region in the sample image to determine the multiple enhanced target regions.

3. The method for determining the training sample set according to claim 1, characterized in that, Before performing the movement operation on the target region in the sample image to determine multiple enhanced target regions, the method further includes: For each current target region among multiple target regions in the sample image, The current target area is pre-moved to determine multiple pre-enhanced target areas corresponding to the current target area, wherein the multiple pre-enhanced target areas include the current target area and the pre-moved target area corresponding to the current target area; Based on the multiple candidate regions corresponding to the sample image and the multiple pre-enhanced target regions corresponding to the current target region, the degree of overlap corresponding to the current target region is determined. Based on the degree of overlap of the multiple target regions, the movement order of the multiple target regions is determined, so that the multiple target regions in the sample image are moved in accordance with the movement order.

4. A model training method, characterized in that, include: A target region in a sample image is moved to determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; Based on the N candidate regions corresponding to the sample image and the plurality of enhancement target regions, a training sample set corresponding to the sample image is determined, wherein the training sample set corresponding to the sample image includes at least one candidate region among the N candidate regions, and N is a positive integer; Based on the training sample set corresponding to the sample image, an initial network model is trained to generate a target detection model, wherein the target detection model is used to detect targets in the image to be detected; The step of determining the training sample set corresponding to the sample image based on the N candidate regions and the plurality of enhancement target regions corresponding to the sample image includes: For each of the plurality of enhanced target regions, Calculate the intersection-union ratio (IUU) of each of the N candidate regions with the current enhanced target region, and determine the overlap degree of each of the N candidate regions; Based on the overlap degree and overlap threshold of each of the N candidate regions, the training sample set corresponding to the current enhancement target region is determined; Based on the training sample sets corresponding to each of the multiple enhanced target regions, the training sample set corresponding to the sample image is determined; The overlap threshold includes a fixed overlap threshold and a dynamic overlap threshold. Determining the training sample set corresponding to the current enhancement target region based on the overlap and overlap threshold of each of the N candidate regions includes: Based on the overlap degree and fixed overlap threshold of each of the N candidate regions, a first training sample set corresponding to the current enhanced target region is determined. The first training sample set includes at least one candidate region among the N candidate regions. Based on the overlap of each of the N candidate regions, the overlap of the current enhanced target region is determined, and based on the overlap of each of the multiple enhanced target regions, the mean and standard deviation of the overlap are determined. Based on the first training sample set corresponding to each of the plurality of augmented target regions, the stability coefficient of the current augmented target region is determined; Based on the mean of the overlap, the standard deviation of the overlap, and the stability coefficient of the current enhanced target region, a dynamic overlap threshold for the current enhanced target region is determined. Based on the overlap of each of the N candidate regions and the dynamic overlap threshold of the current enhanced target region, a second training sample set corresponding to the current enhanced target region is determined. Based on the first training sample set and the second training sample set, the training sample set corresponding to the current enhanced target region is determined.

5. A target detection method, characterized in that, include: Identify the image to be detected; The target detection model is used to detect the image to be detected and to determine the target region in the image to be detected, wherein the target detection model is trained based on the model training method described in claim 4.

6. A training sample set determination device, characterized in that, include: The moving module is configured to perform a moving operation on a target region in a sample image and determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; The sample determination module is configured to determine a training sample set corresponding to the sample image based on multiple candidate regions and multiple enhanced target regions corresponding to the sample image, and to train a network model based on the training sample set corresponding to the sample image to obtain a target detection model, wherein the training sample set corresponding to the sample image includes at least one candidate region among the multiple candidate regions corresponding to the sample image. The step of determining the training sample set corresponding to the sample image based on multiple candidate regions and multiple enhancement target regions corresponding to the sample image includes: For each of the plurality of enhanced target regions, Calculate the intersection-union ratio (IUU) of the multiple candidate regions with the current enhanced target region, and determine the overlap degree of each of the multiple candidate regions. Based on the overlap degree and overlap threshold of each of the multiple candidate regions, the training sample set corresponding to the current enhancement target region is determined; Based on the training sample sets corresponding to each of the multiple enhanced target regions, the training sample set corresponding to the sample image is determined; The overlap threshold includes a fixed overlap threshold and a dynamic overlap threshold. Determining the training sample set corresponding to the current enhancement target region based on the overlap and overlap threshold corresponding to each of the multiple candidate regions includes: Based on the overlap degree and fixed overlap threshold corresponding to each of the multiple candidate regions, a first training sample set corresponding to the current enhanced target region is determined, wherein the first training sample set includes at least one candidate region among the multiple candidate regions; Based on the overlap of each of the multiple candidate regions, the overlap of the current enhancement target region is determined, and based on the overlap of each of the multiple enhancement target regions, the mean and standard deviation of the overlap are determined. Based on the first training sample set corresponding to each of the plurality of augmented target regions, the stability coefficient of the current augmented target region is determined; Based on the mean of the overlap, the standard deviation of the overlap, and the stability coefficient of the current enhanced target region, a dynamic overlap threshold for the current enhanced target region is determined. Based on the overlap degree of each of the multiple candidate regions and the dynamic overlap degree threshold of the current enhanced target region, a second training sample set corresponding to the current enhanced target region is determined. Based on the first training sample set and the second training sample set, the training sample set corresponding to the current enhanced target region is determined.

7. A model training device, characterized in that, include: The moving module is configured to perform a moving operation on a target region in a sample image and determine multiple enhanced target regions, wherein the multiple enhanced target regions include the moved target region; The sample determination module is configured to determine the training sample set corresponding to the sample image based on N candidate regions corresponding to the sample image and the plurality of enhancement target regions, wherein the training sample set corresponding to the sample image includes at least one candidate region among the N candidate regions, and N is a positive integer; The training module is configured to train an initial network model based on the training sample set corresponding to the sample image to generate a target detection model, wherein the target detection model is used to detect targets in the image to be detected; The step of determining the training sample set corresponding to the sample image based on the N candidate regions and the plurality of enhancement target regions corresponding to the sample image includes: For each of the plurality of enhanced target regions, Calculate the intersection-union ratio (IUU) of each of the N candidate regions with the current enhanced target region, and determine the overlap degree of each of the N candidate regions; Based on the overlap degree and overlap threshold of each of the N candidate regions, the training sample set corresponding to the current enhancement target region is determined; Based on the training sample sets corresponding to each of the multiple enhanced target regions, the training sample set corresponding to the sample image is determined; The overlap threshold includes a fixed overlap threshold and a dynamic overlap threshold. Determining the training sample set corresponding to the current enhancement target region based on the overlap and overlap threshold of each of the N candidate regions includes: Based on the overlap degree and fixed overlap threshold of each of the N candidate regions, a first training sample set corresponding to the current enhanced target region is determined. The first training sample set includes at least one candidate region among the N candidate regions. Based on the overlap of each of the N candidate regions, the overlap of the current enhanced target region is determined, and based on the overlap of each of the multiple enhanced target regions, the mean and standard deviation of the overlap are determined. Based on the first training sample set corresponding to each of the plurality of augmented target regions, the stability coefficient of the current augmented target region is determined; Based on the mean of the overlap, the standard deviation of the overlap, and the stability coefficient of the current enhanced target region, a dynamic overlap threshold for the current enhanced target region is determined. Based on the overlap of each of the N candidate regions and the dynamic overlap threshold of the current enhanced target region, a second training sample set corresponding to the current enhanced target region is determined. Based on the first training sample set and the second training sample set, the training sample set corresponding to the current enhanced target region is determined.

8. A target detection device, characterized in that, include: The image determination module is configured to determine the image to be detected; The detection module is configured to use a target detection model to detect the image to be detected and determine the target region in the image to be detected, wherein the target detection model is trained based on the model training method described in claim 4.

Images