Training method and device of target detection network, electronic equipment and storage medium

Abstract

The application discloses a target detection network training method and device, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. In the method, each image sample in a global sample set is segmented to obtain a plurality of image blocks, and each image block containing a target is labeled to obtain a local sample set by using the label of each image sample. Any label contains category information and position information of the target. First, each image block in the local sample set and a descriptive sentence of the category information corresponding to each image block are taken as input, and the label of each image block is taken as output, and the target detection network is trained for multiple rounds. When the number of training rounds reaches a first set value, each image sample in the global sample set and a descriptive sentence of the category information corresponding to each image sample are taken as input, and the label of each image sample is taken as output, and the target detection network is trained for multiple rounds until the number of training rounds reaches a second set value. The considered target information is more abundant, and the detection accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a target detection network training method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, target detection tasks in many scenarios are performed by using target detection networks. However, the target detection networks in the related art identify and locate targets by using features such as colors, shapes, and textures of objects in images. This way of detecting targets is still one-sided, and therefore the accuracy needs to be improved. SUMMARY

[0003] Embodiments of the present application provide a target detection network training method and device, an electronic device, and a storage medium to improve the detection accuracy of a target detection network.

[0004] In a first aspect, the embodiments of the present application provide a target detection network training method, comprising:

[0005] segmenting each image sample in the obtained global sample set to obtain a plurality of image blocks;

[0006] adding a label to each image block containing a target by using the label of each image sample to obtain a local sample set, and any label contains category information and position information of the target;

[0007] performing multi-round training on the target detection network by taking each image block in the local sample set and a descriptive sentence of the category information corresponding to each image block as input and taking the label of each image block as output;

[0008] when the number of training rounds reaches a first set value, continuing to perform multi-round training on the target detection network by taking each image sample in the global sample set and a descriptive sentence of the category information corresponding to each image sample as input and taking the label of each image sample as output, until the number of training rounds reaches a second set value, and stopping training.

[0009] In some embodiments, the target detection network includes an image encoder, a text encoder, and an attention mechanism-based fusion layer, and the multi-round training on the target detection network by taking each image block in the local sample set and a descriptive sentence of the category information corresponding to each image block as input and taking the label of each image block as output includes:

[0010] performing one round of training on the target detection network according to the following steps in the multi-round training:

[0011] inputting each image block in the local sample set into the image encoder to obtain image features of the image block;

[0012] Inputting a descriptive sentence of the category information corresponding to the image block into the text encoder to obtain text features of the image block;

[0013] fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image block;

[0014] Performing target detection based on the fusion features to obtain a target detection result of the image block;

[0015] Based on the object detection results and labels of each image block, the network parameters of the object detection network are adjusted.

[0016] In some embodiments, the target detected by the target detection network has at least two categories, and adjusting the network parameters of the target detection network based on the target detection results and labels of each image block includes:

[0017] Determining a detection error for each image block based on the object detection result and the label of the image block;

[0018] According to the rule that the error weight corresponding to the image block containing one type of target is greater than the error weight corresponding to the image block containing at least two types of targets, the detection errors of each image block are weightedly summed;

[0019] According to the weighted summation results, the network parameters of the target detection network are adjusted.

[0020] In some embodiments, the object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism. The object detection network is trained for multiple rounds using each image sample in the global sample set and a descriptive statement of category information corresponding to each image sample as input and a label of each image sample as output, including:

[0021] Perform one of the multiple rounds of training on the object detection network according to the following steps:

[0022] Inputting each image sample in the global sample set into the image encoder to obtain image features of the image sample;

[0023] Inputting a descriptive sentence of the category information corresponding to the image sample into the text encoder to obtain text features of the image sample;

[0024] fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image sample;

[0025] Performing target detection based on the fusion features to obtain a target detection result of the image sample;

[0026] Based on the object detection results and labels of each image, the network parameters of the object detection network are adjusted.

[0027] In some embodiments, after stopping the training, the method further includes:

[0028] Inputting each acquired test image and test target category descriptive statement into the target detection network to obtain the target detection result of the test image;

[0029] Determining whether the target detection of the test image is correct based on the target detection result and the test label of the test image;

[0030] Based on the detection accuracy of each test image, determine whether to continue training the object detection network.

[0031] In some embodiments, it further includes:

[0032] The obtained image to be detected and the category descriptive statement of the target to be detected are input into the target detection network finally trained to obtain the target detection result of the image to be detected.

[0033] In a second aspect, an embodiment of the present application provides a training device for an object detection network, comprising:

[0034] An acquisition module, configured to segment each image sample in the acquired global sample set to obtain a plurality of image blocks;

[0035] An adding module is used to add labels to each image block containing the target using the labels of each image sample to obtain a local sample set, where any label contains the category information and location information of the target;

[0036] A first training module is configured to perform multiple rounds of training on the object detection network using the image blocks in the local sample set and the descriptive statements of the category information corresponding to the image blocks as input and the labels of the image blocks as output;

[0037] The second training module is used to, when the number of training rounds reaches a first set value, continue to train the target detection network for multiple rounds with the descriptive statements of each image sample in the global sample set and the category information corresponding to each image sample as input and the label of each image sample as output, until the number of training rounds reaches a second set value, and then stop training.

[0038] In some embodiments, the object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism, and the first training module is specifically configured to perform one round of the multiple rounds of training on the object detection network according to the following steps:

[0039] Inputting each image block in the local sample set into the image encoder to obtain image features of the image block;

[0040] Inputting a descriptive sentence of the category information corresponding to the image block into the text encoder to obtain text features of the image block;

[0041] fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image block;

[0042] Performing target detection based on the fusion features to obtain a target detection result of the image block;

[0043] Based on the object detection results and labels of each image block, the network parameters of the object detection network are adjusted.

[0044] In some embodiments, the target detection network has at least two categories of detection targets, and the first training module is specifically used to:

[0045] Determining a detection error for each image block based on the object detection result and the label of the image block;

[0046] According to the rule that the error weight corresponding to the image block containing one type of target is greater than the error weight corresponding to the image block containing at least two types of targets, the detection errors of each image block are weightedly summed;

[0047] According to the weighted summation results, the network parameters of the target detection network are adjusted.

[0048] In some embodiments, the object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism, and the second training module is specifically configured to perform one round of the multiple rounds of training on the object detection network according to the following steps:

[0049] Inputting each image sample in the global sample set into the image encoder to obtain image features of the image sample;

[0050] Inputting a descriptive sentence of the category information corresponding to the image sample into the text encoder to obtain text features of the image sample;

[0051] fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image sample;

[0052] Performing target detection based on the fusion features to obtain a target detection result of the image sample;

[0053] Based on the object detection results and labels of each image, the network parameters of the object detection network are adjusted.

[0054] In some embodiments, a testing module is further included for:

[0055] After stopping the training, each test image and the category descriptive statement of the test target are input into the target detection network to obtain the target detection result of the test image;

[0056] Determining whether the target detection of the test image is correct based on the target detection result and the test label of the test image;

[0057] Based on the detection accuracy of each test image, determine whether to continue training the object detection network.

[0058] In some embodiments, a detection module is further included for:

[0059] The obtained image to be detected and the category descriptive statement of the target to be detected are input into the target detection network finally trained to obtain the target detection result of the image to be detected.

[0060] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0061] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform any of the above-mentioned training methods for the target detection network.

[0062] In a fourth aspect, an embodiment of the present application provides a storage medium. When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned training methods for the target detection network.

[0063] In an embodiment of the present application, each image sample in the acquired global sample set is segmented to obtain multiple image blocks, and the labels of each image sample are used to add labels to each image block containing the target to obtain a local sample set, wherein any label contains the category information and location information of the target. Afterwards, the target detection network is trained for multiple rounds using the descriptive statements of the category information corresponding to each image block in the local sample set as input and the labels of each image block as output. When the number of training rounds reaches a first set value, the target detection network is trained for multiple rounds using the descriptive statements of the category information corresponding to each image sample in the global sample set as input and the labels of each image sample as output. The training of the target detection network is stopped until the number of training rounds reaches a second set value. In this way, image blocks are used for training in the early stage to capture the local features of the target, and image samples are used for training in the later stage to focus on the overall features of the target. This not only helps to improve the detection accuracy of the target detection network, but also can be better applied to target detection tasks in different scenarios. In addition, during the training process, the target category descriptive statement is additionally used as an input, which considers richer target information and helps improve the detection accuracy of the target detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0065] Figure 1 A schematic diagram of the structure of a target detection network provided in an embodiment of the present application;

[0066] Figure 2 A schematic diagram of the structure of a target detection network provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of the training process of an object detection network provided in an embodiment of the present application;

[0068] Figure 4 A flowchart of performing a round of training on a local sample set provided in an embodiment of the present application;

[0069] Figure 5 A flowchart of performing a round of training on a global sample set provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the structure of a training device for an object detection network provided in an embodiment of the present application;

[0071] Figure 7A schematic diagram of the hardware structure of an electronic device for implementing a training method for a target detection network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to improve the detection accuracy of a target detection network, embodiments of the present application provide a method, apparatus, electronic device, and storage medium for training a target detection network.

[0073] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0074] First of all, it should be noted that the solution of the embodiment of the present application is applicable to all scenarios that require the use of a target detection network to detect targets, such as using a target detection network to detect pedestrians, vehicles, faces, license plates, production dates, etc. in images.

[0075] The target detection network in the related art only extracts image features to determine the location and category of the target, and considers relatively little information, so the accuracy needs to be improved. To this end, this solution introduces a multimodal detection method. On the basis of a one-stage target detection network (that is, detecting the category and location of the target at the same time), a text prompt project is added. The text prompt project is used to obtain text information of the target category in the image, and the text information is combined with the image information itself to better learn the characteristics of the target, so as to achieve the purpose of more accurate target detection. Considering that an image may contain multiple targets, although the text prompt project can describe the text information of multiple targets at the same time, it is difficult to connect with the targets in the image. Therefore, this solution divides the image into multiple image blocks (patches). In the first half of the training process, the image blocks are used for training. This can reduce the number of target types contained in each image block, making the matching of image information and text information more accurate.

[0076] The following first introduces the structure of the target detection network in the embodiment of the present application.

[0077] See also Figure 1 The target detection network may include an image encoder, a text encoder, and a fusion layer based on an attention mechanism. The text encoder and the image encoder are pre-learned through comparative learning based on a large number of images and descriptions of target categories in the images. The text encoder is already able to extract the semantic information of the image well, and the image encoder is also able to extract the feature information of the image well. When training the target detection network in the embodiment of the present application, the parameters of the two pre-trained large models of the text encoder and the image encoder are fine-tuned, and the parameters of the fusion layer based on the attention mechanism are adjusted.

[0078] Next, the training process of the target detection network is introduced.

[0079] See also Figure 2 , Figure 2 The training process diagram of a target detection network provided in the embodiment of the present application mainly includes the first half training process, the second half training process and the test process. Figure 2 These three training processes are introduced separately.

[0080] 1. The first half of the training process.

[0081] 1. Divide the image samples in the global sample set into multiple image blocks: i0, i1…i n , n≥4.

[0082] In practical applications, the number of image patches n into which each image sample is divided can be determined based on the target number x of patches contained in an image sample in the global sample set. For example, if x is an odd number, then n = x-1; if x is an even number, then n = x. Each image sample is then randomly divided into n patches: m+1 rows and k+1 columns, where n = (m+1)*(k+1).

[0083] For example, based on the number of targets contained in all image samples, it is determined that the average number of targets contained in an image sample is x=5, then n=x-1=4, and each image sample can be cut into 4 image blocks with 2 rows and 2 columns.

[0084] 2. Using the labels of the image samples, add labels to the image blocks containing the target. The labeled image blocks form a local sample set, where any label contains the category information and location information of the target.

[0085] Assume that the horizontal coordinate list of the cutting points of the image sample is: [x c0 ,x c1 …x ck ], the vertical coordinate list is: [y c0 ,y c1 …y cm ], and assuming that the image sample contains an object and the coordinate box of the object is: (x1, y1), (x2, y2), then:

[0086] The coordinates of the upper left corner of the image block containing the target after the image sample is cut are: [x c0 +x c1 …+x ci ,y c0 +y c1 …+y cj ],

[0087] The coordinates of the target in the image block are:

[0088]

[0089]

[0090] Assume that an image sample is 500 pixels wide and 400 pixels high and is cut into 4 image blocks, 2 rows and 2 columns. Then, the horizontal coordinate of the cut point of the image sample is 300 and the vertical coordinate is 200. Also, assume that the original position of the target in the image sample is [450, 300], then the position of the target in the image block is [450-300, 300-200]. This position and the target's category, mirror, can be added to the label of the image block to obtain a new training sample.

[0091] 3. Use the local sample set to perform multiple rounds of training on the target detection network, such as 100 rounds of training.

[0092] First, for the image blocks in the local sample set, the category information in the label of the image block can be hinted, that is, the category information in the label of the image block can be expanded into a category descriptive statement.

[0093] For example, expand the target category in the label into a description statement: The pixels of CLASS_NAME.

[0094] Then, perform a round of training as follows:

[0095] In the first step, each image block in the local sample set passes through the image encoder to obtain the image features of the image block.

[0096] In the second step, the category descriptive sentence corresponding to the image block is input into the text encoder to obtain the text features of the image block.

[0097] The third step is to fuse the image features and text features through a fusion layer based on the attention mechanism to obtain the fusion features of the image block.

[0098] Generally, Among them, d k is a constant.

[0099] When text features and image features are fused through the fusion layer based on the attention mechanism, the image feature F img As Q in the attention mechanism, the text feature F text As K in the attention mechanism, the text feature and the image feature are multiplied to obtain the relationship feature F text-img-realtion (F text *F img T), the text features are obtained through a layer of convolution to obtain V in the attention mechanism. That is, in the embodiment of the present application, the image features and text features are fused using the following formula:

[0100]

[0101] The fourth step is to perform target detection based on the fusion features and obtain the target detection results of the image block.

[0102] Step 5: Based on the target detection result of the image block and the label of the image block, the detection error of the image block is determined.

[0103] Step 6: Based on the detection error of each image block, adjust the network parameters of the target detection network.

[0104] Considering that an image block may contain one or more types of targets, and that the sample quality of an image block containing one type of target is better than that of an image block containing at least two types of targets, in order to highlight the importance of image blocks containing one type of target, the detection errors of each image block can be weighted and summed according to the rule that the error weight corresponding to an image block containing one type of target is greater than the error weight corresponding to an image block containing at least two types of targets. The network parameters of the target detection network can then be adjusted based on the weighted summation result. In this way, the target detection network can pay more attention to these high-quality image blocks containing only one type of target, learn the local features of the target as quickly as possible, and improve the training speed.

[0105] When the number of training rounds reaches 100, the second half of the training process can be entered.

[0106] 2. The second half of the training process uses the global sample set to perform multiple rounds of training on the target detection network, such as the 101st to 300th rounds of training.

[0107] First, for the image samples in the global sample set, the category information in the labels of the image samples can be hinted, that is, the category information in the labels of the image samples can be expanded into category descriptive sentences.

[0108] For example, expand the target category in the label into a description statement: The pixels of CLASS_NAME.

[0109] Then, perform a round of training as follows:

[0110] In the first step, each image sample in the global sample set passes through the image encoder to obtain the image features of the image sample.

[0111] In the second step, the category descriptive sentence corresponding to the image sample is input into the text encoder to obtain the text features of the image sample.

[0112] The third step is to fuse the image features and text features through a fusion layer based on the attention mechanism to obtain the fusion features of the image sample.

[0113] Generally, Among them, d k is a constant.

[0114] When text features and image features are fused through the fusion layer based on the attention mechanism, the image feature F img As Q in the attention mechanism, the text feature F text As K in the attention mechanism, the text feature and the image feature are multiplied to obtain the relationship feature F text-img-realtion (F text *F img T ), the text features are obtained through a layer of convolution to obtain V in the attention mechanism. That is, in the embodiment of the present application, the image features and text features are fused using the following formula:

[0115]

[0116] The fourth step is to perform target detection based on the fusion features to obtain the target detection results of the image samples.

[0117] Step 5: Based on the target detection results of the image samples and the labels of the image samples, the detection errors of the image samples are determined.

[0118] Step 6: Based on the detection error of each image sample, adjust the network parameters of the target detection network.

[0119] For example, the detection errors of each image sample are summed up, and then the network parameters of the target detection network are adjusted according to the summation result.

[0120] When the number of training rounds reaches 300, the test process can be started.

[0121] 3. Testing process

[0122] Each test image and the category description of the target to be tested are input into the trained target detection network to obtain the target detection result of the test image. Based on this target detection result and the test label of the test image, it is determined whether the target detection of the test image is correct. After that, the detection accuracy rate for all test images can be calculated. Based on the detection accuracy rate, it is determined whether to continue training the target detection network. For example, if the detection accuracy rate reaches the set accuracy rate, the training can be terminated. If the detection accuracy rate does not reach the set accuracy rate, the training can be continued and the second half of the training process can be restarted. In addition, new image samples can be added. In short, any method that can quickly improve the accuracy of the target detection network can be adopted.

[0123] Finally, after the object detection network is trained, it can be used to perform object detection. Specifically, the image to be detected and the category description statement of the object to be detected are input into the trained object detection network to obtain the object detection result of the image to be detected.

[0124] The training method of the target detection network proposed in this application is explained below with reference to a flowchart.

[0125] Figure 3 A flowchart of a method for training a target detection network provided in an embodiment of the present application, the method comprising the following steps.

[0126] In step 301, each image sample in the acquired global sample set is segmented to obtain a plurality of image blocks.

[0127] Among them, the global sample set contains multiple image samples, each image sample has a label, and the label contains the category information and location information of the target.

[0128] For example, if the number of image blocks into which an image sample is divided is n, and the average number of target objects contained in the image sample is x, then when x is an odd number, n = x – 1, and when x is an even number, n = x. Then, each image sample can be randomly divided into n patches: m + 1 rows and k + 1 columns, where n = (m + 1) * (k + 1).

[0129] In this way, determining the number of image blocks based on the average number of targets contained in the image samples can increase the probability that a segmented image block contains only one target, thereby making the subsequent matching of image features and text features more accurate.

[0130] In step 302, labels of the image samples are used to add labels to the image blocks containing the target to obtain a local sample set, where any label contains the category information and location information of the target.

[0131] After splitting an image sample into multiple image blocks, labels can be added to these blocks based on the label of the image sample. Specifically, the position information in the label of the image sample is updated according to the split position to obtain a new position. This new position and the target category are used as the label of the corresponding image block.

[0132] Considering that after the image samples are segmented, some image blocks may not contain the target, such image blocks will not be labeled and therefore do not need to participate in training. The labeled image blocks constitute the local sample set.

[0133] In step 303, the object detection network is trained for multiple rounds using the image blocks in the local sample set and the descriptive statements of the category information corresponding to the image blocks as input and the labels of the image blocks as output.

[0134] refer to Figure 1 , the target detection network in the embodiment of the present application includes an image encoder, a text encoder and a fusion layer based on the attention mechanism. Figure 4 The process shown here trains the object detection network in one of several rounds of training. The process includes the following steps:

[0135] In step 3031, each image block in the local sample set is input into an image encoder to obtain image features of the image block.

[0136] In step 3032, the descriptive sentence of the category information corresponding to the image block is input into a text encoder to obtain the text features of the image block.

[0137] Among them, the descriptive statement of the category information corresponding to the image block is obtained through the prompt project.

[0138] In step 3033, the image features and text features are fused through a fusion layer based on the attention mechanism to obtain the fusion features of the image block.

[0139] Compared with image samples, the probability of an image block containing one type of target is higher. When an image block contains only one type of target, its image features and text features are unique, which helps the fusion layer to better match the two.

[0140] In step 3034, target detection is performed based on the fused features to obtain a target detection result for the image block.

[0141] In step 3035, based on the object detection results and labels of each image block, the network parameters of the object detection network are adjusted.

[0142] In some embodiments, the target detection network has only one type of detection target. In this case, the detection error of the image block can be determined based on the target detection result and label of each image block, and then the detection errors of each image block can be summed. According to the summation result, the network parameters of the target detection network can be adjusted.

[0143] In some embodiments, the target detection network detects at least two categories of targets. In this case, the detection error of each image block can be determined based on the target detection results and labels of each image block. Then, according to the rule that the error weight corresponding to the image block containing only one category of targets is greater than the error weight corresponding to the image block containing at least two categories of targets, the detection errors of each image block are weighted and summed. Then, based on the weighted summation result, the network parameters of the target detection network are adjusted. In this way, the influence of image blocks containing only one category of targets on the network parameters can be increased, causing the target detection network to pay more attention to the detection of image blocks containing only one category of targets, thereby improving the training speed and effect with the help of high-quality image blocks containing only one category of targets.

[0144] In step 304, when the number of training rounds reaches a first set value, the target detection network is trained for multiple rounds with each image sample in the global sample set and a descriptive statement of the category information corresponding to each image sample as input and the label of each image sample as output until the number of training rounds reaches a second set value, at which point the training is stopped.

[0145] Among them, the first set value and the second set value both represent specific values ​​of the number of training rounds, and the first set value is greater than the second set value. The specific values ​​of the first set value and the second set value can be pre-set by technical personnel according to actual conditions.

[0146] Still refer to Figure 1 , the target detection network in the embodiment of the present application includes an image encoder, a text encoder and a fusion layer based on the attention mechanism. Figure 5 The process shown here trains the object detection network in one of several rounds of training. The process includes the following steps:

[0147] In step 3041, each image sample in the global sample set is input into an image encoder to obtain image features of the image sample.

[0148] In step 3042, the descriptive sentence of the category information corresponding to the image sample is input into a text encoder to obtain the text features of the image sample.

[0149] Among them, the descriptive statements of the category information corresponding to the image samples are obtained through prompt engineering.

[0150] In step 3043, the image features and text features are fused through a fusion layer based on the attention mechanism to obtain the fusion features of the image sample.

[0151] In step 3044, target detection is performed based on the fused features to obtain a target detection result of the image sample.

[0152] In step 3045 , the network parameters of the object detection network are adjusted based on the object detection results and labels of each image sample.

[0153] In step 305, the obtained category descriptive statements of each test image and test target are input into the target detection network to obtain the target detection result of the test image.

[0154] In step 306 , based on the object detection result and the test label of the test image, it is determined whether the object detection of the test image is correct.

[0155] In step 307, based on the detection accuracy of each test image, it is determined whether to continue training the object detection network.

[0156] For example, if the detection accuracy reaches the set accuracy, training can be terminated. If the detection accuracy does not reach the set accuracy, training can be continued, starting from the second half of the training process. In addition, new image samples can be added. In short, any method that can quickly improve the accuracy of the object detection network can be adopted.

[0157] In step 308, the acquired image to be detected and the category descriptive statement of the target to be detected are input into the target detection network finally trained to obtain the target detection result of the image to be detected.

[0158] That is, the target detection network obtained by final training is used for target detection.

[0159] In the embodiment of the present application, during the training process, not only the image information is considered, but also the text information of the target in the image is additionally considered. The target information considered is richer, which is conducive to improving the detection accuracy of the target detection network. In addition, the fusion of image information and text information is achieved through the attention mechanism. The image samples are cut and the image blocks obtained are first used for training, which can better match the image information and text information, and can further improve the detection accuracy of the target detection network. In addition, using image blocks for training in the early stage to capture the local features of the target, and using image samples for training in the later stage to focus on the overall features of the target, not only helps to improve the detection accuracy, but also can better adapt to target detection tasks in different scenarios.

[0160] Based on the same technical concept, an embodiment of the present application also provides a training device for a target detection network. The principle of solving the problem by the training device for the target detection network is similar to that of the above-mentioned training method for the target detection network. Therefore, the implementation of the training device for the target detection network can refer to the implementation of the training method for the target detection network, and the repeated parts will not be repeated.

[0161] Figure 6A structural diagram of a training device for a target detection network provided in an embodiment of the present application includes an acquisition module 601, an adding module 602, a first training module 603, and a second training module 604.

[0162] An acquisition module 601 is configured to segment each image sample in the acquired global sample set to obtain a plurality of image blocks;

[0163] An adding module 602 is configured to add a label to each image block containing an object using the label of each image sample to obtain a local sample set, wherein any label contains category information and location information of the object;

[0164] A first training module 603 is configured to perform multiple rounds of training on the object detection network using the image blocks in the local sample set and the descriptive statements of the category information corresponding to the image blocks as input and the labels of the image blocks as output;

[0165] The second training module 604 is used to, when the number of training rounds reaches a first set value, continue to train the target detection network for multiple rounds using the descriptive statements of each image sample in the global sample set and the category information corresponding to each image sample as input and the label of each image sample as output, until the number of training rounds reaches a second set value, and then stop training.

[0166] In some embodiments, the object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism. For the local sample set, the first training module 603 is specifically configured to perform one round of the multiple rounds of training on the object detection network according to the following steps:

[0167] Inputting each image block in the local sample set into the image encoder to obtain image features of the image block;

[0168] Inputting a descriptive sentence of the category information corresponding to the image block into the text encoder to obtain text features of the image block;

[0169] fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image block;

[0170] Performing target detection based on the fusion features to obtain a target detection result of the image block;

[0171] Based on the object detection results and labels of each image block, the network parameters of the object detection network are adjusted.

[0172] In some embodiments, the target detection network has at least two categories of detection targets, and the first training module 603 is specifically used to:

[0173] Determining a detection error for each image block based on the object detection result and the label of the image block;

[0174] According to the rule that the error weight corresponding to the image block containing one type of target is greater than the error weight corresponding to the image block containing at least two types of targets, the detection errors of each image block are weightedly summed;

[0175] According to the weighted summation results, the network parameters of the target detection network are adjusted.

[0176] In some embodiments, the object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism. For the global sample set, the second training module 604 is specifically configured to perform one round of the multiple rounds of training on the object detection network according to the following steps:

[0177] Inputting each image sample in the global sample set into the image encoder to obtain image features of the image sample;

[0178] Inputting a descriptive sentence of the category information corresponding to the image sample into the text encoder to obtain text features of the image sample;

[0179] fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image sample;

[0180] Performing target detection based on the fusion features to obtain a target detection result of the image sample;

[0181] Based on the object detection results and labels of each image, the network parameters of the object detection network are adjusted.

[0182] In some embodiments, a testing module 605 is further included for:

[0183] After stopping the training, each test image and the category descriptive statement of the test target are input into the target detection network to obtain the target detection result of the test image;

[0184] Determining whether object detection of the test image is correct based on the object detection result and the test label of the test image;

[0185] Based on the detection accuracy of each test image, determine whether to continue training the object detection network.

[0186] In some embodiments, a detection module 606 is further included for:

[0187] The obtained image to be detected and the category descriptive statement of the target to be detected are input into the target detection network finally trained to obtain the target detection result of the image to be detected.

[0188] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The coupling between the modules can be achieved through some interfaces, which are usually electrical communication interfaces, but it is not ruled out that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0189] After introducing the object detection network training method and apparatus according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application is introduced next.

[0190] Refer to the following Figure 7 The electronic device 130 implemented according to this embodiment of the present application is described. Figure 7 The electronic device 130 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0191] like Figure 7 As shown, the electronic device 130 is a general electronic device. Components of the electronic device 130 may include, but are not limited to, the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0192] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0193] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0194] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0195] The electronic device 130 may also communicate with one or more external devices 134 (e.g., a keyboard, pointing device, etc.), one or more devices that enable a user to interact with the electronic device 130, and / or any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules of the electronic device 130 via a bus 133. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0196] In an exemplary embodiment, a storage medium is also provided. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned object detection network training method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

[0197] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may execute the steps of any target detection network training method provided in the embodiments of the present application.

[0198] In an exemplary embodiment, a computer program product is further provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in this application.

[0199] Furthermore, the computer program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0200] In the embodiments of the present application, the program product for training the target detection network may be a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0201] A readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries readable program code. Such a transmitted data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0202] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0203] Program code implementing the application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider. The application is related to the use of computer system 100 for online analytics, online content, and online advertising.

[0204] It should be noted that, although several units or sub-units of the apparatus are mentioned in the above detailed description, such division into units or sub-units is merely exemplary and not mandatory. Indeed, according to an embodiment of the application, features and functions of two or more units described above can be embodied in a single unit. Conversely, features and functions of one unit described above can be split into multiple units.

[0205] Moreover, while operations of the methods of the present application are described in a particular order in the figures, this is not required or implied in any particular order of operations or that all operations be performed to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined, performed in a different order, and / or split into multiple steps.

[0206] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code.

[0207] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0208] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0210] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0211] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also includes these modifications and variations.

Claims

1. A training method for a target detection network, characterized in that: include: Segmenting each image sample in the acquired global sample set to obtain multiple image blocks; Using the labels of each image sample, add labels to each image block containing the target to obtain a local sample set. Any label contains the category information and location information of the target; Performing multiple rounds of training on the object detection network using the image blocks in the local sample set and the descriptive statements of the category information corresponding to the image blocks as input and the labels of the image blocks as output; When the number of training rounds reaches a first set value, the object detection network is continuously trained for multiple rounds using the image samples in the global sample set and the descriptive statements of the category information corresponding to the image samples as input and the labels of the image samples as output until the number of training rounds reaches a second set value, and then the training is stopped; The object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism. The object detection network is trained multiple times using the image blocks in the local sample set and the descriptive statements of the category information corresponding to each image block as input and the labels of each image block as output, including: Perform one of the multiple rounds of training on the object detection network according to the following steps: Inputting each image block in the local sample set into the image encoder to obtain image features of the image block; Inputting a descriptive sentence of the category information corresponding to the image block into the text encoder to obtain text features of the image block; fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image block; Performing target detection based on the fusion features to obtain a target detection result of the image block; Based on the object detection results and labels of each image block, the network parameters of the object detection network are adjusted.

2. The method according to claim 1, wherein The target detection network has at least two types of detection targets. Based on the target detection results and labels of each image block, the network parameters of the target detection network are adjusted, including: Determining a detection error for each image block based on the object detection result and the label of the image block; According to the rule that the error weight corresponding to the image block containing one type of target is greater than the error weight corresponding to the image block containing at least two types of targets, the detection errors of each image block are weightedly summed; According to the weighted summation results, the network parameters of the target detection network are adjusted.

3. The method according to claim 1, wherein The object detection network is trained for multiple rounds using the image samples in the global sample set and the descriptive statements of the category information corresponding to the image samples as input and the labels of the image samples as output, including: Perform one of the multiple rounds of training on the object detection network according to the following steps: Inputting each image sample in the global sample set into the image encoder to obtain image features of the image sample; Inputting a descriptive sentence of the category information corresponding to the image sample into the text encoder to obtain text features of the image sample; fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image sample; Performing target detection based on the fusion features to obtain a target detection result of the image sample; Based on the object detection results and labels of each image, the network parameters of the object detection network are adjusted.

4. The method according to claim 1, wherein After stopping training, also includes: Inputting each acquired test image and test target category descriptive statement into the target detection network to obtain the target detection result of the test image; Determining whether the target detection of the test image is correct based on the target detection result and the test label of the test image; Based on the detection accuracy of each test image, determine whether to continue training the object detection network.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: The obtained image to be detected and the category descriptive statement of the target to be detected are input into the target detection network finally trained to obtain the target detection result of the image to be detected.

6. A training device for a target detection network, characterized in that: include: An acquisition module, configured to segment each image sample in the acquired global sample set to obtain a plurality of image blocks; An adding module is used to add labels to each image block containing the target using the labels of each image sample to obtain a local sample set, where any label contains the category information and location information of the target; A first training module is configured to perform multiple rounds of training on the object detection network using the image blocks in the local sample set and the descriptive statements of the category information corresponding to the image blocks as input and the labels of the image blocks as output; a second training module, configured to, when the number of training rounds reaches a first set value, continue training the object detection network for multiple rounds using the image samples in the global sample set and the descriptive statements of the category information corresponding to the image samples as input and the labels of the image samples as output, until the number of training rounds reaches a second set value, and then stop training; The object detection network includes an image encoder, a text encoder, and a fusion layer based on an attention mechanism. The first training module is specifically used to perform one round of the multiple rounds of training on the object detection network according to the following steps: Inputting each image block in the local sample set into the image encoder to obtain image features of the image block; Inputting a descriptive sentence of the category information corresponding to the image block into the text encoder to obtain text features of the image block; fusing the image features and the text features through the fusion layer based on the attention mechanism to obtain a fusion feature of the image block; Performing target detection based on the fusion features to obtain a target detection result of the image block; Based on the object detection results and labels of each image block, the network parameters of the object detection network are adjusted.

7. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the method according to any one of claims 1 to 5.

Images