Method, device and electronic device for updating target detection network

Abstract

An embodiment of the present invention provides a method, device and electronic device for updating a target detection network. The method includes: obtaining visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model and a target detection network to be updated, performing target detection on the visible light image samples, thermal imaging image samples and fused image samples respectively to obtain a first target set, a second target set and a third target set, determining sample targets based on the overlapping relationship of the identification boxes corresponding to each target in the first target set, the second target set and the third target set and the category of each target, adjusting the model parameters of the target detection network to be updated based on the fused image samples corresponding to the sample targets and the corresponding pseudo labels, the pseudo labels including the identification boxes and categories of the sample targets, thereby realizing the update of the target detection network to be updated and improving its detection capability for different scenarios and the accuracy of the detection results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and electronic equipment for updating a target detection network. Background Art

[0002] With the development of computer vision technology, object detection using visible light images has become a very important target detection method. However, visible light images may not be clear enough under certain conditions, such as at night or in foggy environments, which can lead to a decrease in target detection accuracy. Since the image quality of thermal images is not affected by these conditions, object detection based on visible light and thermal images has been widely used in recent years.

[0003] In the current target detection method based on visible light images and thermal imaging images, a dual-light camera is used to collect visible light images and thermal imaging images containing the target to be detected, and then the target detection results are obtained through the target detection network based on the visible light images and thermal imaging images.

[0004] However, due to reasons such as the possible changes in the scene of the dual-light camera, the accuracy of the target detection results output by the target detection network may not be high enough, so a method for updating the target detection network is needed. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, and electronic device for updating a target detection network to improve the accuracy of target detection results. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for updating a target detection network, the method comprising:

[0007] Obtain visible light image samples, thermal imaging image samples, fused image samples, visible light detection models, thermal imaging detection models, and target detection networks to be updated;

[0008] performing target detection on the visible light image samples based on the visible light detection model to obtain a first target set corresponding to the visible light image samples, performing target detection on the thermal imaging image samples based on the thermal imaging detection model to obtain a second target set corresponding to the thermal imaging image samples, and performing target detection on the fused image samples based on the target detection network to be updated to obtain a third target set corresponding to the fused image samples;

[0009] Determine a sample target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set and a category of each target;

[0010] Based on the fused image sample and the corresponding pseudo label corresponding to the sample target, the model parameters of the target detection network to be updated are adjusted, wherein the pseudo label includes the identification box and category of the sample target.

[0011] Optionally, the step of determining the sample target based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the category of each target includes:

[0012] Determine a detection score corresponding to each target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set and a preset scoring rule;

[0013] Selecting candidate targets from all targets according to the detection score corresponding to each target to obtain a candidate target set;

[0014] For each target in the candidate target set, performing category recognition on the visible light image sample, thermal imaging image sample, and fused image sample of the target to obtain the category corresponding to the target;

[0015] Based on the detection score and the category, a sample target is determined from the candidate targets.

[0016] Optionally, the step of determining a detection score corresponding to each target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set and a preset scoring rule includes:

[0017] For each target, determining whether the identification frames corresponding to the target overlap;

[0018] If the three identification boxes corresponding to the target in the first target set, the second target set, and the third target set all overlap, determining the detection score corresponding to the target to be a first score;

[0019] If two identification boxes corresponding to the target in the first target set, the second target set, and the third target set overlap, determining the detection score corresponding to the target to be the second score;

[0020] If there is no overlap between the identification boxes corresponding to the target in the first target set, the second target set, and the third target set, determining the detection score corresponding to the target to be the third score;

[0021] The first score is greater than the second score and greater than the third score.

[0022] Optionally, the step of determining whether the identification frames corresponding to the target overlap includes:

[0023] Calculate the intersection-over-union ratio of the identification box corresponding to the target;

[0024] If the intersection-over-union ratio is greater than a preset value, determining that the identification frames overlap;

[0025] If the intersection-over-union ratio is not greater than the preset value, it is determined that the identification frames do not overlap.

[0026] Optionally, the step of determining a sample target from the candidate targets based on the detection score and the category includes:

[0027] If the category is a preset category, the detection score is increased by the preset score to obtain a final score;

[0028] Sorting the candidate targets according to the final scores to obtain a sorting result;

[0029] Based on preset screening rules and the sorting result, a sample target is determined from the candidate targets.

[0030] Optionally, before the step of adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target, the method further includes:

[0031] For each of the sample objects, calculating the bounding box of the identification box corresponding to the sample object as the identification box of the sample object;

[0032] If the categories corresponding to the visible light image sample, thermal imaging image sample, and fused image sample where the sample target is located are different, the category with the highest confidence is determined as the category of the sample target.

[0033] Optionally, the method further includes:

[0034] Acquire visible light images and thermal imaging images to be detected;

[0035] Inputting the visible light image and the thermal imaging image into a pre-trained target detection model, fusing the image features of the visible light image and the thermal imaging image to obtain fused features, generating a fused image based on the fused features, and performing target detection based on the fused features to obtain target detection results, wherein the target detection model includes the target detection network to be updated;

[0036] Based on the target detection result, the fused image is displayed, wherein the fused image includes an identification frame and a category for identifying the target detection result.

[0037] Optionally, the target detection model further includes a feature alignment network and an image fusion network;

[0038] The steps of fusing the image features of the visible light image and the image features of the thermal imaging image to obtain fused features, generating a fused image based on the fused features, and performing target detection based on the fused features to obtain target detection results include:

[0039] The feature alignment network extracts features of the visible light image and features of the thermal image respectively to obtain visible light image features and thermal image features; performs deformable convolution processing on the thermal image features to obtain deformed thermal image features, and aligns the deformed thermal image features with the visible light image features;

[0040] The image fusion network performs feature splicing on the aligned thermal imaging image features and the visible light image features to obtain fusion features; generates a fusion image based on the fusion features, and outputs the fusion image;

[0041] The target detection network to be updated determines an identification frame and a category of the target to be detected in the fused image based on the fusion features, and outputs the identification frame and the category.

[0042] In a second aspect, an embodiment of the present invention provides a target detection device, the device comprising:

[0043] A first acquisition module is used to acquire visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated;

[0044] a first detection module, configured to perform target detection on the visible light image samples based on the visible light detection model to obtain a first target set corresponding to the visible light image samples, perform target detection on the thermal imaging image samples based on the thermal imaging detection model to obtain a second target set corresponding to the thermal imaging image samples, and perform target detection on the fused image samples based on the target detection network to be updated to obtain a third target set corresponding to the fused image samples;

[0045] A first determination module is configured to determine a sample target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and a category of each target;

[0046] An adjustment module is used to adjust the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target, wherein the pseudo label includes the identification box and category of the sample target.

[0047] Optionally, the first determining module includes:

[0048] a first determining unit, configured to determine a detection score corresponding to each target in the first target set, the second target set, and the third target set based on an overlapping relationship between identification boxes corresponding to each target and a preset scoring rule;

[0049] A selection unit is used to select candidate targets from all targets according to the detection score corresponding to each target to obtain a candidate target set;

[0050] an identification unit, configured to perform category identification on each target in the candidate target set using the visible light image sample, the thermal imaging image sample, and the fused image sample of the target to obtain a category corresponding to the target;

[0051] A second determining unit is configured to determine a sample target from the candidate targets based on the detection score and the category.

[0052] Optionally, the first determining unit includes:

[0053] A judging subunit, configured to judge, for each target, whether the identification frames corresponding to the target overlap;

[0054] a first determining subunit, configured to determine a detection score corresponding to the target as a first score if three identification boxes corresponding to the target in the first target set, the second target set, and the third target set all overlap;

[0055] a second determining subunit, configured to determine a detection score corresponding to the target as a second score if two identification boxes corresponding to the target in the first target set, the second target set, and the third target set overlap;

[0056] a third determining subunit, configured to determine a detection score corresponding to the target as a third score if there is no overlap between the identification boxes corresponding to the target in the first target set, the second target set, and the third target set;

[0057] The first score is greater than the second score and greater than the third score.

[0058] Optionally, the judgment subunit includes:

[0059] A calculation subunit, used to calculate the intersection-over-union ratio of the identification box corresponding to the target;

[0060] a fourth determining subunit, configured to determine that the identification frames overlap if the intersection-over-union ratio is greater than a preset value;

[0061] The fifth determining subunit is configured to determine that the identification frames do not overlap if the intersection-over-union ratio is not greater than the preset value.

[0062] Optionally, the second determining unit includes:

[0063] an adding subunit, configured to increase the detection score by a preset score to obtain a final score if the category is a preset category;

[0064] A sorting subunit, configured to sort the candidate targets according to the final scores to obtain a sorting result;

[0065] The screening subunit is used to determine a sample target from the candidate targets based on preset screening rules and the sorting result.

[0066] Optionally, the device further includes:

[0067] a calculation module configured to calculate, for each sample target, a bounding box of an identification box corresponding to the sample target as the identification box of the sample target, before adjusting the model parameters of the target detection network to be updated based on the fused image samples and the corresponding pseudo labels corresponding to the sample targets;

[0068] The third determination unit is used to determine the category with the highest confidence as the category of the sample target if the categories corresponding to the visible light image sample, thermal imaging image sample and fused image sample of the sample target are different before the step of adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo-label corresponding to the sample target.

[0069] Optionally, the device further includes:

[0070] A second acquisition module is used to acquire a visible light image and a thermal imaging image to be detected;

[0071] a second detection module, configured to input the visible light image and the thermal imaging image into a pre-trained target detection model, fuse the image features of the visible light image and the thermal imaging image to obtain a fused feature, generate a fused image based on the fused feature, and perform target detection based on the fused feature to obtain a target detection result, wherein the target detection model includes the target detection network to be updated;

[0072] The display module is configured to display the fused image based on the target detection result, wherein the fused image includes an identification frame and a category for identifying the target detection result.

[0073] Optionally, the target detection model further includes a feature alignment network and an image fusion network;

[0074] The second detection module includes:

[0075] a feature alignment unit configured to extract features of the visible light image and the thermal image using the feature alignment network to obtain visible light image features and thermal image features; perform deformable convolution processing on the thermal image features to obtain deformed thermal image features; and align the deformed thermal image features with the visible light image features;

[0076] An image fusion unit, configured to perform feature splicing on the aligned thermal imaging image features and visible light image features in the image fusion network to obtain fusion features; generate a fusion image based on the fusion features, and output the fusion image;

[0077] The first target detection unit is configured to determine, by the target detection network to be updated, an identification frame and a category of the target to be detected in the fused image based on the fusion features, and output the identification frame and the category.

[0078] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0079] Memory for storing computer programs;

[0080] The processor is configured to implement any of the method steps described in the first aspect when executing a program stored in the memory.

[0081] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0082] Beneficial effects of the embodiments of the present invention:

[0083] In the solution provided by an embodiment of the present invention, an electronic device can obtain visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated. Based on the visible light detection model, the electronic device can perform target detection on the visible light image samples to obtain a first target set corresponding to the visible light image samples. Based on the thermal imaging detection model, the electronic device can perform target detection on the thermal imaging image samples to obtain a second target set corresponding to the thermal imaging image samples. Based on the target detection network to be updated, the electronic device can perform target detection on the fused image samples to obtain a third target set corresponding to the fused image samples. Based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and the category of each target, the electronic device can determine the sample target. Based on the fused image samples corresponding to the sample target and the corresponding pseudo labels, the electronic device can adjust the model parameters of the target detection network to be updated, wherein the pseudo labels include the identification boxes and categories of the sample targets. Through the above solution, the electronic device can update the target detection network to be updated based on the fused image samples corresponding to the sample targets and the corresponding pseudo labels, thereby improving the target detection capability of the target detection network to be updated, thereby improving the accuracy of the target detection results. Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0085] Figure 1 A flowchart of a method for updating a target detection network provided by an embodiment of the present invention;

[0086] Figure 2 for Figure 1 A specific flow chart of step S103 in the embodiment shown;

[0087] Figure 3 Based on Figure 2 A schematic diagram of a method for determining a detection score in the illustrated embodiment;

[0088] Figure 4 Based on Figure 2 A schematic diagram of a method for determining overlapping identification frames in the illustrated embodiment;

[0089] Figure 5 for Figure 2 A specific flow chart of step S204 in the embodiment shown;

[0090] Figure 6 A flowchart of another target detection network updating method provided by an embodiment of the present invention;

[0091] Figure 7 A flowchart of another target detection network updating method provided by an embodiment of the present invention;

[0092] Figure 8 for Figure 7 A specific flow chart of step S702 in the embodiment shown;

[0093] Figure 9 A schematic diagram of the structure of an updating device for a target detection network provided by an embodiment of the present invention;

[0094] Figure 10 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0095] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention are within the scope of protection of the present invention.

[0096] In order to improve the accuracy of target detection results, an embodiment of the present invention provides a method, device, electronic device, computer-readable storage medium and computer program product for updating a target detection network. The following first introduces a method for updating a target detection network provided by an embodiment of the present invention.

[0097] A target detection network update method provided in an embodiment of the present invention can be applied to any electronic device that updates the target detection network, which can be a server or a terminal, for example, a computer, a mobile phone, a tablet computer, etc., without specific limitation here. For the sake of clarity of description, it will be referred to as an electronic device in the following.

[0098] like Figure 1 As shown, a method for updating a target detection network, the method comprising:

[0099] S101, obtaining visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated;

[0100] S102: performing target detection on the visible light image samples based on the visible light detection model to obtain a first target set corresponding to the visible light image samples; performing target detection on the thermal imaging image samples based on the thermal imaging detection model to obtain a second target set corresponding to the thermal imaging image samples; and performing target detection on the fused image samples based on the target detection network to be updated to obtain a third target set corresponding to the fused image samples.

[0101] S103, determining a sample target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set and a category of each target;

[0102] S104: Adjust the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target.

[0103] The pseudo label includes the identification box and category of the sample target.

[0104] As can be seen, in the solution provided by the embodiments of the present invention, the electronic device can obtain visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated. Based on the visible light detection model, the electronic device can perform target detection on the visible light image samples to obtain a first target set corresponding to the visible light image samples. Based on the thermal imaging detection model, the electronic device can perform target detection on the thermal imaging image samples to obtain a second target set corresponding to the thermal imaging image samples. Based on the target detection network to be updated, the electronic device can perform target detection on the fused image samples to obtain a third target set corresponding to the fused image samples. Based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and the category of each target, the electronic device can determine the sample target. Based on the fused image samples corresponding to the sample target and the corresponding pseudo labels, the electronic device can adjust the model parameters of the target detection network to be updated, wherein the pseudo labels include the identification boxes and categories of the sample targets. Through the above solution, the electronic device can update the target detection network to be updated based on the fused image samples corresponding to the sample targets and the corresponding pseudo labels, thereby improving the target detection capability of the target detection network to be updated, thereby improving the accuracy of the target detection results.

[0105] To update the target detection network to be updated, the electronic device may obtain visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and the target detection network to be updated. In one embodiment, the target detection network to be updated may be a pre-trained target detection network, and the electronic device may update the target detection network to be updated online while the target detection network to be updated is being used in actual scenarios.

[0106] The visible light image sample and the thermal image sample can be, respectively, the visible light image and thermal image to be detected during use of the target detection network to be updated in an actual scenario. The fused image sample can be a fused image obtained by fusing the visible image sample and the thermal image sample, or it can be the fused image to be detected during use of the target detection network to be updated. Both are reasonable. In this case, the visible light image sample and thermal image sample corresponding to the fused image sample are the visible light image and thermal image based on which the fused image sample is obtained.

[0107] The above-mentioned visible light detection model can be a detection model with strong performance that can accurately detect visible light images; the above-mentioned thermal imaging detection model can be a detection model with strong performance that can accurately detect thermal imaging images.

[0108] Furthermore, the electronic device can perform target detection on the visible light image samples based on a high-performance visible light detection model, obtaining a target set corresponding to the visible light image samples as a first target set. Furthermore, the electronic device can perform target detection on the thermal imaging image samples based on a high-performance thermal imaging detection model, obtaining a target set corresponding to the thermal imaging image samples as a second target set.

[0109] Furthermore, the electronic device may perform target detection on the fused image sample based on the target detection network to be updated, obtaining a target set corresponding to the fused image sample as a third target set. For each target in the first target set, the second target set, and the third target set, an identification box may be used to identify the location of each target in the corresponding image sample.

[0110] In one embodiment, the electronic device may classify each target in the visible light image sample based on a high-performance visible light classification model, obtaining a category corresponding to each target in the visible light image sample as a first category. Furthermore, the electronic device may classify each target in the thermal imaging image sample based on a high-performance thermal imaging classification model, obtaining a category corresponding to each target in the thermal imaging image sample as a second category. Furthermore, the electronic device may classify each target in the fused image sample based on a high-performance dual-light classification model, obtaining a category corresponding to each target in the fused image sample as a third category.

[0111] Since the position of each target in the first target set, the second target set and the third target set can be identified in the corresponding image sample with an identification box, the overlapping relationship of the identification boxes can identify the accuracy of target detection, and the ability to identify the category of the target can represent a high accuracy in detecting the target.

[0112] Therefore, after obtaining the first target set, the second target set, and the third target set, the electronic device can perform the above step S103, that is, the electronic device can determine sample targets based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set, as well as the category of each target. The sample targets are targets with higher target detection accuracy in the first target set, the second target set, and the third target set.

[0113] After obtaining the sample target, the electronic device may perform the above step S104, that is, adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target.

[0114] A pseudo-label for the sample object can be generated based on the sample object's category and the corresponding identification box. The pseudo-label includes the sample object's identification box and category. Furthermore, the electronic device can use the pseudo-label for the sample object as the true value signal (i.e., the sample label) of the fused image sample corresponding to the sample object.

[0115] Furthermore, for each fused image sample corresponding to a sample target, the electronic device can calculate the target detection loss function value based on the difference between the target detection result of the fused image sample output by the target detection network to be updated and the true value signal of the fused image sample. The electronic device can then adjust the model parameters of the target detection network to be updated based on the target detection loss function value, thereby updating the target detection network to be updated, thereby improving the target detection capability of the target detection network to be updated, and thus improving the accuracy of the target detection results. In particular, the target detection network to be updated can be updated in a targeted manner for different usage scenarios, so that the updated target detection network can have excellent detection capabilities for the usage scenario, thereby improving the accuracy of the target detection results.

[0116] As an implementation method of the embodiment of the present invention, Figure 2 As shown, the step of determining the sample target based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the category of each target may include:

[0117] S201, determining a detection score corresponding to each target in the first target set, the second target set, and the third target set based on an overlapping relationship between identification boxes corresponding to each target and a preset scoring rule;

[0118] Since the position of each target in the first target set, the second target set, and the third target set can be identified by an identification box in the corresponding image sample, and the overlapping relationship of the identification boxes can identify the accuracy of target detection, the electronic device can determine the detection score corresponding to the target based on the overlapping relationship of the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the preset scoring rules.

[0119] Since the detection score corresponds to the above-mentioned overlapping relationship, and the overlapping relationship can represent the accuracy of the target detection, the size of the detection score can represent the accuracy of the target detection.

[0120] S202, selecting candidate targets from all targets based on the detection score corresponding to each target, to obtain a candidate target set;

[0121] After obtaining the detection score corresponding to each target, since the size of the detection score can represent the accuracy of the target detection, that is, it can identify the quality of the corresponding image sample, the electronic device can select alternative targets from all targets included in the first target set, the second target set, and the third target set according to the detection score corresponding to each target to obtain an alternative target set.

[0122] In one embodiment, the electronic device may select targets with detection scores as candidate targets, thereby obtaining a set of candidate targets. In another embodiment, the electronic device may compare the detection score corresponding to each target with a preset threshold, and select targets with detection scores greater than the first preset threshold as candidate targets, thereby obtaining a set of candidate targets.

[0123] S203, for each target in the candidate target set, performing category recognition on the visible light image sample, thermal imaging image sample, and fused image sample of the target to obtain a category corresponding to the target;

[0124] After obtaining the candidate target set, for each target in the candidate target set, the electronic device may input the visible light image sample containing the target into a high-performance visible light classification model to obtain the first category corresponding to the target in the visible light image sample. The electronic device may input the thermal image sample containing the target into a high-performance thermal imaging classification model to obtain the second category corresponding to the target in the thermal image sample. The electronic device may also input the fused image sample containing the target into a high-performance dual-light classification model to obtain the third category corresponding to the target in the fused image sample.

[0125] In one embodiment, if the target does not have an identification frame in the visible light image sample, thermal image sample, or fused image sample, that is, if the target cannot be identified from the visible light image sample, thermal image sample, or fused image sample, the electronic device can cut out the corresponding position in the image sample where the target cannot be identified based on the position, shape, and size of the identification frame corresponding to the target in the image sample where the target has been identified, to obtain a classified image of the target in the image sample where the target cannot be identified. This classified image can then be input into the corresponding classification model to obtain the category corresponding to the target.

[0126] For example, if target A is identified in the thermal image sample and the fused image sample, but not in the visible light image sample, the electronic device can cut out the corresponding position of the visible light image sample based on the position, shape, and size of the identification box of target A in the thermal image sample to obtain a classified image of target A. This classified image can then be input into the visible light classification model to obtain the first category corresponding to target A.

[0127] S204, determining a sample target from the candidate targets based on the detection score and the category;

[0128] After obtaining the category corresponding to each target in the candidate target set, the electronic device can determine a sample target from the candidate targets based on the detection score and category. In one embodiment, if the above category belongs to a preset category and the detection score is greater than a second preset threshold, the candidate target is determined as a sample target.

[0129] For example, the preset category is human, the second preset threshold is 10, the category of alternative target 1 is car, and the detection score is 18, then alternative target 1 cannot be used as a sample target, and the category of alternative target 2 is human, and the detection score is 15, then alternative target 2 can be used as a sample target.

[0130] It can be seen that in this embodiment, the electronic device can determine the detection score corresponding to the target based on the overlapping relationship of the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the preset scoring rules, select alternative targets from all targets according to the detection score corresponding to each target, and obtain an alternative target set. For each target in the alternative target set, category recognition is performed on the visible light image sample, thermal imaging image sample, and fused image sample where the target is located to obtain the category corresponding to the target, and sample targets are determined from the alternative targets based on the detection score and category. Through the above scheme, the electronic device can filter out sample targets based on the detection score and category to facilitate the subsequent update of the target detection network to be updated, thereby improving the accuracy of the target detection results.

[0131] As an implementation manner of an embodiment of the present invention, the step of determining the detection score corresponding to each target based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the preset scoring rule may include:

[0132] For each target, determine whether there is overlap in the identification box corresponding to the target; if there is overlap in the three identification boxes corresponding to the target in the first target set, the second target set, and the third target set, determine that the detection score corresponding to the target is the first score; if there is overlap in the two identification boxes corresponding to the target in the first target set, the second target set, and the third target set, determine that the detection score corresponding to the target is the second score; if there is no overlap in the identification boxes corresponding to the target in the first target set, the second target set, and the third target set, determine that the detection score corresponding to the target is the third score.

[0133] The electronic device may determine, for each target in the first target set, the second target set, and the third target set, whether the identification frames corresponding to the target overlap, where the overlapping of the identification frames may include at least three situations:

[0134] In the first case, the three identification boxes corresponding to the target in the first target set, the second target set, and the third target set all overlap. In the second case, the two identification boxes corresponding to the target in the first target set, the second target set, and the third target set overlap. In the third case, the identification boxes corresponding to the target in the first target set, the second target set, and the third target set do not overlap.

[0135] Among them, the above-mentioned second situation can have two sub-situations. In the first sub-situation, if the target in the first target set, the second target set and the third target set has only two corresponding identification frames and there is overlap, then this situation is regarded as the second situation.

[0136] In the second sub-case, the target in the first target set, the second target set and the third target set all has a corresponding identification frame. When any two identification frames corresponding to the target in the first target set, the second target set and the third target set overlap and do not overlap with another identification frame, this situation is regarded as the second situation.

[0137] Similarly, the third case mentioned above may have three sub-cases. In the first sub-case, if there is only one corresponding identification box for the target in the first target set, the second target set, and the third target set, this case is regarded as the third case.

[0138] In the second sub-case, if there are only two corresponding identification boxes for the target in the first target set, the second target set, and the third target set, and the two identification boxes do not overlap, then this situation is considered as the third case. In the third sub-case, if there are three corresponding identification boxes for the target in the first target set, the second target set, and the third target set, and none of the three identification boxes overlap, then this situation is considered as the third case.

[0139] For the first case mentioned above, that is, if the three identification boxes corresponding to the target in the first target set, the second target set and the third target set all overlap, the detection score corresponding to the target can be determined to be the first score according to the preset scoring rules.

[0140] For the second case mentioned above, that is, if the two identification boxes corresponding to the target in the first target set, the second target set and the third target set overlap, the detection score corresponding to the target can be determined to be the second score according to the preset scoring rules.

[0141] For the third scenario described above, if the identification boxes corresponding to the target in the first target set, the second target set, and the third target set do not overlap, then the detection score corresponding to the target is determined to be the third score based on the preset scoring rules. Since a greater number of overlapping identification boxes indicates more accurate detection of the target corresponding to the identification box, the first score is greater than the second score and the third score.

[0142] For example, the scores corresponding to different overlapping situations of the identification boxes corresponding to the target can be shown in the following table:

[0143] Overlap The first case Second case The third case Fraction 5 3 1

[0144] The following combination Figure 3 The diagram shown below illustrates the three cases of overlapping identification boxes and the detection scores:

[0145] The first circle 310 represents the first target set, the second circle 320 represents the second target set, and the third circle 330 represents the third target set. The portion with a score of 5 corresponds to the first case mentioned above, that is, the three identification frames corresponding to the target in the first target set, the second target set, and the third target set all overlap, and the detection score corresponding to the target is determined to be 5 points. The portion with a score of 3 corresponds to the second case mentioned above, that is, the two identification frames corresponding to the target in the first target set, the second target set, and the third target set overlap, and the detection score corresponding to the target is determined to be 3 points. The portion with a score of 1 corresponds to the third case mentioned above, that is, the identification frames corresponding to the target in the first target set, the second target set, and the third target set do not overlap, and the detection score corresponding to the target is determined to be 1 point.

[0146] It can be seen that in this embodiment, the electronic device determines whether there is any overlap in the identification frame corresponding to each target, and different detection scores can be determined for different overlapping situations. Through the above scheme, since different overlapping situations correspond to different detection scores, and different overlapping situations also represent whether the recognition of the target is accurate, the electronic device can judge whether the recognition of the target is accurate based on different detection scores, so that it can select sample targets with more accurate detection results, thereby improving the accuracy of the target detection results.

[0147] As an implementation method of the embodiment of the present invention, Figure 4 As shown, the above step of determining whether the identification frames corresponding to the targets overlap may include:

[0148] S401, calculating the intersection-over-union (IoU) of the identification box corresponding to the target; if the IoU is greater than a preset value, executing step S402; if the IoU is not greater than the preset value, executing step S403;

[0149] To determine whether the target's corresponding identification frames overlap, the electronic device can calculate the intersection-and-union (IoU) of the target's corresponding identification frames. The formula for calculating IoU is: IoU = area of ​​the overlapping portions of the identification frames / maximum area of ​​the identification frame markers. If the IoU is greater than a preset value, it indicates that the identification frames overlap significantly, and step S402 is executed. If the IoU is not greater than the preset value, it indicates that the identification frames overlap minimally, and step S403 is executed.

[0150] S402, determining whether the identification frames overlap;

[0151] If the intersection-over-union ratio is greater than the preset value, it means that the identification frames have a large overlap, and the electronic device can determine that the identification frames are overlapping, that is, the targets identified by the identification frames are overlapping, which means that the detection of the target is relatively accurate.

[0152] S403: Determine whether the identification frames overlap.

[0153] If the intersection-over-union ratio is not greater than the preset value, it means that the overlapping parts of the identification frames are small, and the electronic device can determine that there is no overlap in the identification frames, that is, there is no overlap in the targets identified by the identification frames, which means that the detection of the target may be a false detection.

[0154] It can be seen that in this embodiment, the electronic device can calculate the intersection-and-union ratio of the identification frame corresponding to the target. If the intersection-and-union ratio is greater than the preset value, it is determined that the identification frames are overlapping. If the intersection-and-union ratio is not greater than the preset value, it is determined that the identification frames are not overlapping. Through the above scheme, the electronic device can calculate the intersection-and-union ratio of the identification frames, and then determine whether the identification frames are overlapping based on the relationship between the intersection-and-union ratio and the preset value, that is, whether the targets corresponding to the identification frames are overlapping, and then determine whether the detection of the target is accurate, thereby improving the accuracy of the target detection results.

[0155] As an implementation method of the embodiment of the present invention, Figure 5 As shown, the step of determining the sample target from the candidate targets based on the detection score and the category may include:

[0156] S501, if the category is a preset category, increase the detection score by a preset score to obtain a final score;

[0157] After obtaining the detection score and category of each candidate target, the electronic device can determine whether the category of the candidate target is a preset category in order to select a sample target. If the category of the candidate target is a preset category, the electronic device can increase the detection score corresponding to the candidate target by the preset score to obtain the final score of the candidate target. If the category of the candidate target is not a preset category, the electronic device can use the detection score corresponding to the candidate target as the final score of the candidate target.

[0158] For example, the detection score corresponding to alternative target 1 is 5 points, the preset category is human, and the preset score is 1. The electronic device can determine whether the category of alternative target 1 is human. If so, the final score of alternative target 1 is 5+1=6 points. If not, the final score corresponding to alternative target 1 is 5 points.

[0159] S502, sorting the candidate targets according to the final scores to obtain a sorting result;

[0160] The final score can represent whether the detection of the target is accurate and whether the category corresponding to the target is a preset category. After obtaining the final score of each alternative target, the electronic device can sort the alternative targets according to the final score to obtain a sorting result. The sorting result can represent whether the detection of the target is accurate and whether the category corresponding to the target is a preset category.

[0161] Among them, the sorting method can be to sort the candidate targets in order from small to large according to the final scores, or the sorting method can be to sort the candidate targets in order from large to small according to the final scores, which is reasonable.

[0162] S503: Determine a sample target from the candidate targets based on preset screening rules and the ranking result.

[0163] Because the ranking result can indicate whether the detection of the target is accurate, the electronic device can determine sample targets from the candidate targets based on the preset screening rules and the ranking result. In one embodiment, the electronic device can select the candidate targets whose final scores in the ranking result are greater than a preset value as sample targets based on the ranking result. In another embodiment, the electronic device can also select a preset number of candidate targets from the ranking result as sample targets based on the ranking result, the ranking method, and the number of candidate targets included in the ranking result.

[0164] For example, the sorting results are sorted from large to small according to the size of the final score. The number of alternative targets included in the sorting results is 50, and the preset number is 10. Then, the electronic device can select the top 10 alternative targets in the sorting results as sample targets.

[0165] It can be seen that in this embodiment, if the category is a preset category, the electronic device can increase the detection score by the preset score to obtain a final score, sort the alternative targets according to the final score to obtain a sorting result, and determine the sample target from the alternative targets based on the preset screening rules and the sorting result. Through the above scheme, the electronic device can screen out sample targets with higher accuracy, thereby improving the accuracy of the target detection results.

[0166] As an implementation method of the embodiment of the present invention, Figure 6 As shown, before the step of adjusting the model parameters of the target detection network based on the fused image sample and the corresponding pseudo label corresponding to the sample target, the method may further include:

[0167] S601, for each sample object, calculating the bounding box of the identification box corresponding to the sample object as the identification box of the sample object;

[0168] Due to environmental and weather factors, the sizes of the corresponding identification frames of the sample target in the visible light image sample, thermal image sample, and fused image sample may be inconsistent. To more accurately identify the identification frame of the sample target, the electronic device can calculate, for each sample target, a bounding box of the identification frame of the sample target in the visible light image sample, thermal image sample, and fused image sample, as the identification frame of the sample target. The bounding box is the minimum rectangular frame that can include the identification frame of the sample target in the visible light image sample, thermal image sample, and fused image sample.

[0169] S602: If the categories corresponding to the visible light image sample, the thermal imaging image sample, and the fused image sample where the sample target is located are different, the category with the highest confidence is determined as the category of the sample target.

[0170] Due to the existence of detection errors, the corresponding categories of the sample target in the visible light image sample, thermal imaging image sample and fused image sample may be different. In this case, since a higher confidence level indicates a higher accuracy in identifying the category of the sample target, the electronic device can determine the category with the highest confidence level as the category of the sample target, thereby improving the accuracy of the target detection network.

[0171] For example, the first category corresponding to the sample target a in the visible light image sample is human, with a confidence level of 0.9; the second category corresponding to the sample target a in the thermal imaging image sample is car, with a confidence level of 0.6; the third category corresponding to the sample target a in the fused image sample is human, with a confidence level of 0.8; then, the electronic device can determine the first category with a corresponding confidence level of 0.9, i.e., human, as the category of the sample target a.

[0172] It can be seen that in this embodiment, the electronic device can calculate the circumscribed frame of the identification frame corresponding to each sample target as the identification frame of the sample target. If the categories corresponding to the visible light image sample, thermal imaging image sample and fused image sample where the sample target is located are different, the category with the highest confidence is determined as the category of the sample target. In this way, the electronic device can obtain a more accurate identification frame of the sample target and the category of the sample target, thereby improving the accuracy of the target detection result.

[0173] As an implementation method of the embodiment of the present invention, Figure 7 As shown, the method may further include:

[0174] S701, acquiring a visible light image and a thermal imaging image to be detected;

[0175] In order to perform target detection on visible light images and thermal imaging images, the electronic device may obtain the visible light image and thermal imaging image to be detected captured by an image acquisition device, wherein the image acquisition device may be a dual-light camera.

[0176] S702: Input the visible light image and the thermal image into a pre-trained target detection model, fuse the visible light image and the thermal image based on image features to obtain fused features, generate a fused image based on the fused features, and perform target detection based on the fused features to obtain a target detection result.

[0177] After obtaining the visible light image and thermal imaging image to be detected, the electronic device can input the visible light image and thermal imaging image into a pre-trained target detection model, wherein the target detection model includes the target detection network to be updated, and the target detection network to be updated is used to output the target detection result.

[0178] In this way, the target detection model can fuse the image features of the visible light image and the image features of the thermal image to obtain a fused feature. Since the fused feature includes the features of both the visible light image and the thermal image, when environmental factors cause the visible light image corresponding to the target to be unclear, for example, at night or in foggy environments, the image features of the thermal image corresponding to the target can help better detect the target.

[0179] That is, the fused features can complement the image features of the visible light image and the thermal image, giving full play to their respective advantages. As a result, the image quality of the fused image generated based on the fused features is better.

[0180] The fusion feature is used to perform target detection using the target detection network to be updated. Since the target detection network to be detected can be updated online according to different implementation scenarios during the use of the target detection model, the target detection results output by the target detection network to be updated are also more accurate.

[0181] S703, displaying the fused image based on the target detection result;

[0182] After obtaining the target detection result, the electronic device may display a fused image based on the target detection result, wherein the fused image includes an identification frame and a category for identifying the target detection result.

[0183] The electronic device can display a fused image based on the target detection results output by the target detection network to be updated, wherein the fused image includes an identification box and a category for identifying the target detection results, wherein the identification box can be a rectangular box that selects the target box in the fused image, etc., and the category can include people, vehicles, etc.

[0184] It can be seen that in this embodiment, the electronic device can use the visible light image and the thermal imaging image as the input of the target detection network. The target detection network obtains the fusion features based on the extracted image features of the visible light image and the image features of the thermal imaging image, and then generates a fused image based on the fusion features. At the same time, the target detection result can be obtained based on the fusion features. Then, the electronic device displays the fused image based on the target detection result output by the target detection network, thereby realizing the end-to-end output of the fused image. The target detection result can be displayed while displaying the fused image, and the accuracy and display effect of the target detection result are better.

[0185] As an implementation method of the embodiment of the present invention, Figure 8 As shown, the above-mentioned target detection model may include a feature alignment network 810, an image fusion network 820 and a target detection network 830 to be updated.

[0186] The feature alignment network 810 can be used to align the extracted image features of the visible light image 801 and the image features of the thermal imaging image 802, the image fusion network 820 can be used to fuse the image features of the visible light image 801 and the image features of the thermal imaging image 802, and the target detection network 830 to be updated can be used to perform target detection based on the fused features after fusion.

[0187] Therefore, the electronic device can input the visible light image 801 and the thermal imaging image 802 into the target detection model, so that the feature alignment network 810, the image fusion network 820 and the target detection network 830 to be updated in the target detection model perform target detection on the visible light image 801 and the thermal imaging image 802, and thus obtain the target detection result.

[0188] Accordingly, the steps of fusing the image features of the visible light image and the image features of the thermal imaging image to obtain fused features, generating a fused image based on the fused features, and performing target detection based on the fused features to obtain target detection results may include:

[0189] The feature alignment network 810 extracts features of the visible light image 801 and the thermal image 802 respectively to obtain visible light image features and thermal image features; performs deformable convolution processing on the thermal image features to obtain deformed thermal image features, and aligns the deformed thermal image features with the visible light image features; the image fusion network 820 performs feature splicing on the aligned thermal image features and visible light image features to obtain fused features; generates a fused image based on the fused features and outputs the fused image; the target detection network 830 to be updated determines the identification box and category of the target to be detected in the fused image based on the fused features, and outputs the identification box and category.

[0190] Because visible light image 801 and thermal image 802 captured by the image acquisition device have different fields of view and resolutions, feature alignment network 810 can extract features of visible light image 801 and features of thermal image 802, respectively, to enable subsequent image fusion network 820 to fuse them. Specifically, Encoder 1 (encoder) 8031 ​​- Encoder N 803N in feature alignment network 810 can extract features from visible light image 801 to obtain first-order visible light image features and N-order visible light image features. Encoder 1 8041 - Encoder N 804N in feature alignment network 810 can extract features from thermal image 802 to obtain first-order thermal image features and N-order thermal image features.

[0191] After obtaining the 1st-order thermal imaging image features and the Nth-order thermal imaging image features, Deformable1 (deformer) 8051 - DeformableN 805N in the feature alignment network 210 can perform a deformable convolution process on the 1st-order thermal imaging image features and the Nth-order thermal imaging image features to obtain the 1st-order deformed thermal imaging image features and the Nth-order deformed thermal imaging image features. At this point, the 1st-order deformed thermal imaging image features are roughly aligned with the 1st-order visible light image features, the 2nd-order deformed thermal imaging image features are roughly aligned with the 2nd-order visible light image features, and so on. The Nth-order deformed thermal imaging image features are roughly aligned with the Nth-order visible light image features. That is, the 1st-order deformed thermal imaging image features and the Nth-order deformed thermal imaging image features are roughly aligned with the 1st-order visible light image features and the Nth-order visible light image features, respectively.

[0192] In order to strictly align the 1st-order deformed thermal imaging image features and the Nth-order deformed thermal imaging image features with the 1st-order visible light image features and the Nth-order visible light image features, respectively, Eltwise1 8061 in the feature alignment network 810 can perform Eltwise processing on the 1st-order deformed thermal imaging image features and the 1st-order visible light image features, Eltwise2 8062 can perform Eltwise processing on the 2nd-order deformed thermal imaging image features and the 2nd-order visible light image features, and so on. EltwiseN 806N can perform Eltwise processing on the Nth-order deformed thermal imaging image features and the Nth-order visible light image features, that is, Eltwise1 8061-EltwiseN 806N can perform Eltwise processing on the 1st-order deformed thermal imaging image features and the Nth-order deformed thermal imaging image features, respectively, to obtain aligned features of different orders. Thus, the field of view and resolution of the visible light image 801 and the thermal imaging image 802 can be strictly matched, which prepares the subsequent image fusion network 820 to fuse the visible light image 801 and the thermal imaging image 802.

[0193] Since the visible light image 801 and the thermal imaging image 802 are processed by the above-mentioned feature alignment network 810, alignment features of different orders, namely alignment features 8071-alignment features 807M, are obtained, and the sizes of the feature maps corresponding to the alignment features of different orders are different, in order to enable subsequent feature fusion and target detection, the image fusion network 820 can upsample the feature maps of different sizes to obtain feature maps of the same resolution, wherein the alignment features include the aligned 1st-order deformed thermal imaging image features-Nth-order deformed thermal imaging image features and the aligned 1st-order visible light image features-Nth-order visible light image features, wherein M is generally twice of N.

[0194] That is, the image fusion network 820 can upsample the aligned features 8071-807M to the same resolution through transposed convolution to obtain aligned features 8081-808M at the same resolution. The aligned first-order deformed thermal imaging image features at the same resolution are feature-concatenated with the aligned first-order visible light image features at the same resolution, and the aligned second-order deformed thermal imaging image features at the same resolution are feature-concatenated with the aligned second-order visible light image features at the same resolution. Similarly, the aligned N-order deformed thermal imaging image features at the same resolution are feature-concatenated with the aligned N-order visible light image features at the same resolution. That is, the aligned first-order deformed thermal imaging image features at the same resolution - the aligned N-order deformed thermal imaging image features at the same resolution, and the aligned first-order visible light image features at the same resolution - the aligned N-order visible light image features at the same resolution are feature-concatenated to obtain fused features.

[0195] Furthermore, Decoder1 (decoder) 8091 -DecoderN 809N in the image fusion network 820 can decode the fusion features to generate a fused image, so that the image fusion network 820 can output the fused image 8010 .

[0196] After the above-mentioned image fusion network 820 obtains the fusion feature, the fusion feature can be input into the target detection network 830 to be updated, so that the target detection network 830 to be updated can determine the identification box and category of the target to be detected in the above-mentioned fusion image based on the above-mentioned fusion feature, and output the identification box and category as the target detection result 8011, so that the electronic device can obtain the target detection result output by the target detection model.

[0197] As can be seen, in this embodiment, the target detection model can include a feature alignment network, an image fusion network, and a target detection network to be updated. The feature alignment network can extract features of the visible light image and the thermal image, respectively, to obtain visible light image features and thermal image features. The thermal image features are subjected to deformable convolution processing to obtain deformed thermal image features, and the deformed thermal image features are aligned with the visible light image features. The image fusion network can perform feature concatenation of the aligned thermal image features and the visible light image features to obtain fused features. A fused image is generated based on the fused features and outputted. The target detection network to be updated can determine the identification box and category of the target to be detected in the fused image based on the fused features and output the identification box and category. Through the above scheme, the electronic device can use the visible light image and the thermal image as input to the target detection model, and the target detection model can then generate the fused image and obtain the target detection result at the same time, thus achieving end-to-end output of the fused image. The target detection result can be displayed simultaneously with the fused image, and the target detection result has good accuracy and display effect.

[0198] Corresponding to the above-mentioned method for updating a target detection network, an embodiment of the present invention further provides an updating device for a target detection network. The following introduces an updating device for a target detection network provided by an embodiment of the present invention.

[0199] like Figure 9 As shown, a device for updating a target detection network may include:

[0200] A first acquisition module 910 is used to acquire visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated;

[0201] a first detection module 920 configured to perform target detection on the visible light image samples based on the visible light detection model to obtain a first target set corresponding to the visible light image samples, perform target detection on the thermal imaging image samples based on the thermal imaging detection model to obtain a second target set corresponding to the thermal imaging image samples, and perform target detection on the fused image samples based on the target detection network to be updated to obtain a third target set corresponding to the fused image samples;

[0202] A first determination module 930 is configured to determine a sample target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and a category of each target;

[0203] The adjustment module 940 is used to adjust the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target.

[0204] The pseudo label includes the identification box and category of the sample target.

[0205] As can be seen, in the solution provided by the embodiments of the present invention, the electronic device can obtain visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated. Based on the visible light detection model, the electronic device can perform target detection on the visible light image samples to obtain a first target set corresponding to the visible light image samples. Based on the thermal imaging detection model, the electronic device can perform target detection on the thermal imaging image samples to obtain a second target set corresponding to the thermal imaging image samples. Based on the target detection network to be updated, the electronic device can perform target detection on the fused image samples to obtain a third target set corresponding to the fused image samples. Based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and the category of each target, the electronic device can determine the sample target. Based on the fused image samples corresponding to the sample target and the corresponding pseudo labels, the electronic device can adjust the model parameters of the target detection network to be updated, wherein the pseudo labels include the identification boxes and categories of the sample targets. Through the above solution, the electronic device can update the target detection network to be updated based on the fused image samples corresponding to the sample targets and the corresponding pseudo labels, thereby improving the target detection capability of the target detection network to be updated, thereby improving the accuracy of the target detection results.

[0206] As an implementation of an embodiment of the present invention, the first determining module 930 may include:

[0207] a first determining unit, configured to determine a detection score corresponding to each target in the first target set, the second target set, and the third target set based on an overlapping relationship between identification boxes corresponding to each target and a preset scoring rule;

[0208] A selection unit is used to select candidate targets from all targets according to the detection score corresponding to each target to obtain a candidate target set;

[0209] an identification unit, configured to perform category identification on each target in the candidate target set using the visible light image sample, the thermal imaging image sample, and the fused image sample of the target to obtain a category corresponding to the target;

[0210] A second determining unit is configured to determine a sample target from the candidate targets based on the detection score and the category.

[0211] As an implementation manner of an embodiment of the present invention, the first determining unit may include:

[0212] A judging subunit, configured to judge, for each target, whether the identification frames corresponding to the target overlap;

[0213] a first determining subunit, configured to determine a detection score corresponding to the target as a first score if three identification boxes corresponding to the target in the first target set, the second target set, and the third target set all overlap;

[0214] a second determining subunit, configured to determine a detection score corresponding to the target as a second score if two identification boxes corresponding to the target in the first target set, the second target set, and the third target set overlap;

[0215] The third determining subunit is configured to determine that the detection score corresponding to the target is a third score if there is no overlap between the identification boxes corresponding to the target in the first target set, the second target set, and the third target set.

[0216] The first score is greater than the second score and greater than the third score.

[0217] As an implementation manner of an embodiment of the present invention, the above-mentioned judgment subunit may include:

[0218] A calculation subunit, used to calculate the intersection-over-union ratio of the identification box corresponding to the target;

[0219] a fourth determining subunit, configured to determine that the identification frames overlap if the intersection-over-union ratio is greater than a preset value;

[0220] The fifth determining subunit is configured to determine that the identification frames do not overlap if the intersection-over-union ratio is not greater than the preset value.

[0221] As an implementation manner of an embodiment of the present invention, the second determining unit may include:

[0222] an adding subunit, configured to increase the detection score by a preset score to obtain a final score if the category is a preset category;

[0223] A sorting subunit, configured to sort the candidate targets according to the final scores to obtain a sorting result;

[0224] The screening subunit is used to determine a sample target from the candidate targets based on preset screening rules and the sorting result.

[0225] As an implementation of an embodiment of the present invention, the above-mentioned device may include:

[0226] a calculation module configured to calculate, for each sample target, a bounding box of an identification box corresponding to the sample target as the identification box of the sample target, before adjusting the model parameters of the target detection network to be updated based on the fused image samples and the corresponding pseudo labels corresponding to the sample targets;

[0227] The second determination module is used to determine the category with the highest confidence as the category of the sample target if the categories corresponding to the visible light image sample, thermal imaging image sample and fused image sample of the sample target are different before the step of adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo-label corresponding to the sample target.

[0228] As an implementation manner of the embodiment of the present invention, the above-mentioned device may further include:

[0229] A second acquisition module is used to acquire a visible light image and a thermal imaging image to be detected;

[0230] a second detection module, configured to input the visible light image and the thermal imaging image into a pre-trained target detection model, fuse the image features of the visible light image and the thermal imaging image to obtain a fused feature, generate a fused image based on the fused feature, and perform target detection based on the fused feature to obtain a target detection result, wherein the target detection model includes the target detection network to be updated;

[0231] The display module is configured to display the fused image based on the target detection result, wherein the fused image includes an identification frame and a category for identifying the target detection result.

[0232] As an implementation method of an embodiment of the present invention, the above-mentioned target detection model may further include a feature alignment network and an image fusion network;

[0233] The second module may include:

[0234] a feature alignment unit configured to extract features of the visible light image and the thermal image using the feature alignment network to obtain visible light image features and thermal image features; perform deformable convolution processing on the thermal image features to obtain deformed thermal image features; and align the deformed thermal image features with the visible light image features;

[0235] An image fusion unit, configured to perform feature splicing on the aligned thermal imaging image features and visible light image features in the image fusion network to obtain fusion features; generate a fusion image based on the fusion features, and output the fusion image;

[0236] The first target detection unit is configured to determine, by the target detection network to be updated, an identification frame and a category of the target to be detected in the fused image based on the fusion features, and output the identification frame and the category.

[0237] The embodiment of the present invention further provides an electronic device, such as Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0238] Memory 1003, used for storing computer programs;

[0239] The processor 1001 is configured to implement the steps of the target detection network updating method described in any of the above embodiments when executing the program stored in the memory 1003.

[0240] As can be seen, in the solution provided by the embodiments of the present invention, the electronic device can obtain visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated. Based on the visible light detection model, the electronic device can perform target detection on the visible light image samples to obtain a first target set corresponding to the visible light image samples. Based on the thermal imaging detection model, the electronic device can perform target detection on the thermal imaging image samples to obtain a second target set corresponding to the thermal imaging image samples. Based on the target detection network to be updated, the electronic device can perform target detection on the fused image samples to obtain a third target set corresponding to the fused image samples. Based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and the category of each target, the electronic device can determine the sample target. Based on the fused image samples corresponding to the sample target and the corresponding pseudo labels, the electronic device can adjust the model parameters of the target detection network to be updated, wherein the pseudo labels include the identification boxes and categories of the sample targets. Through the above solution, the electronic device can update the target detection network to be updated based on the fused image samples corresponding to the sample targets and the corresponding pseudo labels, thereby improving the target detection capability of the target detection network to be updated, thereby improving the accuracy of the target detection results.

[0241] The communication bus mentioned in the electronic devices mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into address buses, data buses, control buses, etc. For ease of illustration, only a single thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0242] The communication interface is used for communication between the above electronic device and other devices.

[0243] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0244] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0245] In another embodiment provided by the present invention, a computer-readable storage medium is further provided, which stores a computer program. When the computer program is executed by a processor, the steps of the target detection network update method in any of the above embodiments are implemented.

[0246] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes any target detection network updating method of the above embodiments.

[0247] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state disk (SSD)).

[0248] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0249] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, since the apparatus, electronic device, computer-readable storage medium, and computer program product are generally similar to the method embodiments, their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0250] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for updating a target detection network, characterized in that: The method comprises: Obtain visible light image samples, thermal imaging image samples, fused image samples, visible light detection models, thermal imaging detection models, and target detection networks to be updated; performing target detection on the visible light image samples based on the visible light detection model to obtain a first target set corresponding to the visible light image samples, performing target detection on the thermal imaging image samples based on the thermal imaging detection model to obtain a second target set corresponding to the thermal imaging image samples, and performing target detection on the fused image samples based on the target detection network to be updated to obtain a third target set corresponding to the fused image samples; Determine a sample target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set and a category of each target; Adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target, wherein the pseudo label includes the identification box and category of the sample target; The step of determining the sample target based on the overlapping relationship between the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the category of each target includes: Based on the overlapping relationship of the identification boxes corresponding to each target in the first target set, the second target set, and the third target set and the preset scoring rules, the detection score corresponding to the target is determined; according to the detection score corresponding to each target, alternative targets are selected from all targets to obtain an alternative target set; for each target in the alternative target set, category identification is performed on the visible light image samples, thermal imaging image samples, and fused image samples where the target is located to obtain the category corresponding to the target; based on the detection score and the category, a sample target is determined from the alternative targets.

2. The method according to claim 1, characterized in that The step of determining a detection score corresponding to each target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set and a preset scoring rule includes: For each target, determining whether the identification frames corresponding to the target overlap; If the three identification boxes corresponding to the target in the first target set, the second target set, and the third target set all overlap, determining the detection score corresponding to the target to be a first score; If two identification boxes corresponding to the target in the first target set, the second target set, and the third target set overlap, determining the detection score corresponding to the target to be the second score; If there is no overlap between the identification boxes corresponding to the target in the first target set, the second target set, and the third target set, determining the detection score corresponding to the target to be the third score; The first score is greater than the second score and greater than the third score.

3. The method according to claim 2, characterized in that The step of determining whether the identification frames corresponding to the targets overlap includes: Calculate the intersection-over-union ratio of the identification box corresponding to the target; If the intersection-over-union ratio is greater than a preset value, determining that the identification frames overlap; If the intersection-over-union ratio is not greater than the preset value, it is determined that the identification frames do not overlap.

4. The method according to claim 1, wherein The step of determining a sample target from the candidate targets based on the detection score and the category includes: If the category is a preset category, the detection score is increased by the preset score to obtain a final score; Sorting the candidate targets according to the final scores to obtain a sorting result; Based on preset screening rules and the sorting result, a sample target is determined from the candidate targets.

5. The method according to any one of claims 1 to 4, characterized in that Before the step of adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target, the method further includes: For each of the sample objects, calculating the bounding box of the identification box corresponding to the sample object as the identification box of the sample object; If the categories corresponding to the visible light image sample, thermal imaging image sample, and fused image sample where the sample target is located are different, the category with the highest confidence is determined as the category of the sample target.

6. The method according to claim 1, characterized in that The method further comprises: Acquire visible light images and thermal imaging images to be detected; Inputting the visible light image and the thermal imaging image into a pre-trained target detection model, fusing the image features of the visible light image and the thermal imaging image to obtain fused features, generating a fused image based on the fused features, and performing target detection based on the fused features to obtain target detection results, wherein the target detection model includes the target detection network to be updated; Based on the target detection result, the fused image is displayed, wherein the fused image includes an identification frame and a category for identifying the target detection result.

7. The method according to claim 6, characterized in that The target detection model also includes a feature alignment network and an image fusion network; The steps of fusing the image features of the visible light image and the image features of the thermal imaging image to obtain fused features, generating a fused image based on the fused features, and performing target detection based on the fused features to obtain target detection results include: The feature alignment network extracts features of the visible light image and features of the thermal image respectively to obtain visible light image features and thermal image features; performs deformable convolution processing on the thermal image features to obtain deformed thermal image features, and aligns the deformed thermal image features with the visible light image features; The image fusion network performs feature splicing on the aligned thermal imaging image features and the visible light image features to obtain fusion features; generates a fusion image based on the fusion features, and outputs the fusion image; The target detection network to be updated determines an identification frame and a category of the target to be detected in the fused image based on the fusion features, and outputs the identification frame and the category.

8. An updating device for a target detection network, characterized in that: The device comprises: A first acquisition module is used to acquire visible light image samples, thermal imaging image samples, fused image samples, a visible light detection model, a thermal imaging detection model, and a target detection network to be updated; a first detection module, configured to perform target detection on the visible light image samples based on the visible light detection model to obtain a first target set corresponding to the visible light image samples, perform target detection on the thermal imaging image samples based on the thermal imaging detection model to obtain a second target set corresponding to the thermal imaging image samples, and perform target detection on the fused image samples based on the target detection network to be updated to obtain a third target set corresponding to the fused image samples; A first determination module is configured to determine a sample target based on an overlapping relationship between identification boxes corresponding to each target in the first target set, the second target set, and the third target set, and a category of each target; An adjustment module, configured to adjust the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo label corresponding to the sample target, wherein the pseudo label includes an identification box and a category of the sample target; The first determining module includes: a first determining unit, configured to determine a detection score corresponding to each target in the first target set, the second target set, and the third target set based on an overlapping relationship between identification boxes corresponding to each target and a preset scoring rule; A selection unit is used to select candidate targets from all targets according to the detection score corresponding to each target to obtain a candidate target set; an identification unit, configured to perform category identification on each target in the candidate target set using the visible light image sample, the thermal imaging image sample, and the fused image sample of the target to obtain a category corresponding to the target; A second determining unit is configured to determine a sample target from the candidate targets based on the detection score and the category.

9. The device according to claim 8, characterized in that The first determining unit includes: A judging subunit, configured to judge, for each target, whether the identification frames corresponding to the target overlap; a first determining subunit, configured to determine a detection score corresponding to the target as a first score if three identification boxes corresponding to the target in the first target set, the second target set, and the third target set all overlap; a second determining subunit, configured to determine a detection score corresponding to the target as a second score if two identification boxes corresponding to the target in the first target set, the second target set, and the third target set overlap; a third determining subunit, configured to determine a detection score corresponding to the target as a third score if there is no overlap between the identification boxes corresponding to the target in the first target set, the second target set, and the third target set; The first score is greater than the second score and greater than the third score.

10. The device according to claim 9, characterized in that The judgment subunit includes: A calculation subunit, used to calculate the intersection-over-union ratio of the identification box corresponding to the target; a fourth determining subunit, configured to determine that the identification frames overlap if the intersection-over-union ratio is greater than a preset value; The fifth determining subunit is configured to determine that the identification frames do not overlap if the intersection-over-union ratio is not greater than the preset value.

11. The device according to claim 8, characterized in that The second determining unit includes: an adding subunit, configured to increase the detection score by a preset score to obtain a final score if the category is a preset category; A sorting subunit, configured to sort the candidate targets according to the final scores to obtain a sorting result; The screening subunit is used to determine a sample target from the candidate targets based on preset screening rules and the sorting result.

12. The device according to any one of claims 8 to 11, characterized in that The device further comprises: a calculation module configured to calculate, for each sample target, a bounding box of an identification box corresponding to the sample target as the identification box of the sample target, before adjusting the model parameters of the target detection network to be updated based on the fused image samples and the corresponding pseudo labels corresponding to the sample targets; The second determination module is used to determine the category with the highest confidence as the category of the sample target if the categories corresponding to the visible light image sample, thermal imaging image sample and fused image sample of the sample target are different before the step of adjusting the model parameters of the target detection network to be updated based on the fused image sample and the corresponding pseudo-label corresponding to the sample target.

13. The device according to claim 8, characterized in that The device further comprises: A second acquisition module is used to acquire a visible light image and a thermal imaging image to be detected; a second detection module, configured to input the visible light image and the thermal imaging image into a pre-trained target detection model, fuse the image features of the visible light image and the thermal imaging image to obtain a fused feature, generate a fused image based on the fused feature, and perform target detection based on the fused feature to obtain a target detection result, wherein the target detection model includes the target detection network to be updated; The display module is configured to display the fused image based on the target detection result, wherein the fused image includes an identification frame and a category for identifying the target detection result.

14. The device according to claim 13, characterized in that The target detection model also includes a feature alignment network and an image fusion network; The second detection module includes: a feature alignment unit configured to extract features of the visible light image and the thermal image using the feature alignment network to obtain visible light image features and thermal image features; perform deformable convolution processing on the thermal image features to obtain deformed thermal image features; and align the deformed thermal image features with the visible light image features; An image fusion unit, configured to perform feature splicing on the aligned thermal imaging image features and visible light image features in the image fusion network to obtain fusion features; generate a fusion image based on the fusion features, and output the fusion image; The first target detection unit is configured to determine, by the target detection network to be updated, an identification frame and a category of the target to be detected in the fused image based on the fusion features, and output the identification frame and the category.

15. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images