Multi-target detection model integration method and device, program product and electronic equipment

Abstract

The invention relates to the technical field of image detection, and provides a multi-target detection model integration method and device, a program product and electronic equipment. The integration method comprises the following steps: acquiring information of all detection frames output by more than two target detection models after processing a to-be-detected image; constructing a first list and a second list; determining a global importance weight of each target category; determining a dynamic weight of each target detection model for each target category; weighting the initial confidence coefficient of the detection frame i by using the global importance weight and the dynamic weight to obtain a weighted confidence coefficient; circularly executing the following steps until the first list is empty: removing the detection box with the maximum weighted confidence from the first list, and adding the removed detection box to the second list; and fusing all the detection frames meeting the preset overlapping condition in the first list to obtain a new detection frame, and adding the new detection frame into the first list. The high-quality detection result inhibition risk can be reduced, and the comprehensive detection performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a multi-target detection model integration method, device, program product and electronic equipment. Background Art

[0002] In the field of video behavior detection, it is necessary to cover the detection and behavior recognition of multiple targets. For example, in security monitoring scenarios, it is necessary to detect targets such as objects thrown from high places and fires, and identify behaviors such as pedestrians wandering in restricted areas and vehicles parked illegally; in camera video intelligent monitoring scenarios, it is necessary to detect targets such as vehicles, pedestrians, and traffic signs on the road, and identify dynamic behaviors such as vehicles suddenly changing lanes and pedestrians crossing the road. In order to meet the needs of complex image detection, it is usually necessary to call two or more target detection models to handle different tasks separately. The two or more target detection models can include R-CNN series, YOLO series (such as YOLOv4, YOLOv8 and YOLOv10, etc.), Transformer detectors, etc.

[0003] When two or more target detection models are called to handle different tasks, each target detection model will output detection results for multiple targets, including the target category, detection box location, and confidence level. Differences in detection logic and threshold settings among target detection models may lead to inconsistent confidence standards across different target detection models, resulting in different detection results for the same target. Using simple fusion strategies such as voting or averaging often makes it difficult to fully leverage the strengths of each target detection model, and high-quality detection results may be suppressed. Summary of the Invention

[0004] The present application aims to at least solve the technical problems existing in the prior art and provide a multi-target detection model integration method, device, program product and electronic device.

[0005] In a first aspect, the present application provides a method for integrating multi-target detection models, including: obtaining information on all detection frames output by more than two target detection models after processing an image to be detected, the information on each detection frame including an initial confidence; constructing a first list, the first list being initialized to include all detection frames; constructing a second list, the second list being initialized to be empty; determining the global importance weight of each target category; determining the dynamic weight of each target detection model for each target category; using the global importance weight of the target category to which detection frame i belongs and the dynamic weight of the target detection model that outputs detection frame i to the target category to which detection frame i belongs, weighting the initial confidence of detection frame i to obtain the weighted confidence of detection frame i, where i is the detection frame index; looping through the following steps until the first list is empty, and when the first list is empty, outputting the detection frames included in the second list: removing the detection frame with the largest weighted confidence from the first list, and adding the removed detection frame to the second list; fusing one or more detection frames in the first list whose intersection-over-union ratio with the removed detection frame meets a preset overlap condition to obtain a new detection frame, and adding the new detection frame to the first list.

[0006] In the second aspect, the present application provides a multi-target detection model integration device for implementing the multi-target detection model integration method described in the first aspect of the present application, including: a first list construction module, which obtains information on all detection frames output after two or more target detection models process the image to be detected, and the information on each detection frame includes an initial confidence; constructs a first list, and the first list is initialized to include all detection frames; a second list construction module, which constructs a second list, and the second list is initialized to be empty; a weight determination module, which determines the global importance weight of each target category; determines the dynamic weight of each target detection model for each target category; and a confidence weighting module, which uses the detection frame i to which it belongs. The global importance weight of the target category and the dynamic weight of the target detection model of the output detection box i on the target category to which the detection box i belongs are weighted, and the initial confidence of the detection box i is weighted to obtain the weighted confidence of the detection box i, where i is the detection box index; the detection box fusion output module loops through the following steps until the first list is empty, and when the first list is empty, outputs the detection boxes included in the second list: removes the detection box with the largest weighted confidence from the first list, and adds the removed detection box to the second list; fuses one or more detection boxes in the first list whose intersection-over-union ratio with the removed detection box meets the preset overlap condition to obtain a new detection box, and adds the new detection box to the first list.

[0007] In a third aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the multi-target detection model integration method described in the present application.

[0008] In a fourth aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-target detection model integration method described in the present application.

[0009] The beneficial technical effects of the present application are: assigning a global importance weight to each target category according to actual task requirements, and obtaining the dynamic weight of each target detection model for each target category, and performing double weighting on the initial confidence of each detection frame using the corresponding two weights to obtain a weighted confidence, fully tapping the advantages of each target detection model, and the weighted confidence of the detection frames output by different target detection models has better standard consistency than the initial confidence, thereby reducing the risk of high-quality detection results being suppressed during the fusion of detection results; in the fusion process of all detection frames, the present application improves the existing non-maximum suppression algorithm NMS, and fuses the detection frames in the first list whose intersection-over-union ratio with the removed detection frames meets the preset overlap conditions into a new detection frame, and adds the new detection frame to the first list, which can avoid the loss of the detection frame after fusion and improve the comprehensive detection performance of the target detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 1 is a flow chart of a multi-target detection model integration method in a preferred embodiment of the present invention;

[0011] Figure 2 2 is a schematic structural diagram of a multi-target detection model integration device in a preferred embodiment of the present invention;

[0012] Figure 3 It is a structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0014] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0015] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0016] The execution subject of the multi-target detection model integration method provided by the present invention includes but is not limited to at least one of the electronic devices such as the server and the terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the multi-target detection model integration method can be executed by software or hardware installed on the terminal device or the server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0017] The multi-target detection model integration method provided by the present invention, in a preferred embodiment, is as follows Figure 1 As shown, including:

[0018] Step S1: Obtain information about all detection frames output by two or more target detection models after processing the image to be detected, where the information about each detection frame includes an initial confidence level; and construct a first list, where the first list is initialized to include all detection frames.

[0019] In this embodiment, each target detection model is used to process the image to be detected to obtain the detection results of each target detection model, and the detection results of all target detection models are integrated to obtain the information of all detection frames. The information of each detection frame output by each target detection model includes the target category, initial confidence level, and detection frame position (such as the coordinates of the upper left corner and lower right corner). After integration, the information of each detection frame also includes the model identifier of the target detection model that outputs the detection frame. Exemplarily, the information of detection frame i is expressed as: in, Indicates the position coordinates of the upper left corner of the detection box i, Indicates the coordinates of the lower right corner of the detection box i, s i represents the initial confidence of the detection box i, c i represents the target category of the detection box i, m i The model identifier of the target detection model that generates the detection box i. The information of the integrated detection box is added to the first list to complete the initialization of the first list.

[0020] In this embodiment, depending on the application field, the image to be detected can be a traffic image, and the target detection model is used to detect target categories such as pedestrians, vehicles, and traffic signs in the traffic image; the image to be detected can also be a field image captured by a field camera, and the target detection model is used to detect various wild animals in the field image; the image to be detected can also be a production image from industrial production, and the target detection model is used to detect target categories such as equipment, parts, and workers in the production image. The target detection model is not limited to the R-CNN series, the YOLO series (such as YOLOv4, YOLOv8, and YOLOv10), Transformer, etc.

[0021] Step S2: construct a second list, which is initially empty.

[0022] Step S3: Determine the global importance weight of each target category; determine the dynamic weight of each target detection model for each target category.

[0023] In this implementation, the global importance weight of each target type can be determined based on the core requirements of the target recognition task to adjust the overall priority of different target category detection boxes in the sorting process. The value range of the global importance weight of each target type is [0, 1]. For example, for the pedestrian behavior recognition task, the global importance weight of pedestrians can be set to 0.7, the global importance weight of vehicles can be set to 0.5, and the global importance weight of trees can be set to 0.3.

[0024] In this embodiment, in step S3, illustratively, the dynamic weight of each target detection model for each target category can be pre-stored, and the value interval of the dynamic weight is (0, 1], and the value can be determined according to the detection accuracy of each target detection model for each target category. When executing the multi-target detection model integration method, the dynamic weight of each target detection model for each target category is read from the memory.

[0025] Step S4, using the global importance weight of the target category to which detection box i belongs and the dynamic weight of the target category to which detection box i belongs by the target detection model of the output detection box i, weight the initial confidence of detection box i to obtain the weighted confidence of detection box i, where i is the detection box index and i is a positive integer.

[0026] For example, the weighted confidence of the detection box i is for:

[0027]

[0028] in, The target detection model m represents the output detection box i i For the target category c to which the detection box i belongs i Dynamic weight of α(c i ) indicates the target category c to which the detection box i belongs i The global importance weight of .

[0029] Step S5: loop through steps S51 and S52 until the first list is empty. When the first list is empty, output the information of the detection boxes included in the second list:

[0030] Step S51: remove the detection box with the largest weighted confidence from the first list, and add the removed detection box to the second list.

[0031] In this embodiment, the detection boxes in the first list are sorted in descending order of weighted confidence. After sorting, the detection box with the largest weighted confidence is removed from the first list, and the removed detection box with the largest weighted confidence is added to the second list.

[0032] Step S52 , fusing one or more detection frames in the first list whose intersection-over-union ratios with the removed detection frames meet a preset overlapping condition to obtain a new detection frame, and adding the new detection frame to the first list.

[0033] Exemplarily, the preset overlapping condition is that the intersection-and-union (IoU) of the detection frame in the first list and the removed detection frame is greater than an IoU threshold, and the IoU threshold can be set in advance. After removing the detection frame with the largest weighted confidence, the IoU of each detection frame in the first list and the removed detection frame is calculated one by one, and the detection frames in the first list whose IoU is greater than the IoU threshold are identified, and all the identified detection frames are fused to obtain a new detection frame. The fusion method is not limited to selecting the detection frame with the largest weighted confidence as the new detection frame. Alternatively, the position average value and the weighted confidence average value of all identified detection frames can be calculated as the position and weighted confidence of the new detection frame, respectively, and the target category of the detection frame with the largest initial confidence among all identified detection frames is used as the target category of the new detection frame.

[0034] In this embodiment, by obtaining weighted confidence and fusing all detection frames according to the weighted confidence, the problem of suppressing high-quality detection results caused by inconsistent confidence standards of different target detection models can be reduced.

[0035] In a preferred embodiment, in order to balance the differences in detection capabilities of different target detection models for specific target types, in step S3, the step of determining the dynamic weight of each target detection model for each target category includes:

[0036] Step S3a1, obtain the average precision (AP) of each target detection model for each target category, and normalize the obtained original average precision.

[0037] In this embodiment, specifically, the average accuracy of each target detection model for each target category on the test sample set during the training process of each target detection model is obtained. For example, the i'th target detection model m i' For the j'th target category c j' The average precision is expressed as

[0038] Step S3a2: Select the maximum value among the average precisions of the target category j' of two or more target detection models, and use the selected maximum value as the maximum average precision of the target category j', where j' is the target category index and is a positive integer.

[0039] For example, the maximum average precision of target category j' is expressed as: M represents the number of target detection models, m k represents the k-th object detection model.

[0040] Step S3a3, divide the average precision of each target detection model for target category j' by the maximum average precision of target category j' to obtain the dynamic weight of each target detection model for target category j'. i' For the j'th target category c j' The dynamic weight of is expressed as:

[0041]

[0042] In a preferred embodiment, dynamic weights are updated in multiple rounds. After each test sample set is used to test two or more target detection models, the dynamic weights are updated based on the obtained test data. This ensures that the dynamic weights are set according to the detection capabilities of the target detection models, achieving a better model balancing effect. Therefore, in step S3, the step of determining the dynamic weight of each target detection model for each target category includes:

[0043] Step S3b1, obtain the test data of the current round. The test data of the current round is obtained by testing two or more target detection models using the test sample set of the current round. The test data of the current round includes the average test accuracy of each target detection model for each target category on the test sample set. Testing two or more target detection models once using the test sample set is considered a round. For example, when the current round is t, the i'th target detection model m i' For the j'th target category c j' The average test accuracy is expressed as

[0044] Step S3b2, obtaining the current round dynamic weight increment of each target detection model for each target category based on the current round test data.

[0045] In this embodiment, specifically, step S3b2 includes:

[0046] Step 1: Obtain the dynamic weight of each target detection model for each target category in the current round of test data. For example, let the current round be t, and the i'th target detection model m i' For the j'th target category c j' The dynamic weight of the current round of test is expressed as:

[0047] Step 2: Subtract the dynamic weight of each target detection model for each target category in the current round of test from the dynamic weight of the target detection model for the target category in the previous round, and multiply the difference obtained by the subtraction by the learning rate δ to obtain the dynamic weight of each target detection model for each target category.

[0048] The dynamic weight increment of the current round. For example, the i'th target detection model m i' For the j'th

[0049] Target category c j' The current round dynamic weight increment is: δ·Δω (t) (m i' ,c j' )=δ·[testω (t) (m i' ,c j' )-ω (t-1) (m i' ,c j' )]. (t-1) (m i' ,c j' ) represents the i'th target detection model m in the previous round (t-1) i' For the j'th target category c j' The dynamic weight of δ represents the learning rate, which is usually in the range of 0.01-0.05.

[0050] Step S3b3, calculate the sum of the current round dynamic weight increment of each target detection model for each target category and the dynamic weight of each target detection model for each target category in the previous round, and use the sum as the dynamic weight of each target detection model for each target category in the current round.

[0051] For example, the current round i'th target detection model m i' For the j'th target ω (t) (m i' ,c j' )=ω (t-1) (m i' ,c j' )+γ·Δω (t) (m i' ,c j' ). The value range of γ is usually 0.01-0.05.

[0052] In a preferred embodiment, in step S52, fusing one or more detection frames in the first list whose intersection-over-union ratios with the removed detection frames satisfy a preset overlap condition to obtain a new detection frame includes:

[0053] Step S521 : forming a set of overlapping detection frames, where the set of overlapping detection frames includes one or more detection frames in the first list whose intersection-over-union ratio with the removed detection frame satisfies a preset overlapping condition.

[0054] For example, the set of overlapping detection boxes B overlap Including K detection boxes: Β overlap ={b1,b2,…,b K}.

[0055] Step S522: determine the position weight of each detection frame in the set of overlapping detection frames, where the position weight of detection frame i is the dynamic weight of the target detection model that outputs detection frame i for the target category to which detection frame i belongs. and the initial confidence s of the detection box i i The product of , the position weight of the detection box i can be expressed as:

[0056] Step S523: sum the position weights of the detection frames in the set of overlapping detection frames to obtain the cumulative sum of the position weights:

[0057] In step S524, the position information of the detection frames in the set of overlapping detection frames is weighted and summed based on the position weights of the detection frames to obtain a weighted sum of the detection frame positions. Specifically, the weighted sums are performed on the x-axis coordinate of the upper left corner, the y-axis coordinate of the upper left corner, the x-axis coordinate of the lower right corner, and the y-axis coordinate of the lower right corner, respectively, which are expressed as: and

[0058] Step S525: Divide the weighted sum of the detection frame positions by the accumulated sum of the position weights to obtain the coordinates of the upper left corner of the new detection frame position information. and the lower right corner coordinates

[0059]

[0060] In step S526, the initial confidence scores of the detection boxes in the set of overlapping detection boxes are fused to obtain a weighted confidence score for the new detection box. The fusion method is not limited to taking the average or selecting the maximum. Using the initial confidence score here can prevent detection boxes of high-priority target categories from excessively influencing the fusion result, thereby improving the accuracy of the fusion result.

[0061] In step S527 , the target category of the detection frame with the largest initial confidence in the set of overlapping detection frames is used as the target category of the new detection frame.

[0062] In this embodiment, the position of the new detection frame is obtained by weighted averaging the positions of the detection frames in the set of overlapping detection frames, which can well represent the set of overlapping detection frames and avoid the loss of high-quality detection results.

[0063] In this embodiment, further preferably, in step S521, forming a set of overlapping detection frames includes:

[0064] Step S521a: Obtain an IoU threshold between each detection frame in the first list that has not been removed and the removed detection frame. Setting an IoU threshold between any two detection frames can more accurately identify detection frames that effectively overlap with the removed detection frame.

[0065] Further preferably, the IoU threshold of the two detection frames is adaptively obtained based on the dynamic weights of the target categories to which the target detection models that output the two detection frames belong, so as to balance the performance differences of the two target detection models. Therefore, step S521a includes:

[0066] Step S521a1: Obtain the dynamic weight of the target detection model for each detection box that has not been removed from the first list, and record it as the first dynamic weight. Let detection box j be a detection box that has not been removed from the first list, then the first dynamic weight is the dynamic weight of the target detection model for the target type of detection box j.

[0067] Step S521a2: Obtain the dynamic weight of the target detection model outputting the removed detection frame for the target type to which the removed detection frame belongs, recorded as the second dynamic weight. Let detection frame i be the removed detection frame, then the second dynamic weight is the dynamic weight of the target detection model outputting detection frame i for the target type to which detection frame i belongs.

[0068] Step S521a3, obtain the absolute value of the difference between the first dynamic weight and the second dynamic weight, and record it as the absolute value of the weight difference, such as

[0069] Step S521a1: multiply the absolute value of the weight difference by the weight difference sensitivity coefficient β to obtain the intersection-over-combination ratio threshold adjustment value, such as

[0070] Step S521a1: Set the basic intersection-over-union threshold τ base Subtract the IoU threshold adjustment amount to obtain the IoU threshold of each detection frame in the first list that has not been removed and the removed detection frame. That is, the IoU threshold τ of the detection frame j in the first list that has not been removed and the removed detection frame i i,j for:

[0071] In this embodiment, β is the weight difference sensitivity coefficient, which represents the adjustment amount of the intersection-over-union ratio threshold per unit model weight difference, and its value range is [0.1, 0.3]; τ base The intersection-over-union threshold is based on , and its value range is [0,1], which can generally be initialized to 0.5.

[0072] Step S521b, calculate the intersection-over-union (IoU) of each detection frame in the first list that has not been removed and the removed detection frame. If the IoU is greater than the IoU threshold of the detection frame and the removed detection frame, then add the detection frame to the set of overlapping detection frames. Specifically, it can be expressed as: the set of overlapping detection frames B overlap ={b j |IoU(b i ,b j )>τ i,j}. IoU(b i ,b j ) represents the intersection-over-union ratio of the detection box j that has not been removed in the first list and the removed detection box i.

[0073] In this embodiment, IoU(b i ,b j ) is calculated as follows:

[0074] For the detection boxes removed from the first list and the detection boxes that were not removed The coordinates of the upper left corner of the overlapping area are:

[0075]

[0076] The coordinates of the lower right corner of the overlapping area are:

[0077]

[0078] The overlapping area is:

[0079] overlap_area=

[0080] max(0,overlap_x2-overlap_x1)×max(0,overlap_y2-overlap_y1)

[0081] The area of ​​the union region is:

[0082] union_area=area_b i +area_b j -overlap_area;

[0083]

[0084] In a preferred embodiment, to obtain a more accurate and reasonable weighted confidence of a new detection frame and avoid the problem of losing high-quality detection results, in step S526, the initial confidences of the detection frames in the set of overlapping detection frames are fused to obtain the weighted confidence of the new detection frame, including:

[0085] Step S5261: From the overlapping detection frame set Boverlap Extract the maximum initial confidence from the initial confidence corresponding to the detection frame (assuming there are K detection frames in total)

[0086] Step S5262: According to the comparison result of the initial confidence of each detection frame in the set of overlapping detection frames and the preset confidence filtering threshold θ, the indicator function value of each detection frame in the set of overlapping detection frames is obtained. Specifically, the indicator function f(s i >θ), indicating the condition s i >θ holds true when f(s i >θ) takes 1, condition s i >θ does not hold when f(s i >θ) is set to 0. θ is generally set to 0.3-0.5.

[0087] Step S5263: sum the indicator function values ​​of the detection frames in the set of overlapping detection frames to obtain the indicator function cumulative value, which can be expressed as:

[0088] In step S5264, the indicator function accumulated value is multiplied by the multi-model collaborative reward coefficient γ to obtain the indicator function accumulated correction value, which can be expressed as: The multi-model collaborative reward coefficient γ is generally in the range of 0.05-0.1.

[0089] Step S5265: The sum of the maximum initial confidence and the indicator function cumulative correction value is used as the weighted confidence of the new detection box.

[0090] The present invention also discloses a multi-target detection model integration device, the structural diagram of which is shown in FIG. Figure 2 As shown, a method for implementing a multi-target detection model integration method for fir trees is provided. In a preferred embodiment, the device includes:

[0091] The first list construction module obtains information of all detection boxes output after two or more target detection models process the image to be detected, where the information of each detection box includes an initial confidence level; and constructs a first list, where the first list is initialized to include all detection boxes.

[0092] The second list construction module constructs the second list, and the second list is initialized to be empty.

[0093] The weight determination module determines the global importance weight of each target category and the dynamic weight of each target detection model for each target category.

[0094] The confidence weighting module uses the global importance weight of the target category to which detection box i belongs and the dynamic weight of the target category to which detection box i belongs by the target detection model of the output detection box i to weight the initial confidence of detection box i and obtain the weighted confidence of detection box i, where i is the detection box index.

[0095] The detection frame fusion output module loops through the following steps until the first list is empty, and when the first list is empty, outputs the detection frames included in the second list: removing the detection frame with the largest weighted confidence from the first list and adding the removed detection frame to the second list; fusing one or more detection frames in the first list whose intersection-over-union ratio with the removed detection frame meets a preset overlap condition to obtain a new detection frame, and adding the new detection frame to the first list.

[0096] The present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned multi-target detection model integration method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0097] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor; wherein,

[0098] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-target detection model integration method provided by the present invention.

[0099] like Figure 3 FIG2 is a schematic diagram of the structure of an electronic device for implementing a multi-target detection model integration method according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a multi-target detection model integration method program.

[0100] Among them, in some embodiments, the processor 10 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits packaged with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device. It executes or executes programs or modules stored in the memory 11 (for example, executing a multi-target detection model integration method, etc.), and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0101] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device. Furthermore, the memory 11 can also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the multi-target detection model integration method program, but can also be used to temporarily store data that has been output or is to be output.

[0102] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0103] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0104] Figure 3 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0105] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0106] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0107] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0108] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. Multi-target detection model integration method, characterized by: include: Obtain information about all detection boxes output by two or more object detection models after processing the image to be detected, where the information about each detection box includes an initial confidence level. Construct the first list, which is initialized to include all detection boxes; Construct the second list, which is initially empty; Determine the global importance weight of each target category; Determine the dynamic weight of each object detection model for each object category; The global importance weight of the target category to which detection box i belongs and the dynamic weight of the target category to which detection box i belongs by the target detection model of the output detection box i are used to weight the initial confidence of detection box i to obtain the weighted confidence of detection box i, where i is the detection box index; Loop through the following steps until the first list is empty, and when the first list is empty, output the detection boxes included in the second list: Remove the detection box with the largest weighted confidence from the first list and add the removed detection box to the second list; Fuse one or more detection frames in the first list whose intersection-over-union ratios with the removed detection frames meet a preset overlap condition to obtain a new detection frame, and add the new detection frame to the first list.

2. The multi-target detection model integration method according to claim 1, wherein: Determining the dynamic weight of each target detection model for each target category includes: Get the average precision of each target detection model for each target category; Select the maximum value among the average precisions of target category j' by two or more target detection models, and use the selected maximum value as the maximum average precision of target category j', where j' is the target category index; The dynamic weight of each object detection model for target category j' is obtained by dividing the average precision of each object detection model for target category j' by the maximum average precision of target category j'.

3. The multi-target detection model integration method according to claim 1, wherein: Determining the dynamic weight of each target detection model for each target category includes: Acquire test data of a current round, where the test data of the current round is obtained by testing two or more object detection models using a test sample set of the current round; Obtain the current round dynamic weight increment of each target detection model for each target category based on the current round test data; Calculate the sum of the current round dynamic weight increment of each target detection model for each target category and the dynamic weight of each target detection model for each target category in the previous round, and use the sum as the dynamic weight of each target detection model for each target category in the current round.

4. The multi-target detection model integration method according to any one of claims 1 to 3, characterized in that: The step of fusing one or more detection frames in the first list whose intersection-over-union ratios with the removed detection frames satisfy a preset overlap condition to obtain a new detection frame includes: assembling an overlapping detection frame set, the overlapping detection frame set including one or more detection frames in the first list whose intersection-over-union ratio with the removed detection frame satisfies a preset overlapping condition; Determine a position weight for each detection frame in the set of overlapping detection frames, where the position weight of detection frame i is the product of a dynamic weight of the target category to which detection frame i belongs, output by the object detection model for detection frame i, and an initial confidence score of detection frame i; Sum the position weights of the detection boxes in the set of overlapping detection boxes to obtain the cumulative sum of the position weights; Perform weighted summation on the position information of the detection frames in the set of overlapping detection frames based on the position weights of the detection frames to obtain a weighted sum of the detection frame positions; The weighted sum of the detection frame positions is divided by the accumulated sum of the position weights to obtain the new detection frame position information; The initial confidence of the detection frame in the set of overlapping detection frames is fused to obtain the weighted confidence of the new detection frame; The target category of the detection frame with the largest initial confidence in the set of overlapping detection frames is used as the target category of the new detection frame.

5. The multi-target detection model integration method according to claim 4, characterized in that: The forming of the overlapping detection frame set includes: Obtaining an intersection-over-union ratio threshold between each detection box in the first list that has not been removed and the removed detection box; Calculate the intersection-of-union (IoU) of each detection box in the first list that has not been removed with the removed detection box. If the IoU is greater than the IoU threshold of the detection box and the removed detection box, add the detection box to the set of overlapping detection boxes.

6. The multi-target detection model integration method according to claim 5, characterized in that: Obtaining the intersection-over-union (IoU) threshold of each detection box in the first list that has not been removed and the removed detection box, including: Obtain the dynamic weight of the target type to which the target detection model of each detection box that has not been removed in the output first list belongs, recorded as the first dynamic weight; Obtain the dynamic weight of the target detection model that outputs the removed detection box for the target type to which the removed detection box belongs, recorded as the second dynamic weight; Obtaining the absolute value of the difference between the first dynamic weight and the second dynamic weight, and recording it as the absolute value of the weight difference; Multiply the absolute value of the weight difference by the weight difference sensitivity coefficient to obtain the intersection-over-union ratio threshold adjustment amount; The IoU threshold of each detection box not removed in the first list and the removed detection box is obtained by subtracting the IoU threshold adjustment amount from the base IoU threshold.

7. The multi-target detection model integration method according to claim 4, wherein: The initial confidences of the detection frames in the fused set of overlapping detection frames are used to obtain the weighted confidence of the new detection frame, including: Extract the maximum initial confidence from the initial confidences corresponding to the detection frames of the overlapping detection frame set; Obtaining an indicator function value for each detection frame in the set of overlapping detection frames based on a comparison result of the initial confidence of each detection frame in the set of overlapping detection frames with a preset confidence filtering threshold; Sum the indicator function values ​​of the detection boxes in the set of overlapping detection boxes to obtain the indicator function accumulated value; Multiplying the indicator function cumulative value by the multi-model collaborative reward coefficient to obtain the indicator function cumulative correction value; The sum of the maximum initial confidence and the cumulative correction value of the indicator function is used as the weighted confidence of the new detection box.

8. A multi-target detection model integration device, characterized in that: A method for implementing a multi-target detection model integration method according to any one of claims 1 to 7, comprising: A first list construction module is configured to obtain information about all detection boxes output by two or more object detection models after processing the image to be detected, wherein the information about each detection box includes an initial confidence level; and to construct a first list, wherein the first list is initialized to include all detection boxes. The second list construction module constructs the second list, and the second list is initialized to be empty; The weight determination module determines the global importance weight of each target category; determines the dynamic weight of each target detection model for each target category; The confidence weighting module uses the global importance weight of the target category to which the detection box i belongs and the dynamic weight of the target category to which the target detection model outputs the detection box i to weight the initial confidence of the detection box i to obtain the weighted confidence of the detection box i, where i is the detection box index; The detection frame fusion output module loops through the following steps until the first list is empty, and when the first list is empty, outputs the detection frames included in the second list: removing the detection frame with the largest weighted confidence from the first list and adding the removed detection frame to the second list; fusing one or more detection frames in the first list whose intersection-over-union ratio with the removed detection frame meets a preset overlap condition to obtain a new detection frame, and adding the new detection frame to the first list.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multi-target detection model integration method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-target detection model integration method according to any one of claims 1 to 7.

Images