Target object detection method and device, electronic equipment, storage medium and product

Abstract

Embodiments of the present application provide a target object detection method and device, electronic equipment, storage medium and product. The method comprises: acquiring a plurality of image frames; for any image frame, determining the confidence of the boundary box corresponding to each target object in the image frame; for any first target object, at least one second target object is selected from the target objects other than the first target object, and the detection probability of the second target object is greater than a first preset threshold on the premise that the first target object is detected, and the first target object is a target object with a confidence greater than or equal to a preset confidence; the confidence corresponding to each second target object is improved to obtain an improved confidence; according to the current confidence corresponding to each target object, at least one object to be presented is determined from the plurality of target objects, and the boundary box corresponding to each object to be presented is presented to the user, so as to improve the efficiency and accuracy of detecting target objects.

Description

Technical Field

[0001] This application relates to the field of target detection, and more particularly to a method, apparatus, electronic device, storage medium, and product for detecting target objects. Background Technology

[0002] Substations contain numerous pieces of equipment, most of which operate under energized conditions. Any defect can potentially lead to safety issues. Therefore, ensuring the integrity of the equipment and monitoring its status are crucial tasks in the daily operation of substations.

[0003] However, existing inspection methods mainly rely on fixed-point cameras and inspection robots to photograph the equipment's exterior. While this provides basic image data, it still requires manual identification and analysis. However, manual identification and analysis of image data is typically time-consuming, especially when dealing with large amounts of equipment and image data. This can lead to inefficiency, a high probability of missed detections, and the identification process heavily depends on the experience and expertise of the inspection personnel, potentially resulting in misjudgments of equipment status and incorrect maintenance decisions. Summary of the Invention

[0004] This application provides a target object detection method, apparatus, electronic device, storage medium, and product to improve the efficiency and accuracy of target object detection.

[0005] In a first aspect, embodiments of this application provide a target object detection method, including:

[0006] Acquire multiple image frames obtained by capturing images of the device to be inspected;

[0007] For any image frame among multiple image frames, determine the confidence level of the bounding box corresponding to each target object in the image frame;

[0008] For any first target object, at least one second target object is selected from target objects other than the first target object, wherein, provided that the first target object is detected, the probability of the second target object being detected is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level;

[0009] For any second target object, the confidence level corresponding to the second target object is increased to obtain the increased confidence level;

[0010] Based on the improved confidence levels corresponding to each second target object, and the confidence levels corresponding to each first target object and each third target object, at least one object to be presented is determined from multiple target objects, and the bounding boxes corresponding to each object to be presented are presented to the user. The third target object is a target object other than the second target object and the first target object.

[0011] In one possible implementation, selecting at least one second target object from target objects other than the first target object includes:

[0012] By querying the co-occurrence matrix, the detection probability of each fourth target object is determined under the premise that the first target object is detected; the co-occurrence matrix is ​​used to indicate the detection probability of each target object under the premise that any target object is detected; the fourth target object is a target object other than the first target object.

[0013] For any fourth target object, if the detection probability of the fourth target object is greater than a first preset threshold, the fourth target object is determined to be the second target object.

[0014] In one possible implementation, the confidence level corresponding to the second target object is increased to obtain an increased confidence level, including:

[0015] Based on the detection probability of the second target object and the first difference, the confidence level increase value corresponding to the second target object is determined; the first difference is used to indicate the difference between the confidence level corresponding to the first target object and the confidence level corresponding to the second target object.

[0016] The increased confidence level is determined based on the increase in confidence level corresponding to the second target object.

[0017] In one possible implementation, determining the confidence boost value corresponding to the second target object based on the detection probability of the second target object and the first difference includes:

[0018] The confidence boost value corresponding to the second target object is determined according to the following formula:

[0019] Δscore i =α·a(h i ,h j )·(score j -score i )

[0020] Wherein, Δscore i This represents the confidence level increase, where α is the adjustment factor, ranging from 0 to 1. α(h)i h j The score represents the probability of detecting the second target object given that the first target object has been detected. j The score represents the confidence level corresponding to the first target object. i This indicates the confidence level corresponding to the second target object.

[0021] In one possible implementation, based on the enhanced confidence levels corresponding to each second target object, and the confidence levels corresponding to each first target object and each third target object, at least one object to be presented is determined from a plurality of target objects, including:

[0022] The improved confidence, coordinate range, and category of the bounding boxes corresponding to each second target object, and the confidence, coordinate range, and category of the bounding boxes corresponding to each first and third target object are input into the trained confidence adjustment model to obtain the adjusted confidence value for each target object.

[0023] Based on the current and adjusted confidence values ​​of each target object, at least one object to be presented is determined from multiple target objects.

[0024] In one possible implementation, the method further includes:

[0025] A training sample set is constructed, wherein each training sample in the training sample set includes: the confidence score, coordinate range, and category of the predicted bounding box corresponding to each target object in a certain image frame; the label corresponding to the training sample is: the true adjusted value of the confidence score of the predicted bounding box corresponding to each target object; the true adjusted value is determined according to the following formula:

[0026] Δscore = β(IoU - score)

[0027] Wherein, Δscore is used to indicate the true adjusted value of the confidence score, β is an adjustment factor used to control the magnitude of the adjustment, IoU is used to indicate the intersection-union ratio of the predicted bounding box and the true bounding box, and the IoU is determined based on the coordinate range of the predicted bounding box and the coordinate range of the true bounding box, and score is used to indicate the confidence score of the predicted bounding box.

[0028] The training sample set is input into the initial confidence adjustment model to obtain the predicted adjusted confidence value for each training sample.

[0029] For any training sample, the loss value is determined based on the predicted adjustment value and the actual adjustment value of the confidence level corresponding to the training sample.

[0030] The parameters of the initial confidence adjustment model are adjusted based on the loss value corresponding to each training sample.

[0031] In one possible implementation, for each training sample, a loss value is determined based on the predicted adjustment value and the actual adjustment value of the confidence level corresponding to the training sample, including:

[0032] Based on the predicted and actual adjusted confidence values ​​corresponding to the training samples, co-occurrence loss, and / or confidence loss, and / or ranking loss are determined; the ranking loss is used to measure the difference between confidence ranking and intersection-over-union (IoU) ranking, the IoU ranking is used to indicate the ranking of the IoU ratios of the predicted bounding boxes and the actual bounding boxes corresponding to each target object, and the co-occurrence loss is used to indicate the difference between the occurrence probability of a target object determined by the co-occurrence matrix and the confidence value corresponding to the target object; the co-occurrence matrix is ​​used to indicate the detection probability of each target object given that any target object is detected.

[0033] The loss value is determined based on co-occurrence loss, and / or confidence loss, and / or ranking loss.

[0034] In one possible implementation, the co-occurrence loss is determined based on the predicted and actual adjusted values ​​of the confidence levels corresponding to the training samples, including:

[0035] For any target object in the training samples, the adjusted confidence level of the target object is determined based on the confidence level of the target object and the predicted adjusted value of the confidence level; the joint co-occurrence probability of the target object in the current detection result is determined according to the following formula;

[0036]

[0037] Where p(label) i |Context) represents the joint co-occurrence probability corresponding to target object i, Context is the set of target objects in the training samples, n represents the number of target objects in the training samples, and p(label) i |label j ) represents the probability of target object i being detected, assuming target object j is detected.

[0038] Based on the adjusted confidence level and joint co-occurrence probability of each target object, the co-occurrence loss is determined by the following formula;

[0039]

[0040] in, The adjusted confidence level corresponds to the target object i. l(x) is an indicator function that takes the value 1 when x>0 and zero otherwise. θ represents the second preset threshold.

[0041] In one possible implementation, the confidence loss is determined based on the predicted and actual adjusted values ​​of the confidence scores corresponding to the training samples, including:

[0042] The confidence loss is determined using the following formula:

[0043]

[0044] Where l(x) is the indicator function, and the IoU i Used to indicate the intersection-union ratio (IUGR) of the predicted bounding box and the true bounding box of target object i; represents the adjusted confidence level corresponding to target object i, and n represents the number of target objects in the training samples.

[0045] In one possible implementation, the ranking loss is determined based on the predicted and actual adjusted values ​​of the confidence levels corresponding to the training samples, including:

[0046] The sorting loss is determined using the following formula:

[0047]

[0048] Where H(x) is the Herveside step function, This represents the adjusted confidence level corresponding to target object i. The IoU represents the adjusted confidence level corresponding to target object j. i The IoU is used to indicate the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box of target object i. j The cross-union ratio (CUP) is used to indicate the predicted bounding box and the ground truth bounding box of target object j; n represents the number of target objects in the training samples.

[0049] In one possible implementation, determining the confidence level of the bounding box corresponding to each target object in the image frame includes:

[0050] The image frame is input into the improved YOLOv8 model to determine the confidence level of the bounding box corresponding to each target object in the image frame. The improved YOLOv8 model includes a C3STR module and an SPPCSPC module.

[0051] Secondly, embodiments of this application provide a target object detection device, comprising:

[0052] The acquisition module is used to acquire multiple image frames obtained by capturing images of the device under test;

[0053] The first determining module is used to determine the confidence level of the bounding box corresponding to each target object in any one of the multiple image frames;

[0054] The filtering module is used to filter at least one second target object from target objects other than the first target object for any first target object, wherein, provided that the first target object is detected, the probability of the second target object being detected is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level.

[0055] The boosting module is used to boost the confidence level of any second target object to obtain the boosted confidence level.

[0056] The second determining module is used to determine at least one object to be presented from multiple target objects based on the improved confidence levels corresponding to each second target object, the confidence levels corresponding to each first target object and each third target object, and to present the bounding boxes corresponding to each object to be presented to the user. The third target object is a target object other than the second target object and the first target object.

[0057] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0058] The memory stores computer-executed instructions;

[0059] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0060] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0061] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0062] The target object detection method, apparatus, electronic device, storage medium, and product provided in this application embodiment acquire multiple image frames captured by a device to be detected; for any image frame, determine the confidence level of the bounding box corresponding to each target object in the image frame; for any first target object, select at least one second target object from target objects other than the first target object, wherein, provided that the first target object is detected, the detection probability of the second target object is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level; for any second target object, determine the confidence level of the bounding box corresponding to the second target object. The confidence level is improved to obtain the improved confidence level. Based on the improved confidence levels corresponding to each second target object, and the confidence levels corresponding to each first and third target object, at least one object to be presented is determined from multiple target objects. The bounding boxes corresponding to each object to be presented are then shown to the user. The third target object is any target object other than the second and first target objects. By selecting second target objects with a high correlation to the first target object (i.e., given that the first target object is detected, the probability of detecting the second target object is greater than a certain threshold), and increasing the confidence level of the second target object, the correlation between target objects can be used to improve detection accuracy. This method can effectively reduce missed detections and improve recognition efficiency and accuracy. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0064] Figure 1 An application scenario diagram provided for an embodiment of this application;

[0065] Figure 2 A flowchart illustrating a target object detection method provided in an embodiment of this application;

[0066] Figure 3 This application provides a schematic diagram of the structure of a C3STR module according to an embodiment.

[0067] Figure 4 This application provides a schematic diagram of the structure of an SPPC module.

[0068] Figure 5 A schematic diagram of the backbone network structure of a YOLOv8 model provided in this application embodiment;

[0069] Figure 6 A schematic diagram of the backbone network structure of an improved YOLOv8 model provided in this application embodiment;

[0070] Figure 7 A flowchart illustrating another target object detection method provided in an embodiment of this application;

[0071] Figure 8 A schematic diagram of the target object detection device provided in this application;

[0072] Figure 9 A schematic diagram of the structure of the electronic device provided in this application.

[0073] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] Substations contain numerous pieces of equipment, most of which operate under energized conditions. Any defect can potentially lead to safety issues. Therefore, ensuring the integrity of equipment and monitoring its status are crucial tasks in the daily operation of substations. To ensure the stable operation of the power system, substations need to conduct regular inspections and maintenance of equipment to promptly identify and resolve potential problems, preventing power outages or other safety hazards caused by equipment failures.

[0076] However, existing inspection methods primarily rely on fixed-point cameras and inspection robots to photograph the equipment's exterior. While this provides basic image data, manual identification and analysis are still essential. Although this traditional inspection method improves automation to some extent, it still has several shortcomings. First, fixed-point cameras, limited by their installation location, may not be able to fully cover all critical parts of the equipment, leading to some potential hazards going undetected. Second, the movement paths and shooting angles of inspection robots may be limited by the environment and equipment layout, making them unable to flexibly handle complex on-site situations.

[0077] Manually identifying and analyzing image data is typically time-consuming, especially when dealing with large amounts of equipment and image data, which can lead to inefficiency and a high probability of missed detections. Even experienced inspectors are susceptible to fatigue and lapses in concentration during long hours of intense work, increasing the risk of missed detections and misjudgments. Furthermore, the identification process heavily relies on the experience and expertise of the inspectors, which may result in misjudgments of equipment status and incorrect maintenance decisions. This can not only lead to unnecessary increased maintenance costs but also potentially cause more serious consequences due to the failure to address genuine equipment malfunctions in a timely manner.

[0078] In view of this, this application provides a target object detection method. The method involves capturing images of a device to be detected, obtaining multiple image frames, determining the confidence level of the bounding box corresponding to each target object in each image frame, identifying target objects with a confidence level greater than or equal to a preset confidence level as first target objects, and for each first target object, selecting second target objects from among the target objects other than the first target object. Given that the first target object is detected, the detection probability of the second target object is greater than a first preset threshold. Then, the confidence level of each second target object is increased. Finally, based on the increased confidence levels of each second target object, and the confidence levels of each first target object and each third target object, at least one object to be presented is determined from the multiple target objects, and the bounding boxes corresponding to each object to be presented are presented to the user. The third target object is a target object other than the second and first target objects. Thus, by selecting second target objects with a high correlation to the first target object (i.e., given that the first target object is detected, the detection probability of the second target object is greater than a certain threshold), and increasing the confidence level of the second target object, the correlation between target objects can be utilized to improve detection accuracy. This method can effectively reduce missed detections and improve identification efficiency and accuracy.

[0079] Figure 1 An application scenario diagram provided for an embodiment of this application, such as... Figure 1As shown, the camera first captures images of the device to be detected, obtaining multiple image frames, and then sends these image frames to the server. The server first determines the confidence level of the bounding box corresponding to each target object in each image frame. For example, the image frame contains target object 1, target object 2, target object 3, and target object 4. The confidence levels of the bounding boxes corresponding to the four target objects are 0.8, 0.4, 0.5, and 0.2, respectively. The preset confidence level is 0.7, so target object 1 is determined to be the first target object. It is known that if target object 1 is detected, target object 2, target object 3, and target object 4 are also detected. The detection probabilities of object 3 and target object 4 are 0.3, 0.4 and 0.8 respectively. The first preset threshold is 0.6. Since 0.8 > 0.6, target object 4 is determined to be the second target object. The confidence level of target object 4 is increased. Finally, based on the confidence levels of target objects 1, 2 and 3 and the increased confidence level of target object 4, the object to be presented is determined from target objects 1, 2, 3 and 4. The image frame containing the bounding box of the object to be presented is sent to the user terminal. The user terminal will present the image frame containing the bounding box of the object to be presented to the user on the display interface.

[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Figure 2 This is a flowchart illustrating a target object detection method provided in an embodiment of this application. The execution entity in this embodiment can be any device with data processing capabilities. This application uses a server as the execution entity for specific description, such as... Figure 2 As shown in the embodiments of this application, a target object detection method may include:

[0082] Step 201: Acquire multiple image frames obtained by capturing images of the device to be tested.

[0083] Specifically, the camera takes pictures of the device to be inspected, obtains multiple image frames, and sends the multiple image frames to the server, which then receives the multiple image frames.

[0084] Step 202: For any image frame among multiple image frames, determine the confidence level of the bounding box corresponding to each target object in the image frame.

[0085] The target object to be detected can be equipment or defects on the equipment. For example, in a substation scenario, when the target object is equipment, it can be the main transformer equipment, such as oil tanks, submersible pumps, bushings, etc., or other equipment, such as insulators, equalizing rings, grounding down conductors, etc. When the target object is defects on the equipment, it can be oil leakage defects corresponding to the main transformer equipment such as oil tanks, submersible pumps, bushings, or damage and deformation defects of other equipment such as insulators, equalizing rings, grounding down conductors.

[0086] Specifically, for any image frame among multiple image frames, the server can determine the confidence level of the bounding box corresponding to each target object in that image frame. The bounding box indicates the location of the target object, and the confidence level of the bounding box typically refers to the degree of confidence that the target object is indeed contained within that bounding box. Specifically, the confidence score usually consists of two parts:

[0087] The first is the existence probability, which predicts the probability that a target object exists within the bounding box. This value is usually between 0 and 1, where 1 indicates that it is very certain that a target object exists in the region, and 0 indicates that it is believed that no target object exists in the region.

[0088] The second is the class probability, which predicts the probability that the target object belongs to a specific class. For multi-class detection tasks, a probability value is provided for each possible class.

[0089] Combining these two parts, the confidence level of a bounding box is usually expressed as the product of the existence probability and the class probability.

[0090] Optionally, determining the confidence level of the bounding box corresponding to each target object in the image frame includes:

[0091] The image frame is input into the improved YOLOv8 model to determine the confidence level of the bounding box corresponding to each target object in the image frame. The improved YOLOv8 model includes a C3STR module and an SPPCSPC module.

[0092] The C3STR module stands for C3 with Self-attention Transformer module, meaning a C3 module with a self-attention mechanism. C3STR consists of a standard C3 module and an embedded self-attention Transformer. The C3 module is responsible for extracting efficient local features, while the Transformer module can capture long-range contextual information, compensating for the limitations of convolutional networks.

[0093] The input feature map is X, and the output after passing through C3STR is Y. The calculation process is as follows:

[0094] Y = C3(X) + Transformer(X)

[0095] The C3 module performs computations following traditional convolution, while the Transformer part can be implemented using a self-attention mechanism, as detailed below:

[0096]

[0097] Where Q (query), K (key), and V (value) are generated from the input features, d k This is the dimension scaling factor.

[0098] Figure 3 This application provides a schematic diagram of the structure of a C3STR module, as shown in the embodiment. Figure 3 As shown, the input feature map is passed through the first branch and the second branch to obtain the first feature map and the second feature map, respectively. The first branch includes a CBS combination module and a Swin Transformer module, and the second branch includes a CBS combination module. The first feature map and the second feature map are input into the concatenation layer to obtain the concatenation result. The concatenation result is then passed through the CBS combination module, the batch normalization layer, and the activation layer in sequence to obtain the final output feature map.

[0099] The full name of the SPPCSPC module is Spatial Pyramid Pooling with Cross Stage Partial Connections module. The SPPCSPC module includes multiple parallel pooling layers, which acquire multi-scale information through different kernel sizes (3x3, 5x5, 9x9), and then fuse these features through CSPC.

[0100] The output features of SPPCSPC are:

[0101] Y = CSPC(Concat(MaxPool) 3×3 (X),MaxPool 5×5 (X),MaxPool 9×9 (X)))

[0102] Among them, MaxPool n×n (X) indicates the use of max pooling with a kernel size of n*n, Concat (concatenation) indicates concatenating features of different scales, and CSPC is responsible for reducing redundant information and improving computational efficiency during the fusion process.

[0103] Figure 4 This application provides a schematic diagram of the structure of an SPPC module, as shown in the embodiment. Figure 4As shown, the input feature map first passes through three CBS combining modules to obtain the first feature map. The first feature map then passes through three max pooling layers to obtain three second feature maps. The three second feature maps and the first feature map are then input into a concatenation layer to obtain the third feature map. The third feature map passes through two CBS combining modules to obtain the fourth feature map. The initial input feature map and the fourth feature map are then input into the concatenation layer, and the output of the concatenation layer is input into the CBS combining module to obtain the final output feature map.

[0104] Figure 5 A schematic diagram of the backbone network structure of a YOLOv8 model provided in this application embodiment is shown below. Figure 5 As shown, the backbone network of the YOLOv8 model consists of six functional modules from top to bottom: P1, P2, P3, P4, P5, and P6. P1 includes one convolutional layer (Conv), P2, P3, P4, and P5 each include one convolutional layer (Conv) and a C2f module, and P6 includes one SPPF module. When a feature image passes through any of the functional modules P2, P3, P4, or P5, it first passes through the convolutional layer and then through the C2f module. The numbers 0, 1, 3, 5, 7, and 9 above P1, P2, P3, P4, P5, and P6 in the diagram correspond to the accumulated layer numbers.

[0105] Figure 6 A schematic diagram of the backbone network structure of an improved YOLOv8 model provided in this application embodiment is shown below. Figure 6 As shown, the improved YOLOv8 model's backbone network comprises seven functional modules from top to bottom: P1, P2, P3, P4, P5, P6, and P7. P1 includes a Focus layer, while P2, P3, P4, P5, and P6 each include a convolutional layer (Conv) and a C3STR module. P7 includes an SPPFSPC module. When a feature image passes through any of the functional modules P2, P3, P4, P5, or P6, it first passes through the convolutional layer and then the C3STR module. The numbers 0, 1, 3, 5, 7, 9, and 11 above P1, P2, P3, P4, P5, P6, and P7 in the diagram correspond to the accumulated layer numbers.

[0106] Specifically, the server inputs the image frame into the improved YOLOv8 model to determine the confidence level of the bounding boxes corresponding to each target object in the image frame. The improved YOLOv8 model includes the C3STR module and the SPPCSPC module.

[0107] Thus, the introduction of the C3STR module improves the YOLOv8 model's ability to capture long-range dependencies, especially demonstrating stronger robustness against small defects in complex scenarios. The SPPCSPC module further improves the model's detection accuracy and speed through multi-scale feature extraction and efficient information fusion, significantly reducing computational overhead, particularly during real-time inference on resource-constrained devices.

[0108] Step 203: For any first target object, select at least one second target object from the target objects other than the first target object, wherein, on the premise that the first target object is detected, the detection probability of the second target object is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level.

[0109] Specifically, the server determines the target object with a confidence level greater than or equal to a preset confidence level as the first target object. For any first target object, at least one second target object is selected from the target objects other than the first target object. The selection criterion is that, given that the first target object is detected, the probability of being detected is greater than a first preset threshold.

[0110] It should be noted that the target objects here, excluding the first target object, refer to all target objects other than the first target objects.

[0111] Optionally, at least one second target object may be selected from the target objects other than the first target object, including:

[0112] By querying the co-occurrence matrix, the detection probability of each fourth target object is determined under the premise that the first target object is detected; the co-occurrence matrix is ​​used to indicate the detection probability of each target object under the premise that any target object is detected; the fourth target object is a target object other than the first target object.

[0113] For any fourth target object, if the detection probability of the fourth target object is greater than a first preset threshold, the fourth target object is determined to be the second target object.

[0114] Specifically, the server first needs to construct a co-occurrence matrix, and the specific steps are as follows:

[0115] 1. Calculate the co-occurrence frequency among different categories from the training data;

[0116] 2. For target categories A and B, the probability of target category A occurring given that target category B occurs can be calculated using the following formula:

[0117]

[0118] P(A|B) and P(B|A) have different meanings. According to the formula, the denominator of P(A|B) is the total number of times B appears, and the denominator of P(B|A) is the number of times A appears.

[0119] For example, target category A is a cooler and target category B is an oil tank. The number of times the cooler and the oil tank are detected simultaneously is 50, the total number of times the cooler is detected is 70, and the total number of times the oil tank is detected is 100.

[0120] P(A|B)=50 / 100=0.5, which means that if the oil tank (B) is detected, there is a 50% probability that the cooler (A) will also be detected at the same time.

[0121] P(B|A)=50 / 70=0.71, which means that under the condition that the cooler (A) is detected, there is a 71% probability that the oil tank (B) will also be detected at the same time.

[0122] The fact that P(A|B) is smaller than P(B|A) indicates that the scenario in which the oil tank (B) appears is not always accompanied by the cooler (A).

[0123] P(B|A) is larger than P(A|B), indicating that the oil conservator (B) is more likely to be present in the scenario where the cooler (A) is present.

[0124] 3. Construct a co-occurrence matrix based on the above calculation results. For example, the title of each row in the co-occurrence matrix can represent target category A, the title of each column in the co-occurrence matrix can represent target category B, and each element in the co-occurrence matrix can be represented as P(A|B), which is the probability of target category A appearing given that target category B appears, or the probability of target category A being detected given that target category B is detected.

[0125] Table 1 is a schematic table of a co-occurrence matrix provided in an embodiment of this application. As shown in Table 1, for example, if category 3 is detected, the probability of category 1 being detected is 0.2271, and if category 4 is detected, the probability of category 5 being detected is 0.

[0126] Table 1

[0127] Category 1 Category 2 Category 3 Category 4 Category 5 Category 1 1 0.2994 0.2957 0.3753 0 Category 2 0.3969 1 0.2426 0.5349 0 Category 3 0.2271 0.1372 1 0.047 0 Category 4 0.2991 0.3378 0.0525 1 0 Category 5 0 0 0 0 1

[0128] Specifically, by querying the co-occurrence matrix, the server can determine the detection probability of each fourth target object, assuming the first target object is detected; whereby the fourth target object is any target object other than the first target object. For any fourth target object, if the detection probability of the fourth target object is greater than a first preset threshold, the server determines that the fourth target object as the second target object.

[0129] Step 204: For any second target object, increase the confidence level corresponding to the second target object to obtain the increased confidence level.

[0130] Specifically, for any second target object, the server increases the confidence level corresponding to that second target object to obtain the increased confidence level.

[0131] Optionally, the confidence level corresponding to the second target object is boosted to obtain the boosted confidence level, including:

[0132] Based on the detection probability of the second target object and the first difference, the confidence level increase value corresponding to the second target object is determined; the first difference is used to indicate the difference between the confidence level corresponding to the first target object and the confidence level corresponding to the second target object.

[0133] The increased confidence level is determined based on the increase in confidence level corresponding to the second target object.

[0134] Specifically, the server determines the boost value of the confidence level of the second target object based on the probability of the second target object being detected given that the first target object is detected, and the difference between the confidence level of the first target object and the confidence level of the second target object; the boost value is added to the confidence level of the second target object to obtain the boosted confidence level.

[0135] In this way, by dynamically adjusting the confidence level based on the difference between the detection probability and the confidence level, the accuracy of the confidence level adjustment can be improved.

[0136] Optionally, based on the detection probability of the second target object and the first difference, the confidence enhancement value corresponding to the second target object is determined, including:

[0137] The confidence boost value corresponding to the second target object is determined according to the following formula:

[0138] Δscore i =α·a(h i ,h j )·(score j -score i )

[0139] Wherein, Δscore i This represents the confidence level increase, where α is the adjustment factor, ranging from 0 to 1. α(h) i h j The score represents the probability of detecting the second target object given that the first target object has been detected. jThe score represents the confidence level corresponding to the first target object. i This indicates the confidence level corresponding to the second target object.

[0140] For example, the confidence score of the first target object with the category "oil storage tank" is 1. j =0.8; the confidence score of the second target object with the category "cooler" is 0.8. i =0.6; According to the co-occurrence matrix, P(cooler|oil tank) = 0.7, that is, α(h i h j The value is 0.7, and the adjustment factor α is 0.5.

[0141] Substituting into the calculation formula, we can see that: Δscore i =0.5*0.7*(0.8-0.6)=0.07, that is, the confidence level of the second target object cooler is increased by 0.07, and the increased confidence level of the second target object is 0.6+0.07=0.67.

[0142] Thus, given that the first target object is detected, the higher the probability of the second target object being detected, the greater the confidence boost; or the greater the difference in confidence between the first and second target objects, the greater the confidence boost, which can improve the accuracy of the boost determination.

[0143] Step 205: Based on the improved confidence levels of each second target object, and the confidence levels of each first target object and each third target object, determine at least one object to be presented from the multiple target objects, and present the bounding boxes corresponding to each object to be presented to the user. The third target object is a target object other than the second target object and the first target object.

[0144] This application does not limit the specific method by which at least one object to be presented is determined from a plurality of target objects.

[0145] In one possible implementation, the server determines the target object to be presented as an object based on the enhanced confidence level corresponding to each second target object, the confidence level corresponding to each first target object, and the confidence level corresponding to each third target object, i.e., the current confidence level corresponding to each target object. If the current confidence level corresponding to any target object exceeds a preset threshold, the target object is then sent to the user terminal as an image frame containing the bounding box corresponding to the target object. The user terminal then displays the image frame containing the bounding box corresponding to the target object on the display interface.

[0146] The target object detection method provided in this application can acquire multiple image frames captured by a device to be detected; for any image frame, determine the confidence level of the bounding box corresponding to each target object in the image frame; for any first target object, filter at least one second target object from target objects other than the first target object, wherein, provided that the first target object is detected, the detection probability of the second target object is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level; for any second target object, increase the confidence level corresponding to the second target object to obtain... The improved confidence level is used to determine at least one target object from multiple target objects based on the improved confidence levels of each second target object, each first target object, and each third target object. The bounding boxes corresponding to each target object are then presented to the user. The third target object is any target object other than the second and first target objects. By selecting second target objects with a high correlation to the first target object (i.e., given that the first target object is detected, the probability of detecting the second target object is greater than a certain threshold), and increasing the confidence level of the second target object, the correlation between target objects can be used to improve detection accuracy. This method effectively reduces missed detections and improves recognition efficiency and accuracy.

[0147] Optionally, based on the enhanced confidence levels corresponding to each second target object, and the confidence levels corresponding to each first target object and each third target object, at least one object to be presented is determined from the multiple target objects, including:

[0148] The improved confidence, coordinate range, and category of the bounding boxes corresponding to each second target object, and the confidence, coordinate range, and category of the bounding boxes corresponding to each first and third target object are input into the trained confidence adjustment model to obtain the adjusted confidence value for each target object.

[0149] Based on the current and adjusted confidence values ​​of each target object, at least one object to be presented is determined from multiple target objects.

[0150] The confidence adjustment model can be any model capable of confidence adjustment. For example, it can be a transformer model. The input of the confidence adjustment model is the improved confidence, coordinate range, and category of the bounding box corresponding to each second target object, and the confidence, coordinate range, and category of the bounding box corresponding to each first target object and each third target object. That is, the current value of the confidence, coordinate range, and category of the bounding box corresponding to each target object. The output is the adjusted value of the confidence corresponding to each target object.

[0151] Specifically, the server inputs the current confidence value, coordinate range, and category of the bounding box corresponding to each target object into the trained confidence adjustment model to obtain the adjusted confidence value for each target object. For example, the adjusted value can be positive or negative. A positive adjusted value indicates that the current confidence value should be increased, and a negative adjusted value indicates that the current confidence value should be decreased.

[0152] For any target object, the current confidence value and the adjusted confidence value corresponding to the target object are added together to obtain the updated confidence value corresponding to the target object. When the updated confidence value is greater than a preset threshold, the target object is determined to be the object to be presented.

[0153] In this way, by using a trained confidence adjustment model, the confidence adjustment value of each target object can be evaluated more quickly and accurately, thereby improving the accuracy of the final selected objects to be presented.

[0154] Optionally, the target object detection method provided in this application further includes:

[0155] A training sample set is constructed, wherein each training sample in the training sample set includes: the confidence score, coordinate range, and category of the predicted bounding box corresponding to each target object in a certain image frame; the label corresponding to the training sample is: the true adjusted value of the confidence score of the predicted bounding box corresponding to each target object; the true adjusted value is determined according to the following formula:

[0156] Δscore = β(IoU - score)

[0157] Wherein, Δscore is used to indicate the true adjusted value of the confidence score, β is an adjustment factor used to control the magnitude of the adjustment, IoU is used to indicate the intersection-union ratio of the predicted bounding box and the true bounding box, and the IoU is determined based on the coordinate range of the predicted bounding box and the coordinate range of the true bounding box, and score is used to indicate the confidence score of the predicted bounding box.

[0158] The training sample set is input into the initial confidence adjustment model to obtain the predicted adjusted confidence value for each training sample.

[0159] For any training sample, the loss value is determined based on the predicted adjustment value and the actual adjustment value of the confidence level corresponding to the training sample.

[0160] The parameters of the initial confidence adjustment model are adjusted based on the loss value corresponding to each training sample.

[0161] In this way, the difference between the IoU (Intersection over Union) and the confidence level is considered when determining the true adjusted value of the confidence level, which can effectively reduce the error of the confidence level. Especially when the confidence level is too high but the positioning is inaccurate, the adjusted value of the confidence level will be relatively large, thereby reducing the possibility of misjudgment. Furthermore, the introduction of an adjustment factor provides a flexible mechanism to control the magnitude of the adjustment.

[0162] Optionally, for each training sample, a loss value is determined based on the predicted adjusted value and the actual adjusted value of the confidence level corresponding to the training sample, including:

[0163] Based on the predicted and actual adjusted confidence values ​​corresponding to the training samples, co-occurrence loss, and / or confidence loss, and / or ranking loss are determined; the ranking loss is used to measure the difference between confidence ranking and intersection-over-union (IoU) ranking, the IoU ranking is used to indicate the ranking of the IoU ratios of the predicted bounding boxes and the actual bounding boxes corresponding to each target object, and the co-occurrence loss is used to indicate the difference between the occurrence probability of a target object determined by the co-occurrence matrix and the confidence value corresponding to the target object; the co-occurrence matrix is ​​used to indicate the detection probability of each target object given that any target object is detected.

[0164] The loss value is determined based on co-occurrence loss, and / or confidence loss, and / or ranking loss.

[0165] In this system, each training sample corresponds to multiple target objects. The target objects are first sorted from highest to lowest confidence level based on their corresponding bounding boxes. Then, they are second sorted from highest to lowest intersection-over-union (IoU) ratio. The sorting loss measures the difference between the two sorting processes. The IoU ratio for each target object is the intersection-over-union ratio between the predicted bounding box and the ground truth bounding box.

[0166] Co-occurrence loss can represent the difference between the probability of occurrence of a target object determined by the co-occurrence matrix and the confidence level corresponding to the target object.

[0167] Confidence loss is used to measure the loss of confidence.

[0168] In this way, by combining co-occurrence loss, confidence loss, and ranking loss, the performance of the model can be optimized from multiple dimensions. Each loss function focuses on different model characteristics, and using them in combination can more comprehensively improve the model's capabilities.

[0169] Optionally, the co-occurrence loss is determined based on the predicted and actual adjusted confidence values ​​corresponding to the training samples, including:

[0170] For any target object in the training samples, the adjusted confidence level of the target object is determined based on the confidence level of the target object and the predicted adjusted value of the confidence level; the joint co-occurrence probability of the target object in the current detection result is determined according to the following formula;

[0171]

[0172] Where p(label) i |Context) represents the joint co-occurrence probability corresponding to target object i, Context is the set of target objects in the training samples, n represents the number of target objects in the training samples, and p(label) i |label j ) represents the probability of target object i being detected, assuming target object j is detected.

[0173] Based on the adjusted confidence level and joint co-occurrence probability of each target object, the co-occurrence loss is determined by the following formula;

[0174]

[0175] in, The adjusted confidence level corresponds to the target object i. l(x) is an indicator function that takes the value 1 when x>0 and zero otherwise. θ represents the second preset threshold.

[0176] Specifically, for any target object in the training samples, the server adds the confidence level of the target object to the predicted adjusted value of the confidence level to obtain the adjusted confidence level corresponding to the target object; and determines the joint co-occurrence probability of the target object in the current detection result according to the following formula;

[0177]

[0178] Where p(label) i |Context) represents the joint co-occurrence probability corresponding to target object i, Context is the set of target objects in the training sample, n represents the number of target objects in the training sample, and p(label) i |label j Let be the probability of target object i being detected given that target object j is detected.

[0179] For example, the training sample includes three target objects: target object 1, target object 2, and target object 3. To find the joint co-occurrence probability of target object 1 in the current detection results, it is known that the detection probability of target object 1 is 0.7 when target object 2 is detected, the detection probability of target object 1 is 0.1 when target object 3 is detected, and the detection probability of target object 1 is 1 when target object 1 is detected. Therefore, the joint co-occurrence probability of target object 1 is (1 + 0.7 + 0.1) / 3 = 0.6.

[0180] Then, the server determines the co-occurrence loss based on the adjusted confidence level and joint co-occurrence probability of each target object using the following formula;

[0181]

[0182] in, The adjusted confidence level corresponds to the target object i. l(x) is an indicator function that takes the value 1 when x>0 and zero otherwise. θ represents the second preset threshold.

[0183] Thus, when the joint co-occurrence probability is greater than the second preset threshold, the confidence level is encouraged to increase, prompting the model to adjust the confidence level by increasing it when the joint co-occurrence probability is high. Conversely, when the joint co-occurrence probability is low, the model adjusts the confidence level by decreasing it. The purpose of this loss term is to encourage increasing the latent confidence level of the missed report category, thereby improving the accuracy of the determined confidence level adjustment value.

[0184] Optionally, the confidence loss is determined based on the predicted and actual adjusted values ​​of the confidence scores corresponding to the training samples, including:

[0185] The confidence loss is determined using the following formula:

[0186]

[0187] Where l(x) is the indicator function, and the IoU i Used to indicate the intersection-union ratio (IUGR) of the predicted bounding box and the true bounding box of target object i; represents the adjusted confidence level corresponding to target object i, and n represents the number of target objects in the training samples.

[0188] Thus, for false alarm detection boxes, the IoU is close to 0, and the confidence loss is determined by the above formula to reduce the probability of false alarms.

[0189] Optionally, the ranking loss is determined based on the predicted and actual adjusted values ​​of the confidence levels corresponding to the training samples, including:

[0190] The sorting loss is determined using the following formula:

[0191]

[0192] Where H(x) is the Herveside step function, This represents the adjusted confidence level corresponding to target object i. The IoU represents the adjusted confidence level corresponding to target object j. i The IoU is used to indicate the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box of target object i. j The cross-union ratio (CUP) is used to indicate the predicted bounding box and the ground truth bounding box of target object j; n represents the number of target objects in the training samples.

[0193] H(x) is used to measure whether the confidence ranking is consistent with the IoU ranking. If the ranking after confidence adjustment is inconsistent with the IoU ranking, the loss will increase, encouraging the model to maintain higher confidence in high-quality detection boxes.

[0194] This ensures higher confidence levels for high-quality detection frames, which are used to maintain or optimize the ranking of confidence levels, thereby improving the accuracy of the determined confidence adjustment values.

[0195] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0196] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0197] Figure 7 A flowchart illustrating another target object detection method provided in this application embodiment is shown below. Figure 7 As shown, the specific implementation process includes the following steps:

[0198] 1. Data Preparation

[0199] Images of substation equipment defects and status were acquired, manually labeled, and then divided into training and validation sets. Specifically, high-definition cameras and drones were used to collect images of equipment in the substation operating environment, covering the normal operating status and defects of various equipment. The collected images were manually labeled using LabelImg. To improve the model's recognition ability in complex environments, data augmentation processing was performed on the original image data, including random rotation, horizontal flipping, cropping, brightness adjustment, and adding Gaussian noise, thereby generating more diverse training samples and enhancing the model's robustness.

[0200] 2. Improve the YOLOv8 model for detection.

[0201] The improved YOLO model, after training, is used to extract features from the input image and directly predict the location, class, and confidence score of the detection boxes. The model outputs a series of detection results, each containing:

[0202] Category: The category label of the target;

[0203] Confidence score: The confidence score of the prediction;

[0204] Bounding box coordinates: [xmin, ymin, xmax, ymax].

[0205] Specifically, to enhance the feature extraction capabilities of the YOLOv8 model for substation equipment defects and status images, this invention introduces the C3STR (C3 with Self-attention Transformer) module and the SPPCSPC (Spatial Pyramid Pooling with Cross Stage Partial Connections) module into the YOLOv8 backbone network. These modules strengthen the network's feature extraction at multiple scales and its long-range dependency modeling, thereby improving the model's robustness and detection accuracy.

[0206] The C3STR module consists of a standard C3 module and an embedded self-attention Transformer. The C3 module is responsible for extracting efficient local features, while the Transformer module can capture long-range contextual information, compensating for the limitations of convolutional networks.

[0207] Calculation formula: The input feature map is X, and the output after passing through C3STR is Y. The calculation process is as follows:

[0208] Y = C3(X) + Transformer(X)

[0209] The C3 module performs computations following traditional convolution, while the Transformer part can be implemented using a self-attention mechanism, as detailed below:

[0210]

[0211] Where Q (query), K (key), and V (value) are generated from the input features, d k This is the dimension scaling factor.

[0212] The SPPC module includes multiple parallel pooling layers that acquire multi-scale information using different kernel sizes (3x3, 5x5, 9x9), and then fuse these features using CSPC.

[0213] Calculation formula: The output characteristics of SPPC are:

[0214] Y = CSPC(Concat(MaxPool) 3×3 (X),MaxPool 5×5 (X),MaxPool 9×9 (X)))

[0215] Among them, MaxPool n×n (X) represents the use of max pooling with a kernel size of n*n, Concat represents the concatenation of features at different scales, and CSPC is responsible for reducing redundant information and improving computational efficiency during the fusion process.

[0216] The introduction of the C3STR module improves the YOLOv8 model's ability to capture long-range dependencies, especially demonstrating stronger robustness against small defects in complex scenarios. The SPPCSPC module further improves the model's detection accuracy and speed through multi-scale feature extraction and efficient information fusion, significantly reducing computational overhead, particularly during real-time inference on resource-constrained devices.

[0217] 3. Optimized detection results of co-occurrence matrix

[0218] The detection results of the model are input into the co-occurrence inference module, and the detection results are further optimized based on the established co-occurrence matrix.

[0219] Specifically:

[0220] 3.1 Construction of the co-occurrence matrix

[0221] The co-occurrence inference module proposed in this embodiment constructs a co-occurrence matrix to record the co-occurrence relationships between target categories, thereby inferring whether there are missed or false detections of the target. The specific steps are as follows:

[0222] 3.1.1 Data Preparation

[0223] The co-occurrence frequency among different target categories is calculated from the training data. Co-occurring target categories can include critical equipment defects or equipment status in substation equipment.

[0224] 3.1.2 Calculation of co-occurrence probability

[0225] For target categories A and B, calculate the probability of them co-occurring in the same image. The formula is as follows:

[0226]

[0227] Here, A and B are the target categories in the substation data. P(A|B) represents the probability that category A also occurs given that category B occurs.

[0228] 3.1.3 Generation of Co-occurrence Matrix

[0229] Based on the above calculation results, a co-occurrence matrix is ​​constructed, where each element represents the co-occurrence probability of a pair of target categories appearing in the same image. Each row of the co-occurrence matrix represents category A, each column represents category B, and the matrix elements represent P(A|B), as detailed in Table 1.

[0230] 3.1.4 Setting the co-occurrence probability threshold

[0231] To effectively utilize co-occurrence relationships, a co-occurrence probability threshold θ is set. When P(A|B) is greater than this threshold, the system will infer a strong co-occurrence relationship between categories A and B.

[0232] 3.2 Co-occurrence Reasoning Process

[0233] When target category A is detected but category B, which has a high co-occurrence probability with it, is not detected, the system infers that there may be a missed detection. The specific steps are as follows:

[0234] Step 1: Query the co-occurrence matrix to obtain the value of P(B|A). If P(B|A) is greater than the threshold θ, it indicates that A and B usually occur simultaneously.

[0235] Step 2: Check if there are any targets of category B in the current detection results. If no targets of category B are detected, it is assumed that there may be a missed detection of category B.

[0236] Step 3: Increase the confidence level related to category B, that is, increase the confidence level of the detection boxes related to category B, or prompt the system to pay further attention to the possible existence of category B.

[0237] 3.3 Confidence Adjustment Mechanism

[0238] The co-occurrence inference module in this embodiment can not only predict missed detections and false detections, but also adjust the confidence level of the detection box to optimize the final detection result. The confidence level adjustment formula is:

[0239] Δscore i =α·a(h i ,h j )·(score j -score i )

[0240] Where: score_i and score_j are the confidence scores of detection boxes i and j, respectively; a(h_i,h_j) is the co-occurrence weight between detection boxes, which is derived from P(A|B) in the co-occurrence matrix; α is an adjustment factor used to control the magnitude of confidence adjustment, ranging from 0 to 1.

[0241] Adjustment process:

[0242] Step 1: Calculate the co-occurrence relationship between detection boxes to obtain the relevance weight a(h_i,h_j).

[0243] Step 2: Compare the confidence scores score_i and score_j. If score_j is greater than score_i and the correlation between the two is strong, increase the value of confidence_i. If the co-occurrence relationship between the two is weak (below the threshold θ), do not adjust the confidence score. i =0.

[0244] Step 3: Update the confidence of the detection box.

[0245] 4. Optimization and detection results of the Transform model

[0246] The results adjusted by the co-occurrence inference module are serialized and input into the trained Transformer model. A self-attention mechanism is used to capture long-distance dependencies between detection boxes, dynamically adjusting the confidence of each detection box to improve the system's robustness and detection accuracy in complex scenarios. Specifically:

[0247] 4.1 Input Serialization

[0248] To facilitate inputting the results into the Transformer model for subsequent processing, the detection results generated by the YOLOv8 object detection module and co-occurrence inference module are serialized. Each detection box information in the result is represented as a vector h. i Specifically:

[0249] h i =[label i ′,scorei ,xmin i ′,ymin i ′,xmax i ′,ymax i ′]

[0250] Where: label i ' represents the result of one-hot encoding of the detection box category, xmin i ′、ymin i ′、xmax i ′、ymax i ′ represents the result of global normalization of the original coordinates.

[0251] For a result with n detection boxes, the entire detection result forms a serialized feature vector set H, represented as:

[0252] H = [h1, h2, ..., h n ]

[0253] 4.2 Transformer Model

[0254] This embodiment introduces a detection result optimization model based on the Transformer self-attention mechanism. This model needs to be trained in advance to learn the long-distance dependencies between detection boxes using the training set, thereby optimizing the detection model's results.

[0255] 4.2.1 Input / Output Design

[0256] In this embodiment, the input / output design focuses only on optimizing the confidence level of the detection boxes, but it is not limited to confidence level. It can also be an optimization design for location information and category information. The input to the Transformer model is the serialized detection box information H output by the detection model, including the category, location, confidence level, and manually annotated target information for each box, including the category and location of each box. The output is an adjustment value for the confidence level of the detection boxes, calculated as follows:

[0257] Δscore i =β·(IoU i -score i )

[0258] Among them, IoU i IoU between predicted bounding box i and ground truth bounding box i

[0259] B: Adjustment factor, which controls the magnitude of the adjustment value.

[0260] The adjusted threshold is calculated as follows:

[0261] 4.2.2 Model Mechanism

[0262] To capture long-range dependencies between bounding boxes, the Transformer uses a self-attention mechanism to calculate the correlation between each bounding box and other bounding boxes. The calculation steps of the self-attention mechanism are as follows:

[0263] Step 1: Linear Transformation

[0264] Each input vector h i The query vector q is generated after three sets of linear transformations. i Key vector k i Sum vector v i :

[0265] q i =W Q h i ,k i =W K h i ,v i =W V h i

[0266] Among them, W Q W k W v , which are the linear transformation matrices of query, key, and value, respectively.

[0267] Step 2: Attention Weight Calculation

[0268] Calculate the query vector q using the dot product. i With key vector k i Based on the similarity, the attention weights of each detection box i to detection box j are obtained:

[0269]

[0270] Step 3: Weighted summation

[0271] Based on attention weight α(h) i h j ), for the value vector v i We perform a weighted summation to obtain the updated feature representation z. i :

[0272]

[0273] This feature z i It contains global dependency information between detection boxes, which is used to optimize confidence and location.

[0274] 4.2.3 Loss Function

[0275] When training a Transformer to optimize bounding box confidence using indirect supervision based on category and IoU, false negatives (FN) and false positives (FP) are significant issues, especially when the predicted bounding boxes do not perfectly match the ground truth bounding boxes. To address these situations, more refined supervision mechanisms can be designed, allowing the Transformer to appropriately adjust for these false negatives and FPs.

[0276] To prevent false negatives, category co-occurrence information is incorporated into the loss function design. Co-occurrence matrix inference serves as a regularization term, helping the model improve the confidence of certain targets. The co-occurrence loss calculation process is as follows:

[0277]

[0278] p(label i |Context) represents the joint co-occurrence probability of detection box i in the current detection result, where Context is the set of detection boxes in the current result, and n is the number of detection boxes included in Context.

[0279]

[0280] Where: Δscore i It is the confidence level of detection box i;

[0281] 1(x) is an indicator function that takes the value 1 when x > 0 and 0 otherwise;

[0282] θ is the threshold for co-occurrence probability. When the co-occurrence probability is greater than the threshold, it encourages the increase of confidence and pushes the model to adjust the confidence level to increase when the co-occurrence probability is high. Conversely, it pushes the model to adjust the confidence level to decrease when the co-occurrence probability is low.

[0283] The purpose of this loss term is to encourage an increase in the potential confidence level of the underreporting category.

[0284] For false positive detection boxes, the IoU is close to 0. The following confidence loss is designed to handle false positives:

[0285]

[0286] In addition to handling false positives and false negatives, the relative ranking information between detection boxes is also crucial. A ranking loss function is designed to maintain or optimize the ranking relationship of confidence scores, specifically:

[0287]

[0288] Here, H(x) is the Herveside step function, which measures whether the confidence ranking is consistent with the IoU ranking. If the ranking after confidence adjustment is inconsistent with the IoU ranking, the loss will increase, encouraging the model to maintain higher confidence in high-quality detection boxes.

[0289] The loss function for comprehensive optimization of confidence is:

[0290] L=λ score L score +λ rank L rank +λ co-srcor L co-scroe +λ FP L FP

[0291] Where, λ score The weights for the confidence loss term based on IoU guide the matching of confidence with IoU; λ rank The confidence ranking loss ensures higher confidence for high-quality detection boxes; λ co-srcor To improve the potential confidence of missed categories by using regularized loss based on category co-occurrence; λ FP To mitigate the loss from false alarms, the confidence level of the false alarm detection box is reduced.

[0292] Corresponding to the above-described target object detection method, this application also provides a target object detection device. Figure 8 A schematic diagram of the target object detection device provided in this application is shown below. Figure 9 As shown, the target object detection device provided in this embodiment includes:

[0293] The acquisition module 801 is used to acquire multiple image frames obtained by capturing images of the device to be inspected.

[0294] The first determining module 802 is used to determine the confidence level of the bounding box corresponding to each target object in any one of the multiple image frames;

[0295] The filtering module 803 is used to filter at least one second target object from target objects other than the first target object for any first target object, wherein, on the premise that the first target object is detected, the detection probability of the second target object is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level.

[0296] The boosting module 804 is used to boost the confidence level of any second target object to obtain the boosted confidence level.

[0297] The second determining module 805 is used to determine at least one object to be presented from multiple target objects based on the improved confidence levels corresponding to each second target object, the confidence levels corresponding to each first target object and each third target object, and to present the bounding boxes corresponding to each object to be presented to the user. The third target object is a target object other than the second target object and the first target object.

[0298] Optionally, when filtering module 803 filters at least one second target object from target objects other than the first target object, it is specifically used to:

[0299] By querying the co-occurrence matrix, the detection probability of each fourth target object is determined under the premise that the first target object is detected; the co-occurrence matrix is ​​used to indicate the detection probability of each target object under the premise that any target object is detected; the fourth target object is a target object other than the first target object.

[0300] For any fourth target object, if the detection probability of the fourth target object is greater than a first preset threshold, the fourth target object is determined to be the second target object.

[0301] Optionally, when the boosting module 804 boosts the confidence level corresponding to the second target object to obtain the boosted confidence level, it is specifically used for:

[0302] Based on the detection probability of the second target object and the first difference, the confidence level increase value corresponding to the second target object is determined; the first difference is used to indicate the difference between the confidence level corresponding to the first target object and the confidence level corresponding to the second target object.

[0303] The increased confidence level is determined based on the increase in confidence level corresponding to the second target object.

[0304] Optionally, when determining the confidence boost value corresponding to the second target object based on the detection probability of the second target object and the first difference, the boosting module 804 is specifically used for:

[0305] The confidence boost value corresponding to the second target object is determined according to the following formula:

[0306] Δscore i =α·a(h i ,h j )·(score j -score i )

[0307] Wherein, Δscore iThis represents the confidence level increase, where α is the adjustment factor, ranging from 0 to 1. α(h) i h j The score represents the probability of detecting the second target object given that the first target object has been detected. j The score represents the confidence level corresponding to the first target object. i This indicates the confidence level corresponding to the second target object.

[0308] Optionally, when the second determining module 805 determines at least one object to be presented from multiple target objects based on the improved confidence levels corresponding to each second target object, and the confidence levels corresponding to each first target object and each third target object, it is specifically used for:

[0309] The improved confidence, coordinate range, and category of the bounding boxes corresponding to each second target object, and the confidence, coordinate range, and category of the bounding boxes corresponding to each first and third target object are input into the trained confidence adjustment model to obtain the adjusted confidence value for each target object.

[0310] Based on the current and adjusted confidence values ​​of each target object, at least one object to be presented is determined from multiple target objects.

[0311] Optionally, the second determining module 805 is also used for:

[0312] A training sample set is constructed, wherein each training sample in the training sample set includes: the confidence score, coordinate range, and category of the predicted bounding box corresponding to each target object in a certain image frame; the label corresponding to the training sample is: the true adjusted value of the confidence score of the predicted bounding box corresponding to each target object; the true adjusted value is determined according to the following formula:

[0313] Δscore = β(IoU - score)

[0314] Wherein, Δscore is used to indicate the true adjusted value of the confidence score, β is an adjustment factor used to control the magnitude of the adjustment, IoU is used to indicate the intersection-union ratio of the predicted bounding box and the true bounding box, and the IoU is determined based on the coordinate range of the predicted bounding box and the coordinate range of the true bounding box, and score is used to indicate the confidence score of the predicted bounding box.

[0315] The training sample set is input into the initial confidence adjustment model to obtain the predicted adjusted confidence value for each training sample.

[0316] For any training sample, the loss value is determined based on the predicted adjustment value and the actual adjustment value of the confidence level corresponding to the training sample.

[0317] The parameters of the initial confidence adjustment model are adjusted based on the loss value corresponding to each training sample.

[0318] Optionally, when determining the loss value for each training sample based on the predicted adjustment value and the actual adjustment value of the confidence level corresponding to the training sample, the second determining module 805 is specifically used for:

[0319] Based on the predicted and actual adjusted confidence values ​​corresponding to the training samples, co-occurrence loss, and / or confidence loss, and / or ranking loss are determined; the ranking loss is used to measure the difference between confidence ranking and intersection-over-union (IoU) ranking, the IoU ranking is used to indicate the ranking of the IoU ratios of the predicted bounding boxes and the actual bounding boxes corresponding to each target object, and the co-occurrence loss is used to indicate the difference between the occurrence probability of a target object determined by the co-occurrence matrix and the confidence value corresponding to the target object; the co-occurrence matrix is ​​used to indicate the detection probability of each target object given that any target object is detected.

[0320] The loss value is determined based on co-occurrence loss, and / or confidence loss, and / or ranking loss.

[0321] Optionally, when determining the co-occurrence loss based on the predicted and actual adjusted values ​​of the confidence levels corresponding to the training samples, the second determining module 805 is specifically used for:

[0322] For any target object in the training samples, the adjusted confidence level of the target object is determined based on the confidence level of the target object and the predicted adjusted value of the confidence level; the joint co-occurrence probability of the target object in the current detection result is determined according to the following formula;

[0323]

[0324] Where p(labeli|Context) is the joint co-occurrence probability corresponding to target object i, Context is the set of target objects in the training sample, n represents the number of target objects in the training sample, and p(labeli|labelj) is the detection probability of target object i given that target object j is detected.

[0325] Based on the adjusted confidence level and joint co-occurrence probability of each target object, the co-occurrence loss is determined by the following formula;

[0326]

[0327] in, The adjusted confidence level corresponds to the target object i. l(x) is an indicator function that takes the value 1 when x>0 and zero otherwise. θ represents the second preset threshold.

[0328] Optionally, when determining the confidence loss based on the predicted and actual adjusted values ​​of the confidence corresponding to the training samples, the second determining module 805 is specifically used for:

[0329] The confidence loss is determined using the following formula:

[0330]

[0331] Where l(x) is the indicator function, and the IoU i Used to indicate the intersection-union ratio (IUGR) of the predicted bounding box and the true bounding box of target object i; represents the adjusted confidence level corresponding to target object i, and n represents the number of target objects in the training samples.

[0332] Optionally, when determining the ranking loss based on the predicted and actual adjusted values ​​of the confidence levels corresponding to the training samples, the second determining module 805 is specifically used for:

[0333] The sorting loss is determined using the following formula:

[0334]

[0335] Where H(x) is the Herveside step function, This represents the adjusted confidence level corresponding to target object i. The IoU represents the adjusted confidence level corresponding to target object j. i The IoU is used to indicate the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box of target object i. j The cross-union ratio (CUP) is used to indicate the predicted bounding box and the ground truth bounding box of target object j; n represents the number of target objects in the training samples.

[0336] Optionally, when determining the confidence level of the bounding box corresponding to each target object in the image frame, the first determining module 802 is specifically used for:

[0337] The image frame is input into the improved YOLOv8 model to determine the confidence level of the bounding box corresponding to each target object in the image frame. The improved YOLOv8 model includes a C3STR module and an SPPCSPC module.

[0338] The target object detection device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0339] Figure 9 A schematic diagram of the structure of the electronic device provided in this application. Figure 9 As shown, the electronic device 90 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the device 90 further includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus 904.

[0340] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to perform the above-described method.

[0341] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0342] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0343] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0344] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0345] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0346] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0347] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0348] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0349] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0350] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0351] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0352] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0353] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0354] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for detecting a target object, characterized in that, include: Acquire multiple image frames obtained by capturing images of the device to be inspected; For any image frame among multiple image frames, determine the confidence level of the bounding box corresponding to each target object in the image frame; For any first target object, at least one second target object is selected from target objects other than the first target object, wherein, provided that the first target object is detected, the probability of the second target object being detected is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level; For any second target object, the confidence level corresponding to the second target object is increased to obtain the increased confidence level; Based on the improved confidence levels corresponding to each second target object, and the confidence levels corresponding to each first target object and each third target object, at least one object to be presented is determined from multiple target objects, and the bounding boxes corresponding to each object to be presented are presented to the user. The third target object is a target object other than the second target object and the first target object.

2. The method according to claim 1, characterized in that, Select at least one second target object from the target objects other than the first target object, including: By querying the co-occurrence matrix, the detection probability of each fourth target object is determined under the premise that the first target object is detected; the co-occurrence matrix is ​​used to indicate the detection probability of each target object under the premise that any target object is detected; the fourth target object is a target object other than the first target object. For any fourth target object, if the detection probability of the fourth target object is greater than a first preset threshold, the fourth target object is determined to be the second target object.

3. The method according to claim 1, characterized in that, The confidence level corresponding to the second target object is increased to obtain the increased confidence level, including: Based on the detection probability of the second target object and the first difference, the confidence level increase value corresponding to the second target object is determined; the first difference is used to indicate the difference between the confidence level corresponding to the first target object and the confidence level corresponding to the second target object. The increased confidence level is determined based on the increase in confidence level corresponding to the second target object.

4. The method according to claim 3, characterized in that, Based on the detection probability of the second target object and the first difference, the confidence enhancement value corresponding to the second target object is determined, including: The confidence boost value corresponding to the second target object is determined according to the following formula: Δscore i =α·a(h i ,h j )·(score j -score i ) Wherein, Δscore i This represents the confidence level increase, where α is the adjustment factor, ranging from 0 to 1. α(h) i h j The score represents the probability of detecting the second target object given that the first target object has been detected. j The score represents the confidence level corresponding to the first target object. i This indicates the confidence level corresponding to the second target object.

5. The method according to claim 1, characterized in that, Based on the increased confidence levels corresponding to each second target object, and the confidence levels corresponding to each first and third target object, at least one object to be presented is determined from the multiple target objects, including: The improved confidence, coordinate range, and category of the bounding boxes corresponding to each second target object, and the confidence, coordinate range, and category of the bounding boxes corresponding to each first and third target object are input into the trained confidence adjustment model to obtain the adjusted confidence value for each target object. Based on the current and adjusted confidence values ​​of each target object, at least one object to be presented is determined from multiple target objects.

6. The method according to claim 5, characterized in that, The method further includes: A training sample set is constructed, wherein each training sample in the training sample set includes: the confidence score, coordinate range, and category of the predicted bounding box corresponding to each target object in a certain image frame; the label corresponding to the training sample is: the true adjusted value of the confidence score of the predicted bounding box corresponding to each target object; the true adjusted value is determined according to the following formula: Δscore = β(IoU - score) Wherein, Δscore is used to indicate the true adjusted value of the confidence score, β is an adjustment factor used to control the magnitude of the adjustment, IoU is used to indicate the intersection-union ratio of the predicted bounding box and the true bounding box, and the IoU is determined based on the coordinate range of the predicted bounding box and the coordinate range of the true bounding box, and score is used to indicate the confidence score of the predicted bounding box. The training sample set is input into the initial confidence adjustment model to obtain the predicted adjusted confidence value for each training sample. For any training sample, the loss value is determined based on the predicted adjustment value and the actual adjustment value of the confidence level corresponding to the training sample. The parameters of the initial confidence adjustment model are adjusted based on the loss value corresponding to each training sample.

7. The method according to claim 6, characterized in that, For each training sample, the loss value is determined based on the predicted adjusted value and the actual adjusted value of the confidence level corresponding to the training sample, including: Based on the predicted and actual adjusted confidence values ​​corresponding to the training samples, co-occurrence loss, and / or confidence loss, and / or ranking loss are determined; the ranking loss is used to measure the difference between confidence ranking and intersection-over-union (IoU) ranking, the IoU ranking is used to indicate the ranking of the IoU ratios of the predicted bounding boxes and the actual bounding boxes corresponding to each target object, and the co-occurrence loss is used to indicate the difference between the occurrence probability of a target object determined by the co-occurrence matrix and the confidence value corresponding to the target object; the co-occurrence matrix is ​​used to indicate the detection probability of each target object given that any target object is detected. The loss value is determined based on co-occurrence loss, and / or confidence loss, and / or ranking loss.

8. The method according to claim 7, characterized in that, Based on the predicted and actual adjusted confidence levels corresponding to the training samples, the co-occurrence loss is determined, including: For any target object in the training samples, the adjusted confidence level of the target object is determined based on the confidence level of the target object and the predicted adjusted value of the confidence level; the joint co-occurrence probability of the target object in the current detection result is determined according to the following formula; Where p(labeli|Context) is the joint co-occurrence probability corresponding to target object i, Context is the set of target objects in the training sample, n represents the number of target objects in the training sample, and p(labeli|labelj) is the detection probability of target object i given that target object j is detected. Based on the adjusted confidence level and joint co-occurrence probability of each target object, the co-occurrence loss is determined by the following formula; Among them, score i adjusted The adjusted confidence level corresponds to the target object i. l(x) is an indicator function that takes the value 1 when x>0 and zero otherwise. θ represents the second preset threshold.

9. The method according to claim 7, characterized in that, Based on the predicted and actual adjusted confidence values ​​corresponding to the training samples, the confidence loss is determined, including: The confidence loss is determined using the following formula: Where l(x) is the indicator function, and the IoU i The score is used to indicate the intersection-union ratio (IU) between the predicted bounding box and the ground truth bounding box of target object i. i adjusted represents the adjusted confidence level corresponding to target object i, and n represents the number of target objects in the training samples.

10. The method according to claim 7, characterized in that, Based on the predicted and actual adjusted confidence levels corresponding to the training samples, the ranking loss is determined, including: The sorting loss is determined using the following formula: Where H(x) is the Herveside step function, score i adjusted The score represents the adjusted confidence level corresponding to target object i. j adjusted The IoU represents the adjusted confidence level corresponding to target object j. i The IoU is used to indicate the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box of target object i. j The cross-union ratio (CUP) is used to indicate the predicted bounding box and the ground truth bounding box of target object j; n represents the number of target objects in the training samples.

11. The method according to any one of claims 1-10, characterized in that, Determining the confidence level of the bounding box corresponding to each target object in the image frame includes: The image frame is input into the improved YOLOv8 model to determine the confidence level of the bounding box corresponding to each target object in the image frame. The improved YOLOv8 model includes a C3STR module and an SPPCSPC module.

12. A target object detection device, characterized in that, include: The acquisition module is used to acquire multiple image frames obtained by capturing images of the device under test; The first determining module is used to determine the confidence level of the bounding box corresponding to each target object in any one of the multiple image frames; The filtering module is used to filter at least one second target object from target objects other than the first target object for any first target object, wherein, provided that the first target object is detected, the probability of the second target object being detected is greater than a first preset threshold, and the first target object is a target object whose corresponding confidence level is greater than or equal to a preset confidence level. The boosting module is used to boost the confidence level of any second target object to obtain the boosted confidence level. The second determining module is used to determine at least one object to be presented from multiple target objects based on the improved confidence levels corresponding to each second target object, the confidence levels corresponding to each first target object and each third target object, and to present the bounding boxes corresponding to each object to be presented to the user. The third target object is a target object other than the second target object and the first target object.

13. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

Images