Multi-mode fusion target detection method, device and electronic equipment for target tracking

Abstract

The embodiments of the present disclosure disclose a multi-mode fusion target detection method, device and electronic device for target tracking. A specific implementation of the method includes: acquiring an image to be detected; inputting the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set; performing target overlap detection on the first initial detection frame information set; determining each first initial detection frame information within a highly overlapping area as a second initial detection frame information set; clustering the second initial detection frame information set to obtain a first detection frame information set; determining each first initial detection frame information outside a highly overlapping area as a third initial detection frame information set; suppressing the third initial detection frame information set to obtain a second detection frame information set; generating detection frame information. This implementation improves the accuracy of the generated detection frame information and reduces the waste of computer computing resources.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a multi-mode fusion target detection method, device, and electronic device for target tracking. Background Art

[0002] Detection frame information refers to the information used to represent the location and confidence of the rectangular box containing the object in image object detection. Currently, the method commonly used to determine the detection frame information is to generate the detection frame information based on the confidence threshold method.

[0003] However, when using the above method to generate detection box information, the following technical problems often occur:

[0004] First, when generating detection frame information using a confidence threshold-based method, if objects in an image are densely packed, the generated initial detection frame information corresponds to a significant amount of overlap, and only one detection frame has a confidence score greater than the threshold, the confidence threshold-based method may only select the detection frame information corresponding to one detection frame in the highly overlapping area as the final result, while removing the detection frame information corresponding to the remaining objects in the highly overlapping area. This results in low accuracy of the generated detection frame information, necessitating re-detection and wasting computing resources.

[0005] Second, when generating detection frames using confidence threshold-based methods, if multiple overlapping detection frames for an object in an image exist and their confidence levels are all above the threshold, all of the detection frames are retained as the final result, making it impossible to determine the unique detection frame corresponding to the target object. Furthermore, the redundant detection frames may contain background or other non-target objects, resulting in lower accuracy in the generated detection frames. This necessitates re-detection, wasting computing resources.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the Invention

[0007] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0008] Some embodiments of the present disclosure propose a multi-modal fusion target detection method, device, electronic device and computer-readable medium for target tracking to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a multi-modal fusion target detection method for target tracking, the method comprising: acquiring an image to be detected; inputting the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and a target existence probability value; performing target overlap detection on the first initial detection frame information set to obtain a target overlap detection result; in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, determining each first initial detection frame information in the highly overlapping area as a second initial detection frame information set; The above-mentioned second initial detection frame information set is clustered to obtain individual initial detection frame information clusters; the initial detection frame information in each initial detection frame information cluster that meets the first preset condition is determined as the first detection frame information to obtain the first detection frame information set; in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, the first initial detection frame information outside the highly overlapping area is determined as the third initial detection frame information set; the above-mentioned third initial detection frame information set is suppressed to generate second detection frame information to obtain the second detection frame information set; based on the above-mentioned first detection frame information set and the above-mentioned second detection frame information set, detection frame information is generated.

[0010] In a second aspect, some embodiments of the present disclosure provide a multi-mode fusion target detection device for target tracking, the device comprising: an acquisition unit, configured to acquire an image to be detected; an input unit, configured to input the above-mentioned image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the above-mentioned first initial detection frame information set includes bounding box coordinate information and a target existence probability value; a detection unit, configured to perform target overlap detection on the above-mentioned first initial detection frame information set to obtain a target overlap detection result; a first determination unit, configured to determine each first initial detection frame information in the highly overlapping area as a second initial detection frame information set in response to determining that the target overlap detection result indicates the existence of a highly overlapping area; a first processing unit, configured to The second initial detection frame information set is configured to be clustered to obtain individual initial detection frame information clusters; the second determining unit is configured to determine the initial detection frame information in each initial detection frame information cluster in the individual initial detection frame information clusters that meets the first preset condition as the first detection frame information to obtain the first detection frame information set; the third determining unit is configured to, in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, determine the individual first initial detection frame information outside the highly overlapping area as the third initial detection frame information set; the second processing unit is configured to suppress the third initial detection frame information set to generate the second detection frame information to obtain the second detection frame information set; the generating unit is configured to generate the detection frame information based on the first detection frame information set and the second detection frame information set.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0013] The above-described embodiments of the present disclosure have the following beneficial effects: The multimodal fusion target detection method for target tracking according to some embodiments of the present disclosure improves the accuracy of generated detection frame information and reduces the waste of computer computing resources. Specifically, the low accuracy of generated detection frame information and the waste of computer resources are caused by the following reasons: when the detection frame information is generated using a confidence threshold-based method, when objects in an image are densely packed, the detection frames corresponding to the generated initial detection frame information have a high degree of overlap, and only one detection frame has a confidence score greater than the threshold, the confidence threshold-based method may only select the detection frame information corresponding to one detection frame in the highly overlapping area as the final result, while removing the other detection frame information corresponding to the remaining objects in the highly overlapping area. This results in low accuracy of the generated detection frame information, necessitating re-detection, and wasting computer computing resources. Based on this, the multimodal fusion target detection method for target tracking according to some embodiments of the present disclosure first obtains an image to be detected. Then, the image to be detected is input into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and a target presence probability value. Thus, a first initial detection frame information set for generating an object overlap detection result can be obtained. Next, an object overlap detection is performed on the first initial detection frame information set to obtain a target overlap detection result. Thus, a target overlap detection result can be obtained that indicates the presence of a highly overlapping area. Then, in response to determining that the target overlap detection result indicates the presence of a highly overlapping area, each piece of first initial detection frame information within the highly overlapping area is determined as a second initial detection frame information set. Thus, a second initial detection frame information set representing each piece of first initial detection frame information contained in the highly overlapping area can be obtained. Next, clustering is performed on the second initial detection frame information set to obtain initial detection frame information clusters. Thus, initial detection frame information clusters can be obtained, each containing pieces of second initial detection frame information with similar distance features. Then, in each of the initial detection frame information clusters, the initial detection frame information that meets a first preset condition is determined as first detection frame information, thereby obtaining a first detection frame information set. Thus, a first detection frame information set representing each piece of detection frame information corresponding to each object within the highly overlapping area can be obtained. Next, in response to determining that the target overlap detection result indicates the presence of a highly overlapping area, each piece of first initial detection frame information outside the highly overlapping area is determined as a third initial detection frame information set. This generates a third initial detection frame information set for generating each piece of detection frame information included in the area outside the highly overlapping area. This third initial detection frame information set is then subjected to suppression processing to generate second detection frame information, resulting in a second detection frame information set.In this way, the detection frame information of overlapping detection frames with high target presence probabilities can be suppressed in areas where objects are sparsely present, generating individual detection frame information corresponding to each target object. Finally, detection frame information is generated based on the first and second detection frame information sets. Clustering is performed on the detection frame information within highly overlapping areas to obtain initial detection frame information clusters, and the first detection frame information set is generated based on each cluster. The first detection frame information in the first detection frame information set represents the detection frame information with high target presence probabilities corresponding to objects in different clusters within the highly overlapping areas. When objects are densely present in an image, clustering is performed on the detection frame information within the dense areas to generate detection frame information with high target presence probabilities corresponding to each object, reducing the probability of generating detection frame information for only one object within the dense area. This improves the accuracy of the generated detection frame information, reduces the number of re-detections and detection frame information generation, and reduces the waste of computer computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flowchart of some embodiments of a multi-modal fusion target detection method for target tracking according to the present disclosure;

[0016] Figure 2 is a schematic structural diagram of some embodiments of a multi-mode fusion target detection device for target tracking according to the present disclosure;

[0017] Figure 3 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the multi-mode fusion target detection method for target tracking according to the present disclosure is shown. The multi-mode fusion target detection method for target tracking includes the following steps:

[0025] Step 101: Acquire an image to be detected.

[0026] In some embodiments, the execution entity (e.g., a computing device) of the multi-modal fusion target detection method for target tracking can obtain the image to be detected through a wired connection or a wireless connection. In practice, the execution entity can obtain an image captured by a camera as the image to be detected.

[0027] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0028] Step 102: Input the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set.

[0029] In some embodiments, the execution entity may input the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set. The first initial detection frame information in the first initial detection frame information set may represent the detection frame position information and target existence probability value corresponding to a detection frame in the detection image that has a higher target existence probability value corresponding to an object. The detection frame may be a directed rectangular frame used to represent the position and size of an object in target detection. The first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and target existence probability value. The target existence probability value may be the confidence level of the object within the detection frame. The initial detection frame information generation model includes an initial feature capture network and a detection frame information generation network. The initial feature capture network may be a convolutional neural network (CNN). The detection frame information generation network may be a YOLOv5 convolutional neural network (You Only Look Once version 5, YOLOv5).

[0030] In some optional implementations of some embodiments, the execution entity may input the image to be detected into a preset initial detection frame information generation model through the following steps to obtain a first initial detection frame information set:

[0031] The first step is to input the above-mentioned image to be detected into the input layer of the above-mentioned initial feature capture network to obtain the initial image data corresponding to the above-mentioned image to be detected, wherein the above-mentioned initial feature capture network includes the above-mentioned input layer, the first initial feature capture layer, the second initial feature conversion layer and the third initial feature enhancement layer.

[0032] The second step is to input the initial image data into the first initial feature capture layer to obtain feature capture information. The first initial feature capture layer may be a layer of a neural network that takes the initial image data as input and outputs the feature capture information. For example, the feature capture information may be feature map information. The first initial feature capture layer may be a convolutional layer.

[0033] In the third step, the first feature capture information is input into a second initial feature conversion layer to obtain nonlinear feature capture information. The second initial feature conversion layer may be a layer of a neural network that uses the first feature capture information as input data and outputs the nonlinear feature capture information. For example, the second initial feature conversion layer may be an activation function layer. The nonlinear feature capture information may be nonlinear feature map information processed by the activation function.

[0034] In the fourth step, the nonlinear feature capture information is input into a third initial feature enhancement layer to obtain feature enhancement information. The third initial feature enhancement layer may be a layer of a neural network that uses the nonlinear feature capture information as input data and outputs the feature enhancement information. For example, the third initial feature enhancement layer may be a pooling layer. The feature enhancement information may be the nonlinear feature map information obtained by performing dimensionality reduction and feature enhancement on the nonlinear feature map information.

[0035] In the fifth step, the feature enhancement information is input into the initial detection frame information generation layer of the detection frame information generation network to obtain an initial detection frame information set. The detection frame information generation network includes the initial detection frame information generation layer, the initial detection frame information classification layer, the initial detection frame information regression layer, and the output layer. The initial detection frame information generation layer may be a layer of a neural network that takes the feature enhancement information as input data and the initial detection frame information set as output data. The initial detection frame information in the initial detection frame information set may be information used to represent the coordinates of each bounding box. For example, the initial detection frame information generation layer may be a region proposal network (RPN) layer.

[0036] In the sixth step, the initial detection frame information set and the feature enhancement information are input into the initial detection frame information classification layer to obtain object presence probability information. The initial detection frame information classification layer may be a neural network layer that takes the initial detection frame information set as input and outputs the object presence probability information. For example, the initial detection frame information classification layer may be a fully connected layer. The object presence probability information may be information indicating the confidence level of the object within the detection frame.

[0037] In the seventh step, the initial detection frame information set and the feature enhancement information are input into the initial detection frame information regression layer to obtain the detection frame information set. The initial detection frame information regression layer may be a neural network layer that takes the initial detection frame information set and the feature enhancement information as input data and the detection frame information set as output data. For example, the initial detection frame information regression layer may be a neural network regression (Quantile Regression Neural Network, QRNN) layer. The detection frame information in the detection frame information set may be more accurate initial detection frame information obtained by processing the bounding box coordinates of the initial detection frame information through the neural network regression layer.

[0038] In the eighth step, the detection frame information set and the target existence probability information are input into the output layer to obtain a first initial detection frame information set.

[0039] Step 103: Perform target overlap detection on the first initial detection frame information set to obtain a target overlap detection result.

[0040] In some embodiments, the execution entity may perform target overlap detection on the first initial detection frame information set to obtain a target overlap detection result.

[0041] In some optional implementations of some embodiments, the execution entity may perform target overlap detection on the first initial detection frame information set through the following steps to obtain a target overlap detection result:

[0042] In the first step, the bounding box coordinate information included in each first initial detection box information in the above-mentioned first initial detection box information set is determined as the target coordinate information to obtain the target coordinate information set. The above-mentioned bounding box coordinate information can be the coordinates of the four vertices of the bounding box. For example, the coordinates of the lower left corner of the bounding box can be (x1=1, y1=1), the coordinates of the upper right corner can be (x2=2, y2=4), the coordinates of the upper left corner can be (x3=1, y3=4), and the coordinates of the lower right corner can be (x4=2, y4=1). The above x1, y1 represent the horizontal coordinate and vertical coordinate of the lower left corner coordinate. The above x2, y2 represent the horizontal coordinate and vertical coordinate of the upper right corner coordinate. The above x3, y3 represent the horizontal coordinate and vertical coordinate of the upper left corner coordinate. The above x4, y4 represent the horizontal coordinate and vertical coordinate of the lower right corner coordinate.

[0043] In the second step, for each piece of first initial detection frame information in the first initial detection frame information set, the following steps are performed:

[0044] In the first sub-step, the target coordinate information corresponding to the first initial detection frame information is determined as the current target coordinate information.

[0045] In the second sub-step, each target coordinate information in the target coordinate information set except the current target coordinate information is determined as the target coordinate information set to be processed.

[0046] The third sub-step is to generate the overlapping areas corresponding to the first initial detection frame information based on the current target coordinate information and the set of target coordinate information to be processed. In practice, the execution entity can solve the overlapping intersection areas of the detection frame corresponding to the current target coordinate information and the detection frame corresponding to each target coordinate information to be processed in the set of target coordinate information to be processed, and then determine the intersection areas obtained as the overlapping areas corresponding to the first initial detection frame information. As an example, the current target coordinate information corresponding to the first initial detection frame information can be A information, and the A information can be {the lower left corner coordinate can be (x1=1, y1=1), the upper right corner coordinate can be (x2=2, y2=4), the upper left corner coordinate can be (x3=1, y3=4), and the lower right corner coordinate can be (x4=2, y4=1)}. The target coordinate information set to be processed may be {B information, C information}, where {B information, C information} may be {{upper left corner coordinates may be (3, 4), lower right corner coordinates may be (4, 1), lower left corner coordinates may be (3, 1), upper right corner coordinates may be (4, 4)}, {upper left corner coordinates may be (1, 4), lower right corner coordinates may be (3, 1), lower left corner coordinates may be (1, 1), upper right corner coordinates may be (3, 4)}}. Determine whether the detection frame corresponding to the current target coordinate information (A information) intersects with the detection frame corresponding to the target coordinate information to be processed (B information). Since min(2, 4)-max(1, 3)>=0 and min(4, 4)-max(1, 1)>=0 are not satisfied, the detection frame corresponding to the current target coordinate information (A information) and the detection frame corresponding to the target coordinate information to be processed (B information) do not intersect, and the overlapping area between the detection frame corresponding to the current target coordinate information (A information) and the detection frame corresponding to the target coordinate information to be processed (B information) is 0. Determine whether the detection frame corresponding to the current target coordinate information (A information) intersects with the detection frame corresponding to the target coordinate information to be processed (C information). Since min(2, 4)-max(1, 1)>=0 and min(4, 4)-max(1, 1)>=0 are satisfied, the detection frame corresponding to the current target coordinate information (A information) intersects with the detection frame corresponding to the target coordinate information to be processed (C information). The overlapping area of ​​the detection frame corresponding to the current target coordinate information (A information) and the detection frame corresponding to the target coordinate information to be processed (C information) can be (min(2, 3)-max(1, 1))*((min(4, 4)-max(1, 1))=3. The respective overlapping areas corresponding to the first initial detection frame information can be 0 and 3.

[0047] In a fourth sub-step, in response to determining that an overlapping area among the overlapping areas satisfies a second preset condition, at least one overlapping area satisfying the second preset condition is determined as an initial target overlapping area set. The second preset condition may be that the overlapping area is greater than a first preset threshold.

[0048] In the third step, in response to determining the existence of at least one initial target overlapping area set, information indicating the presence of a highly overlapping area is determined as a target overlap detection result. The highly overlapping area may be an area containing each first detection frame. Each first detection frame may be a detection frame corresponding to the at least one initial target overlapping area set. The detection frame may be a directed rectangular frame used to represent the position and size of an object in target detection. In practice, the execution entity may determine the highly overlapping area through the following steps. First, the execution entity may determine the union of the initial target overlapping area sets as the target overlapping area set. Then, the execution entity may determine each first initial detection frame information corresponding to each target overlapping area in the target overlapping area set as each target detection frame information, thereby obtaining a target detection frame information set. Next, the execution entity may determine the bounding box coordinate information within each target detection frame information within the target detection frame information set as target bounding box coordinate information, thereby obtaining each target bounding box coordinate information. Furthermore, the execution entity may determine each coordinate corresponding to each target bounding box coordinate information as a coordinate set. Finally, the execution entity may determine the minimum horizontal coordinate in the coordinate set as the horizontal coordinate of the lower left vertex of the highly overlapping area. The execution entity may determine the minimum horizontal coordinate in the coordinate set as the horizontal coordinate of the upper left vertex of the highly overlapping area. The execution entity may determine the minimum vertical coordinate in the coordinate set as the vertical coordinate of the lower left vertex of the highly overlapping area. The execution entity may determine the maximum vertical coordinate in the coordinate set as the vertical coordinate of the upper left vertex of the highly overlapping area. The execution entity may determine the maximum horizontal coordinate in the coordinate set as the horizontal coordinate of the lower right vertex of the highly overlapping area. The execution entity may determine the maximum horizontal coordinate in the coordinate set as the horizontal coordinate of the upper right vertex of the highly overlapping area. The execution entity may determine the minimum vertical coordinate in the coordinate set as the vertical coordinate of the lower right vertex of the highly overlapping area. The execution entity may determine the maximum vertical coordinate in the coordinate set as the vertical coordinate of the upper right vertex of the highly overlapping area. Finally, the execution entity may determine a rectangular area with the above vertex coordinates as vertices as the highly overlapping area. As an example, the coordinate set may be {(1, 1), (1, 4), (2, 4), (2, 1), (1, 1), (1, 4), (3, 1), (3, 4)}. The horizontal coordinate of the lower left vertex of the highly overlapping area may be 1, the vertical coordinate of the lower left vertex of the highly overlapping area may be 1, and the coordinates of the lower left vertex of the highly overlapping area may be (1, 1). The horizontal coordinate of the upper left vertex of the highly overlapping area may be 1, the vertical coordinate of the upper left vertex of the highly overlapping area may be 4, and the coordinates of the upper left vertex of the highly overlapping area may be (1, 4).The horizontal coordinate of the lower right vertex of the highly overlapping area may be 3, the vertical coordinate of the lower right vertex of the highly overlapping area may be 1, and the coordinates of the lower left vertex of the highly overlapping area may be (3, 1). The horizontal coordinate of the upper right vertex of the highly overlapping area may be 3, the vertical coordinate of the upper right vertex of the highly overlapping area may be 4, and the coordinates of the lower left vertex of the highly overlapping area may be (3, 4). The highly overlapping area may be a rectangular area with {(1, 1), (1, 4), (3, 1), (3, 4)} as vertex coordinates.

[0049] Step 104 : In response to determining that the target overlap detection result indicates that a highly overlapping area exists, each piece of first initial detection frame information within the highly overlapping area is determined as a second initial detection frame information set.

[0050] In some embodiments, the execution entity may determine each piece of first initial detection frame information within the highly overlapping area as a second initial detection frame information set in response to determining that the target overlap detection result indicates the existence of a highly overlapping area.

[0051] Step 105 : performing clustering processing on the second initial detection frame information set to obtain initial detection frame information clusters.

[0052] In some embodiments, the execution entity may perform clustering processing on the second initial detection frame information set to obtain various initial detection frame information clusters.

[0053] In some optional implementations of some embodiments, the execution entity may perform clustering processing on the second initial detection box information set through the following steps to obtain initial detection box information clusters:

[0054] In the first step, a preset number of second initial detection frame information pieces are selected from the second initial detection frame information set as initial cluster centers. In practice, the execution entity may randomly select a preset number of second initial detection frame information pieces as initial cluster centers. The preset number may be 10.

[0055] In the second step, each of the aforementioned initialization cluster centers is added to the preset information cluster corresponding to the aforementioned initialization cluster center to generate a target preset information cluster, thereby obtaining each target preset information cluster. The preset information cluster may be an information set corresponding to the initialization cluster center. In practice, the execution entity may determine the preset information cluster resulting from the addition of the initialization cluster center to the preset information cluster corresponding to the aforementioned initialization cluster center as the target preset information cluster.

[0056] In the third step, the center coordinates of the detection box corresponding to each of the above-mentioned initialized cluster centers are determined as the first center coordinates to obtain a first center coordinate set.

[0057] In the fourth step, the selected preset number of second initial detection frame information are deleted from the above second initial detection frame information set to update the second initial detection frame information set.

[0058] In the fifth step, the updated second initial detection frame information set is determined as the initial detection frame information set to be allocated.

[0059] Step 6: For each piece of initial detection frame information to be allocated in the above set of initial detection frame information to be allocated, perform the following processing steps:

[0060] Sub-step 1: Determine the center coordinates of the detection frame corresponding to the initial detection frame information to be assigned as the second center coordinates.

[0061] Sub-step two, based on the above-mentioned second center coordinates and the above-mentioned first center coordinate set, determine the distance between the above-mentioned initial detection frame information to be assigned and each of the above-mentioned initialized cluster centers to obtain a distance set. In practice, the above-mentioned execution entity can determine the distance between the second center coordinate and each first center coordinate in the first center coordinate set as the distance between the corresponding initial detection frame information to be assigned and the corresponding initialized cluster center. Then, the above-mentioned execution entity can determine each determined distance as a distance set. The above-mentioned distance can be a Euclidean distance. For example, the above-mentioned second center coordinate can be (2, 2), and the above-mentioned first center coordinate can be (4, 2), then the Euclidean distance between the above-mentioned corresponding initial detection frame information to be assigned and the corresponding initialized cluster center can be 2.

[0062] Sub-step three: determining the distance that meets the first preset screening condition as the target distance. The first preset screening condition may be the smallest distance in the distance set.

[0063] Sub-step 4: Determine the target preset information cluster where the initialization cluster center corresponding to the target distance is located. In practice, the execution entity may determine the preset information cluster where the initialization cluster center corresponding to the target distance is located as the target preset information cluster.

[0064] Sub-step five: adding the above-mentioned information of the initial detection frame to be allocated to the determined target preset information cluster to update the target preset information cluster.

[0065] In the seventh step, each updated target preset information cluster is determined as each initial detection frame information cluster.

[0066] Step 106 : Determine the initial detection frame information that meets the first preset condition in each of the initial detection frame information clusters as first detection frame information to obtain a first detection frame information set.

[0067] In some embodiments, the execution entity may determine the initial detection frame information in each of the initial detection frame information clusters that satisfies a first preset condition as the first detection frame information, thereby obtaining a first detection frame information set. The first preset condition may be that the target presence probability value of the initial detection frame information is the largest.

[0068] Step 107 : In response to determining that the target overlap detection result indicates that a highly overlapping area exists, each piece of first initial detection frame information outside the highly overlapping area is determined as a third initial detection frame information set.

[0069] In some embodiments, the execution entity may determine each piece of first initial detection frame information outside the highly overlapping area as a third initial detection frame information set in response to determining that the target overlap detection result indicates the existence of a highly overlapping area.

[0070] Step 108 : Suppression processing is performed on the third initial detection frame information set to generate second detection frame information, thereby obtaining a second detection frame information set.

[0071] In some embodiments, the execution entity may perform suppression processing on the third initial detection frame information set to generate second detection frame information and obtain a second detection frame information set.

[0072] In some optional implementations of some embodiments, the execution entity may suppress the third initial detection box information set to generate second detection box information by the following steps:

[0073] In the first step, the target existence probability value in each third initial detection frame information in the third initial detection frame information set is determined as a confidence value to obtain a confidence value set.

[0074] In the second step, the following suppression processing steps are performed on the above confidence value set:

[0075] Sub-step 1: In response to determining that a confidence value that satisfies a second preset screening condition exists in the confidence value set, determining the confidence value that satisfies the second preset screening condition as the first target confidence value. The second preset screening condition may be a maximum confidence value.

[0076] Sub-step two: determining the third initial detection box information corresponding to the first target confidence value as the second detection box information.

[0077] Sub-step three: determining the bounding box coordinate information in the second detection box information as the first target bounding box coordinate information.

[0078] Sub-step four: deleting the first target confidence value from the confidence value set to update the confidence value set.

[0079] Sub-step five: determining the updated confidence value set as the first confidence value set.

[0080] Sub-step six: determining the bounding box coordinate information in the third initial detection box information corresponding to each first confidence value in the above-mentioned first confidence value set as the second target bounding box coordinate information to obtain the second target bounding box coordinate information set.

[0081] Sub-step seven: Generate a set of overlap degrees based on the first target bounding box coordinate information and the second target bounding box coordinate information set. In practice, for each second target bounding box coordinate information in the set of second target bounding box coordinate information, the execution entity may generate, based on the first target bounding box coordinate information and each second target bounding box coordinate information, the area of ​​each bounding box intersection and the area of ​​each bounding box union. The execution entity may then determine the ratio of each bounding box intersection area to the corresponding area of ​​each bounding box union as each overlap degree, thereby obtaining a set of overlap degrees. As an example, the first target bounding box coordinate information can be expressed as {the lower left corner coordinate can be (1, 1), the upper right corner coordinate can be (2, 4), the upper left corner coordinate can be (1, 4), and the lower right corner coordinate can be (2, 1)}, and the second target bounding box coordinate information set can be {(the lower left corner coordinate can be (1, 1), the upper right corner coordinate can be (3, 4), the upper left corner coordinate can be (1, 4), and the lower right corner coordinate can be (3, 1)}. The intersection area can be 3. The union area can be 9. The overlap can be 1 / 3. The overlap set can be {1 / 3}.

[0082] Sub-step eight: In response to determining that an overlap degree that satisfies a third preset screening condition exists in the overlap degree set, at least one overlap degree that satisfies the third preset screening condition is determined as a target overlap degree set. The third preset screening condition may be that the overlap degree is greater than a second preset threshold. The second preset threshold may be 1 / 6.

[0083] Sub-step nine: determining each confidence value corresponding to each target overlap degree in the target overlap degree set as each second target confidence value to obtain a second target confidence value set.

[0084] Sub-step ten: deleting each confidence value corresponding to the second target confidence value set from the updated confidence value set to update the confidence value set again.

[0085] In the third step, in response to determining that the updated confidence value set is not an empty set, the above suppression processing step is performed again on the updated confidence value set.

[0086] In the fourth step, in response to determining that the confidence value set after the re-update is an empty set, the determined second detection box information is determined as a second detection box information set.

[0087] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology: "When generating detection frame information by a method based on a confidence threshold, when there are multiple overlapping detection frames for an object in an image and the confidence of the detection frames is higher than the threshold, all the detection frame information will be retained as the final result, resulting in the inability to determine the unique detection frame information corresponding to the target object. At the same time, the redundant detection frames may contain background or other non-target objects, resulting in low accuracy of the generated detection frame information. Re-detection is required, wasting computer computing resources." The factors that lead to low accuracy of the generated detection frame information and waste of computer computing resources are often as follows: When generating detection frame information by a method based on a confidence threshold, when there are multiple overlapping detection frames for an object in an image and the confidence of the detection frames is higher than the threshold, all the detection frame information will be retained as the final result, resulting in the inability to determine the unique detection frame information corresponding to the target object. At the same time, the redundant detection frames may contain background or other non-target objects, resulting in low accuracy of the generated detection frame information. Re-detection is required, wasting computer computing resources. If the above factors are resolved, the accuracy of the generated detection frame information can be improved and the waste of computer computing resources can be reduced. To achieve this effect, the present disclosure adopts the following steps: First, the target existence probability value in each third initial detection frame information in the third initial detection frame information set is determined as a confidence value to obtain a confidence value set. Thus, a confidence value set for determining the second detection frame information can be obtained. Second, the following suppression processing steps are performed on the confidence value set: Sub-step 1: In response to determining that there is a confidence value in the confidence value set that meets the second preset filtering condition, the confidence value that meets the second preset filtering condition is determined as the first target confidence value. Sub-step 2: The third initial detection frame information corresponding to the first target confidence value is determined as the second detection frame information. Thus, the second detection frame information corresponding to the first target confidence value can be determined. Sub-step 3: The bounding box coordinate information in the second detection frame information is determined as the first target bounding box coordinate information. Sub-step 4: The first target confidence value is deleted from the confidence value set to update the confidence value set. Thus, the first target confidence value for the confirmed second detection frame information can be deleted from the confidence value set to avoid generating duplicate detection information. Sub-step five, the updated confidence value set is determined as the first confidence value set. Thus, a first confidence value set for generating the second target bounding box coordinate information set can be obtained. Sub-step six, the bounding box coordinate information in the third initial detection frame information corresponding to each first confidence value in the above-mentioned first confidence value set is determined as the second target bounding box coordinate information, to obtain the second target bounding box coordinate information set. Thus, a second target bounding box coordinate information set for generating the overlap degree set can be obtained.Sub-step 7: Generate an overlap degree set based on the first target bounding box coordinate information and the second target bounding box coordinate information set. Sub-step 8: In response to determining that an overlap degree that satisfies a third preset screening condition exists in the overlap degree set, determine at least one overlap degree that satisfies the third preset screening condition as a target overlap degree set. This results in a target overlap degree set representing a high degree of overlap. Sub-step 9: Determine each confidence value corresponding to each target overlap degree in the target overlap degree set as a second target confidence value, thereby obtaining a second target confidence value set. Sub-step 10: Delete each confidence value corresponding to the second target confidence value set from the updated confidence value set to update the confidence value set again. This allows the confidence values ​​corresponding to the target overlap degree set representing a high degree of overlap to be deleted from the updated confidence value set, thereby suppressing the third initial detection box information with a high degree of overlap and no longer confirming its retention. In step 3: In response to determining that the updated confidence value set is not an empty set, re-perform the suppression processing step on the updated confidence value set. In the fourth step, in response to determining that the updated confidence value set is empty, each piece of second detection frame information determined is determined as a second detection frame information set. This results in detection frame information representing areas outside of highly overlapping regions, namely, the second detection frame information set. Because each piece of second detection frame information is continuously generated based on the confidence value set after suppression processing, when an object in an image has multiple overlapping detection frames and the target presence probabilities (i.e., confidence values) of the detection frames are all high, the suppression process retains the third initial detection frame information with the highest confidence value as the second detection frame information. The remaining third initial detection frame information with a high degree of overlap with the third initial detection frame with the highest confidence value is then suppressed and no longer confirmed for retention. This ensures that each object in the image has a unique corresponding detection frame information, namely, the second detection frame information set, improving the accuracy of the generated detection frame information. Furthermore, the number of re-detections and generation of detection frame information is reduced, thereby reducing the waste of computer computing resources.

[0088] Step 109 : Generate detection box information based on the first detection box information set and the second detection box information set.

[0089] In some embodiments, the execution entity may generate the detection frame information based on the first detection frame information set and the second detection frame information set. In practice, the execution entity may determine the first detection frame information set and the second detection frame information set as the detection frame information.

[0090] Optionally, after performing target overlap detection on the first initial detection frame information set to obtain a target overlap detection result, the method further includes:

[0091] In the first step, in response to determining that the target overlap detection result indicates the absence of a high overlap area, the first initial detection frame information set is suppressed to generate third detection frame information, resulting in a third detection frame information set. The third target detection frame information in the third detection frame information set may represent the detection frame position information and target presence probability value of the detection frame of the object contained in the detection image when the high overlap area is absent.

[0092] Optionally, after generating the detection box information based on the first detection box information set and the second detection box information set, the method further includes:

[0093] In the first step, the bounding box coordinate information of each detection box in the above detection box information is determined as a bounding box coordinate information set.

[0094] The second step is to determine the pixel locations of the detection frame in the image to be detected based on the bounding box coordinate information set. In practice, the execution entity can first determine the coordinate information of each vertex in the bounding box coordinate information in the image to be detected. Then, a linear interpolation algorithm can be used to determine the pixel locations of the detection frame in the image to be detected based on the coordinate information of each vertex in the bounding box coordinate information in the image to be detected.

[0095] The third step is to update the pixel value of each pixel point to a preset value, so as to update each pixel point in the image to be detected. The preset value may be (255, 0, 0).

[0096] The fourth step is to determine the image to be detected after each pixel is updated as the detection frame display image.

[0097] The above-described embodiments of the present disclosure have the following beneficial effects: The multimodal fusion target detection method for target tracking according to some embodiments of the present disclosure improves the accuracy of generated detection frame information and reduces the waste of computer computing resources. Specifically, the low accuracy of generated detection frame information and the waste of computer resources are caused by the following reasons: when the detection frame information is generated using a confidence threshold-based method, when objects in an image are densely packed, the detection frames corresponding to the generated initial detection frame information have a high degree of overlap, and only one detection frame has a confidence score greater than the threshold, the confidence threshold-based method may only select the detection frame information corresponding to one detection frame in the highly overlapping area as the final result, while removing the other detection frame information corresponding to the remaining objects in the highly overlapping area. This results in low accuracy of the generated detection frame information, necessitating re-detection, and wasting computer computing resources. Based on this, the multimodal fusion target detection method for target tracking according to some embodiments of the present disclosure first obtains an image to be detected. Then, the image to be detected is input into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and a target presence probability value. Thus, a first initial detection frame information set for generating an object overlap detection result can be obtained. Next, an object overlap detection is performed on the first initial detection frame information set to obtain a target overlap detection result. Thus, a target overlap detection result can be obtained that indicates the presence of a highly overlapping area. Then, in response to determining that the target overlap detection result indicates the presence of a highly overlapping area, each piece of first initial detection frame information within the highly overlapping area is determined as a second initial detection frame information set. Thus, a second initial detection frame information set representing each piece of first initial detection frame information contained in the highly overlapping area can be obtained. Next, clustering is performed on the second initial detection frame information set to obtain initial detection frame information clusters. Thus, initial detection frame information clusters can be obtained, each containing pieces of second initial detection frame information with similar distance features. Then, in each of the initial detection frame information clusters, the initial detection frame information that meets a first preset condition is determined as first detection frame information, thereby obtaining a first detection frame information set. Thus, a first detection frame information set representing each piece of detection frame information corresponding to each object within the highly overlapping area can be obtained. Next, in response to determining that the target overlap detection result indicates the presence of a highly overlapping area, each piece of first initial detection frame information outside the highly overlapping area is determined as a third initial detection frame information set. This generates a third initial detection frame information set for generating each piece of detection frame information included in the area outside the highly overlapping area. This third initial detection frame information set is then subjected to suppression processing to generate second detection frame information, resulting in a second detection frame information set.In this way, the detection frame information of overlapping detection frames with high target presence probabilities can be suppressed in areas where objects are sparsely present, generating individual detection frame information corresponding to each target object. Finally, detection frame information is generated based on the first and second detection frame information sets. Clustering is performed on the detection frame information within highly overlapping areas to obtain initial detection frame information clusters, and the first detection frame information set is generated based on each cluster. The first detection frame information in the first detection frame information set represents the detection frame information with high target presence probabilities corresponding to objects in different clusters within the highly overlapping areas. When objects are densely present in an image, clustering is performed on the detection frame information within the dense areas to generate detection frame information with high target presence probabilities corresponding to each object, reducing the probability of generating detection frame information for only one object within the dense area. This improves the accuracy of the generated detection frame information, reduces the number of re-detections and detection frame information generation, and reduces the waste of computer computing resources.

[0098] Further references Figure 2 As an implementation of the methods shown in the figures, the present disclosure provides some embodiments of a multi-mode fusion target detection device for target tracking. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0099] like Figure 2As shown, some embodiments of the multi-mode fusion target detection device 200 for target tracking include: an acquisition unit 201, an input unit 202, a detection unit 203, a first determination unit 204, a first processing unit 205, a second determination unit 206, a third determination unit 207, a second processing unit 208, and a generation unit 209. The acquisition unit 201 is configured to acquire an image to be detected; the input unit 202 is configured to input the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and a target existence probability value; the detection unit 203 is configured to perform target overlap detection on the first initial detection frame information set to obtain a target overlap detection result; the first determination unit 204 is configured to, in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, determine each first initial detection frame information in the highly overlapping area as a second initial detection frame information set; the first processing unit 205 is configured to perform a target overlap detection on the second initial detection frame information set. Clustering processing is performed to obtain each initial detection frame information cluster; the second determination unit 206 is configured to determine the initial detection frame information in each initial detection frame information cluster that meets the first preset condition as the first detection frame information, and obtain the first detection frame information set; the third determination unit 207 is configured to, in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, determine each first initial detection frame information outside the highly overlapping area as the third initial detection frame information set; the second processing unit 208 is configured to suppress the third initial detection frame information set to generate the second detection frame information, and obtain the second detection frame information set; the generation unit 209 is configured to generate detection frame information based on the first detection frame information set and the second detection frame information set.

[0100] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units included therein, and will not be repeated here.

[0101] Reference below Figure 3 , which shows a structural diagram of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0102] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0103] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0104] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the functions defined in the methods of some embodiments of the present disclosure are performed.

[0105] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0106] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0107] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the image to be detected; inputs the above-mentioned image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the above-mentioned first initial detection frame information set includes bounding box coordinate information and target existence probability value; performs target overlap detection on the above-mentioned first initial detection frame information set to obtain a target overlap detection result; in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, each first initial detection frame information in the highly overlapping area is determined as the second initial detection frame information set; clustering the above-mentioned second initial detection frame information set to obtain each initial detection frame information cluster; determining the initial detection frame information that meets the first preset condition in each initial detection frame information cluster in the above-mentioned each initial detection frame information cluster as the first detection frame information to obtain the first detection frame information set; in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, determining each first initial detection frame information outside the highly overlapping area as the third initial detection frame information set; suppressing the above-mentioned third initial detection frame information set to generate second detection frame information to obtain the second detection frame information set; generating detection frame information based on the above-mentioned first detection frame information set and the above-mentioned second detection frame information set.

[0108] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The units described in some embodiments of the present disclosure may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor includes an acquisition unit, an input unit, a detection unit, a first determination unit, a first processing unit, a second determination unit, a third determination unit, a second processing unit, and a generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves. For example, the acquisition unit may also be described as a "unit for acquiring an image to be detected."

[0111] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0112] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multi-modal fusion target detection method for target tracking, comprising: Obtain the image to be detected; Inputting the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and a target existence probability value; Performing target overlap detection on the first initial detection frame information set to obtain a target overlap detection result; In response to determining that the target overlap detection result indicates that a highly overlapping area exists, determining each piece of first initial detection frame information within the highly overlapping area as a second initial detection frame information set; performing clustering processing on the second initial detection frame information set to obtain initial detection frame information clusters; determining the initial detection frame information that meets a first preset condition in each of the initial detection frame information clusters as first detection frame information to obtain a first detection frame information set; In response to determining that the target overlap detection result indicates that a highly overlapping area exists, determining each piece of first initial detection frame information outside the highly overlapping area as a third initial detection frame information set; performing suppression processing on the third initial detection frame information set to generate second detection frame information, thereby obtaining a second detection frame information set; Detection frame information is generated based on the first detection frame information set and the second detection frame information set.

2. The method according to claim 1, wherein After performing target overlap detection on the first initial detection frame information set to obtain a target overlap detection result, the method further includes: In response to determining that the target overlap detection result indicates that there is no highly overlapping area, the first initial detection frame information set is suppressed to generate third detection frame information, thereby obtaining a third detection frame information set.

3. The method according to claim 1, wherein The initial detection frame information generation model includes an initial feature capture network and a detection frame information generation network; as well as The step of inputting the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set includes: Inputting the image to be detected into the input layer of the initial feature capture network to obtain initial image data corresponding to the image to be detected, wherein the initial feature capture network includes the input layer, a first initial feature capture layer, a second initial feature conversion layer, and a third initial feature enhancement layer; Inputting the initial image data into the first initial feature capture layer to obtain first feature capture information; Inputting the first feature capture information into a second initial feature conversion layer to obtain nonlinear feature capture information; Inputting the nonlinear feature capture information into a third initial feature enhancement layer to obtain feature enhancement information; Inputting the feature enhancement information into an initial detection box information generation layer of a detection box information generation network to obtain an initial detection box information set, wherein the detection box information generation network includes the initial detection box information generation layer, an initial detection box information classification layer, an initial detection box information regression layer, and an output layer; Inputting the initial detection frame information set and the feature enhancement information into the initial detection frame information classification layer to obtain target existence probability information; Inputting the initial detection frame information set and the feature enhancement information into the initial detection frame information regression layer to obtain a detection frame information set; The detection frame information set and the target existence probability information are input into the output layer to obtain a first initial detection frame information set.

4. The method according to claim 1, wherein The performing target overlap detection on the first initial detection frame information set to obtain a target overlap detection result includes: determining the bounding box coordinate information included in each first initial detection box information in the first initial detection box information set as target coordinate information to obtain a target coordinate information set; The following steps are performed on each piece of first initial detection frame information in the first initial detection frame information set: Determining the target coordinate information corresponding to the first initial detection frame information as the current target coordinate information; Determine each target coordinate information in the target coordinate information set except the current target coordinate information as the target coordinate information set to be processed; generating overlapping areas corresponding to the first initial detection frame information based on the current target coordinate information and the set of target coordinate information to be processed; In response to determining that there is an overlapping area satisfying a second preset condition among the overlapping areas, determining at least one overlapping area satisfying the second preset condition as an initial target overlapping area set; In response to determining that at least one initial target overlapping area set exists, information indicating that a highly overlapping area exists is determined as a target overlap degree detection result.

5. The method according to claim 4, wherein The clustering process of the second initial detection frame information set to obtain each initial detection frame information cluster includes: Selecting a preset number of second initial detection frame information from the second initial detection frame information set as respective initialization cluster centers; adding each of the initialized cluster centers to the preset information cluster corresponding to the initialized cluster center to generate a target preset information cluster, thereby obtaining each target preset information cluster; Determining the center coordinates of the detection box corresponding to each of the initialized cluster centers as first center coordinates to obtain a first center coordinate set; Deleting the selected preset number of second initial detection frame information from the second initial detection frame information set to update the second initial detection frame information set; Determine the updated second initial detection frame information set as the initial detection frame information set to be allocated; For each piece of initial detection frame information to be allocated in the set of initial detection frame information to be allocated, perform the following processing steps: Determining the center coordinates of the detection frame corresponding to the to-be-assigned initial detection frame information as second center coordinates; Determining, based on the second center coordinates and the first center coordinate set, a distance between the to-be-assigned initial detection box information and each of the initialized cluster centers to obtain a distance set; Determine the distance that meets the first preset screening condition as the target distance; Determine the target preset information cluster where the initialization cluster center corresponding to the target distance is located; Adding the to-be-allocated initial detection frame information to the determined target preset information cluster to update the target preset information cluster; The updated target preset information clusters are determined as initial detection frame information clusters.

6. The method according to claim 1, wherein After generating the detection box information based on the first detection box information set and the second detection box information set, the method further includes: Determining bounding box coordinate information of each detection box in the detection box information as a bounding box coordinate information set; Determine, based on the bounding box coordinate information set, each pixel of the detection box in the image to be detected; Updating the pixel value of each pixel point to a preset value to update each pixel point in the image to be detected; The image to be detected after each pixel is updated is determined as the detection frame display image.

7. A multi-mode fusion target detection device for target tracking, comprising: an acquisition unit, configured to acquire an image to be detected; an input unit configured to input the image to be detected into a preset initial detection frame information generation model to obtain a first initial detection frame information set, wherein the first initial detection frame information in the first initial detection frame information set includes bounding box coordinate information and a target existence probability value; a detection unit configured to perform target overlap detection on the first initial detection frame information set to obtain a target overlap detection result; a first determining unit configured to, in response to determining that the target overlap detection result indicates the existence of a highly overlapping area, determine each piece of first initial detection frame information within the highly overlapping area as a second initial detection frame information set; a first processing unit configured to perform clustering processing on the second initial detection frame information set to obtain initial detection frame information clusters; a second determining unit configured to determine the initial detection frame information that satisfies a first preset condition in each of the initial detection frame information clusters as first detection frame information, to obtain a first detection frame information set; a third determining unit configured to, in response to determining that the target overlap detection result indicates the presence of a highly overlapping area, determine each piece of first initial detection frame information outside the highly overlapping area as a third initial detection frame information set; a second processing unit configured to perform suppression processing on the third initial detection frame information set to generate second detection frame information, thereby obtaining a second detection frame information set; The generating unit is configured to generate detection frame information based on the first detection frame information set and the second detection frame information set.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images