Target detection method, electronic device, and medium

Abstract

The application provides a target detection method, an electronic device and a medium, relates to the field of computer vision, and first accurately corresponds a to-be-detected image to imaging state parameters when the to-be-detected image is collected, then predicts a pixel size interval of a to-be-detected target according to the imaging state parameters, and dynamically adjusts the fusion weight of image features with different spatial resolutions according to the pixel size interval. Therefore, when the size of the target in the image changes due to changes in the shooting distance, the shooting posture or the imaging parameters, the target detection model can adaptively select more suitable detail features and semantic features for the current target scale, reduce the small target feature deficiency or scale mismatch problem caused by the fixed feature fusion mode, thereby reducing the probability of target missed detection and false detection, and improving the accuracy of target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a target detection method, electronic device, and medium. Background Technology

[0002] With the development of computer vision and artificial intelligence technologies, target detection technology has been widely applied in scenarios such as aerial inspection, environmental monitoring, security surveillance, and emergency search and rescue. During image acquisition by imaging platforms such as drones, targets are typically far from the imaging equipment, occupying a small pixel area in the image, resulting in relatively limited discernible information such as texture and edges. Furthermore, factors such as observation distance, observation angle, imaging platform motion, and changes in the surrounding environment can significantly affect the size and appearance of the same target in different images. Complex backgrounds, lighting variations, and motion blur can further increase the difficulty of distinguishing targets from the background, making existing target detection methods prone to missed or false detections.

[0003] Therefore, how to improve the accuracy of target detection has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The purpose of this application is to provide a target detection method, electronic device, and medium, which aim to improve the accuracy of target detection.

[0005] In a first aspect, embodiments of this application provide a target detection method, the method comprising: The image to be detected and the imaging state parameters corresponding to the image to be detected are obtained. The imaging state parameters are used to characterize the shooting distance, shooting posture and imaging parameters of the imaging device used to acquire the image to be detected when the image to be detected is acquired. Multi-scale feature extraction is performed on the image to be detected to obtain image features corresponding to multiple scale levels, where image features at different scale levels have different spatial resolutions. Based on the imaging state parameters, the predicted pixel scale of the target to be detected in the image to be detected is determined, wherein the predicted pixel scale is used to characterize the expected pixel size of the target to be detected in the image to be detected. Based on the predicted pixel scale, determine the fusion weights corresponding to each of the multiple image features; Based on multiple fusion weights, multiple image features are weighted and fused to obtain fused image features; Target detection is performed based on the features of the fused image to obtain the target detection results.

[0006] In one embodiment of this application, the image to be detected is an aerial image captured by an imaging device installed on a flight device. The imaging state parameters include the flight altitude, flight speed, heading angle, pitch angle and roll angle of the flight device, as well as the camera intrinsic parameters, camera extrinsic parameters and gimbal attitude parameters of the imaging device. Based on the imaging state parameters, the predicted pixel scale of the target object in the image to be detected is determined, including: Based on the flight altitude, pitch angle, and gimbal attitude parameters, and combined with the preset reference plane, the imaging distance between the imaging device and the preset reference plane is determined. The preset reference plane is used to characterize the plane where the target to be detected is located. Obtain the target size corresponding to the target to be detected, and determine the camera focal length from the camera intrinsic parameters; Determine the product of the camera focal length and the target size, and use the ratio of the product to the imaging distance as the predicted pixel scale.

[0007] In one embodiment of this application, determining the fusion weights corresponding to multiple image features based on the predicted pixel scale includes: The imaging state parameters are vectorized and encoded to obtain state features; Determine the scale adjustment factor based on the predicted pixel scale; Based on a preset weight generation function, the fusion weights corresponding to multiple image features are obtained according to the scale adjustment factor and state features.

[0008] In one embodiment of this application, multi-scale feature extraction is performed on the image to be detected to obtain image features corresponding to multiple scale levels, including: Input the image to be detected into the feature extraction model; By using different network layers in the feature extraction model to extract features from the image to be detected, multiple basic image features with different spatial resolutions are obtained. Shallow basic image features are determined from multiple basic image features, wherein the shallow basic image features are output by the network layer near the input end in the backbone network, and the spatial resolution of the shallow basic image features is higher than the spatial resolution of at least one other basic image feature. Convolutional transformation is performed on shallow basic image features to obtain detailed features of small targets; Spatial attention and channel attention are applied sequentially to the detailed features of small targets to obtain enhanced detailed features. Enhanced detail features and basic image features other than shallow basic image features among multiple basic image features are identified as image features corresponding to each of multiple scale levels.

[0009] In one embodiment of this application, target detection is performed based on fused image features to obtain target detection results, including: Multiple candidate targets are identified based on the features of the fused image using a detection model. Each candidate target includes a two-dimensional bounding box, target category, and detection confidence. Each candidate target is mapped to a three-dimensional space to obtain the three-dimensional spatial position of each candidate target; The three-dimensional spatial position of each candidate target is verified according to the preset spatial constraints. The preset spatial constraints are used to characterize the physical range of the target of the corresponding target category in three-dimensional space. Candidate targets that meet the preset spatial constraints are retained, and the two-dimensional bounding boxes and detection confidence of the retained candidate targets are corrected according to their corresponding three-dimensional spatial positions. Based on the retained and corrected candidate targets, the target detection results are obtained.

[0010] In one embodiment of this application, the image to be detected is the current image frame in the video stream acquired by the imaging device; After obtaining the target detection result by performing target detection based on the fused image features, the method also includes: The retained and revised candidate targets are designated as reference targets; The historical target detection results of multiple historical image frames that are located before the current image frame in the video stream in time sequence are obtained. The multiple historical target detection results jointly represent multiple historical targets. A historical target is a target object detected in at least one historical image frame. The detection results of the same target object in different historical image frames belong to the same historical target. For each historical target, based on the detection results of multiple historical targets, the target state of the historical target in the current image frame is predicted, where the target state includes the predicted position and the predicted motion state. Based on the target states corresponding to multiple historical targets, each reference target in the current image frame is associated with multiple historical targets to obtain target association results. The target association results are used to characterize whether each reference target corresponds to the same target object as each historical target. Based on the target association results, the target detection results corresponding to the current image frame are corrected to obtain the temporally optimized detection results.

[0011] In one embodiment of this application, correcting the target detection result corresponding to the current image frame based on the target association result includes: For each historical target, determine whether there is a reference target in the current image frame that corresponds to the same target object as the historical target, based on the target association results; If there is a reference target in the current image frame that corresponds to the same target object as the historical target, the target position of the corresponding reference target is corrected according to the predicted position of the historical target, and the reference target with the corrected target position is retained in the target detection result corresponding to the current image frame. If there is no reference target in the current image frame that corresponds to the same target object as the historical target, the predicted position of the historical target is determined as the target position of the historical target in the current image frame, and the historical target is added to the target detection result corresponding to the current image frame. For each reference target, if the target association results show that the reference target does not correspond to the same target object as any historical target, and the reference target only appears in the current image frame, the reference target will be deleted from the target detection results corresponding to the current image frame.

[0012] In one embodiment of this application, the method further includes: Based on the time-series optimization results, the target positions of multiple target objects in the current image frame are determined, and the target positions of the same target object in multiple historical image frames and the current image frame are connected in chronological order to determine the target trajectory corresponding to the target object.

[0013] Secondly, embodiments of this application provide an electronic device, including: The memory is configured to store instructions; and The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the target detection method provided in the first aspect of the embodiments of this application.

[0014] Thirdly, embodiments of this application provide a machine-readable storage medium storing instructions that, when executed by a processor, cause the processor to implement the target detection method according to the first aspect of embodiments of this application.

[0015] In this embodiment, the image to be detected is first accurately matched with its imaging state parameters at the time of acquisition. Then, the pixel size range of the target to be detected is predicted based on the imaging state parameters, and the fusion weights of image features at different spatial resolutions are dynamically adjusted accordingly. Therefore, when the size of the target in the image changes due to changes in shooting distance, shooting posture, or imaging parameters, the target detection model can adaptively select more suitable detailed features and semantic features for the current target scale. This reduces the problem of insufficient features or scale mismatch for small targets caused by fixed feature fusion methods, thereby reducing the probability of missed and false detections and improving the accuracy of target detection. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the target detection method provided in an embodiment of this application; Figure 2 This is another schematic flowchart of the target detection method provided in the embodiments of this application; Figure 3 This is another schematic flowchart of the target detection method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the image feature fusion process provided in an embodiment of this application; Figure 5 This is another schematic diagram of the image feature fusion process provided in the embodiments of this application; Figure 6 This is a schematic diagram of the target detection process provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the process of optimizing target detection results provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] With the development of computer vision and artificial intelligence technologies, target detection technology has been widely applied in scenarios such as aerial inspection, environmental monitoring, security surveillance, and emergency search and rescue. During image acquisition by imaging platforms such as drones, targets are typically far from the imaging equipment, occupying a small pixel area in the image, resulting in relatively limited discernible information such as texture and edges. Furthermore, factors such as observation distance, observation angle, imaging platform motion, and changes in the surrounding environment can significantly affect the size and appearance of the same target in different images. Complex backgrounds, lighting variations, and motion blur can further increase the difficulty of distinguishing targets from the background, making existing target detection methods prone to missed or false detections.

[0020] Therefore, how to improve the accuracy of target detection has become a technical problem that urgently needs to be solved in this field.

[0021] Based on this, embodiments of this application provide a target detection method, electronic device, and medium, aiming to improve the accuracy of target detection.

[0022] Figure 1 This is a schematic flowchart of the target detection method provided in the embodiments of this application. Figure 2 This is another schematic flowchart of the target detection method provided in the embodiments of this application. Figure 3 This is another schematic flowchart of the target detection method provided in the embodiments of this application, please refer to it as well. Figures 1-3 .

[0023] In a first aspect, this application provides a target detection method that can be used in electronic devices, including the following steps S100-S600.

[0024] The electronic device can be an electronic device deployed on or integrated into a drone (e.g., an airborne edge computing device, mission computer, artificial intelligence computing module, or flight control computing unit) or the drone itself; the electronic device can also be a separate electronic device that receives images transmitted by other flying devices or mobile devices such as drones, such as a ground station computer, edge server, or cloud server. The electronic device may include a processor, memory, and communication interface. The memory stores a computer program, and the processor executes the computer program to implement the target detection method provided in the embodiments of this application.

[0025] In one implementation, an object detection model is deployed in the electronic device. This object detection model may include a feature extraction model, a scale estimation structure, a scale-aware feature fusion structure, and a detection model.

[0026] The feature extraction model, also known as the backbone network, is used to extract image features with different spatial resolutions from the image to be detected. The scale estimation structure is used to predict the pixel size range of the target in the image to be detected based on imaging state parameters and generate fusion weights for different image features. The scale-aware feature fusion structure is used to perform weighted fusion of multiple image features according to the fusion weights. The detection model, also known as the detection head, is used to output the target category, two-dimensional bounding box, and detection confidence based on the fused image features. Furthermore, the target detection model can also include a three-dimensional spatial constraint processing structure and a temporal optimization structure, used to perform three-dimensional spatial verification of candidate targets and temporal optimization of the detection results using consecutive image frames, respectively.

[0027] Step S100: Obtain the image to be detected and the imaging state parameters corresponding to the image to be detected; Among them, the imaging state parameters are used to characterize the shooting distance, shooting posture, and imaging parameters of the imaging device used to acquire the image to be detected when acquiring the image to be detected.

[0028] In this step, the electronic device acquires the original image captured by the imaging device and performs necessary preprocessing on the original image to obtain the image to be detected that can be input into the target detection model; at the same time, the electronic device acquires the imaging state parameters when the original image is acquired and maps the original image or the image to be detected to the imaging state parameters according to the acquisition time.

[0029] The imaging device can be a visible light camera, an infrared camera, a multispectral camera, or other cameras with video acquisition capabilities. The imaging device can be mounted on flying equipment such as drones, or on vehicles, robots, fixed monitoring equipment, or other mobile devices.

[0030] Imaging state parameters can characterize at least the following three types of information: The first category is shooting distance information, which characterizes the distance between the imaging device and the area where the target to be detected is located. The shooting distance can be directly measured by a laser rangefinder or depth sensor, or it can be calculated based on the flight altitude, the shooting attitude of the imaging device, and the preset reference plane where the target to be detected is located.

[0031] The second category is shooting posture information, which characterizes the orientation and tilt of the imaging device when acquiring the image to be detected, such as yaw angle, pitch angle, roll angle, and the azimuth, pitch angle, and roll angle of the gimbal. Shooting posture information can reflect the optical axis direction of the imaging device and the geometric relationship between the imaging plane and the spatial surface of the target.

[0032] The third category consists of imaging parameters of the imaging device, such as focal length, principal point coordinates, pixel size, image resolution, distortion parameters, camera intrinsic parameters, camera extrinsic parameters, exposure parameters, and zoom ratio. Camera intrinsic parameters are used to establish the relationship between camera coordinates and pixel coordinates, while camera extrinsic parameters are used to establish the relationship between the camera coordinate system and the world coordinate system.

[0033] In one specific implementation, the preprocessing of the original image may include at least one of distortion correction, size normalization, brightness enhancement, noise suppression, and motion blur correction. For example, the original image may be distorted based on the radial and tangential distortion parameters of the imaging device; bilinear interpolation may be used to adjust original images of different resolutions to the input size required by the target detection model; histogram equalization, gamma correction, or brightness normalization may be used to improve image quality under low-light conditions; spatial filtering or neural network denoising may be used to suppress image noise; and motion blur may be corrected by combining the angular velocity or flight speed of the flight device.

[0034] When the original image is scaled, cropped, or padded, camera intrinsics can also be updated synchronously. For example, the scaling ratios of the original image in the horizontal and vertical directions are respectively... and Then the updated focal length and principal point coordinates can be expressed as:

[0035]

[0036] in, and These represent the horizontal and vertical focal lengths of the original image, respectively. and These represent the coordinates of the principal points in the original image. , , and This indicates the camera intrinsic parameters used after image scaling. This avoids scaling or 3D backprojection errors that can occur when using the original camera intrinsic parameters after image size changes.

[0037] In one specific implementation, the imaging device is a camera mounted on the flight equipment. The electronic equipment can acquire images and image timestamps from the camera, and obtain imaging state parameters corresponding to multiple parameter sampling times from the flight control system, inertial measurement unit, positioning module, ranging module, and gimbal control module.

[0038] When the image timestamp and the parameter sampling time are not exactly the same, the nearest neighbor matching method can be used to select the parameter sampling time with the smallest time difference from the image timestamp from multiple parameter sampling times, and the imaging state parameter corresponding to this parameter sampling time is determined as the imaging state parameter corresponding to the image to be detected. The parameter sampling time corresponding to the nearest neighbor matching can be expressed as:

[0039] in, Represents the image timestamp. Indicates the first Each parameter sampling time, This indicates the parameter sampling time that is closest to the image timestamp.

[0040] When imaging state parameters change rapidly or the image sampling frequency differs significantly from the parameter sampling frequency, linear interpolation can also be used. Assuming an image timestamp... Located at adjacent parameter sampling times and Between these, any scalar form of the imaging state parameter It can be represented as:

[0041] in, and These represent the parameter values ​​at two adjacent sampling times. For angle parameters such as heading angle that exhibit periodic jumps, the angle can be converted into sine and cosine components before interpolation. For shooting attitudes represented by quaternions, quaternion interpolation can be used to avoid abrupt attitude changes caused by directly interpolating Euler angles.

[0042] In this way, each image to be detected has a temporally matched shooting distance, shooting posture, and imaging parameters, thus providing accurate geometric input for subsequent prediction of pixel scale and three-dimensional spatial position; image preprocessing can reduce the impact of distortion, noise, and motion blur on feature extraction results.

[0043] For example, the camera acquires an aerial image at timestamp 12.500, while the flight control system records the flight altitude, attitude angle, and gimbal attitude at 12.490 and 12.510, respectively. The electronic equipment can perform linear interpolation on the parameters at the two sampling times to obtain a flight altitude of 120 meters, a flight speed of 8 meters per second, a pitch angle of -5 degrees, and the corresponding gimbal attitude at 12.500. Subsequently, the electronic equipment performs distortion correction and size normalization on the aerial image and simultaneously adjusts the camera's intrinsic parameters to obtain the image to be detected and its corresponding imaging state parameters.

[0044] Step S200: Perform multi-scale feature extraction on the image to be detected to obtain image features corresponding to each of the multiple scale levels; Image features at different scale levels have different spatial resolutions.

[0045] In this step, the electronic device inputs the image to be detected into the feature extraction model, which then extracts image features using network layers of varying depths. As the network layer depth increases, the spatial resolution of the image features typically decreases gradually, while the receptive field and semantic expressive power gradually increase.

[0046] In this embodiment, the scale level is used to distinguish image features with different spatial resolutions. Let the scale level be... The image features corresponding to each scale level are:

[0047] in, and They represent the first The height and width of an image feature together characterize the spatial resolution of that image feature; This indicates the number of channels. Different scale levels correspond to different... and .

[0048] Feature extraction models can employ convolutional neural networks, visual Transformer networks, or a combination of both. For convolutional neural networks, multiple basic image features can be obtained through network layers with different downsampling factors; for visual Transformer networks, multiple image features with different spatial resolutions can be obtained through hierarchical token merging or window partitioning.

[0049] Image features with higher spatial resolution typically retain more edge, texture, and positional details, making them suitable for representing small targets that occupy fewer pixels in the image to be detected. Image features with lower spatial resolution have a larger receptive field and stronger semantic expressive power, making them suitable for distinguishing larger targets or targets with similar appearances from the background. At this point, each image feature still corresponds to a different scale level, and weighted fusion based on fusion weights has not yet been performed.

[0050] In this way, a multi-scale feature set that simultaneously covers detailed information and high-level semantic information is formed, providing a foundation for subsequently selectively enhancing the corresponding scale level according to the predicted pixel scale.

[0051] For example, the input size of the image to be detected is 1024×1024 pixels, and the feature extraction model can output four image features with spatial resolutions of 256×256, 128×128, 64×64, and 32×32, respectively. These four image features correspond to four scale levels in sequence. Among them, the 256×256 image feature is more conducive to preserving the outline details of small vehicles or pedestrians, while the 32×32 image feature is more conducive to expressing the overall semantics of the target category and the surrounding scene.

[0052] Step S300: Determine the predicted pixel scale of the target to be detected in the image to be detected based on the imaging state parameters; The predicted pixel scale is used to characterize the expected pixel size of the target in the image to be detected.

[0053] In this step, the electronic device establishes the imaging geometric relationship between the actual size of the target to be detected and its image size based on the shooting distance, shooting posture and imaging parameters of the imaging device, and predicts the pixel size (which can be a size range) that the target to be detected may occupy in the image to be detected.

[0054] The predicted pixel scale can be represented as a single size or as a range of sizes:

[0055] in, This indicates the lower limit of the expected pixel size of the target to be detected. This indicates the upper limit of the expected pixel size of the target to be detected. This indicates the acquisition time of the image to be detected. The predicted pixel scale can represent the expected width (or width range) and expected height (or height range) of the target, or it can represent a representative size determined by the expected width and expected height, such as the longest side, the shortest side, or a range of geometric mean sizes.

[0056] In one implementation, the electronic device pre-stores target size priors corresponding to one or more categories to be detected. For different categories such as vehicles, pedestrians, and ships, separate true width and true height ranges can be set. Since the specific target category may not be determined before detection is performed, the electronic device can calculate the predicted pixel scale separately for each category to be detected, or merge the predicted pixel scales corresponding to each category into a single coverage area.

[0057] In a simplified pinhole imaging model, the representative pixel size of the target can be represented as:

[0058] in, Indicates the pixel size of the target. This represents the camera focal length in pixels. Indicates the actual size of the target. This represents the imaging distance between the imaging device and the target location. When both the target size and imaging distance are ranges, the lower limit of the predicted pixel scale can be determined according to the lower limit of the target size and the upper limit of the imaging distance, and the upper limit of the predicted pixel scale can be determined according to the upper limit of the target size and the lower limit of the imaging distance.

[0059] When the shooting posture changes significantly or the target plane is not parallel to the imaging plane, multiple 3D reference points representing the true size of the target can be projected onto the image based on the camera projection relationship. The camera projection relationship can be expressed as:

[0060] in, This represents the coordinates of a 3D reference point in the world coordinate system. This indicates the pixel coordinates of the 3D reference point projected onto the image to be detected. This represents the camera intrinsic parameter matrix. and These represent the rotation and translation relationships between the camera coordinate system and the world coordinate system, respectively. This represents the scaling factor for homogeneous coordinates. By calculating the pixel distances between multiple projection points, the predicted pixel scale, taking into account the shooting pose, can be obtained.

[0061] Thus, before actually detecting the target, the range of pixel sizes that the target may present can be determined in advance based on the objective imaging conditions at the time of image acquisition, so that the subsequent feature fusion process can selectively choose image features that are suitable for the current target size.

[0062] For example, if the category to be detected is a vehicle, the preset vehicle length range is 4 to 5 meters, the camera focal length is 1200 pixels, and the imaging distance is determined to be 140 to 160 meters based on imaging state parameters, then the predicted pixel scale in the vehicle length direction can be:

[0063] This means the estimated length of the vehicle in the image to be detected is approximately 30 to 43 pixels. Based on this, the electronic device can determine that the vehicle is more suitable for detection using image features with higher spatial resolution.

[0064] Step S400: Determine the fusion weights corresponding to each of the multiple image features based on the predicted pixel scale.

[0065] In this step, the electronic device matches the predicted pixel scale with the target pixel size suitable for representation at different scale levels, and determines the contribution of each image feature to the subsequent fused image features based on the degree of matching.

[0066] In one implementation, a reference pixel scale can be set for each scale level. Electronic devices can first determine the representative scale based on the predicted pixel scale:

[0067] in, This represents a representative value for the predicted pixel scale. Using the geometric mean of the upper and lower limits of the interval can reduce the influence between different size orders of magnitude.

[0068] No. The matching value between each scale level and the predicted pixel scale can be expressed as:

[0069] in, Indicates the first The degree of matching between each scale level and the predicted pixel scale. Indicates the first Reference pixel scale at each scale level This represents the scale parameter that controls the width of the matching range. The closer the predicted pixel scale is to the reference pixel scale at a certain scale level, the better the corresponding... The larger.

[0070] Subsequently, the multiple matching values ​​can be normalized to obtain the fusion weight:

[0071] in, Indicates the first The fusion weights corresponding to each image feature Indicates the number of image features. This represents a preset positive number used to prevent the denominator from being zero. After normalization, each fusion weight can be between 0 and 1, and their sum is 1.

[0072] In addition to the analytical calculation method described above, the predicted pixel scale can also be input into a trained weight generation network, which will directly output multiple fusion weights. Generally, the smaller the predicted pixel scale, the larger the fusion weight corresponding to the image features with higher spatial resolution; the larger the predicted pixel scale, the larger the fusion weight corresponding to the deeper image features with stronger semantic expressive power can be.

[0073] In this way, the target scale changes caused by imaging conditions are transformed into weights that can be directly applied to different image features, so that the target detection model no longer always uses a fixed combination of multi-scale features.

[0074] For example, the four image features are suitable for characterizing targets with pixel sizes of approximately 16 pixels, 32 pixels, 64 pixels, and 128 pixels, respectively. When the predicted pixel scale is between 30 and 43 pixels, the electronic device can determine the fusion weights of the four image features to be 0.45, 0.35, 0.15, and 0.05, respectively, so that the first two image features with higher spatial resolution occupy a larger proportion in the fusion result.

[0075] Step S500: Based on multiple fusion weights, perform weighted fusion of multiple image features to obtain fused image features.

[0076] In this step, since multiple image features have different spatial resolutions and number of channels, the electronic device can first perform size alignment and channel alignment on each image feature, and then perform weighted calculations according to the corresponding fusion weights.

[0077] Let the first Each image feature is This can be achieved through alignment operations. Obtain the aligned image features:

[0078] in, It can include at least one of upsampling, downsampling, convolution transformation, and channel mapping. For example, bilinear interpolation can be used to upsample low spatial resolution image features to a preset spatial resolution, and 1×1 convolution can be used to map each image feature to the same number of channels.

[0079] The fused image features can be represented as:

[0080] in, This represents the fused image features corresponding to the image to be detected. Indicates the first The fusion weights corresponding to each image feature This indicates the first [stage] after spatial resolution and channel alignment are completed. Image features.

[0081] In another implementation, each image feature can be multiplied by its corresponding fusion weight, and then the weighted image features can be concatenated along the channel direction and passed through a convolutional layer to obtain the fused image features. Regardless of whether summation or concatenation is used, the fusion weights are used to adjust the contribution of image features at different scales to the final fusion result.

[0082] In this way, while preserving multi-scale information, image features that match the current predicted pixel scale are utilized in a key way, thereby reducing background interference caused by features that do not match the current target scale.

[0083] For example, the electronic device uniformly adjusts the four image features to a size of 256×256×256, and then multiplies them by fusion weights of 0.45, 0.35, 0.15, and 0.05 respectively before summing them. The resulting fused image features retain both vehicle edges and textures from the high spatial resolution image features and road and vehicle category semantics from the deep image features.

[0084] Step S600: Perform target detection based on the fused image features to obtain the target detection result.

[0085] In this step, the electronic device will integrate image features into the detection model. The detection model can be an anchor-box detection model, an anchor-free detection model, or a target query-based detection model. The detection model can be configured with classification branches, location regression branches, and confidence branches.

[0086] The classification branch is used to determine the target category; the location regression branch is used to determine the target's two-dimensional bounding box in the image to be detected; and the confidence branch is used to determine the reliability of the corresponding detection result. An initial detection result can be represented as:

[0087] in, Indicates the first One test result, Represents a two-dimensional bounding box. Indicates the target category. This represents the detection confidence level. A two-dimensional bounding box can be represented as:

[0088] in, Indicates the center position of the two-dimensional bounding box. and These represent the width and height of the two-dimensional bounding box, respectively.

[0089] The electronic device can further filter out detection results with low confidence based on a preset confidence threshold, and eliminate duplicate two-dimensional bounding boxes generated for the same target through non-maximum suppression, soft non-maximum suppression, or set matching, ultimately obtaining the target detection result. The target detection result may include the target category, two-dimensional bounding box, and detection confidence of one or more targets; in subsequent embodiments, it may further include the target's three-dimensional spatial location and target trajectory.

[0090] In this way, target classification and location determination are completed based on the fused image features after adaptive adjustment of imaging state parameters, enabling the detection model to obtain feature inputs that are more consistent with the current target pixel size.

[0091] For example, the fused image features are enhanced with high spatial resolution information suitable for representing vehicles ranging from 30 to 43 pixels. The detection model identifies three vehicles in the road area based on these fused image features, outputting three two-dimensional bounding boxes, vehicle categories, and corresponding detection confidence scores, and forming the target detection result after removing overlapping duplicate two-dimensional bounding boxes.

[0092] Through steps S100 to S600, the electronic device first ensures that the image to be detected accurately corresponds to its imaging state parameters at the time of acquisition. Then, it predicts the pixel size range of the target to be detected based on the imaging state parameters and dynamically adjusts the fusion weights of image features at different spatial resolutions accordingly. Therefore, when changes in shooting distance, shooting posture, or imaging parameters cause changes in the size of the target in the image, the target detection model can adaptively select more suitable detailed features and semantic features for the current target scale. This reduces the problem of insufficient features or scale mismatch for small targets caused by fixed feature fusion methods, thereby reducing the probability of missed and false detections and improving the accuracy of target detection.

[0093] Figure 4 This is a schematic diagram of the image feature fusion process provided in the embodiments of this application. Please refer to [link / reference]. Figure 4 .

[0094] In one embodiment of this application, the image to be detected is an aerial image captured by an imaging device mounted on a flight device. The imaging state parameters include the flight altitude, flight speed, heading angle, pitch angle, and roll angle of the flight device, as well as the camera intrinsic parameters, camera extrinsic parameters, and gimbal attitude parameters of the imaging device.

[0095] Specifically, flight altitude can be obtained from a barometer, laser altimeter, satellite positioning module, or visual positioning module; flight speed, heading angle, pitch angle, and roll angle can be obtained from the flight control system and inertial measurement unit; gimbal attitude parameters can be obtained from the gimbal control module. Camera intrinsic parameters may include focal length, principal point coordinates, and distortion parameters, while camera extrinsic parameters may include rotation and translation parameters of the imaging device relative to the flight device's body coordinate system or world coordinate system.

[0096] Flight speed can be used to determine the degree of motion blur that may occur when acquiring aerial images, and to extrapolate the position between the image timestamp and the sampling time of the imaging state parameters; heading angle, pitch angle, roll angle and gimbal attitude parameters can be used together to determine the actual optical axis direction of the imaging device; camera extrinsic parameters are used to convert the attitude of the flight device into the shooting attitude of the imaging device.

[0097] Based on the imaging state parameters, the predicted pixel scale of the target object in the image to be detected is determined, including: Step A1: Based on the flight altitude, pitch angle, and gimbal attitude parameters, and in conjunction with the preset reference plane, determine the imaging distance between the imaging device and the preset reference plane; The preset reference plane is used to characterize the plane where the target to be detected is located.

[0098] Specifically, based on the flight altitude, pitch angle, and gimbal attitude parameters, and in conjunction with a preset reference plane, the imaging distance between the imaging device and the preset reference plane is determined. The preset reference plane can be a horizontal ground surface, road surface, water surface, building facade, or a locally fitted plane established based on terrain data, used to approximately represent the plane where the target to be detected is located.

[0099] Let the preset reference plane be:

[0100] in, This represents the unit normal vector of the preset reference plane. Represents a three-dimensional point in the world coordinate system. This indicates the plane parameters of the preset reference plane.

[0101] Based on the attitude of the flight equipment, the attitude of the gimbal, and the camera's extrinsic parameters, the unit direction vector of the imaging equipment's optical axis in the world coordinate system can be obtained. And based on the flight altitude or positioning results, the position of the imaging device in the world coordinate system is obtained. The spatial ray corresponding to the optical axis of the imaging device can be represented as:

[0102] The intersection parameters of the spatial ray and the preset reference plane can be expressed as:

[0103] Among them, In this case, This represents the distance parameter along the optical axis from the imaging device to the preset reference plane. When When the vector is a unit vector, the imaging distance can be determined as:

[0104] With the preset reference plane being a horizontal ground and the angle between the optical axis of the imaging device and the vertically downward direction being... In this case, the above calculation can be simplified to:

[0105] in, This indicates the vertical height of the imaging device relative to a preset reference plane. It is determined by the pitch angle, roll angle, camera extrinsic parameters, and gimbal attitude parameters of the flight equipment.

[0106] Step A2: Obtain the target size corresponding to the target to be detected, and determine the camera focal length from the camera intrinsic parameters.

[0107] Specifically, the electronic device acquires the target size corresponding to the target to be detected. The target size can be a preset true length, width, or height for the corresponding target category, or it can be a range of values ​​for the corresponding true size. For detection tasks that include multiple categories to be detected, target size priors can be set for categories such as vehicles, pedestrians, ships, or buildings.

[0108] The camera focal length can be obtained from the camera's intrinsic parameters. When using a horizontally oriented target size, the horizontal focal length can be used. When using the target size in the vertical direction, the vertical focal length can be used. When using only a representative focal length, one can use Alternatively, select the appropriate focal length based on the direction in which the target extends.

[0109] Step A3: Determine the product of the camera focal length and the target size, and use the ratio of the product to the imaging distance as the predicted pixel scale.

[0110] Specifically, electronic devices determine the camera's focal length. With target size The product of the product and the imaging distance. The ratio is determined as the predicted pixel scale:

[0111] in, This represents the predicted pixel scale of the target to be detected in the aerial image. This represents the camera focal length in pixels. Indicates the true size of the target to be detected. This indicates the imaging distance between the imaging device and the preset reference plane.

[0112] If the target size is an interval Furthermore, considering flight altitude, terrain error, or attitude measurement error, the imaging distance is expressed as an interval. When the predicted pixel scale is, it can be expressed as:

[0113] Therefore, the predicted pixel size, which includes measurement error and the difference between the actual size of the target, can be obtained.

[0114] Thus, by combining the altitude and attitude of the flight equipment, the gimbal attitude, camera intrinsic parameters, and prior calculation of the target size to determine the imaging distance and predicted pixel scale, the target scale prediction can be updated as aerial photography conditions change, improving the consistency between the predicted pixel scale and the actual image size of the target.

[0115] For example, if the flying device is flying at an altitude of 120 meters above the ground, and the angle between the optical axis of the imaging device and the vertically downward direction is 30 degrees, then the imaging distance is approximately:

[0116] If the camera focal length is 1200 pixels and the vehicle length ranges from 4 to 5 meters, the predicted pixel scale of the vehicle is approximately 34.6 to 43.3 pixels. Electronic devices can then use this to increase the fusion weights of high spatial resolution image features suitable for detecting vehicles of this pixel size.

[0117] In one embodiment of this application, determining the fusion weights corresponding to multiple image features based on the predicted pixel scale may include vectorizing the imaging state parameters, determining the scale adjustment factor, and generating the fusion weights using a preset weight generation function.

[0118] Step B1: Vectorize and encode the imaging state parameters to obtain state features.

[0119] Specifically, the electronic device arranges at least some of the parameters from flight altitude, flight speed, heading angle, pitch angle, roll angle, camera intrinsic parameters, camera extrinsic parameters, and gimbal attitude parameters in a preset order and performs normalization processing to obtain an initial state vector. For the first... A scalar parameter can be standardized in the following ways:

[0120] in, Indicates the first time when the image to be detected was acquired. One imaging state parameter, and These represent the mean and standard deviation of the parameter in the training data, respectively. This indicates a preset positive number to prevent the denominator from being zero. This represents the standardized parameters.

[0121] For periodic angles such as heading angle, the angle can be... Encoded as This avoids the problem that 0 degrees and 360 degrees are numerically far apart but physically similar. Camera extrinsic parameters and gimbal attitude can also be converted into rotation matrices, rotation vectors, or quaternions before vectorization.

[0122] Electronic devices can input normalized state vectors into parameter encoding networks to obtain state features. For example, state features can be represented as:

[0123] in, This represents the vectorized and normalized imaging state parameters. and This represents the weight matrix in a parameter encoding network. and This represents the bias vector. Represents a non-linear activation function. It represents state characteristics. A parameter encoding network can consist of fully connected layers, normalization layers, and activation layers.

[0124] Step B2: Determine the scale adjustment factor based on the predicted pixel scale.

[0125] Specifically, the electronic device determines a scaling factor based on the predicted pixel scale. The scaling factor can be a scalar or a vector comprising multiple components. When there exists... When there are multiple scale levels, the scale adjustment factor can be expressed as:

[0126] in, Indicates the predicted pixel scale and the first The degree of matching between different scale levels.

[0127] Let the representative value of the predicted pixel scale be... , No. The reference pixel scale for each scale level is Then the scale adjustment factor of the first Each component can be represented as:

[0128] in, Indicates the first The allowable scale deviation range for each scale level. The closer the predicted pixel scale is to the desired scale... Corresponding The larger the scale, the better. By comparing in logarithmic scale space, the scale variations of smaller and larger targets can be handled more evenly.

[0129] Step B3: Based on the preset weight generation function, obtain the fusion weights corresponding to each of the multiple image features according to the scale adjustment factor and state features.

[0130] Specifically, the electronic device will adjust the scale factor. and state characteristics Input preset weight generation function This process yields the fusion weights corresponding to each of the multiple image features. This process can be represented as:

[0131]

[0132] in, This represents the concatenation result of the scale adjustment factor and state features. This represents the parameters of the preset weight generation function. This represents the unnormalized weights corresponding to multiple scale levels. Indicates the first Unnormalized weights, Indicates the first Fusion weights corresponding to each image feature. Preset weight generation function. It can consist of one or more fully connected layers, or it can be a lookup table function based on a preset scale interval. Softmax normalization ensures that the sum of multiple fusion weights is equal to 1.

[0133] State features can further reflect imaging conditions that are difficult to fully represent based solely on target pixel size, such as flight speed, shooting tilt, and gimbal status. For example, when the predicted pixel scale is the same but the flight speed is high, a preset weight generation function can appropriately increase the fusion weight of image features containing stable semantic information to reduce the unreliability of local details caused by motion blur.

[0134] In this way, by using the predicted pixel scale and the complete imaging state parameters together to generate the fusion weights, the fusion weights can adapt to changes in target size and image acquisition state, thereby improving the utilization efficiency of multi-scale image features.

[0135] For example, the reference pixel scales for the four scale levels are 16 pixels, 32 pixels, 64 pixels, and 128 pixels, respectively, while the representative value for the currently predicted pixel scale is 28 pixels. The scale adjustment factor indicates that the first two scale levels match the predicted pixel scale well; meanwhile, the state features indicate that the flying device is at a high flight altitude and moving at a high speed. The preset weight generation function can output fusion weights of 0.50, 0.32, 0.13, and 0.05, giving high spatial resolution image features higher weights while retaining a certain proportion of deep semantic information.

[0136] Figure 5 This is another schematic diagram of the image feature fusion process provided in the embodiments of this application. Please refer to [link / reference]. Figure 5 .

[0137] In one embodiment of this application, multi-scale feature extraction of the image to be detected may further include obtaining multiple basic image features using a feature extraction model and performing small target detail enhancement on the shallow basic image features.

[0138] Multi-scale feature extraction is performed on the image to be detected to obtain image features corresponding to multiple scale levels, including: Step C1: Input the image to be detected into the feature extraction model.

[0139] Specifically, the electronic device inputs the image to be detected into the feature extraction model. The feature extraction model, or backbone network, can consist of multiple sequentially connected convolutional processing stages, or it can consist of both convolutional processing stages and visual Transformer processing stages. Each processing stage can include at least one of the following: convolutional layers, normalization layers, activation layers, residual structures, attention structures, and downsampling structures.

[0140] Step C2: Use different network layers in the feature extraction model to extract features from the image to be detected, and obtain multiple basic image features with different spatial resolutions.

[0141] Specifically, features are extracted from the image to be detected using different network layers in the feature extraction model, resulting in multiple basic image features with different spatial resolutions. These multiple basic image features can be denoted as... The downsampling factor relative to the image to be detected can be 4x, 8x, 16x, and 32x, respectively. These markings are only used to distinguish different network layers and do not constitute a limitation on the number of basic image features or the specific downsampling factor.

[0142] Step C3: Determine shallow basic image features from multiple basic image features, wherein the shallow basic image features are output by the network layer near the input end in the backbone network, and the spatial resolution of the shallow basic image features is higher than the spatial resolution of at least one other basic image feature.

[0143] Specifically, shallow basic image features are determined from multiple basic image features. These shallow basic image features are output by network layers near the input in the feature extraction model and have high spatial resolution. For example, they can be... This is identified as a shallow base image feature. Shallow base image features typically retain more edge, corner, texture, and local location details, but their channel count or semantic expressive power may be weaker than that of deeper base image features.

[0144] Step C4: Perform convolution transformation on the shallow basic image features to obtain the small target detail features.

[0145] Specifically, convolutional transformation is performed on the shallow basic image features to obtain the detail features of small targets. Let the shallow basic image features be... Then the detailed features of the small target can be represented as:

[0146] in, This represents the convolution transformation used to adjust the number of channels. This represents a convolutional transformation used to extract local spatial features. This represents the activation function. This represents the detailed features of small targets. Adjusting the number of channels through 1×1 convolution can reduce the computational cost of subsequent attention processing; further extraction of edges and local textures around small targets can be achieved through 3×3 convolution.

[0147] Step C5: Perform spatial attention processing and channel attention processing on the small target detail features in sequence to obtain enhanced detail features.

[0148] Specifically, spatial attention and channel attention are applied sequentially to the detailed features of small targets. Spatial attention is used to determine which spatial locations in the image are more likely to contain the target to be detected. The spatial attention map can be represented as:

[0149]

[0150] in, and These represent average pooling and max pooling performed along the channel direction, respectively; square brackets indicate feature concatenation. This indicates a convolution operation with a kernel size of 7×7; Represents the Sigmoid function; Represents a spatial attention map; This indicates element-wise multiplication; This represents the detailed features after spatial attention processing. Spatial attention maps can enhance the feature responses of suspected target regions and suppress large areas of irrelevant background regions.

[0151] Subsequently, Channel attention processing is performed. The channel attention weights can be represented as:

[0152]

[0153] in, and These represent global average pooling and global max pooling along the spatial dimension, respectively. This represents a multilayer perceptron used to generate channel weights; Indicates channel attention weights; This indicates enhanced detail features. Channel attention processing can enhance feature channels related to target texture, edges, or shape, while suppressing background noise channels related to road textures, building edges, tree shadows, or water reflections.

[0154] Step C6: Determine the enhanced detail features and the basic image features other than the shallow basic image features from the multiple basic image features as the image features corresponding to each of the multiple scale levels.

[0155] Specifically, the enhanced detail features and the basic image features (excluding shallow basic image features) from multiple basic image features are identified as image features corresponding to multiple scale levels. Specifically, the enhanced detail features can be identified as image features corresponding to one scale level, and... Other basic image features are determined as image features corresponding to the remaining scale levels.

[0156] It should be noted that this step only replaces the original shallow basic image features with enhanced detail features to form multiple image features to be fused subsequently; this step does not perform weighted fusion based on imaging state parameters between different scale levels. Weighted fusion between image features at different scale levels is performed after their respective fusion weights have been determined.

[0157] Thus, by sequentially applying convolutional transformation, spatial attention processing, and channel attention processing to the shallow basic image features with high spatial resolution, it is possible to suppress complex background interference while preserving the location and edge details of small targets, thereby reducing the probability of small target features being lost during deep downsampling.

[0158] For example, for a 1024×1024 pixel aerial image, the feature extraction model outputs four basic image features: 256×256, 128×128, 64×64, and 32×32. The electronic device performs convolutional transformation on the 256×256 shallow basic image features and sequentially enhances pedestrian or vehicle features, which occupy only a dozen or so pixels, through spatial attention and channel attention. Subsequently, the resulting enhanced detail features and the other three basic image features are determined as image features corresponding to four scale levels, respectively, for subsequent scale-aware weighted fusion.

[0159] Figure 6 This is a schematic diagram of the target detection process provided in the embodiments of this application. Please refer to it. Figure 6 .

[0160] In one embodiment of this application, target detection based on fused image features may further include candidate target detection, three-dimensional spatial mapping, three-dimensional spatial verification, and correction of candidate targets based on three-dimensional spatial location.

[0161] Target detection is performed based on the fused image features to obtain target detection results, including: Step D1: Use the detection model to determine multiple candidate targets based on the features of the fused image. Each candidate target includes a two-dimensional bounding box, target category, and detection confidence.

[0162] Specifically, the electronic device uses a detection model to determine multiple candidate targets based on the features of the fused image. The detection model, or detection head, can be an anchor-frame detection head, an anchor-free detection head, or a target query-based detection head. Each candidate target can be represented as:

[0163] in, Indicates the first Two-dimensional bounding boxes of candidate targets, Indicates the target category. This represents the detection confidence level. Electronic devices can initially use a lower initial confidence threshold to retain more candidate targets, and then use three-dimensional spatial constraints to further filter them, thereby reducing the probability of deleting real small targets by relying solely on two-dimensional confidence.

[0164] Step D2: Map each candidate target to three-dimensional space to obtain the three-dimensional spatial position of each candidate target.

[0165] Specifically, each candidate target is mapped to a three-dimensional space. The electronic device can obtain the center pixel coordinates of the two-dimensional bounding box of the candidate target. For targets such as vehicles and pedestrians that are in contact with the ground, the midpoint of the lower edge of the two-dimensional bounding box can be determined as the representative pixel point corresponding to the target and the preset reference plane.

[0166] The camera intrinsic parameter matrix can be represented as:

[0167] in, and These represent the horizontal focal length and the vertical focal length, respectively. and These represent the coordinates of the principal point.

[0168] When the depth corresponding to the candidate target is obtained At this time, we can first perform back projection based on the camera intrinsic parameter matrix and depth to obtain the three-dimensional coordinates of the candidate target in the camera coordinate system:

[0169] in, Indicates the first The three-dimensional coordinates of each candidate target in the camera coordinate system This indicates the depth of the target along the camera's optical axis. When When representing the actual spatial distance between the imaging device and the target, rather than the optical axis depth, one can first... Unitize the result and then multiply it by the actual spatial distance.

[0170] Subsequently, the 3D coordinates in the camera coordinate system are transformed to the world coordinate system based on the camera extrinsic parameters:

[0171] in, Indicates the first The three-dimensional spatial location of each candidate target This represents the rotation matrix from the camera coordinate system to the world coordinate system. This represents the translation vector from the camera coordinate system to the world coordinate system.

[0172] Without directly obtaining the depth of the candidate target, the depth can be estimated by the intersection of a spatial ray and a preset reference plane, based on the flight altitude, the imaging device's shooting attitude, and the preset reference plane. Specifically, the ray direction corresponding to the candidate target in the camera coordinate system is:

[0173] Transforming it to world coordinates and normalizing it, we get:

[0174] Let the position of the imaging device in the world coordinate system be... The preset reference plane is Then the intersection parameters of the space ray and the preset reference plane are:

[0175] The corresponding three-dimensional spatial location is:

[0176] Therefore, even without a depth sensor, the three-dimensional spatial position of a candidate target can be determined using flight altitude, shooting attitude, camera intrinsic parameters, and camera extrinsic parameters.

[0177] Step D3: Verify the three-dimensional spatial position of each candidate target according to the preset spatial constraints. The preset spatial constraints are used to characterize the physical range of the target of the corresponding target category in three-dimensional space.

[0178] Specifically, the electronic device verifies the three-dimensional spatial position of each candidate target according to preset spatial constraints. The preset spatial constraints can be set separately for different target categories and include at least one of the following: Whether the three-dimensional spatial location of the candidate target is within the preset target area. For example, vehicles should be located on roads or in parking areas, boats should be located in water areas, and ground personnel should be located in accessible ground areas.

[0179] Whether the three-dimensional height of the candidate target is within the preset height range of the corresponding target category. For example, the main height of a vehicle should be within a certain height range relative to the ground; if a candidate target identified as a vehicle is projected onto the roof of a building, it can be determined that it does not meet the spatial constraints of the vehicle category.

[0180] The estimated true size of the target, based on the 2D bounding box and 3D spatial location, is determined to be within the preset size range of the corresponding target category. The estimated width and height of the candidate target can be approximated as follows:

[0181] in, and Let represent the pixel width and pixel height of the candidate target's 2D bounding box, respectively. and These represent the estimated true width and the estimated true height, respectively.

[0182] Does the distance between the candidate target and the imaging device conform to the imaging scale relationship between the candidate target image size and its actual size? For example, the size of the observed pixels can be determined. Does it meet the following requirements:

[0183] in, This indicates the prior target size corresponding to the candidate target category. This indicates the permissible scale error.

[0184] In the case of historical image frames, does the 3D spatial position of the candidate target in consecutive image frames satisfy the preset motion continuity condition? For example:

[0185] in, This indicates the three-dimensional spatial location of the candidate target in the current image frame. This indicates the three-dimensional spatial position of the corresponding target in the previous image frame. This indicates the maximum allowable speed for the corresponding target category. This represents the time interval between two image frames. This indicates the position measurement error margin.

[0186] Step D4: Retain candidate targets that meet the preset spatial constraints, and correct the two-dimensional bounding boxes and detection confidence of the retained candidate targets according to their corresponding three-dimensional spatial positions.

[0187] Specifically, for candidate targets that meet preset spatial constraints, the electronic device retains the candidate target and corrects its 2D bounding box and detection confidence based on its 3D spatial location. The electronic device can reproject the 3D target range onto the image to be detected based on the prior knowledge of the candidate target's 3D spatial location and target size, thus obtaining the predicted 2D bounding box corresponding to the 3D spatial constraints. Then, it is weighted and summed with the two-dimensional bounding box output by the detection model:

[0188] in, This represents the corrected 2D bounding box. This represents the position correction factor. When the measurement accuracy of the three-dimensional spatial position is high, it can be increased. When the three-dimensional spatial position is mainly estimated through the assumption of a ground plane and the uncertainty is large, the [efforts] can be reduced. .

[0189] Spatial consistency scores can also be determined based on the degree to which multiple spatial constraints are satisfied. And correct the detection confidence level according to the following formula:

[0190] in, This indicates the corrected detection confidence level. This represents the confidence level correction factor. Indicates the spatial consistency benchmark value, This indicates that the results will be limited to between 0 and 1. For candidate targets that do not meet the preset spatial constraints, they can be removed from the current target detection results, or their detection confidence can be reduced and removed when it falls below the output threshold.

[0191] Step D5: Obtain the target detection results based on the retained and corrected candidate targets.

[0192] Specifically, the electronic device obtains target detection results based on the retained and corrected candidate targets. The electronic device can summarize candidate targets that have passed 3D spatial constraint verification and meet preset spatial constraint conditions, and determine the corrected 2D bounding box, target category, and corrected detection confidence score corresponding to each candidate target as the detection information for that candidate target. Furthermore, the 3D spatial position of the candidate target can also be stored as additional detection information, corresponding to the corrected 2D bounding box, target category, and corrected detection confidence score.

[0193] The target detection result can be expressed as:

[0194] in, This represents the target detection result corresponding to the image to be detected; This represents the set of indices of candidate targets that meet the preset spatial constraints and are retained. Indicates the first A two-dimensional bounding box of the retained and corrected candidate targets; This indicates the target category of the candidate target; This indicates the corrected detection confidence level; This indicates the three-dimensional spatial location of the candidate target.

[0195] In one implementation, since the 2D bounding boxes may form new overlapping relationships between the 2D bounding boxes of different candidate targets after 3D spatial position correction, the electronic device can further perform duplicate target elimination processing on the retained and corrected candidate targets. For example, based on the intersection-union ratio between the corrected 2D bounding boxes and the target category, non-maximum suppression can be performed on multiple candidate targets to merge or delete duplicate candidate targets corresponding to the same target object. The electronic device determines all candidate targets after duplicate target elimination as the target detection result. When there are no candidate targets that meet the preset spatial constraints, the target detection result can be determined as an empty set.

[0196] The obtained target detection results can be output directly, or used as input for subsequent target association and temporal optimization processing when the image to be detected is the current image frame in the video stream. Thus, each target in the target detection results has undergone 3D spatial constraint verification and 2D bounding box and detection confidence correction, which can reduce false detection targets with unreasonable spatial location, true size or imaging scale.

[0197] In this way, by mapping two-dimensional candidate targets to three-dimensional space and verifying them based on the relationship between the target category and its position, height, actual size and imaging scale, candidate targets that look similar in two-dimensional images but have unreasonable physical positions or sizes can be eliminated, thereby reducing false detections caused by complex aerial backgrounds.

[0198] For example, the detection model identifies a highlighted area on a building rooftop as a vehicle and assigns a detection confidence score of 0.86. After back-projecting this candidate target to the world coordinate system, the electronic device determines it is located 12 meters above the ground, thus failing to meet the spatial constraint that a vehicle should be located on a road or near the ground, and the candidate target is deleted. Another vehicle candidate target, after back-projection, is located in the road area, with an estimated true length of 4.6 meters, which falls within the size range for vehicle categories. The electronic device retains this candidate target and corrects its 2D bounding box and detection confidence score based on the 3D projection results.

[0199] Figure 7 This is a schematic diagram illustrating the process of optimizing target detection results according to an embodiment of this application. Please refer to [link / reference]. Figure 7 .

[0200] In one embodiment of this application, the image to be detected is the current image frame in a video stream acquired by an imaging device. After performing target detection based on fused image features and obtaining the target detection result, the electronic device can use multiple historical image frames to perform temporal optimization on the target detection result of the current image frame.

[0201] Step E1: Select the retained and corrected candidate targets as reference targets.

[0202] Specifically, the electronic device identifies candidate targets in the current image frame that have been retained and corrected after undergoing three-dimensional spatial constraint verification as reference targets. Thus, the reference targets participating in temporal correlation have already undergone two-dimensional detection and three-dimensional spatial constraint processing, reducing the probability of falsely detected targets that clearly do not meet spatial physical conditions entering the temporal correlation process.

[0203] Step E2: Obtain the historical target detection results corresponding to each of the multiple historical image frames that are located before the current image frame in the video stream in chronological order. The multiple historical target detection results jointly represent multiple historical targets. A historical target is a target object detected in at least one historical image frame. The detection results of the same target object in different historical image frames belong to the same historical target.

[0204] Specifically, the electronic device acquires the historical target detection results corresponding to multiple historical image frames that are chronologically preceding the current image frame in the video stream. The historical target detection results can be the target detection results after 3D spatial constraint processing of the corresponding historical image frames, or the detection results after temporal optimization of the corresponding historical image frames.

[0205] Each historical image frame corresponds to one historical target detection result, and multiple historical target detection results collectively represent multiple historical targets. A historical target is a target object detected in at least one historical image frame. The detection results of the same actual target object in different historical image frames can have the same target identifier and belong to the same historical target. For example, the five two-dimensional bounding boxes and three-dimensional spatial positions of the same vehicle in five consecutive historical image frames together constitute the historical detection information of a historical target.

[0206] Step E3: For each historical target, based on the detection results of multiple historical targets, predict the target state of the historical target in the current image frame, where the target state includes the predicted position and the predicted motion state.

[0207] Specifically, for each historical target, the electronic device predicts the target state in the current image frame based on the detection results of multiple historical targets. The target state can include predicted position and predicted motion state. The predicted position can be a two-dimensional image coordinate position or a three-dimensional world coordinate position; the predicted motion state can include at least one of velocity, acceleration, motion direction, rate of change of two-dimensional bounding box size, and motion state uncertainty.

[0208] In one implementation, the first The state vector of a historical objective can be represented as:

[0209] in, , and Indicates the three-dimensional position of the target. , and This indicates the target's velocity in three directions. and This represents the width and height of the two-dimensional bounding box after the target is projected onto the image.

[0210] When using Kalman filtering for state prediction, it can be expressed as:

[0211]

[0212] in, This indicates the historical target state updated based on the previous image frame. This indicates the predicted target state of the historical target in the current image frame. Represents the state transition matrix. This represents the state covariance corresponding to the previous image frame. This represents the predicted state covariance. This represents the process noise covariance. The predicted state covariance can characterize the degree of uncertainty in predicting the predicted position and the predicted motion state.

[0213] In addition to Kalman filtering, the target state of historical targets can be predicted based on preset uniform motion models, preset acceleration motion models, or optical flow estimation. When using optical flow estimation, the predicted position of the historical target in the current image frame can be estimated based on the displacement of multiple pixels within the two-dimensional bounding box of the historical target between adjacent image frames; it can also be combined with the motion state of the flight equipment itself to compensate for the global image displacement caused by the motion of the imaging equipment.

[0214] Step E4: Based on the target states corresponding to each of the multiple historical targets, associate each reference target in the current image frame with the multiple historical targets to obtain the target association result. The target association result is used to characterize whether each reference target corresponds to the same target object as each historical target.

[0215] Specifically, based on the target states corresponding to multiple historical targets, each reference target in the current image frame is associated with each of the multiple historical targets to obtain the target association results. When performing target association, the intersection-over-union ratio of two-dimensional bounding boxes, three-dimensional spatial distance, target category consistency, and appearance feature similarity can be used comprehensively.

[0216] No. The first reference target and the first The associated cost between historical goals can be expressed as:

[0217] in, Indicates the first Two-dimensional bounding boxes of a reference target. Indicates the first Predicted two-dimensional bounding boxes for historical targets. Indicates the intersection-union ratio of a two-dimensional bounding box; Indicates the three-dimensional spatial position of the reference target. Indicates the predicted location of historical targets. This represents the three-dimensional distance normalization parameter; and These represent the appearance features of the reference target and historical targets, respectively. Indicates cosine similarity; and These represent the target categories of the two, Indicates an indicator function; to This indicates the weight of each related factor.

[0218] Electronic devices can first use preset distance thresholds, intersection-union ratio thresholds, and category consistency conditions to exclude obviously mismatched target pairs, and then use Hungarian matching algorithm, greedy matching algorithm, or graph-based matching algorithm to determine the one-to-one correspondence.

[0219] The result of the target association can be represented as an association matrix:

[0220] in:

[0221] Therefore, the target association results can clearly characterize whether each reference target corresponds to the same target object as each historical target.

[0222] Step E5: Correct the target detection result corresponding to the current image frame based on the target association result to obtain the temporally optimized detection result.

[0223] Specifically, the electronic device corrects the target detection result corresponding to the current image frame based on the target association result. For a successfully matched reference target, its position can be smoothly corrected based on the predicted position of the corresponding historical target; for historical targets that have not yet matched a reference target, short-term false detection compensation can be implemented using the predicted position; for reference targets that do not match any historical target and only appear in the current image frame, instantaneous false detection suppression can be implemented. After the above processing, a time-optimized detection result is obtained.

[0224] In this way, by using multiple historical image frames to jointly predict the target state of historical targets in the current image frame, and by performing target association between the reference target after verification in three-dimensional space and historical targets, the detection box jumps, missed detections and false detections caused by flight equipment movement, image jitter, motion blur and short-term occlusion can be reduced.

[0225] For example, the same vehicle was detected in the three historical image frames preceding the current image frame. The electronic device predicts the vehicle's position in the current image frame based on its historical 3D position and speed, and then associates each reference target in the current image frame with the vehicle. If one of the reference targets is close to the predicted position, has a high 2D bounding box intersection ratio, and similar appearance features, the target association result indicates that the two correspond to the same vehicle, and the electronic device can correct the position of the current reference target based on the predicted position.

[0226] In one embodiment of this application, correcting the target detection result corresponding to the current image frame based on the target association result may include correcting the position of the matching target, supplementing the unmatched historical targets, and deleting the candidate targets in a single frame.

[0227] Step F1: For each historical target, determine whether there is a reference target in the current image frame that corresponds to the same target object as the historical target, based on the target association results.

[0228] Specifically, for each historical target, the electronic device determines whether there is a reference target in the current image frame that corresponds to the same target object as the historical target, based on the target association results. The electronic device can check whether there is an association identifier with a value of 1 in the column corresponding to the historical target in the target association matrix. If it exists, it indicates that the historical target has a successfully matched reference target in the current image frame; if it does not exist, it indicates that no reference target matching the historical target has been detected in the current image frame.

[0229] Step F2: If there is a reference target in the current image frame that corresponds to the same target object as the historical target, correct the target position of the corresponding reference target according to the predicted position of the historical target, and retain the reference target with the corrected target position in the target detection result corresponding to the current image frame.

[0230] Specifically, if a reference target exists in the current image frame that corresponds to the same target object as the historical target, the target position of the corresponding reference target is corrected based on the predicted position of the historical target. The corrected target position can be represented as:

[0231] in, Indicates the first The original target position of each reference target in the current image frame. Indicates the first matching with it Predicted locations of historical targets, Indicates the corrected target location. This represents the fusion coefficient.

[0232] When the detection confidence of the reference target is high and its three-dimensional spatial position is reliable, the detection confidence can be increased. When the current image frame has motion blur or the detection confidence of the reference target is low, the [measurement level] can be reduced. This allows the correction results to incorporate more of the predicted positions of historical targets. The reference target with the corrected position is retained in the target detection results of the current image frame, and this reference target can also be used to update the motion state and target identifier of the corresponding historical targets.

[0233] Step F3: If there is no reference target in the current image frame that corresponds to the same target object as the historical target, determine the predicted position of the historical target as the target position of the historical target in the current image frame, and add the historical target to the target detection result corresponding to the current image frame.

[0234] Specifically, if there is no reference target in the current image frame that corresponds to the same target object as the historical target, the electronic device determines the predicted position of the historical target as the target position of the historical target in the current image frame and adds the historical target to the target detection result corresponding to the current image frame. The added detection result can inherit the target category of the historical target and generate a two-dimensional bounding box in the current image frame based on the two-dimensional bounding box size, target motion state, and predicted position in the most recent historical image frame.

[0235] To avoid generating false targets by relying on predicted locations over a long period, a maximum number of consecutive compensation frames can be set. And maximum prediction uncertainty. When the number of consecutive unmatched image frames for the historical target does not exceed And if the predicted state covariance does not exceed a preset threshold, then a missed detection compensation is performed. The detection confidence of the supplementary target can also decrease with the number of consecutive unmatched frames:

[0236] in, This indicates the detection confidence level of the supplementary target. This indicates the detection confidence level when the historical target was most recently detected. This indicates the number of consecutive unmatched image frames. This represents the confidence decay coefficient between 0 and 1.

[0237] Step F4: For each reference target, if the target association results show that the reference target does not correspond to the same target object as any historical target, and the reference target only appears in the current image frame, then the reference target is deleted from the target detection results corresponding to the current image frame.

[0238] Specifically, for each reference target, if the target association results show that the reference target does not correspond to the same target object as any historical target, and the reference target only appears in the current image frame, then the reference target is deleted from the target detection results corresponding to the current image frame.

[0239] "Appears only in the current image frame" means that no historical target matching the reference target in terms of category, location, and appearance features exists in a predetermined number of historical image frames preceding the current image frame. The electronic device can temporarily store the reference target in an internal candidate buffer but not output it as the official target detection result of the current image frame; if a candidate target corresponding to the same target object is detected again in a subsequent image frame, it can be converted into a new target confirmed by consecutive frames. Thus, while suppressing false detections in a single frame, it avoids permanently excluding real targets that have just entered the shooting range.

[0240] After completing the position correction of the matched target, the supplementation of unmatched historical targets, and the deletion of single-frame reference targets, the electronic device determines the target detection result corresponding to the current image frame as the temporally optimized detection result.

[0241] In this way, by processing the matching target, the historical target that was not detected for a short time, and the reference target that appears only in a single frame, we can not only compensate for the missed detection caused by short-term occlusion or motion blur, but also remove the instantaneous false detection caused by reflection, shadow or noise, thereby improving the continuity and accuracy of video target detection results.

[0242] For example, multiple historical image frames collectively represent historical target A and historical target B. In the current image frame, there exists a reference target that matches historical target A. The electronic device corrects the position of this reference target based on the predicted position of historical target A. Historical target B, due to temporary occlusion by trees, has no matching reference target. The electronic device adds historical target B to the target detection result of the current image frame based on its predicted position. The current image frame also contains a reference target C formed by reflections from the water surface, but it does not match any historical target and only appears in the current image frame. The electronic device removes reference target C from the target detection result, ultimately obtaining a time-optimized detection result.

[0243] In one embodiment of this application, the electronic device can also determine the target trajectory based on the timing optimization result. In this embodiment, the timing optimization result can be the aforementioned timing optimization detection result, that is, the current image frame detection information after target position correction, historical target supplementation, and single-frame reference target deletion.

[0244] Step G1: Based on the time-series optimization results, determine the target positions of multiple target objects in the current image frame, and connect the target positions of the same target object in multiple historical image frames and the current image frame in chronological order to determine the target trajectory corresponding to the target object.

[0245] Specifically, the electronic device determines the target positions of multiple target objects in the current image frame based on the timing optimization results. The target position can be the center of a two-dimensional bounding box, the midpoint of the lower edge of a two-dimensional bounding box, or a three-dimensional world coordinate position. For imaging devices mounted on flying equipment, it is preferable to use three-dimensional world coordinate positions to generate target trajectories to eliminate the influence of the flying equipment's own movement and changes in the imaging image on the target trajectory.

[0246] For target locations obtained through actual reference targets, their detection source identifier can be retained; for target locations obtained through compensation for missed detections based on predicted locations of historical targets, a compensation source identifier and corresponding confidence level can be set. Thus, when outputting target trajectories, actual detection points and predicted compensation points can be distinguished.

[0247] Specifically, based on the target association results or target identifiers, it is determined whether the detection results in different image frames correspond to the same target object. For each target object, the target positions in multiple historical image frames and the current image frame are connected in chronological order to determine the target trajectory corresponding to that target object. The target object at time The corresponding target trajectory can be represented as:

[0248] in, Indicates the first The target trajectory of a target object This indicates the number of image frames involved in trajectory generation. Indicates the first The acquisition time of each image frame Indicates the first The target position of each target object in the corresponding image frame.

[0249] Electronic devices can directly connect multiple target locations in chronological order to form a broken line trajectory, or they can perform moving averages, exponential smoothing, spline fitting, or Kalman smoothing on multiple target locations. For example, the target locations after exponential smoothing can be represented as:

[0250] in, Indicates the smoothed target position. Indicates the current target location. Indicates the smoothed position at the previous moment. This represents the smoothing coefficient between 0 and 1.

[0251] The target trajectory, along with the target category, two-dimensional bounding box, detection confidence level, and three-dimensional spatial position, can be output to the flight control system, mission planning module, obstacle avoidance module, or ground station display platform of the flight equipment for target tracking, motion trend judgment, anomaly alarm, and auxiliary path planning.

[0252] In this way, based on the target association results, the target positions of the same target object in multiple historical image frames and the current image frame are continuously organized, so that the target detection result of a single frame is transformed into a target trajectory with temporal continuity, thereby improving the reliability of target motion state judgment and subsequent tracking processing.

[0253] For example, in three historical image frames and the current image frame, the target association results show that all four detection results correspond to the same vehicle. The target positions of this vehicle in the world coordinate system are as follows: , , and The electronic device connects the target locations according to the time sequence of the corresponding image frames to obtain the target trajectory of the vehicle moving from west to east. For other vehicles or pedestrians in the same frame, the electronic device generates corresponding target trajectories based on their respective target identifiers.

[0254] Please see Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. A third aspect of this application provides an electronic device 1000, including a processor 1100 and a memory 1200. The memory 1200 stores machine-executable instructions that can be executed by the processor 1100. The processor 1100 can execute the machine-executable instructions to implement the above-mentioned target detection method.

[0255] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, cause the processor to implement the target detection method described above.

[0256] In some embodiments, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the target detection method according to the above embodiments.

[0257] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0258] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0259] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0260] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0261] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include temporary computer-readable media, such as modulated data signals and carrier waves.

[0262] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0263] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0264] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A target detection method, characterized in that, The method includes: Acquire the image to be detected and the imaging state parameters corresponding to the image to be detected, wherein the imaging state parameters are used to characterize the shooting distance, shooting posture and imaging parameters of the imaging device used to acquire the image to be detected when acquiring the image to be detected; Multi-scale feature extraction is performed on the image to be detected to obtain image features corresponding to multiple scale levels, wherein the image features at different scale levels have different spatial resolutions; Based on the imaging state parameters, the predicted pixel scale of the target to be detected in the image to be detected is determined, wherein the predicted pixel scale is used to characterize the expected pixel size of the target to be detected in the image to be detected; Based on the predicted pixel scale, determine the fusion weights corresponding to each of the multiple image features; Based on the multiple fusion weights, the multiple image features are weighted and fused to obtain fused image features; Target detection is performed based on the fused image features to obtain the target detection result.

2. The method according to claim 1, characterized in that, The image to be detected is an aerial image captured by an imaging device installed on the flight equipment. The imaging state parameters include the flight altitude, flight speed, heading angle, pitch angle and roll angle of the flight equipment, as well as the camera intrinsic parameters, camera extrinsic parameters and gimbal attitude parameters of the imaging device. Determining the predicted pixel scale of the target in the image to be detected based on the imaging state parameters includes: Based on the flight altitude, the pitch angle, and the gimbal attitude parameters, and in conjunction with a preset reference plane, the imaging distance between the imaging device and the preset reference plane is determined, wherein the preset reference plane is used to characterize the plane where the target to be detected is located; Obtain the target size corresponding to the target to be detected, and determine the camera focal length from the camera intrinsic parameters; The product of the camera focal length and the target size is determined, and the ratio of the product to the imaging distance is determined as the predicted pixel scale.

3. The method according to claim 1, characterized in that, The step of determining the fusion weights corresponding to each of the multiple image features based on the predicted pixel scale includes: The imaging state parameters are vectorized and encoded to obtain state features; Determine the scale adjustment factor based on the predicted pixel scale; Based on a preset weight generation function, the fusion weights corresponding to each of the multiple image features are obtained according to the scale adjustment factor and the state features.

4. The method according to claim 1, characterized in that, The step of performing multi-scale feature extraction on the image to be detected to obtain image features corresponding to multiple scale levels includes: The image to be detected is input into the feature extraction model; By using different network layers in the feature extraction model, features are extracted from the image to be detected to obtain multiple basic image features with different spatial resolutions. Shallow basic image features are determined from a plurality of said basic image features, wherein the shallow basic image features are output by a network layer near the input end in the backbone network, and the spatial resolution of the shallow basic image features is higher than the spatial resolution of at least one other basic image feature. The shallow basic image features are subjected to convolution transformation to obtain small target detail features; The small target detail features are sequentially processed with spatial attention and channel attention to obtain enhanced detail features. The enhanced detail features and the basic image features other than the shallow basic image features among the multiple basic image features are determined as the image features corresponding to each of the multiple scale levels.

5. The method according to claim 1, characterized in that, The step of performing target detection based on the fused image features to obtain target detection results includes: Multiple candidate targets are determined using a detection model based on the features of the fused image, wherein each candidate target includes a two-dimensional bounding box, a target category, and a detection confidence score; Each candidate target is mapped to a three-dimensional space to obtain the three-dimensional spatial position of each candidate target; According to preset spatial constraints, the three-dimensional spatial position of each candidate target is verified, wherein the preset spatial constraints are used to characterize the physical range of the target of the corresponding target category in three-dimensional space; Candidate targets that satisfy the preset spatial constraints are retained, and the two-dimensional bounding boxes and detection confidence of the retained candidate targets are corrected according to their corresponding three-dimensional spatial positions. The target detection result is obtained based on the retained and corrected candidate targets.

6. The method according to claim 5, characterized in that, The image to be detected is the current image frame in the video stream acquired by the imaging device; After performing target detection based on the fused image features to obtain the target detection result, the method further includes: The retained and corrected candidate targets are determined as reference targets; The historical target detection results corresponding to each of the multiple historical image frames in the video stream that are located before the current image frame in time sequence are obtained. The multiple historical target detection results jointly represent multiple historical targets. The historical target is a target object detected in at least one of the historical image frames. The detection results of the same target object in different historical image frames belong to the same historical target. For each historical target, based on the detection results of multiple historical targets, the target state of the historical target in the current image frame is predicted, wherein the target state includes the predicted position and the predicted motion state; Based on the target states corresponding to the various historical targets, each reference target in the current image frame is associated with the various historical targets to obtain a target association result. The target association result is used to characterize whether each reference target corresponds to the same target object as each of the historical targets. Based on the target association result, the target detection result corresponding to the current image frame is corrected to obtain the temporally optimized detection result.

7. The method according to claim 6, characterized in that, The step of correcting the target detection result corresponding to the current image frame based on the target association result includes: For each historical target, the system determines whether there is a reference target in the current image frame that corresponds to the same target object as the historical target, based on the target association result. If a reference target exists in the current image frame that corresponds to the same target object as the historical target, the target position of the corresponding reference target is corrected according to the predicted position of the historical target, and the reference target with the corrected target position is retained in the target detection result corresponding to the current image frame. If there is no reference target in the current image frame that corresponds to the same target object as the historical target, the predicted position of the historical target is determined as the target position of the historical target in the current image frame, and the historical target is added to the target detection result corresponding to the current image frame. For each reference target, if the target association result indicates that the reference target does not correspond to the same target object as any of the historical targets, and the reference target only appears in the current image frame, the reference target is deleted from the target detection result corresponding to the current image frame.

8. The method according to claim 6, characterized in that, The method further includes: Based on the time-series optimization results, the target positions of multiple target objects in the current image frame are determined, and the target positions of the same target object in multiple historical image frames and the current image frame are connected in chronological order to determine the target trajectory corresponding to the target object.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the target detection method as described in any one of claims 1-8.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions that, when executed by a processor, cause the processor to implement the target detection method as described in any one of claims 1-8.

Images