Open scene object detection method and apparatus

Abstract

The embodiment of the application provides a kind of open scene target detection method and device, method includes: the local granularity feature of input image is extracted by the main network of pre-set convolutional neural network to input image, local perception is carried out to local granularity feature by depth convolution, corresponding local spatial information is extracted, and global context dependency relationship between local spatial information is captured by multi-head self-attention mechanism, and global context dependency relationship and local granularity feature are carried out multi-scale feature fusion, and corresponding multi-level feature representation is obtained;Multi-level feature representation is input into set mobile probability propagation model and carries out target detection operation, and the final position coordinate and category label corresponding to the object identified as dynamic target are output by mobile probability propagation model as target detection result;The precision and real-time performance of target detection under open scene can be effectively improved by the application.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to an open-scene target detection method and apparatus. Background Technology

[0002] Detecting and identifying various targets in open and complex environments is crucial for improving target detection and recognition. Ground targets are diverse, with significant differences within each category. Furthermore, non-target elements in the surrounding background can interfere with target detection. Besides target differences and background influences, numerous time- and environmental variables exist on the ground. In complex environments, targets change with tactics and concealment, further increasing the difficulty of detection and identification. Compared to scenarios like air and sea, open-scene target detection algorithms on the ground require processing far more information to distinguish targets, posing significant challenges to computational resources and algorithm model complexity.

[0003] As the ground situation changes, commanders need to quickly and accurately obtain accurate annotations of on-site data in order to make correct decisions. Therefore, target detection technology in open scenarios has important application prospects in ground command. Summary of the Invention

[0004] To address the problems in the prior art, this application provides an open scene target detection method and apparatus, which can effectively improve the accuracy and real-time performance of target detection in open scenes.

[0005] To solve at least one of the above problems, this application provides the following technical solution:

[0006] Firstly, this application provides an open-scene target detection method, including:

[0007] The input image is fed into the backbone network of a preset convolutional neural network to extract local granular features of the input image. The local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. The global contextual dependencies between the local spatial information are captured through a multi-head self-attention mechanism. The global contextual dependencies and the local granular features are fused at multiple scales to obtain the corresponding multi-level feature representation.

[0008] The multi-level feature representation is input into the set motion probability propagation model for target detection. A dynamic probability value is initialized for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, the dynamic probability value of the target is dynamically updated according to the degree of fit between the target and the camera motion model in multiple consecutive frames. Based on the updated dynamic probability value, low-probability targets are marked as static targets and the static targets are excluded from the target detection operation.

[0009] The motion probability propagation model outputs the final position coordinates and category labels of the remaining objects identified as dynamic targets as the target detection results.

[0010] Further, the step of feeding the input image into the backbone network of a preset convolutional neural network to extract local granular features of the input image includes:

[0011] Obtain the input image to be detected, and input the input image into a pre-defined convolutional neural network;

[0012] The input image is subjected to convolutional pooling operations through the backbone network of the convolutional neural network to extract local granularity features that reflect local granularity information.

[0013] Furthermore, the step of performing local perception on the local granular features through deep convolution to extract the corresponding local spatial information includes:

[0014] The extracted local granular features are obtained and input into a deep convolutional network structure, wherein the deep convolutional network contains multiple convolutional layers and activation layers;

[0015] The local granular features are locally perceived through layer-by-layer deep convolution operations in the deep convolutional network structure, and the corresponding local spatial information is extracted.

[0016] Furthermore, the step of capturing the global contextual dependencies between the local spatial information through a multi-head self-attention mechanism, and performing multi-scale feature fusion on the global contextual dependencies and the local granular features to obtain the corresponding multi-level feature representation includes:

[0017] The extracted local spatial information is input into the multi-head self-attention mechanism module to calculate the correlation weights between them, and the correlation weights are concatenated to generate a feature vector representing the global context dependency.

[0018] The feature vectors of the global context dependency and the local granular features are extracted through a parallel multi-branch network structure on convolutional and pooling layers in different receptive fields and combined in a cascaded or weighted manner to obtain the final multi-level feature representation.

[0019] Further, the step of inputting the multi-level feature representation into a set motion probability propagation model for target detection, and initializing a dynamic probability value for each target detected during the target detection operation to characterize the probability that the target belongs to a dynamic object, includes:

[0020] The multi-level feature representation is input into a probabilistic graph-based mobile probability propagation model for target recognition using a multi-level feature target detector.

[0021] The motion predictor in the motion probability propagation model initializes a dynamic probability value for each detected target to characterize the likelihood that the target is a dynamic object.

[0022] Furthermore, the step of dynamically updating the dynamic probability value of the target in subsequent consecutive frames based on the degree of fit between the target and the camera motion model in multiple consecutive frames, marking low-probability targets as static targets based on the updated dynamic probability value, and excluding the static targets from the target detection operation includes:

[0023] Obtain the dynamic probability value corresponding to each target in the initialization, and dynamically update the probability value of each target according to the degree of fit between each target and the camera motion model in multiple consecutive frames when processing consecutive image frames;

[0024] Obtain the dynamic probability value of each target after probability update. For targets with low probability values, mark them as static targets and exclude them from subsequent target detection operations.

[0025] Furthermore, the step of outputting the final position coordinates and category labels of the remaining objects identified as dynamic targets through the motion probability propagation model as the target detection result includes:

[0026] Obtain the objects that are identified as dynamic targets after static target exclusion in the mobile probability propagation model;

[0027] The final position coordinates and corresponding category labels of the dynamic target are extracted from the output of the mobile probability propagation model as the target detection result.

[0028] Secondly, this application provides an open-scene target detection device, comprising:

[0029] The image processing module is used to feed the input image into the backbone network of a preset convolutional neural network to extract the local granular features of the input image, perform local perception on the local granular features through deep convolution to extract the corresponding local spatial information, capture the global contextual dependencies between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global contextual dependencies and the local granular features to obtain the corresponding multi-level feature representation.

[0030] The motion detection module is used to input the multi-level feature representation into the set motion probability propagation model to perform target detection operations. For each target detected during the target detection operation, a dynamic probability value is initialized to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, the dynamic probability value of the target is dynamically updated according to the degree of fit between the target and the camera motion model in multiple consecutive frames. Based on the updated dynamic probability value, low-probability targets are marked as static targets and the static targets are excluded from the target detection operation.

[0031] The target recognition module is used to output the final position coordinates and category labels of the remaining objects identified as dynamic targets as target detection results through the motion probability propagation model.

[0032] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the open scene target detection method described above.

[0033] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the open scene target detection method described above.

[0034] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the open scene target detection method described above.

[0035] As can be seen from the above technical solution, this application provides an open scene target detection method and apparatus. It extracts local granular features of the input image by feeding the input image into the backbone network of a preset convolutional neural network, performs local perception on the local granular features through deep convolution to extract the corresponding local spatial information, and captures the global contextual dependencies between local spatial information through a multi-head self-attention mechanism. Furthermore, it performs multi-scale feature fusion on the global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. The multi-level feature representation is then input into a set motion probability propagation model for target detection. The motion probability propagation model outputs the final position coordinates and category label of the object identified as a dynamic target as the target detection result. This effectively improves the accuracy and real-time performance of target detection in open scenes. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts illustrating the open scene target detection method in the embodiments of this application;

[0038] Figure 2 This is the second flowchart illustrating the open scene target detection method in this application embodiment;

[0039] Figure 3 This is the third flowchart illustrating the open scene target detection method in this application embodiment;

[0040] Figure 4 This is the fourth flowchart illustrating the open scene target detection method in the embodiments of this application;

[0041] Figure 5 This is the fifth flowchart illustrating the open scene target detection method in the embodiments of this application;

[0042] Figure 6 This is the sixth flowchart illustrating the open scene target detection method in the embodiments of this application;

[0043] Figure 7 This is the seventh flowchart illustrating the open scene target detection method in the embodiments of this application;

[0044] Figure 8 This is a structural diagram of the open scene target detection device in the embodiments of this application;

[0045] Figure 9 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0046] Figure label:

[0047] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0050] In view of the problems existing in the prior art, this application provides an open scene target detection method and apparatus. The method extracts local granular features of the input image by feeding the input image into the backbone of a pre-defined convolutional neural network. Local spatial information is extracted by performing local perception on the local granular features through deep convolution. Global contextual dependencies between local spatial information are captured through a multi-head self-attention mechanism. Multi-scale feature fusion is then performed on the global contextual dependencies and local granular features to obtain corresponding multi-level feature representations. These multi-level feature representations are then input into a motion probability propagation model for target detection. The final position coordinates and category label of the object identified as a dynamic target are output by the motion probability propagation model as the target detection result. This effectively improves the accuracy and real-time performance of target detection in open scenes.

[0051] To effectively improve the accuracy and real-time performance of target detection in open scenes, this application provides an embodiment of an open scene target detection method, see [link to embodiment]. Figure 1 The open scene target detection method specifically includes the following:

[0052] Step S101: The input image is fed into the backbone network of a preset convolutional neural network to extract the local granular features of the input image. The local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. The global contextual dependency between the local spatial information is captured through a multi-head self-attention mechanism. The global contextual dependency and the local granular features are fused at multiple scales to obtain the corresponding multi-level feature representation.

[0053] Optionally, in this embodiment, the input image is first fed into the backbone network of a pre-defined convolutional neural network. This process typically uses a pre-trained deep convolutional neural network, such as ResNet, VGG, or EfficientNet. These networks have been trained on large-scale datasets and are capable of effectively extracting local granular features of the image. The backbone network extracts features progressively from low-level features (such as edges and textures) to high-level features (such as object parts) through multiple layers of convolution and pooling operations. This hierarchical feature extraction capability is a core advantage of convolutional neural networks, enabling them to capture complex patterns in images.

[0054] Next, local spatial information is extracted by using depthwise convolution to perceive local granular features. By increasing the network depth and using small-sized kernels, depthwise convolution can increase the receptive field while maintaining computational efficiency, thereby capturing a wider range of spatial dependencies. This process can be improved using techniques such as dilated convolution or depthwise separable convolution. Extracting local spatial information is crucial for understanding the shape, size, and location of an object.

[0055] Then, a multi-head self-attention mechanism is used to capture global contextual dependencies between local spatial information. Originating from the Transformer model, this mechanism allows the model to learn attention patterns in different representation subspaces. In this process, each local feature interacts with all other features, calculating their correlations. This mechanism effectively captures long-distance dependencies, overcoming the limitations of traditional convolutional networks in processing global information. Through the multi-head mechanism, the model can simultaneously focus on different types of dependencies, such as spatial location and semantic similarity, thus providing richer contextual information.

[0056] Finally, multi-scale feature fusion is performed on global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. This process typically involves a Feature Pyramid Network (FPN) or a similar multi-scale fusion structure. Features at different scales are aligned through upsampling and downsampling operations, and then fused using addition, concatenation, or attention mechanisms. This multi-scale fusion can simultaneously preserve fine-grained local information and large-scale semantic information, which is beneficial for detecting targets of different sizes and scales.

[0057] This step addresses several key technical challenges in open-scene object detection. First, it overcomes the limitations of single feature representations. By combining local granular features, local spatial information, and global contextual dependencies, the model can more comprehensively understand image content. Second, it solves the problem of long-distance dependency modeling. Traditional convolutional networks, due to their limited receptive field, struggle to effectively capture long-distance semantic relationships, while multi-head self-attention mechanisms effectively address this issue. Third, through multi-scale feature fusion, it solves the challenge of detecting objects at different scales, enabling the model to handle objects of varying sizes simultaneously.

[0058] From a technical perspective, this step achieves several important goals. First, it significantly improves the richness and expressiveness of feature representations. Multi-level, multi-scale feature fusion enables the model to better understand targets in complex scenes. Second, it enhances the model's ability to detect targets at different scales and locations. Whether small or large, near or far, targets can be detected more effectively. Third, by introducing global contextual information, it improves the model's overall understanding of the scene, helping to reduce false positives and false negatives.

[0059] Let's illustrate the application of this step with a specific example. Suppose we are developing a target detection system for autonomous driving scenarios. The input image is a complex city street scene containing various vehicles, pedestrians, traffic signs, and buildings.

[0060] First, we use a pre-trained ResNet-101 as the backbone network. The input image (assuming a size of 1920x1080x3) is processed by the backbone network to obtain feature maps at multiple scales, such as feature maps with sizes of 120x67x256, 60x34x512, 30x17x1024, and 15x9x2048. These feature maps contain local granular features from low to high levels.

[0061] Next, we further process these feature maps using a series of dilated convolutional layers. For example, for a 60x34x512 feature map, we might use three parallel 3x3 dilated convolutions with dilation rates of 1, 2, and 4. This expands the receptive field and extracts richer local spatial information without increasing the number of parameters.

[0062] Next, we input the processed features into a multi-head self-attention module. Let's assume we use 8 attention heads, each with a dimension of 64. For a 60x34x512 feature map, we first reshape it into a 2040x512 sequence (2040 = 60x34), and then perform self-attention computation. This process allows features at each location to interact with features at all other locations, capturing global dependencies.

[0063] Finally, we use a Feature Pyramid Network (FPN) structure for multi-scale feature fusion. Starting from the top layer, we perform upsampling and feature fusion layer by layer. For example, a 15x9x2048 feature map is upsampled to 30x17 and then fused with a 30x17x1024 feature map to obtain a new 30x17x256 feature map. This process continues until the bottom layer, ultimately resulting in a series of feature maps of the same size (e.g., all 120x67x256) but containing different levels of information.

[0064] This embodiment demonstrates how step S101 can be implemented in a practical application. Through this complex feature extraction and fusion process, our object detection system can simultaneously focus on large vehicles in a street scene and small traffic signs in the distance, understand the relationship between pedestrians and their surroundings, and may even capture interaction patterns between different vehicles. This comprehensive and detailed scene understanding capability is crucial for the safety and effectiveness of autonomous driving systems.

[0065] In summary, step S101, by combining various advanced techniques in deep learning, provides powerful feature representation capabilities for open-scene object detection. It not only improves the accuracy and robustness of detection but also lays a solid foundation for subsequent object recognition and tracking tasks.

[0066] Step S102: Input the multi-level feature representation into the set motion probability propagation model to perform target detection operation. For each target detected during the target detection operation, initialize a dynamic probability value to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, dynamically update the dynamic probability value of the target according to the degree of fit between the target and the camera motion model in multiple consecutive frames. According to the updated dynamic probability value, mark low-probability targets as static targets and exclude the static targets from the target detection operation.

[0067] Optionally, in this embodiment, the multi-level feature representation is first input into a pre-defined motion probability propagation model. This model is a complex system that combines object detection and motion estimation. It can not only identify targets in a single frame of an image, but also track the motion of targets between consecutive frames. The motion probability propagation model is typically based on a Bayesian framework, which transforms the object detection problem into a probabilistic inference problem. The advantage of this approach is that it can naturally integrate temporal information and uncertainty handling.

[0068] During object detection, the model initializes a dynamic probability value for each detected object. This probability value characterizes the likelihood that the object is a dynamic object. The initialization process may be based on features such as the object's size, position, and category, or it may use a pre-trained motion model for preliminary estimation. For example, in a road scene, vehicles and pedestrians may be assigned higher initial dynamic probability values, while buildings and streetlights may be assigned lower initial values.

[0069] Next, when processing subsequent consecutive frames, the model dynamically updates the dynamic probability value of each target based on its fit with the camera motion model across multiple consecutive frames. This process involves two aspects: camera motion estimation and target motion estimation. The camera motion model can be obtained using methods such as visual odometry or an inertial measurement unit (IMU). Target motion is estimated by comparing the target's positional changes across consecutive frames.

[0070] The core of this step is updating the dynamic probability values. This update process can utilize probabilistic filtering methods such as Kalman filtering or particle filtering. For example, if a target's motion is consistent with the camera's motion (considering perspective transformation), it is likely static, and its dynamic probability value will decrease; conversely, if the target's motion is inconsistent with the camera's motion, its dynamic probability value will increase. This dynamic update mechanism allows the model to accumulate evidence over time, improving the accuracy of its judgments.

[0071] Based on the updated dynamic probability values, the model marks low-probability targets as static targets. A threshold needs to be set here; for example, when the dynamic probability value is below 0.8, the target is marked as static. Targets marked as static are then excluded from the dynamic target detection operation.

[0072] This step addresses several key technical challenges in open-scene object detection. First, it solves the problem of distinguishing between static and dynamic objects. In complex real-world scenarios, differentiating between static and dynamic objects is crucial for many applications; for example, autonomous driving systems need to employ different handling strategies for static obstacles and moving vehicles. Second, it addresses the temporal consistency problem in object detection. By propagating and updating probabilities across consecutive frames, the model can leverage temporal information to improve detection stability and accuracy. Third, it provides a method for handling uncertainty. Target motion in real-world scenarios is often complex and uncertain; using probabilistic models can better represent and handle this uncertainty.

[0073] From a technical perspective, this step achieves several important goals. First, it improves the accuracy of object detection, particularly in distinguishing between static and dynamic objects. By introducing and updating dynamic probability values, the model can accumulate evidence over time and make more reliable judgments. Second, it enhances the system's adaptability to complex scenes. In scenes containing multiple moving objects, this method can better understand the dynamic characteristics of the scene. Third, it improves computational efficiency. By excluding static objects from dynamic object detection, the number of objects that need to be processed can be reduced, thereby lowering the computational burden.

[0074] Let's illustrate the application of this step with a specific example. Suppose we are developing an intelligent camera system for urban environmental monitoring. This system needs to detect and track various objects on the street, including pedestrians, vehicles, bicycles, etc.

[0075] First, we input the multi-level feature representation obtained in the previous step into our designed motion probability propagation model. This model includes a Faster R-CNN-based object detector and a motion predictor based on optical flow estimation. For each detected object, we initialize a dynamic probability value. For example, for a detected car, we might initialize a dynamic probability value of 0.7; for a detected streetlight, we might initialize a dynamic probability value of 0.1.

[0076] When processing consecutive frames, we use visual odometry to estimate camera motion. Simultaneously, we use optical flow algorithms to estimate the motion of each target. Then, we compare the target's motion with the camera's motion. For example, if the motion of a parked car in the image perfectly matches the viewpoint change caused by camera motion, we decrease its dynamic probability value, perhaps from 0.7 to 0.3. Conversely, if the motion of a pedestrian is inconsistent with camera motion, we increase its dynamic probability value, perhaps from 0.6 to 0.9.

[0077] We use a Bayesian update formula to dynamically adjust the probability value of each target. Specifically, we can use a particle filter to represent and update these probability distributions. For each target, we maintain a set of particles, each representing a possible motion state. By observing the target's position changes in consecutive frames, we update the weights of these particles, thus obtaining the updated dynamic probability value.

[0078] Suppose we set the threshold for the dynamic probability value to 0.7. After the update, all targets with a dynamic probability value below 0.7 are marked as static targets. These static targets (such as road signs and buildings) will be excluded from the dynamic target detection operation. In this way, our system can focus on detecting and analyzing dynamic environmental elements, such as pedestrians and vehicles.

[0079] This embodiment demonstrates how step S102 can be implemented in a practical application. Through this dynamic probability update and target classification process, our monitoring system can more accurately understand the dynamic characteristics of the urban environment. It can distinguish between parked and moving vehicles, identify pedestrians crossing the road, and accurately locate static infrastructure elements. This capability is of great value for applications such as urban planning, traffic management, and public safety.

[0080] In summary, step S102, by introducing a mobile probability propagation model and the concept of dynamic probability values, provides an effective method for handling complex dynamic environments in open scene object detection. It not only improves the accuracy and robustness of detection but also provides crucial dynamic information for subsequent scene understanding and decision-making.

[0081] Step S103: Output the final position coordinates and category labels of the remaining objects identified as dynamic targets as the target detection results through the motion probability propagation model.

[0082] Optionally, in this embodiment, step S103 is the final key step in the open scene object detection method, which outputs the final object detection result through a motion probability propagation model. The main goal of this step is to provide accurate dynamic object information, including their location coordinates and category labels. This step is not only a natural continuation of the previous steps but also the endpoint of the entire object detection process. Let's delve into the technical process, principles, problems solved, and effects achieved in this step.

[0083] First, the mobile probability propagation model has already distinguished between dynamic and static targets in the previous steps. At this stage, the model focuses on targets identified as dynamic. These dynamic targets may include vehicles, pedestrians, etc. The model performs final processing and refinement on these targets to ensure the accuracy and reliability of the output.

[0084] In the technical process, the model first calculates the precise location coordinates of each identified dynamic target. This typically involves bounding box regression optimization. The model may use non-maximum suppression (NMS) techniques to eliminate overlapping detection results, ensuring that each actual target has only one corresponding detection box. Location coordinates are usually represented in the image pixel coordinate system, but in some applications, they may also be converted to real-world 3D coordinates.

[0085] Simultaneously, the model assigns a class label to each dynamic target. This process is typically based on the output of the softmax classifier, selecting the class with the highest probability as the final label. In some complex scenarios, the model may output multiple possible classes and their corresponding probabilities, allowing subsequent processing to utilize this uncertainty.

[0086] In addition, the mobile probability propagation model may also output a confidence score for each detection result. This score reflects the model's confidence in its detection and classification results, and is crucial for subsequent decision-making and risk assessment.

[0087] In terms of technical principles, this step primarily relies on the reasoning capabilities of deep learning models and the uncertainty handling capabilities of probabilistic models. Deep neural networks excel in feature extraction and pattern recognition, while probabilistic models provide a framework for handling uncertainty and fusing information from multiple frames. The combination of the two enables the system to make accurate judgments in complex real-world scenarios.

[0088] This step addresses several key technical challenges. First, it solves the problem of accurate localization of dynamic targets. Accurate localization of dynamic targets in complex environments is crucial for many applications, such as autonomous driving and augmented reality. Second, it solves the problem of target classification, enabling accurate identification of different types of dynamic objects. Third, it provides a method for handling uncertainty in detection results by outputting confidence scores, allowing the system to better assess and manage risk.

[0089] From a technical perspective, this step achieves several important goals. First, it improves the accuracy and reliability of dynamic target detection. By combining multi-frame information and a probabilistic model, the system overcomes the limitations of single-frame detection, providing more stable and accurate results. Second, it enhances the system's ability to understand complex scenes. By accurately identifying and locating static targets, the system can construct a more complete scene representation. Third, it provides reliable input for subsequent decision-making and control processes, which is crucial for many practical applications such as autonomous driving and robot navigation.

[0090] As described above, the open-scene target detection method provided in this application can extract local granular features of the input image by feeding the input image into the backbone network of a preset convolutional neural network, perform local perception on the local granular features through deep convolution to extract the corresponding local spatial information, capture the global contextual dependencies between local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. The multi-level feature representation is then input into a set motion probability propagation model for target detection. The final position coordinates and category labels of the objects identified as dynamic targets are output by the motion probability propagation model as the target detection result, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0091] In one embodiment of the open-scene target detection method of this application, see [link to relevant documentation]. Figure 2 It can also specifically include the following:

[0092] Step S201: Obtain the input image to be detected, and input the input image into a pre-defined convolutional neural network;

[0093] Step S202: Perform convolutional pooling operation on the input image through the backbone network of the convolutional neural network to extract local granularity features that reflect local granularity information.

[0094] Optionally, in this embodiment, firstly, in step S201, the system acquires the input image to be detected. This process seems simple, but it actually involves several technical details. The input image may come from various sources, such as video streams captured in real time by a camera, stored still image files, or image data transmitted over a network. When acquiring the image, the system needs to process various possible image formats (such as JPEG, PNG, RAW, etc.) and ensure the integrity and consistency of the image data. In addition, preprocessing operations may be required, such as resizing the image, normalizing pixel values, or performing color space conversion (e.g., from RGB to YUV). These preprocessing steps are crucial to ensuring the proper functioning of the subsequent convolutional neural network.

[0095] The acquired and preprocessed images are then fed into a pre-defined convolutional neural network. This network is the core of the entire object detection system, and its structure and parameters directly affect the detection performance and results. The pre-defined network is usually a model trained on a large amount of data, such as ResNet, VGG, or EfficientNet. These networks are pre-trained on large-scale datasets such as ImageNet and have powerful feature extraction capabilities.

[0096] In step S202, the input image is processed through the backbone of a convolutional neural network. The backbone typically consists of multiple convolutional and pooling layers. Convolution is the core of a convolutional neural network; it extracts features by sliding convolutional kernels (also called filters) across the image. Each convolutional kernel can be viewed as a detector of a specific pattern, capable of recognizing features such as edges, textures, and shapes in the image.

[0097] Pooling operations are primarily used to reduce the spatial dimensionality of feature maps, increase the receptive field, and provide a degree of translation invariance. The most commonly used pooling methods are max pooling and average pooling. Max pooling preserves the most salient features within a region, while average pooling preserves the average information of the region. These operations help the network reduce computational cost and the number of parameters while retaining important information.

[0098] Through a series of convolution and pooling operations, the network extracts more abstract and high-level features layer by layer. Shallow networks mainly extract low-level features, such as edges and textures; while deep networks can extract more complex and abstract features, such as the partial or overall structure of an object. This hierarchical feature extraction process is the key to the power of convolutional neural networks.

[0099] Ultimately, the features extracted in this process are called local granular features. These features reflect the local granular information of the image, including details such as the shape, texture, and color distribution of objects. Local granular features are crucial for subsequent object detection and recognition tasks because they contain rich visual information that helps the system accurately locate and classify objects in the image.

[0100] These two steps address several key technical challenges. First, they solve the problem of effectively extracting useful features from raw images. Traditional hand-designed feature extraction methods often struggle with complex real-world scenes, while convolutional neural networks can automatically learn more robust and representative features. Second, they address the issue of feature representation; through multi-layered feature extraction, the network can simultaneously capture low-level local features and high-level semantic features. Third, they provide an efficient method for processing large-scale visual data; the combination of convolution and pooling operations significantly reduces the amount of data that needs to be processed, making real-time object detection possible in complex scenes.

[0101] From a technical perspective, these two steps achieve several important goals. First, they improve the efficiency and effectiveness of feature extraction. Compared to traditional methods, convolutional neural networks can extract useful information from images more quickly and accurately. Second, they enhance the system's adaptability to complex scenes. Through the learned features, the system can better handle various lighting conditions, occlusion, and complex backgrounds. Third, they provide high-quality feature inputs for subsequent object detection and recognition tasks, which is crucial for improving the overall system performance.

[0102] In one embodiment of the open-scene target detection method of this application, see [link to relevant documentation]. Figure 3 It can also specifically include the following:

[0103] Step S301: Obtain the extracted local granular features and input the local granular features into a deep convolutional network structure, wherein the deep convolutional network contains multiple convolutional layers and activation layers;

[0104] Step S302: The local granular features are locally perceived through the layer-by-layer deep convolution operation of the deep convolutional network structure, and the corresponding local spatial information is extracted.

[0105] Optionally, in this embodiment, in step S301, the system first acquires the local granular features extracted in the previous steps. These features are typically a series of multidimensional tensors, each representing a feature representation of the image at different scales and levels of abstraction. These features contain information such as the local structure, texture, and edges of the image, but may not be sufficient for direct use in complex object detection tasks. Therefore, these features are input into a specially designed deep convolutional network structure.

[0106] This deep convolutional network architecture is specifically designed for further feature extraction and refinement. It contains multiple convolutional and activation layers, the combination of which enables the network to learn more complex and abstract feature representations. Convolutional layers handle feature extraction and transformation, while activation layers introduce non-linearity, increasing the network's expressive power. Commonly used activation functions include ReLU (Rectified Linear Unit), LeakyReLU, and ELU (Exponential Linear Unit). The choice of these activation functions has a significant impact on the network's performance; they help the network overcome the vanishing gradient problem, accelerate the training process, and improve the model's expressive power.

[0107] In step S302, local granular features are subjected to layer-by-layer deep convolution operations through a deep convolutional network structure. This process can be viewed as a local perception process of the input features. Each convolutional operation slides the convolutional kernel across the feature map of the previous layer, thereby capturing local spatial relationships and patterns. A key advantage of this operation is parameter sharing, meaning that the same convolutional kernel shares parameters across the entire feature map, which greatly reduces the number of parameters that need to be learned, improving the model's efficiency and generalization ability.

[0108] Each layer of a deep convolutional network can be viewed as learning a specific feature detector. Shallow networks might learn simple edge or texture detectors, while deep networks might learn more complex patterns, such as the partial or overall structure of an object. By stacking multiple layers of convolutions, the network can progressively build a hierarchical representation of the input data.

[0109] In this process, the network is not only extracting features, but also learning how to most effectively combine and transform these features to suit the object detection task. For example, the network may learn how to effectively fuse information at different scales, or how to highlight the features most critical to object detection.

[0110] In this way, deep convolutional networks can extract rich local spatial information. This information includes not only the appearance features of objects but also the spatial relationships between the different parts of the object. For example, in vehicle detection, the network may learn the relative positional relationships between components such as wheels, body, and windows, which is crucial for accurately identifying and locating vehicles.

[0111] These two steps address several key technical challenges. First, they solve the problem of extracting more abstract and task-relevant features from initial localized, granular features. Through deep convolutional networks, the system learns feature representations more suitable for object detection tasks. Second, they address the problem of effectively capturing local spatial relationships. Through layer-by-layer convolutional operations, the network can progressively expand its receptive field, thereby capturing a wider range of spatial dependencies. Third, they provide an adaptive feature learning method that automatically learns the most relevant features from training data without requiring manually designed feature extractors.

[0112] From a technical perspective, these two steps achieve several important goals. First, they improve the expressive power of features. Through deep convolutional networks, the system can learn richer and more discriminative feature representations, which is crucial for improving the accuracy of object detection. Second, they enhance the system's understanding of complex scenes. By capturing local spatial information, the system can better handle complex situations such as occlusion and deformation between objects. Third, they provide high-quality feature inputs for subsequent object detection and classification tasks, which helps improve the overall performance and robustness of the system.

[0113] In one embodiment of the open-scene target detection method of this application, see [link to relevant documentation]. Figure 4 It can also specifically include the following:

[0114] Step S401: The extracted local spatial information is input into the multi-head self-attention mechanism module to calculate the correlation weights between them, and the correlation weights are concatenated to generate a feature vector representing the global context dependency.

[0115] Step S402: The feature vectors of the global context dependency and the local granular features are extracted through a parallel multi-branch network structure on convolutional and pooling layers in different receptive fields and combined in a cascaded or weighted manner to obtain the final multi-level feature representation.

[0116] Optionally, in this embodiment, in step S401, the system first acquires the local spatial information extracted in the previous steps. This information is typically a series of feature maps, each representing local features of the image at different locations and scales. Although these local features contain rich spatial information, the relationships between them have not been fully utilized. Therefore, these features are input into the multi-head self-attention mechanism module.

[0117] Multi-head self-attention is a powerful technique that captures long-range dependencies within a sequence. In computer vision tasks, it is used to model the relationships between different regions of an image. Specifically, multi-head self-attention first linearly projects the input features into multiple subspaces (called "heads"), and then independently computes attention weights in each subspace. This multi-head design allows the model to focus on different types of relationships simultaneously.

[0118] Within each head, attention weights are typically calculated based on three matrices: query, key, and value. The dot product of the query and key is used to calculate similarity, which is then normalized using a softmax function to obtain the attention weights. These weights are used to sum the weighted matrices to obtain the output of that head. Finally, the outputs of all heads are concatenated to form the final output.

[0119] The key to this process is that it allows the model to learn complex relationships between different regions of an image. For example, in object detection tasks, it might learn the relationships between different parts of an object (such as a person's head and body) or the relationship between an object and its surrounding environment (such as a car and a road). Modeling this global relationship is crucial for understanding complex scenes.

[0120] The output of step S401 is a feature vector representing global contextual dependencies. This feature vector captures the relationships between different regions in the image, providing important contextual information for subsequent object detection tasks.

[0121] In step S402, the system combines the feature vectors of global context dependencies with the previously obtained local granular features. The purpose of this step is to fuse global and local information to generate a more comprehensive and richer feature representation.

[0122] Specifically, this step uses a parallel, multi-branch network structure. Each branch contains convolutional and pooling layers with different receptive fields. The receptive field refers to the size of the input region that each neuron in the network can "see." Different receptive field sizes allow the network to capture features at different scales.

[0123] Within each branch, convolutional layers extract features, while pooling layers reduce the spatial dimensionality of the feature maps, increasing the translation invariance of the features. By using convolutional kernels and pooling windows of different sizes, each branch can focus on features at different scales.

[0124] The outputs of these branches can be combined in two ways: concatenated or weighted. The concatenation method directly joins the outputs of different branches together to form a longer feature vector. The weighting method assigns a weight to the output of each branch and then sums them. Both methods have their advantages; concatenation preserves all information but may lead to excessively high feature dimensionality, while weighting allows the model to learn the importance of features at different scales.

[0125] The result of this step is a final multi-level feature representation. This representation integrates local granular features, global contextual dependencies, and multi-scale information, providing rich information for subsequent object detection and classification tasks.

[0126] These two steps address several key technical challenges. First, they solve the problem of effectively modeling long-range dependencies between different regions in an image. Through a multi-head self-attention mechanism, the system can capture complex global contextual information, which is difficult to achieve in traditional convolutional neural networks. Second, they solve the problem of fusing global and local information. Through a parallel multi-branch network structure, the system can simultaneously process features at different scales and effectively combine them. Third, they provide a flexible feature extraction method that can adapt to targets of different sizes and complexities.

[0127] From a technical perspective, these two steps achieve several important goals. First, they improve the expressive power of features. By fusing global and local information, the system can generate richer and more meaningful feature representations, which is crucial for improving the accuracy and robustness of object detection. Second, they enhance the system's ability to understand complex scenes. By modeling global dependencies and multi-scale features, the system can better handle complex visual scenes, such as occlusion, deformation, and scale variations. Third, they provide high-quality feature inputs for subsequent object detection and classification tasks, which helps improve the overall system performance and generalization ability.

[0128] In one embodiment of the open-scene target detection method of this application, see [link to relevant documentation]. Figure 5 It can also specifically include the following:

[0129] Step S501: Input the multi-level feature representation into the probabilistic graph-based mobile probability propagation model to perform target recognition through a multi-level feature target detector;

[0130] Step S502: Initialize a dynamic probability value for each detected target using the motion predictor in the motion probability propagation model to characterize the likelihood that the target belongs to a dynamic object.

[0131] Optionally, in this embodiment, in step S501, the system first inputs the multi-level feature representation into a probabilistic graph-based motion probability propagation model. This model is a complex probabilistic framework that can effectively handle and propagate uncertainty information. In this model, each region in the image is represented as a node in the probabilistic graph, and the edges between nodes represent the relationships between them. This representation method allows the model to consider the spatial and semantic relationships between targets, thereby improving the accuracy of detection.

[0132] The multi-level feature object detector is one of the core components of this model. It utilizes the rich multi-level features generated in previous steps to perform object recognition. This detector is typically based on deep learning models such as Faster R-CNN, YOLO, or SSD, but is specially modified to adapt to multi-level feature input. The detector first generates a series of candidate regions, and then uses multi-level features to classify these regions and perform bounding box regression.

[0133] A key advantage of this process is its ability to utilize features at different scales and levels of abstraction simultaneously. For example, low-level features may contain detailed texture information, helping to identify small objects or fine structures; mid-level features may contain shape and edge information, helping to locate object boundaries; and high-level features may contain semantic information, helping to understand the object's category and contextual relationships. By comprehensively utilizing these features, the detector can more accurately identify and locate targets of various sizes and complexities.

[0134] Furthermore, probabilistic graphical models allow for the incorporation of prior knowledge and contextual information. For example, we can encode in the model the knowledge that certain objects typically appear together (e.g., a table and a chair), or the probability of certain objects appearing in a specific scene (e.g., a car on a road). This information can be represented by adjusting the edge weights in the probabilistic graph, thereby further improving the accuracy and robustness of detection.

[0135] Step S502 follows immediately after S501, and begins processing the dynamic characteristics of the target. In this step, the system uses the motion predictor in the motion probability propagation model to initialize a dynamic probability value for each detected target. This probability value characterizes the likelihood that the target is a dynamic object.

[0136] A motion predictor is a model specifically designed to predict the motion of objects. It may be based on various techniques, such as Kalman filtering, particle filtering, or more complex deep learning models. This predictor considers not only the target's current position and size, but also its changes over consecutive frames. For example, it might analyze parameters such as the target's position, velocity, and acceleration, as well as the trends of these parameters over time.

[0137] Initializing dynamic probability values ​​is a crucial step. For each detected target, the system estimates its dynamic characteristics based on its features and historical information (if any). For example, for a newly detected target, the system might initialize the dynamic probability value based on its category. People or vehicles might be assigned a higher initial dynamic probability, while buildings or trees might be assigned a lower initial dynamic probability.

[0138] This dynamic probability value is not static, but rather updates continuously with time and new observations. For example, if an object initially considered static begins to move, its dynamic probability value increases rapidly. Conversely, if an object initially considered dynamic remains stationary for an extended period, its dynamic probability value may gradually decrease.

[0139] These two steps address several key technical challenges. First, they solve the problem of effectively utilizing multi-level features for object detection. By inputting rich multi-level features into a probabilistic graphical model, the system can gain a more comprehensive understanding of image content, improving detection accuracy and robustness. Second, they address the problem of distinguishing between static and dynamic targets in complex scenes. By introducing dynamic probability values, the system can better characterize and track dynamic objects, which is crucial in many applications. Third, they provide a flexible framework for integrating prior knowledge and real-time observations, enabling better handling of complex and uncertain situations.

[0140] From a technical perspective, these two steps achieve several important goals. First, they improve the accuracy and reliability of target detection. By utilizing multi-level features and probabilistic graphical models, the system can more accurately identify and locate various targets, performing well even in complex scenes. Second, they enhance the system's ability to understand dynamic scenes. By introducing dynamic probability values, the system can distinguish between static and dynamic targets, which is crucial for many practical applications, such as traffic monitoring and security systems. Third, they lay the foundation for subsequent target tracking and behavior analysis tasks. The introduction of dynamic probability values ​​allows the system to better predict target trajectories, thus supporting more advanced analysis tasks.

[0141] In one embodiment of the open-scene target detection method of this application, see [link to relevant documentation]. Figure 6 It can also specifically include the following:

[0142] Step S601: Obtain the dynamic probability value corresponding to each target after initialization, and dynamically update the probability value of the target according to the degree of fit between each target and the camera motion model in multiple consecutive frames when processing consecutive image frames.

[0143] Step S602: Obtain the dynamic probability value of each target after probability update. For targets with low probability values, mark them as static targets and exclude them from subsequent target detection operations.

[0144] Optionally, in this embodiment, in step S601, the system first obtains the dynamic probability values ​​corresponding to each target initialized in the previous steps. These initial values ​​are estimated based on features such as the target's category, location, and size. Then, when processing consecutive image frames, the system dynamically updates the probability value of each target based on the degree of fit between each target and the camera motion model across multiple consecutive frames. This process involves complex motion estimation and probability update techniques.

[0145] The camera motion model is one of the core components of this step. It describes the camera's motion in three-dimensional space, typically including translation and rotation. This model can be estimated using various methods, such as feature point matching-based methods (e.g., SLAM techniques) or deep learning-based methods. An accurate camera motion model is crucial for distinguishing between real-world object motion and visual changes caused by camera movement.

[0146] When processing consecutive frames, the system predicts the position of each target in the next frame. This prediction is based on the target's current state (position, velocity, etc.) and estimated camera motion. The system then compares this prediction with the actual observed target position. If the difference between the prediction and the observation is small, it indicates that the target's motion is consistent with the camera's motion, and it is likely a static object; if the difference is large, it indicates that the target may have its own motion, and it is a dynamic object.

[0147] Based on this comparison, the system updates the dynamic probability value of the target. For example, if a target closely matches the camera motion model for several consecutive frames, its dynamic probability value may gradually decrease; conversely, if a target's motion continuously deviates from the camera motion model's prediction, its dynamic probability value may increase. This update is typically implemented using Bayesian updates or similar probabilistic update methods.

[0148] This step addresses several key technical challenges. First, it solves the problem of accurately distinguishing between static and dynamic objects in moving camera scenes. By introducing a camera motion model, the system can separate the visual changes caused by camera movement from the object's own motion. Second, it provides a continuous and smooth way to update the target's dynamic characteristics, avoiding the instability that can result from single-frame-based judgments. Third, it can handle complex motion scenes, such as situations where the camera and objects move simultaneously.

[0149] Step S602 follows immediately after S601, and begins target classification and processing using the updated dynamic probability values. In this step, the system first obtains the dynamic probability values ​​of each target after probability updates. Then, the system sets a threshold; targets with dynamic probability values ​​below this threshold are marked as static targets. These targets marked as static are subsequently excluded from subsequent target detection operations.

[0150] The core of this step lies in choosing an appropriate threshold to distinguish between dynamic and static targets. Threshold selection requires a trade-off between detection sensitivity and system efficiency. If the threshold is set too low, some actually static targets may be incorrectly labeled as dynamic; if it is set too high, some dynamic targets may be missed. Therefore, threshold selection usually requires careful tuning based on the specific application scenario and system requirements.

[0151] Excluding dynamic targets from subsequent object detection operations is a key innovation in this step. This approach is based on the assumption that targets identified as dynamic are unlikely to become static targets in a short period of time, thus eliminating the need to re-detect them in every frame. This significantly reduces computational load, especially in systems processing high frame rate video or requiring real-time response.

[0152] However, this exclusion strategy needs to be implemented with caution. The system needs to periodically re-evaluate these dynamic targets to prevent tracking errors over a long period of time. For example, the system may re-detect these dynamic targets completely every certain number of frames, or trigger re-detection when the behavior of the dynamic target changes significantly (such as suddenly stopping).

[0153] These two steps address several key technical challenges. First, they solve the problem of accurately tracking and updating the dynamic characteristics of targets in continuous video streams. By combining camera motion models and continuous frame analysis, the system can more accurately identify truly moving targets. Second, they provide an effective method to balance detection accuracy and computational efficiency. By excluding known static targets, the system can concentrate more computational resources on detecting new targets and analyzing dynamic scenes. Third, this approach provides a flexible framework for handling complex dynamic scenes, adaptable to various application scenarios.

[0154] From a technical perspective, these two steps achieve several important goals. First, they improve the accuracy and reliability of dynamic target recognition. Through continuous frame analysis and probability updates, the system can better handle complex motion scenes and reduce false positives. Second, they significantly improve the system's computational efficiency. By excluding known dynamic targets, the system can process video streams faster, supporting real-time applications. Third, this approach enhances the system's adaptability, enabling it to handle a variety of scenarios, from static monitoring to rapidly changing dynamic environments.

[0155] In one embodiment of the open-scene target detection method of this application, see [link to relevant documentation]. Figure 7 It can also specifically include the following:

[0156] Step S701: Obtain the objects that are identified as dynamic targets after static target exclusion by the motion probability propagation model;

[0157] Step S702: Extract the final position coordinates and corresponding category labels of the dynamic target from the output of the mobile probability propagation model as the target detection result.

[0158] Optionally, in this embodiment, in step S701, the system first acquires the objects identified as dynamic targets remaining after static target exclusion by the motion probability propagation model. This process involves complex probability models and target classification techniques. The motion probability propagation model is an advanced target tracking and classification method that can track the motion of targets in consecutive image frames and estimate the probability of each target being static or dynamic.

[0159] The core idea of ​​this model is to treat the target's motion state as a probability distribution that changes over time. In each frame, the model updates this distribution, taking into account factors such as the target's position, velocity, and acceleration. Simultaneously, the model also considers other environmental factors, such as camera motion and scene complexity. In this way, the model can effectively distinguish between genuine object motion and visual changes caused by camera movement.

[0160] During the static target exclusion process, the system sets a probability threshold. If a target's dynamic probability is below this threshold, it will be marked as a static target and excluded from subsequent processing. The selection of this threshold requires a trade-off between detection sensitivity and system accuracy. Setting it too low may incorrectly mark some static objects as dynamic, while setting it too high may miss some slowly moving objects.

[0161] After this process, what remains are the targets that the system identifies as dynamic. These targets may include vehicles, pedestrians, etc.

[0162] Step S702 then extracts the final position coordinates and corresponding category labels of these dynamic targets from the output of the motion probability propagation model, as the final result of target detection. This step involves accurate coordinate extraction and reliable target classification.

[0163] For each target identified as moving, the system needs to determine its accurate position coordinates. This typically includes the target's two-dimensional coordinates (x, y) in the image, and sometimes depth information (z) is also included, forming a three-dimensional coordinate system. Coordinate extraction may utilize various computer vision techniques, such as edge detection and contour analysis. To improve accuracy, the system may consider the target's position across multiple consecutive frames, using techniques such as Kalman filters to smooth the coordinate estimation.

[0164] Simultaneously, the system needs to assign a category label to each static target. This process typically relies on deep learning models, such as convolutional neural networks (CNNs). These models are able to learn and recognize various categories of objects from the visual features of the target. Category labels may include "car," "pedestrian," "traffic light," "building," etc., depending on the application scenario and training dataset.

[0165] It is worth noting that classifying dynamic targets can be more challenging than classifying static targets. This is because dynamic targets may be partially occluded or lack multi-angle visual information due to maintaining the same pose for extended periods. To address this issue, systems may utilize contextual information and time-series analysis to improve classification accuracy.

[0166] These two steps address several key technical challenges. First, they solve the problem of accurately identifying and locating static targets in complex, dynamic scenes. By using a motion probability propagation model and dynamic target exclusion, the system can effectively filter out moving objects and focus on dynamic elements. This is particularly important in applications such as autonomous driving and augmented reality, where an accurate understanding of the dynamic structure of the environment is crucial.

[0167] Secondly, these steps provide an efficient method for integrating the target's location and semantic information. By simultaneously extracting coordinates and category labels, the system can provide rich scene understanding results. This comprehensive information is crucial for advanced decision-making and scene analysis.

[0168] Furthermore, this approach provides a flexible framework for handling long-term scene changes. Since the location and category of static targets may change slowly over time (e.g., vehicles in a parking lot may leave or new vehicles may arrive), the system needs to be able to adapt to these changes. By continuously updating the movement probability propagation model, the system is able to capture these gradual changes.

[0169] From a technical perspective, these two steps achieve several important goals. First, they improve the accuracy and reliability of dynamic target recognition. By focusing on objects identified as dynamic, the system can allocate more resources to accurately locate and classify these targets. Second, they provide rich scene understanding results, including the precise location and semantic information of the targets. This comprehensive information is essential for many advanced applications. Third, this approach enhances the system's adaptability, enabling it to handle a wide variety of scenarios, from static indoor environments to dynamically changing urban streets.

[0170] To effectively improve the accuracy and real-time performance of target detection in open scenes, this application provides an embodiment of an open scene target detection device for implementing all or part of the open scene target detection method, see [link to embodiment]. Figure 8 The open scene target detection device specifically includes the following components:

[0171] Image processing module 10 is used to feed the input image into the backbone network of a preset convolutional neural network to extract the local granular features of the input image, perform local perception on the local granular features through deep convolution to extract the corresponding local spatial information, capture the global contextual dependency relationship between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global contextual dependency relationship and the local granular features to obtain the corresponding multi-level feature representation.

[0172] The motion detection module 20 is used to input the multi-level feature representation into the set motion probability propagation model to perform target detection operation. For each target detected during the target detection operation, a dynamic probability value is initialized to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, the dynamic probability value of the target is dynamically updated according to the degree of fit between the target and the camera motion model in multiple consecutive frames. Based on the updated dynamic probability value, low-probability targets are marked as static targets and the static targets are excluded from the target detection operation.

[0173] The target recognition module 30 is used to output the final position coordinates and category labels of the remaining objects identified as dynamic targets as target detection results through the motion probability propagation model.

[0174] As described above, the open-scene target detection device provided in this application can extract local granular features of the input image by feeding the input image into the backbone network of a preset convolutional neural network, perform local perception on the local granular features through deep convolution to extract the corresponding local spatial information, capture the global contextual dependencies between local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. The multi-level feature representation is then input into a set motion probability propagation model for target detection. The final position coordinates and category labels of the objects identified as dynamic targets are output by the motion probability propagation model as the target detection result, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0175] From a hardware perspective, in order to effectively improve the accuracy and real-time performance of target detection in open scenes, this application provides an embodiment of an electronic device for implementing all or part of the open scene target detection method, wherein the electronic device specifically includes the following:

[0176] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the open scene target detection device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the open scene target detection method and the open scene target detection device in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0177] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0178] In practical applications, parts of the open-scene target detection method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0179] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0180] Figure 9 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 9As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 9 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0181] In one embodiment, the open scene object detection method functionality can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following controls:

[0182] Step S101: The input image is fed into the backbone network of a preset convolutional neural network to extract the local granular features of the input image. The local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. The global contextual dependency between the local spatial information is captured through a multi-head self-attention mechanism. The global contextual dependency and the local granular features are fused at multiple scales to obtain the corresponding multi-level feature representation.

[0183] Step S102: Input the multi-level feature representation into the set motion probability propagation model to perform target detection operation. For each target detected during the target detection operation, initialize a dynamic probability value to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, dynamically update the dynamic probability value of the target according to the degree of fit between the target and the camera motion model in multiple consecutive frames. According to the updated dynamic probability value, mark low-probability targets as static targets and exclude the static targets from the target detection operation.

[0184] Step S103: Output the final position coordinates and category labels of the remaining objects identified as dynamic targets as the target detection results through the motion probability propagation model.

[0185] As described above, the electronic device provided in this application extracts local granular features of the input image by feeding the input image into the backbone network of a preset convolutional neural network, performs local perception on the local granular features through deep convolution to extract the corresponding local spatial information, captures the global contextual dependencies between local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. The multi-level feature representation is input into a set motion probability propagation model for target detection, and the final position coordinates and category label of the object identified as a dynamic target are output by the motion probability propagation model as the target detection result. This can effectively improve the accuracy and real-time performance of target detection in open scenes.

[0186] In another embodiment, the open scene target detection device can be configured separately from the central processing unit 9100. For example, the open scene target detection device can be configured as a chip connected to the central processing unit 9100, and the open scene target detection method function can be implemented through the control of the central processing unit.

[0187] like Figure 9 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 9 All components shown; in addition, the electronic device 9600 may also include Figure 9 For components not shown, please refer to existing technologies.

[0188] like Figure 9 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0189] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0190] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0191] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0192] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0193] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0194] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0195] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the open scene target detection method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the open scene target detection method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0196] Step S101: The input image is fed into the backbone network of a preset convolutional neural network to extract the local granular features of the input image. The local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. The global contextual dependency between the local spatial information is captured through a multi-head self-attention mechanism. The global contextual dependency and the local granular features are fused at multiple scales to obtain the corresponding multi-level feature representation.

[0197] Step S102: Input the multi-level feature representation into the set motion probability propagation model to perform target detection operation. For each target detected during the target detection operation, initialize a dynamic probability value to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, dynamically update the dynamic probability value of the target according to the degree of fit between the target and the camera motion model in multiple consecutive frames. According to the updated dynamic probability value, mark low-probability targets as static targets and exclude the static targets from the target detection operation.

[0198] Step S103: Output the final position coordinates and category labels of the remaining objects identified as dynamic targets as the target detection results through the motion probability propagation model.

[0199] As described above, the computer-readable storage medium provided in this application extracts local granular features of the input image by feeding the input image into the backbone network of a preset convolutional neural network, performs local perception on the local granular features through deep convolution to extract the corresponding local spatial information, captures the global contextual dependencies between local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. The multi-level feature representation is input into a set motion probability propagation model for target detection, and the final position coordinates and category labels of the objects identified as dynamic targets are output by the motion probability propagation model as the target detection result. This can effectively improve the accuracy and real-time performance of target detection in open scenes.

[0200] Embodiments of this application also provide a computer program product capable of implementing all steps of the open scene target detection method with the execution subject being a server or client in the above embodiments. When the computer program / instructions are executed by a processor, they implement the steps of the open scene target detection method. For example, the computer program / instructions implement the following steps:

[0201] Step S101: The input image is fed into the backbone network of a preset convolutional neural network to extract the local granular features of the input image. The local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. The global contextual dependency between the local spatial information is captured through a multi-head self-attention mechanism. The global contextual dependency and the local granular features are fused at multiple scales to obtain the corresponding multi-level feature representation.

[0202] Step S102: Input the multi-level feature representation into the set motion probability propagation model to perform target detection operation. For each target detected during the target detection operation, initialize a dynamic probability value to characterize the possibility that the target belongs to a dynamic object. In subsequent consecutive frames, dynamically update the dynamic probability value of the target according to the degree of fit between the target and the camera motion model in multiple consecutive frames. According to the updated dynamic probability value, mark low-probability targets as static targets and exclude the static targets from the target detection operation.

[0203] Step S103: Output the final position coordinates and category labels of the remaining objects identified as dynamic targets as the target detection results through the motion probability propagation model.

[0204] As described above, the computer program product provided in this application extracts local granular features of the input image by feeding the input image into the backbone network of a preset convolutional neural network, performs local perception on the local granular features through deep convolution to extract the corresponding local spatial information, captures the global contextual dependencies between local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global contextual dependencies and local granular features to obtain the corresponding multi-level feature representation. The multi-level feature representation is then input into a set motion probability propagation model for target detection. The final position coordinates and category labels of the objects identified as dynamic targets are output by the motion probability propagation model as the target detection result, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0205] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0209] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An open-scene target detection method, characterized in that, The method includes: The input image is fed into the backbone network of a preset convolutional neural network to extract local granular features of the input image. The local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. The global contextual dependencies between the local spatial information are captured through a multi-head self-attention mechanism. The feature vectors of the global contextual dependencies and the local granular features are combined in a cascaded or weighted manner through a parallel multi-branch network structure to extract features on convolution and pooling layers in different receptive fields to obtain the corresponding multi-level feature representation. The multi-level feature representation is input into a probabilistic graph-based motion probability propagation model for target recognition by a multi-level feature target detector. A dynamic probability value is initialized for each detected target by the motion predictor in the motion probability propagation model to characterize the likelihood that the target belongs to a dynamic object. The camera motion model is obtained by visual odometry or inertial measurement unit. In subsequent consecutive frames, the Bayesian update formula is used to dynamically update the dynamic probability value of the target based on the degree of fit between the target and the camera motion model in multiple consecutive frames. The motion of each target is estimated using an optical flow algorithm, and the difference between the target's motion and the camera's motion is compared. If the target's motion in the image completely matches the viewpoint change caused by the camera motion, its dynamic probability value is reduced; if the target's motion is inconsistent with the camera motion, its dynamic probability value is increased. Based on the updated dynamic probability value, low-probability targets are marked as static targets, and these static targets are excluded from the target detection operation. The motion probability propagation model outputs the final position coordinates and category labels of the remaining objects identified as dynamic targets as the target detection results.

2. The open scene target detection method according to claim 1, characterized in that, The step of feeding the input image into the backbone network of a preset convolutional neural network to extract local granular features of the input image includes: Obtain the input image to be detected, and input the input image into a pre-defined convolutional neural network; The input image is subjected to convolutional pooling operations through the backbone network of the convolutional neural network to extract local granularity features that reflect local granularity information.

3. The open-scene target detection method according to claim 1, characterized in that, The step of performing local perception on the local granular features through deep convolution to extract the corresponding local spatial information includes: The extracted local granular features are obtained and input into a deep convolutional network structure, wherein the deep convolutional network contains multiple convolutional layers and activation layers; The local granular features are locally perceived through layer-by-layer deep convolution operations in the deep convolutional network structure, and the corresponding local spatial information is extracted.

4. The open-scene target detection method according to claim 1, characterized in that, The process involves capturing the global contextual dependencies between local spatial information using a multi-head self-attention mechanism, and then combining the feature vectors of these global contextual dependencies and the local granular features through a parallel multi-branch network structure at convolutional and pooling layers in different receptive fields in a cascaded or weighted manner to obtain the corresponding multi-level feature representation, including: The extracted local spatial information is input into the multi-head self-attention mechanism module to calculate the correlation weights between them, and the correlation weights are concatenated to generate a feature vector representing the global context dependency. The feature vectors of the global context dependency and the local granular features are extracted through a parallel multi-branch network structure on convolutional and pooling layers in different receptive fields and combined in a cascaded or weighted manner to obtain the final multi-level feature representation.

5. The open-scene target detection method according to claim 1, characterized in that, The step of outputting the final position coordinates and category labels of the remaining objects identified as dynamic targets through the motion probability propagation model as the target detection result includes: Obtain the objects that are identified as dynamic targets after static target exclusion in the mobile probability propagation model; The final position coordinates and corresponding category labels of the dynamic target are extracted from the output of the mobile probability propagation model as the target detection result.

6. An open-scene target detection device, characterized in that, The device includes: The image processing module is used to feed the input image into the backbone network of a preset convolutional neural network to extract the local granular features of the input image, perform local perception on the local granular features through deep convolution to extract the corresponding local spatial information, capture the global contextual dependencies between the local spatial information through a multi-head self-attention mechanism, and combine the feature vectors of the global contextual dependencies and the local granular features through a parallel multi-branch network structure to extract features on convolution and pooling layers in different receptive fields in a cascaded or weighted manner to obtain the corresponding multi-level feature representation. The motion detection module is used to input the multi-level feature representation into a motion probability propagation model based on a probabilistic graph for target recognition by a multi-level feature target detector. The motion predictor in the motion probability propagation model initializes a dynamic probability value for each detected target to characterize the likelihood that the target is a moving object, and obtains a camera motion model using visual odometry or an inertial measurement unit. In subsequent consecutive frames, a Bayesian update formula is used to dynamically update the dynamic probability value of the target based on its fit with the camera motion model across multiple consecutive frames. Specifically, an optical flow algorithm is used to estimate the motion of each target, comparing the target's motion with the camera motion. If the target's motion in the image perfectly matches the viewpoint change caused by the camera motion, its dynamic probability value is decreased; if the target's motion is inconsistent with the camera motion, its dynamic probability value is increased. Based on the updated dynamic probability values, low-probability targets are marked as static targets and excluded from the target detection operation. The target recognition module is used to output the final position coordinates and category labels of the remaining objects identified as dynamic targets as target detection results through the motion probability propagation model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the open scene target detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the open scene target detection method according to any one of claims 1 to 5.

Images