Open scene target detection method and device

Abstract

The embodiment of the invention provides an open scene target detection method and device, and the method comprises the steps: inputting an input image into a backbone network of a preset convolutional neural network, extracting the local granularity feature of the input image, carrying out the local sensing of the local granularity feature through deep convolution, extracting the corresponding local space information, and carrying out the recognition of the local space information; capturing a global context dependency relationship among the local space information through a multi-head self-attention mechanism, and performing multi-scale feature fusion on the global context dependency relationship and the local granularity features to obtain corresponding multi-level feature representation; inputting the multi-level feature representation into a set movement probability propagation model for target detection operation, and outputting a final position coordinate and a category label corresponding to an object identified as a dynamic target as a target detection result through the movement probability propagation model; according to the invention, the precision and real-time performance of target detection in an open scene can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to an open scene target detection method and device. Background Art

[0002] How to detect and identify various types of targets in an open and complex background is the key to improving target detection and identification. There are many types of ground targets, and there are huge differences within each type of target. Some non-targets in the surrounding background will also interfere with target detection. In addition to the impact of target differences and background, there are also a large number of factors related to time and environmental variables on the ground. In complex environmental conditions, targets will change with tactics and concealment, further increasing the difficulty of detection and identification. Compared with scenes such as sea and air, the ground open scene target detection algorithm needs to process more information to achieve target identification, which brings huge challenges to the computing power resources, algorithm model complexity and other aspects of target detection.

[0003] As the ground situation changes, commanders need to quickly and accurately obtain real annotations of on-site data in order to make correct decisions. Therefore, target detection technology in open scenarios has important application prospects in ground command. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides an open scene target detection method and device, which can effectively improve the accuracy and real-time performance of target detection in open scenes.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides an open scene object detection method, comprising:

[0007] The input image is sent to the preset convolutional neural network backbone network to extract the local granularity features of the input image, the local granularity features are locally perceived through deep convolution, the corresponding local spatial information is extracted, and the global context dependency between the local spatial information is captured through a multi-head self-attention mechanism, and the global context dependency and the local granularity features are multi-scale feature fused to obtain the corresponding multi-level feature representation;

[0008] The multi-level feature representation is input into a moving probability propagation model to perform a target detection operation, a dynamic probability value is initialized for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and the dynamic probability value of the target is dynamically updated in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, and a low-probability target is marked as a static target according to the updated dynamic probability value, and the static target is excluded from the target detection operation;

[0009] The mobile probability propagation model outputs the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets as target detection results.

[0010] Furthermore, the step of sending the input image to a preset convolutional neural network backbone network to extract local granularity features of the input image includes:

[0011] Obtaining an input image to be detected, and inputting the input image into a pre-set convolutional neural network;

[0012] The convolutional pooling operation is performed on the input image through the backbone network of the convolutional neural network to extract local granularity features for reflecting local granularity information.

[0013] Furthermore, the local granularity feature is locally sensed by deep convolution to extract corresponding local spatial information, including:

[0014] Acquire the extracted local granularity features and input the local granularity features into a deep convolutional network structure, wherein the deep convolutional network comprises a plurality of convolutional layers and activation layers;

[0015] The local granularity features are locally perceived through layer-by-layer deep convolution operations of the deep convolutional network structure to extract corresponding local spatial information.

[0016] Furthermore, the global context dependency between the local spatial information is captured by the multi-head self-attention mechanism, and multi-scale feature fusion is performed on the global context dependency and the local granularity features to obtain a corresponding multi-level feature representation, including:

[0017] The extracted local spatial information is input into the multi-head self-attention mechanism module to calculate the mutual correlation weights, and the correlation weights are concatenated to generate a feature vector representing the global context dependency relationship;

[0018] The feature vector of the global context dependency and the local granularity feature are extracted on the convolution and pooling layers of different receptive fields through a parallel multi-branch network structure and combined in a cascade or weighted manner to obtain the final multi-level feature representation.

[0019] Furthermore, the step of inputting the multi-level feature representation into a setting moving probability propagation model to perform a target detection operation, and initializing a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, includes:

[0020] Inputting the multi-level feature representation into a mobile probability propagation model based on a probability graph to perform target recognition through a multi-level feature target detector;

[0021] A dynamic probability value is initialized for each detected target through the motion predictor in the motion probability propagation model to characterize the possibility that the target belongs to a dynamic object.

[0022] Furthermore, dynamically updating the dynamic probability value of the target in subsequent consecutive frames according to the degree of conformity of the target with the camera motion model in multiple consecutive frames, marking the low-probability target as a static target according to the updated dynamic probability value, and excluding the static target from the target detection operation, includes:

[0023] Obtain the dynamic probability value corresponding to each initialized target, and dynamically update the probability value of each target according to the degree of consistency between each target and the camera motion model in multiple consecutive frames when processing consecutive image frames;

[0024] The dynamic probability value of each target after probability update is obtained. For targets with lower probability values, they are marked as static targets and these static targets are excluded from subsequent target detection operations.

[0025] Furthermore, outputting the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets through the mobile probability propagation model as target detection results includes:

[0026] Obtain the remaining objects identified as dynamic targets after the static targets are excluded by the mobile probability propagation model;

[0027] The final position coordinate information of the dynamic target and the corresponding category label are extracted from the output of the mobile probability propagation model as the target detection result.

[0028] In a second aspect, the present application provides an open scene object detection device, comprising:

[0029] An image processing module is used to send an input image into a preset convolutional neural network backbone network to extract local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract corresponding local spatial information, capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain corresponding multi-level feature representation;

[0030] A motion detection module, used to input the multi-level feature representation into a set motion probability propagation model to perform a target detection operation, initialize a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically update the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, mark a low-probability target as a static target according to the updated dynamic probability value, and exclude the static target from the target detection operation;

[0031] The target recognition module is used to output the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets as target detection results through the mobile probability propagation model.

[0032] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the open scene target detection method when executing the program.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the open scene target detection method.

[0034] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the open scene target detection method.

[0035] It can be seen from the above technical scheme that the present application provides a method and device for open scene target detection, which extracts local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, performs local perception of the local granularity features through deep convolution, extracts the corresponding local spatial information, and captures the global context dependency between the local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into a set moving probability propagation model for target detection operation, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the open scene object detection method in the embodiment of the present application;

[0038] Figure 2 This is a second flow chart of the open scene object detection method in an embodiment of the present application;

[0039] Figure 3 The third flowchart of the open scene target detection method in the embodiment of the present application;

[0040] Figure 4 This is a fourth flow chart of the open scene object detection method in an embodiment of the present application;

[0041] Figure 5 FIG5 is a flowchart of the open scene object detection method in the embodiment of the present application;

[0042] Figure 6 This is a sixth flow chart of the open scene object detection method in the embodiment of the present application;

[0043] Figure 7 FIG7 is a flow chart of the open scene object detection method in the embodiment of the present application;

[0044] Figure 8 is a structural diagram of an open scene object detection device in an embodiment of the present application;

[0045] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0046] Reference numerals:

[0047] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0050] In view of the problems existing in the prior art, the present application provides a method and device for open scene target detection, which extracts local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, performs local perception of the local granularity features through deep convolution, extracts the corresponding local spatial information, and captures the global context dependency between the local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into a set moving probability propagation model for target detection operation, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as the target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0051] In order to effectively improve the accuracy and real-time performance of target detection in open scenes, the present application provides an embodiment of an open scene target detection method, see Figure 1 , the open scene target detection method specifically includes the following contents:

[0052] Step S101: Send the input image to the preset convolutional neural network backbone network to extract the local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation;

[0053] Optionally, in this embodiment, first, the input image is fed into a preset convolutional neural network backbone network. This process usually uses pre-trained deep convolutional neural networks, such as ResNet, VGG or EfficientNet. These networks have been trained on large-scale data sets and can effectively extract local granularity features of images. The backbone network gradually extracts from low-level features (such as edges, textures) to high-level features (such as object parts) through multi-layer convolution and pooling operations. This hierarchical feature extraction capability is the core advantage of convolutional neural networks, enabling them to capture complex patterns in images.

[0054] Next, local granular features are locally perceived through deep convolution to extract the corresponding local spatial information. Deep convolution can increase the receptive field while maintaining computational efficiency by increasing the depth of the network and using small-sized convolution kernels, thereby capturing a wider range of spatial dependencies. This process can use techniques such as dilated convolution or depthwise separable convolution to improve efficiency and effectiveness. The extraction of local spatial information is crucial to understanding the shape, size, and position of the target.

[0055] Then, the global contextual dependencies between local spatial information are captured through a multi-head self-attention mechanism. The multi-head self-attention mechanism is derived from the Transformer model, which allows the model to learn attention patterns in different representation subspaces. In this process, each local feature interacts with all other features and calculates the correlation between them. This mechanism can effectively capture long-distance dependencies and overcome the limitations of traditional convolutional networks in processing global information. Through the multi-head mechanism, the model can simultaneously pay attention to different types of dependencies, such as spatial position, semantic similarity, etc., thereby providing richer contextual information.

[0056] Finally, multi-scale feature fusion is performed on the global context dependencies and local granular features to obtain the corresponding multi-level feature representation. This process usually involves a feature pyramid network (FPN) or a similar multi-scale fusion structure. Features of different scales are aligned through upsampling and downsampling operations, and then fused through addition, concatenation, or attention mechanisms. This multi-scale fusion can retain both fine-grained local information and large-scale semantic information, which is conducive to detecting objects of different sizes and scales.

[0057] This step solves several key technical problems in open scene object detection. First, it overcomes the limitations of single feature representation. By combining local granular features, local spatial information, and global contextual dependencies, the model is able to understand the image content more comprehensively. Second, it solves the problem of modeling long-distance dependencies. Traditional convolutional networks are difficult to effectively capture long-distance semantic associations due to the limitations of the receptive field, and the multi-head self-attention mechanism solves this problem well. Third, through multi-scale feature fusion, it solves the problem of object detection at different scales, enabling the model to handle large and small objects at the same time.

[0058] From the technical effect point of view, this step achieves several important goals. First, it significantly improves the richness and expressiveness of feature representation. Multi-level and multi-scale feature fusion enables the model to better understand targets in complex scenes. Second, it enhances the model's ability to detect targets of different scales and positions. Whether it is a small object or a large object, a near or far target, it can be better detected. Third, by introducing global context information, it improves the model's ability to understand the scene as a whole, which helps to reduce false detections and missed detections.

[0059] Let's use a specific example to illustrate the application of this step. Suppose we are developing an object detection system for autonomous driving scenarios. The input image is a complex urban street scene containing various vehicles, pedestrians, traffic signs, and buildings.

[0060] First, we use the pre-trained ResNet-101 as the backbone network. The input image (assuming the size is 1920x1080x3) is processed by the backbone network to obtain feature maps of multiple scales, for example, feature maps of sizes 120x67x256, 60x34x512, 30x17x1024, and 15x9x2048 may be obtained. These feature maps contain local granular features from low-level to high-level.

[0061] Next, we use a series of dilated convolutional layers to further process these feature maps. For example, for a 60x34x512 feature map, we may use three parallel 3x3 dilated convolutions with expansion rates of 1, 2, and 4. This can expand the receptive field without increasing the number of parameters and extract richer local spatial information.

[0062] We then feed the processed features into a multi-head self-attention module. Assume we use 8 attention heads, each with a dimension of 64. For a 60x34x512 feature map, we first reshape it into a sequence of 2040x512 (2040 = 60x34), and then perform self-attention calculations. This process allows the features at each position to interact with the features at all other positions, capturing global dependencies.

[0063] Finally, we use the Feature Pyramid Network (FPN) structure for multi-scale feature fusion. Starting from the top layer, we upsample and fuse features layer by layer. For example, after the 15x9x2048 feature map is upsampled to 30x17, it is fused with the 30x17x1024 feature map to obtain a new 30x17x256 feature map. This process continues until the bottom layer, and finally a series of feature maps with the same size (such as 120x67x256) but containing different levels of information are obtained.

[0064] This embodiment shows how to implement step S101 in a practical application. Through this complex feature extraction and fusion process, our target detection system can simultaneously focus on large vehicles in the street scene and small traffic signs in the distance, understand the relationship between pedestrians and the surrounding environment, and even capture the interaction pattern between different vehicles. This comprehensive and detailed scene understanding capability is crucial to the safety and effectiveness of the autonomous driving system.

[0065] In general, step S101 provides a powerful feature representation capability for open scene object detection by combining multiple advanced technologies in deep learning. It not only improves the accuracy and robustness of detection, but also lays a solid foundation for subsequent object recognition and tracking tasks.

[0066] Step S102: inputting the multi-level feature representation into a set moving probability propagation model to perform a target detection operation, initializing a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically updating the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, marking a low-probability target as a static target according to the updated dynamic probability value, and excluding the static target from the target detection operation;

[0067] Optionally, in this embodiment, first, the multi-level feature representation is input into a pre-set mobile probability propagation model. This model is a complex system that combines target detection and motion estimation. It can not only identify targets in a single frame image, but also track the movement of targets between consecutive frames. The mobile probability propagation model is usually based on a Bayesian framework, which transforms the target detection problem into a probabilistic reasoning problem. The advantage of this method is that it can naturally integrate temporal information and uncertainty processing.

[0068] During the object detection operation, the model initializes a dynamic probability value for each detected object. This probability value is used to characterize the possibility that the object is a dynamic object. The initialization process may be based on the size, position, category and other characteristics of the object, or it may use a pre-trained motion model for preliminary estimation. For example, for road scenes, vehicles and pedestrians may be assigned higher initial dynamic probability values, while buildings and street lights may be assigned lower initial values.

[0069] Next, when processing subsequent consecutive frames, the model will dynamically update the dynamic probability value of each target based on the degree of fit with the camera motion model in multiple consecutive frames. This process involves two aspects: camera motion estimation and target motion estimation. The camera motion model can be obtained through methods such as visual odometry or inertial measurement unit (IMU). The target motion is estimated by comparing the position changes of the target in consecutive frames.

[0070] The updating of dynamic probability values ​​is the core of this step. The updating process can use probabilistic filtering methods such as Kalman filter or particle filter. For example, if the movement of a target is consistent with the movement of the camera (considering the perspective transformation), then it is likely to be static and its dynamic probability value will decrease; conversely, if the movement of the target is inconsistent with the movement of the camera, its dynamic probability value will increase. This dynamic update mechanism enables the model to accumulate evidence over time and improve the accuracy of judgment.

[0071] Based on the updated dynamic probability value, the model marks low-probability targets as static targets. A threshold needs to be set here, for example, when the dynamic probability value is lower than 0.8, the target is marked as static. Targets marked as static are then excluded from the dynamic target detection operation.

[0072] This step solves several key technical problems in open scene object detection. First, it solves the problem of distinguishing static and dynamic targets. In complex real-world scenarios, distinguishing static and dynamic targets is crucial for many applications. For example, autonomous driving systems need to adopt different processing strategies for static obstacles and moving vehicles. Second, it solves the problem of temporal consistency in target detection. By propagating and updating probabilities between consecutive frames, the model is able to utilize temporal information to improve the stability and accuracy of detection. Third, it provides a way to deal with uncertainty. The motion of targets in real scenes is often complex and uncertain, and the use of probabilistic models can better express and handle this uncertainty.

[0073] From the technical effect point of view, this step achieves several important goals. First, it improves the accuracy of target detection, especially in distinguishing static and dynamic targets. By introducing and updating dynamic probability values, the model is able to accumulate evidence over time and make more reliable judgments. Second, it enhances the system's adaptability to complex scenes. In scenes containing multiple moving objects, this method can better understand the dynamic characteristics of the scene. Third, it improves computational efficiency. By excluding static targets from dynamic target detection, the number of targets that need to be processed can be reduced, thereby reducing the computational burden.

[0074] Let's use a specific example to illustrate the application of this step. Suppose we are developing an intelligent camera system for urban environment monitoring. This system needs to detect and track various objects on the street, including pedestrians, vehicles, bicycles, etc.

[0075] First, we use the multi-level feature representation obtained in the previous step and input it into the motion probability propagation model we designed. This model contains an object detector based on Faster R-CNN and a motion predictor based on optical flow estimation. For each detected object, we initialize a dynamic probability value. For example, for a detected car, we may initialize a dynamic probability value of 0.7; for a detected street light, we may initialize a dynamic probability value of 0.1.

[0076] As we process consecutive frames, we estimate the camera's motion using visual odometry techniques. At the same time, we estimate the motion of each object using an optical flow algorithm. We then compare the object's motion to the difference in the camera's motion. For example, if a parked car's motion in the image matches the change in perspective caused by the camera's motion, we would lower its dynamic probability value, perhaps from 0.7 to 0.3. Conversely, if a pedestrian's motion is inconsistent with the camera's motion, we would increase its dynamic probability value, perhaps from 0.6 to 0.9.

[0077] We use the Bayesian update formula to dynamically adjust the probability value of each target. Specifically, we can use particle filters to represent and update these probability distributions. For each target, we maintain a set of particles, each of which represents a possible motion state. By observing the position changes of the target in consecutive frames, we update the weights of these particles to obtain updated dynamic probability values.

[0078] Suppose we set the threshold of the dynamic probability value to 0.7. After the update, all targets with a dynamic probability value lower than 0.7 are marked as static targets. These static targets (such as road signs, buildings) will be excluded from the dynamic target detection operation. In this way, our system can focus on detecting and analyzing dynamic environmental elements such as pedestrians, vehicles, etc.

[0079] This example shows how to implement step S102 in a practical application. Through this process of dynamic probability update and target classification, our monitoring system can more accurately understand the dynamic characteristics of the urban environment. It can distinguish between parked vehicles and moving vehicles, identify pedestrians crossing the road, and accurately locate static infrastructure elements. This capability is of great value for applications such as urban planning, traffic management, and public safety.

[0080] In general, step S102 provides an effective method for handling complex dynamic environments for open scene object detection by introducing the concept of mobile probability propagation model and dynamic probability value. It not only improves the accuracy and robustness of detection, but also provides important dynamic information for subsequent scene understanding and decision making.

[0081] Step S103: Outputting the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets through the mobile probability propagation model as target detection results.

[0082] Optionally, in this embodiment, step S103 is the last key link of the open scene target detection method, which outputs the final target detection result through the mobile probability propagation model. The main goal of this step is to provide accurate dynamic target information, including their position coordinates and category labels. This step is not only a natural continuation of the previous steps, but also the culmination of the entire target detection process. Let us explore the technical process, principles, problems solved and effects achieved in this step.

[0083] First, the mobile probability propagation model has completed the distinction between dynamic and static targets in the previous step. At this stage, the model turns its focus to those targets that are identified as dynamic. These dynamic targets may include vehicles, pedestrians, etc. The model will perform final processing and refinement on these targets to ensure the accuracy and reliability of the output results.

[0084] In the technical process, the model first accurately calculates the position coordinates of each target identified as dynamic. This usually involves regression optimization of the bounding box. The model may use non-maximum suppression (NMS) technology to eliminate overlapping detection results to ensure that each actual target has only one corresponding detection box. The position coordinates are usually expressed in the image pixel coordinate system, but in some applications, they may also be converted to real-world 3D coordinates.

[0085] At the same time, the model assigns a category label to each dynamic object. This process is usually based on the output of the softmax classifier, and the category with the highest probability is selected as the final label. In some complex scenarios, the model may output multiple possible categories and their corresponding probabilities so that subsequent processing can take advantage of this uncertainty information.

[0086] In addition, the mobile probability propagation model may also output a confidence score for each detection result. This score reflects the model's confidence in its detection and classification results, which is very important for subsequent decision making and risk assessment.

[0087] In terms of technical principles, this step mainly relies on the reasoning ability of deep learning models and the uncertainty processing of probabilistic models. Deep neural networks perform well in feature extraction and pattern recognition, while probabilistic models provide a framework for processing uncertainty and fusing multi-frame information. The combination of the two enables the system to make accurate judgments in complex real-world scenarios.

[0088] This step solves several key technical problems. First, it solves the problem of accurate positioning of dynamic targets. Accurately positioning dynamic targets in complex environments is critical for many applications (such as autonomous driving and augmented reality). Second, it solves the problem of target classification and can accurately identify different types of dynamic objects. Third, it provides a way to deal with the uncertainty of detection results, by outputting confidence scores, allowing the system to better assess and manage risks.

[0089] From the technical effect point of view, this step achieves several important goals. First, it improves the accuracy and reliability of dynamic target detection. By combining multi-frame information and probabilistic models, the system is able to overcome the limitations of single-frame detection and provide more stable and accurate results. Second, it enhances the system's ability to understand complex scenes. By accurately identifying and locating static targets, the system is able to build a more complete scene representation. Third, it provides reliable input for subsequent decision-making and control processes, which is very important for many practical applications (such as autonomous driving, robot navigation).

[0090] From the above description, it can be seen that the open scene target detection method provided by the embodiment of the present application can extract the local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, perform local perception of the local granularity features through deep convolution, and extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into the set moving probability propagation model to perform target detection operations, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0091] In one embodiment of the open scene object detection method of the present application, see Figure 2 , and can also include the following:

[0092] Step S201: obtaining an input image to be detected, and inputting the input image into a pre-set convolutional neural network;

[0093] Step S202: performing a convolution pooling operation on the input image through the backbone network of the convolutional neural network to extract local granularity features for reflecting local granularity information.

[0094] Optionally, in this embodiment, first, in step S201, the system obtains an input image to be detected. This process seems simple, but it actually involves multiple technical details. The input image may come from various sources, such as a video stream captured in real time by a camera, a stored static image file, or image data transmitted over a network. When acquiring the image, the system needs to process various possible image formats (such as JPEG, PNG, RAW, etc.) and ensure the integrity and consistency of the image data. In addition, preprocessing operations may also be required, such as adjusting the image size, normalizing pixel values, or performing color space conversion (for example, from RGB to YUV). These preprocessing steps are crucial to ensuring the normal operation of subsequent convolutional neural networks.

[0095] The acquired and preprocessed images are then input into a pre-set convolutional neural network. This network is the core of the entire object detection system, and its structure and parameters directly affect the performance and effect of the detection. The pre-set network is usually a model that has been trained with a large amount of data, such as ResNet, VGG, or EfficientNet. These networks are pre-trained on large-scale datasets such as ImageNet and have powerful feature extraction capabilities.

[0096] In step S202, the input image is processed by the backbone network of the convolutional neural network. The backbone network is usually composed of multiple layers of convolutional layers and pooling layers. The convolution operation is the core of the convolutional neural network, which extracts features by sliding convolution kernels (also called filters) on the image. Each convolution kernel can be regarded as a detector of a specific pattern, which can recognize features such as edges, textures, shapes, etc. in the image.

[0097] The pooling operation is mainly used to reduce the spatial dimension of the feature map, increase the receptive field, and provide a certain degree of translation invariance. The most commonly used pooling methods are maximum pooling and average pooling. Maximum pooling retains the most significant features in the region, while average pooling retains the average information of the region. These operations help the network retain important information while reducing the amount of calculation and the number of parameters.

[0098] Through a series of convolution and pooling operations, the network extracts more abstract and advanced features layer by layer. Shallow networks mainly extract some low-level features, such as edges and textures; while deep networks can extract more complex and abstract features, such as partial or overall structures of objects. This hierarchical feature extraction process is the key to the power of convolutional neural networks.

[0099] Finally, the features extracted by this process are called local granularity features. These features reflect the local granularity information of the image, including details such as the shape, texture, and color distribution of the object. Local granularity features are crucial for subsequent object detection and recognition tasks because they contain rich visual information that can help the system accurately locate and classify objects in the image.

[0100] These two steps solve several key technical problems. First, they solve the problem of how to effectively extract useful features from raw images. Traditional hand-crafted feature methods often have difficulty coping with complex real-world scenes, while convolutional neural networks can automatically learn more robust and representative features. Second, they solve the problem of feature representation. Through multi-level feature extraction, the network can capture both low-level local features and high-level semantic features. Third, they provide an effective method for processing large-scale visual data. The combination of convolution and pooling operations greatly reduces the amount of data that needs to be processed, making real-time target detection in complex scenes possible.

[0101] From the technical effect point of view, these two steps have achieved several important goals. First, they have improved the efficiency and effect of feature extraction. Compared with traditional methods, convolutional neural networks can extract useful information from images more quickly and accurately. Second, they have enhanced the system's ability to adapt to complex scenes. Through the learned features, the system can better handle various lighting conditions, occlusions, and complex backgrounds. Third, they provide high-quality feature inputs for subsequent target detection and recognition tasks, which is crucial to improving the performance of the entire system.

[0102] In one embodiment of the open scene object detection method of the present application, see Figure 3 , and can also include the following:

[0103] Step S301: obtaining the extracted local granularity features and inputting the local granularity features into a deep convolutional network structure, wherein the deep convolutional network includes a plurality of convolutional layers and activation layers;

[0104] Step S302: locally perceive the local granularity features through layer-by-layer deep convolution operations of the deep convolutional network structure to extract corresponding local spatial information.

[0105] Optionally, in this embodiment, in step S301, the system first obtains the local granularity features extracted in the previous step. These features are usually a series of multidimensional tensors, each of which represents the feature representation of the image at different scales and abstract levels. These features contain information such as the local structure, texture, and edges of the image, but may not be sufficient to be directly used for complex target detection tasks. Therefore, these features are input into a specially designed deep convolutional network structure.

[0106] This deep convolutional network structure is specially designed for further feature extraction and refinement. It contains multiple convolutional layers and activation layers. The combination of these layers enables the network to learn more complex and abstract feature representations. The convolutional layer is responsible for feature extraction and transformation, while the activation layer introduces nonlinearity to increase the network's expressiveness. Commonly used activation functions include ReLU (Rectified Linear Unit), LeakyReLU, ELU (Exponential Linear Unit), etc. The choice of these activation functions has an important impact on the performance of the network. They can help the network overcome the gradient vanishing problem, accelerate the training process, and improve the model's expressiveness.

[0107] In step S302, the local granularity features are subjected to layer-by-layer deep convolution operations through a deep convolutional network structure. This process can be viewed as a local perception process of the input features. Each layer of convolution operation slides the convolution kernel on the feature map of the previous layer to capture local spatial relationships and patterns. A key advantage of this operation is parameter sharing, that is, the same convolution kernel shares parameters across the entire feature map, which greatly reduces the number of parameters that need to be learned and improves the efficiency and generalization ability of the model.

[0108] Each layer of a deep convolutional network can be thought of as learning a specific feature detector. A shallow network may learn a simple edge or texture detector, while a deep network may learn more complex patterns, such as the partial or overall structure of an object. By stacking multiple layers of convolution, the network can gradually build a hierarchical representation of the input data.

[0109] In this process, the network is not only extracting features, but also learning how to most effectively combine and transform these features to adapt to the target detection task. For example, the network may learn how to effectively fuse information of different scales, or how to highlight the most critical features for target detection.

[0110] In this way, deep convolutional networks can extract rich local spatial information. This information not only includes the appearance features of the object, but also the spatial relationship between the parts of the object. For example, for vehicle detection, the network may learn the relative position relationship between parts such as wheels, body, and windows, which is crucial for accurately identifying and locating vehicles.

[0111] These two steps solve several key technical problems. First, they solve the problem of how to extract more abstract and task-related features from the initial local granular features. Through deep convolutional networks, the system can learn feature representations that are more suitable for target detection tasks. Second, they solve the problem of how to effectively capture local spatial relationships. Through layer-by-layer convolution operations, the network can gradually expand the receptive field to capture a wider range of spatial dependencies. Third, they provide an adaptive feature learning method that can automatically learn the most relevant features based on training data without the need for manually designed feature extractors.

[0112] From the technical effect point of view, these two steps achieve several important goals. First, they improve the expressiveness of features. Through deep convolutional networks, the system is able to learn richer and more discriminative feature representations, which is crucial to improving the accuracy of target detection. Second, they enhance the system's ability to understand complex scenes. By capturing local spatial information, the system can better handle complex situations such as occlusion and deformation between objects. Third, they provide high-quality feature input for subsequent target detection and classification tasks, which helps to improve the performance and robustness of the entire system.

[0113] In one embodiment of the open scene object detection method of the present application, see Figure 4 , and can also include the following:

[0114] Step S401: obtaining the extracted local spatial information and inputting it into the multi-head self-attention mechanism module to calculate the mutual correlation weights, and performing a splicing operation on the correlation weights to generate a feature vector representing the global context dependency relationship;

[0115] Step S402: extracting features from the feature vector of the global context dependency and the local granularity features on convolution and pooling layers of different receptive fields through a parallel multi-branch network structure and combining them in a cascade or weighted manner to obtain a final multi-level feature representation.

[0116] Optionally, in this embodiment, in step S401, the system first obtains the local spatial information extracted in the previous step. This information is usually a series of feature maps, each of which represents the local features of the image at different positions and scales. Although these local features contain rich spatial information, the relationship between them has not been fully utilized. Therefore, these features are input into the multi-head self-attention mechanism module.

[0117] The multi-head self-attention mechanism is a powerful technique that can capture long-range dependencies within a sequence. In computer vision tasks, it is used to model the relationship between different regions of an image. Specifically, the multi-head self-attention mechanism first linearly projects the input features into multiple subspaces (called "heads"), and then calculates the attention weights independently in each subspace. This multi-head design allows the model to focus on different types of relationships simultaneously.

[0118] In each head, the attention weights are usually calculated based on the query, key, and value matrices. The dot product of the query and key is used to calculate the similarity, and then normalized by the softmax function to get the attention weights. These weights are used to weight the value matrix to get the output of the head. Finally, the outputs of all heads are spliced ​​together to form the final output.

[0119] The key to this process is that it allows the model to learn complex relationships between different regions of an image. For example, in an object detection task, it might learn the relationship between different parts of an object (such as a person's head and body), or the relationship between an object and its surroundings (such as a car and the road). Modeling this kind of global relationship is critical to understanding complex scenes.

[0120] The output of step S401 is a feature vector representing the global context dependency. This feature vector captures the relationship between various regions in the image and provides important context information for subsequent object detection tasks.

[0121] In step S402, the system combines the feature vector of the global context dependency with the local granularity feature obtained previously. The purpose of this step is to fuse the global information with the local information to generate a more comprehensive and rich feature representation.

[0122] Specifically, this step uses a parallel multi-branch network structure. Each branch contains convolutional layers and pooling layers with different receptive fields. The receptive field refers to the size of the input area that each neuron in the network can "see". Receptive fields of different sizes allow the network to capture features of different scales.

[0123] In each branch, the convolution layer is responsible for extracting features, while the pooling layer is used to reduce the spatial dimension of the feature map and increase the translation invariance of the features. By using convolution kernels and pooling windows of different sizes, each branch can focus on features of different scales.

[0124] The outputs of these branches can be combined in two ways: cascaded or weighted. The cascade method directly concatenates the outputs of different branches to form a longer feature vector. The weighted method assigns a weight to the output of each branch and then sums them up. Both methods have their advantages. The cascade retains all information but may result in too high a feature dimension, while the weighted method allows the model to learn the importance of features at different scales.

[0125] The result of this step is a final multi-level feature representation that combines local granular features, global context dependencies, and multi-scale information, providing rich information for subsequent object detection and classification tasks.

[0126] These two steps solve several key technical problems. First, they solve how to effectively model long-range dependencies between different regions in an image. Through the multi-head self-attention mechanism, the system is able to capture complex global contextual information, which is difficult to achieve in traditional convolutional neural networks. Second, they solve the problem of how to fuse global information and local information. Through a parallel multi-branch network structure, the system is able to process features of different scales simultaneously and combine them effectively. Third, they provide a flexible feature extraction method that can adapt to targets of different sizes and complexities.

[0127] From the technical effect point of view, these two steps achieve several important goals. First, they improve the expressiveness of features. By fusing global and local information, the system is able to generate richer and more meaningful feature representations, which is crucial to improving the accuracy and robustness of target detection. Second, they enhance the system's ability to understand complex scenes. By modeling global dependencies and multi-scale features, the system is able to better handle complex visual scenes such as occlusion, deformation, scale changes, etc. Third, they provide high-quality feature inputs for subsequent target detection and classification tasks, which helps improve the performance and generalization ability of the entire system.

[0128] In one embodiment of the open scene object detection method of the present application, see Figure 5 , and can also include the following:

[0129] Step S501: inputting the multi-level feature representation into a probability graph-based mobile probability propagation model to perform target recognition through a multi-level feature target detector;

[0130] Step S502: initializing a dynamic probability value for each detected target through the motion predictor in the motion probability propagation model to characterize the possibility that the target is a dynamic object.

[0131] Optionally, in this embodiment, in step S501, the system first inputs the multi-level feature representation into a mobile probability propagation model based on a probability graph. This model is a complex probabilistic framework that can effectively process and propagate uncertainty information. In this model, each area in the image is represented as a node in a probability graph, and the edges between the nodes represent the relationship between them. This representation method allows the model to consider the spatial and semantic relationships between targets, thereby improving the accuracy of detection.

[0132] The multi-level feature object detector is one of the core components of this model. It uses the rich multi-level features generated in the previous steps to perform object recognition. This detector is usually based on deep learning models such as Faster R-CNN, YOLO or SSD, but is specially modified to adapt to multi-level feature input. The detector first generates a series of candidate regions, and then uses multi-level features to classify and regress bounding boxes for these regions.

[0133] A key advantage of this process is that it can simultaneously utilize features at different scales and levels of abstraction. For example, low-level features may contain detailed texture information, which helps to identify small objects or fine structures; mid-level features may contain shape and edge information, which helps to locate object boundaries; high-level features may contain semantic information, which helps to understand the category and context of objects. By combining these features, the detector can more accurately identify and locate objects of various sizes and complexities.

[0134] In addition, probabilistic graph-based models also allow for the introduction of prior knowledge and contextual information. For example, we can encode in the model the knowledge that certain objects often appear together (such as tables and chairs), or the probability of certain objects appearing in specific scenes (such as cars on the road). This information can be represented by adjusting the edge weights in the probability graph, further improving the accuracy and robustness of detection.

[0135] Step S502 follows step S501 and starts processing the dynamic characteristics of the target. In this step, the system uses the motion predictor in the mobile probability propagation model to initialize a dynamic probability value for each detected target. This probability value is used to characterize the possibility that the target is a dynamic object.

[0136] A motion predictor is a model specifically designed to predict the motion of an object. It may be based on various techniques such as Kalman filtering, particle filtering, or more complex deep learning models. This predictor considers not only the current position and size of the object, but also its changes in consecutive frames. For example, it may analyze the position, velocity, acceleration, and other parameters of the object, as well as the trend of these parameters over time.

[0137] Initialization of dynamic probability values ​​is a critical step. For each detected target, the system estimates its dynamic characteristics based on its features and historical information (if any). For example, for a new target just detected, the system may initialize the dynamic probability value based on its category. A person or a vehicle may be assigned a higher initial dynamic probability, while a building or a tree may be assigned a lower initial dynamic probability.

[0138] This dynamic probability value is not static, but is constantly updated with time and new observations. For example, if an object that was initially considered static begins to move, its dynamic probability value will increase rapidly. Conversely, if an object that was initially considered dynamic remains stationary for a long time, its dynamic probability value may gradually decrease.

[0139] These two steps solve several key technical problems. First, they solve the problem of how to effectively use multi-level features for target detection. By inputting rich multi-level features into the probability map-based model, the system is able to understand the image content more comprehensively and improve the accuracy and robustness of detection. Second, they solve the problem of how to distinguish between static and dynamic targets in complex scenes. By introducing dynamic probability values, the system is able to better characterize and track dynamic objects, which is crucial in many application scenarios. Third, they provide a flexible framework to integrate prior knowledge and real-time observations, so that complex and uncertain situations can be better handled.

[0140] From the technical effect point of view, these two steps achieve several important goals. First, they improve the accuracy and reliability of target detection. By utilizing multi-level features and probabilistic graph models, the system can more accurately identify and locate various targets, and perform well even in complex scenes. Second, they enhance the system's ability to understand dynamic scenes. By introducing dynamic probability values, the system can distinguish between static and dynamic targets, which is very important for many practical applications (such as traffic monitoring, security systems, etc.). Third, they lay the foundation for subsequent target tracking and behavior analysis tasks. The introduction of dynamic probability values ​​enables the system to better predict the motion trajectory of the target, thereby supporting more advanced analysis tasks.

[0141] In one embodiment of the open scene object detection method of the present application, see Figure 6 , and can also include the following:

[0142] Step S601: obtaining dynamic probability values ​​corresponding to each initialized target, and dynamically updating the probability value of each target according to the degree of conformity of each target with the camera motion model in multiple consecutive frames when processing consecutive image frames;

[0143] Step S602: Obtain the dynamic probability value of each target after probability update, mark the target with a lower probability value as a static target, and exclude these static targets from subsequent target detection operations.

[0144] Optionally, in this embodiment, in step S601, the system first obtains the dynamic probability values ​​corresponding to each target initialized in the previous step. These initial values ​​are estimated based on the target's category, position, size and other features. Then, when processing consecutive image frames, the system dynamically updates the probability value of each target based on the degree of fit between each target and the camera motion model in multiple consecutive frames. This process involves complex motion estimation and probability update technology.

[0145] The camera motion model is one of the core components of this step. It describes the movement of the camera in 3D space, usually including translation and rotation. This model can be estimated by various methods, such as feature point matching-based methods (such as SLAM technology) or deep learning-based methods. An accurate camera motion model is crucial to distinguish between real object motion and visual changes caused by camera movement.

[0146] When processing consecutive frames, the system predicts the position of each object in the next frame. This prediction is based on the current state of the object (position, velocity, etc.) and the estimated camera motion. The system then compares this prediction with the actual observed position of the object. If the difference between the prediction and the observation is small, it means that the motion of the object is consistent with the camera motion and it may be a static object; if the difference is large, it means that the object may have its own motion and is a dynamic object.

[0147] Based on this comparison, the system updates the target's dynamic probability value. For example, if a target is highly consistent with the camera motion model for multiple consecutive frames, its dynamic probability value may gradually decrease; conversely, if a target's motion continues to deviate from the prediction of the camera motion model, its dynamic probability value may increase. This update is usually implemented using Bayesian updating or similar probability updating methods.

[0148] This step solves several key technical problems. First, it solves the problem of how to accurately distinguish static and dynamic objects in a moving camera scene. By introducing a camera motion model, the system is able to separate the visual changes caused by camera movement and the movement of the object itself. Second, it provides a continuous and smooth way to update the dynamic characteristics of the target, avoiding the instability that may be caused by single-frame judgment. Third, it can handle complex motion scenes, such as situations where the camera and the object move at the same time.

[0149] Step S602 follows step S601 and starts to classify and process targets using the updated dynamic probability values. In this step, the system first obtains the dynamic probability values ​​of each target after the probability update. Then, the system sets a threshold and marks targets with dynamic probability values ​​below this threshold as static targets. These targets marked as static are then excluded from subsequent target detection operations.

[0150] The core of this step is how to choose an appropriate threshold to distinguish between dynamic and static targets. The selection of the threshold requires a trade-off between the sensitivity of detection and the efficiency of the system. If the threshold is set too low, some actually static targets may be mistakenly marked as dynamic; if it is set too high, some dynamic targets may be missed. Therefore, the selection of the threshold usually requires careful tuning based on the specific application scenario and system requirements.

[0151] A key innovation of this step is to exclude dynamic objects from subsequent object detection operations. This approach is based on the assumption that objects that have been identified as dynamic are unlikely to become static objects in a short period of time, so there is no need to re-detect them in every frame. This can significantly reduce the computational load, especially in systems that process high frame rate videos or require real-time response.

[0152] However, this exclusion strategy also needs to be implemented with caution. The system needs to re-evaluate these dynamic targets regularly to prevent long-term tracking errors. For example, the system may re-detect these dynamic targets every certain number of frames, or trigger re-detection when the behavior of the dynamic target changes significantly (such as suddenly stopping movement).

[0153] These two steps solve several important technical problems. First, they address the problem of how to accurately track and update the dynamic characteristics of targets in a continuous video stream. By combining the camera motion model and continuous frame analysis, the system is able to more accurately identify truly dynamic targets. Second, they provide an effective way to balance detection accuracy and computational efficiency. By excluding known static targets, the system is able to focus more computing resources on the detection of new targets and the analysis of dynamic scenes. Third, this approach provides a flexible framework for handling complex dynamic scenes that can adapt to a variety of different application scenarios.

[0154] From the technical effect point of view, these two steps achieve several important goals. First, they improve the accuracy and reliability of dynamic target recognition. Through continuous frame analysis and probability update, the system can better handle complex motion scenes and reduce misjudgments. Second, they significantly improve the computational efficiency of the system. By excluding known dynamic targets, the system can process video streams faster and support real-time applications. Third, this approach enhances the adaptability of the system, enabling it to handle a variety of different scenarios, from static monitoring to rapidly changing dynamic environments.

[0155] In one embodiment of the open scene object detection method of the present application, see Figure 7 , and can also include the following:

[0156] Step S701: obtaining the objects that are identified as dynamic targets after the static targets are excluded from the mobile probability propagation model;

[0157] Step S702: extracting the final position coordinate information of the dynamic target and the corresponding category label from the output of the mobile probability propagation model as the target detection result.

[0158] Optionally, in this embodiment, in step S701, the system first obtains the objects that are identified as dynamic targets after the motion probability propagation model excludes static targets. This process involves complex probability models and target classification techniques. The motion probability propagation model is an advanced target tracking and classification method that can track the movement of targets in continuous image frames and estimate the probability of each target being static or dynamic.

[0159] The core idea of ​​this model is to regard the motion state of the target as a probability distribution that changes over time. In each frame, the model updates this distribution, taking into account factors such as the position, velocity, acceleration, etc. of the target. At the same time, the model also takes into account other factors in the environment, such as the motion of the camera, the complexity of the scene, etc. In this way, the model can effectively distinguish between real object motion and visual changes caused by camera movement.

[0160] During the static target exclusion process, the system sets a probability threshold. If the dynamic probability of an object is lower than this threshold, it will be marked as a static object and excluded from subsequent processing. The selection of this threshold requires a trade-off between the sensitivity of detection and the accuracy of the system. Setting it too low may mistakenly mark some static objects as dynamic, while setting it too high may miss some slow-moving dynamic objects.

[0161] After this process, what remains are objects that are recognized as dynamic by the system. These objects may include vehicles, pedestrians, etc.

[0162] Step S702 then extracts the final position coordinate information and corresponding category labels of these dynamic targets from the output of the mobile probability propagation model as the final result of target detection. This step involves accurate coordinate extraction and reliable target classification.

[0163] For each target identified as dynamic, the system needs to determine its exact position coordinates. This usually includes the two-dimensional coordinates (x, y) of the target in the image, and sometimes also includes depth information (z) to form a three-dimensional coordinate. Various computer vision techniques may be used to extract coordinates, such as edge detection, contour analysis, etc. In order to improve accuracy, the system may comprehensively consider the position of the target in multiple consecutive frames and use techniques such as Kalman filters to smooth the coordinate estimates.

[0164] At the same time, the system also needs to assign a category label to each static target. This process usually relies on deep learning models, such as convolutional neural networks (CNN). These models are able to learn and recognize various different categories of objects from the visual features of the target. Category labels may include "car", "pedestrian", "traffic light", "building", etc., depending on the application scenario of the system and the training dataset.

[0165] It is worth noting that the classification of dynamic objects may be more challenging than that of static objects. This is because dynamic objects may be partially occluded or lack multi-angle visual information due to maintaining the same posture for a long time. To solve this problem, the system may use contextual information and time series analysis to improve the accuracy of classification.

[0166] These two steps solve several key technical problems. First, they solve the problem of how to accurately identify and locate static targets in complex dynamic scenes. By using the mobile probability propagation model and dynamic target exclusion, the system is able to effectively filter out moving objects and focus on dynamic elements. This is especially important in applications such as autonomous driving and augmented reality, which require accurate understanding of the dynamic structure of the environment.

[0167] Second, these steps provide an efficient way to integrate the location and semantic information of objects. By extracting both coordinates and category labels simultaneously, the system is able to provide rich scene understanding results. This comprehensive information is critical for advanced decision making and scene analysis.

[0168] Again, this approach provides a flexible framework for handling long-term scene changes. Since the location and category of static objects may change slowly over time (for example, vehicles in a parking lot may leave or new vehicles may arrive), the system needs to be able to adapt to these changes. By continuously updating the mobile probability propagation model, the system is able to capture these gradual changes.

[0169] From the technical effect point of view, these two steps achieve several important goals. First, they improve the accuracy and reliability of dynamic target recognition. By focusing on objects judged to be dynamic, the system is able to invest more resources to accurately locate and classify these targets. Second, they provide rich scene understanding results, including the precise location and semantic information of the target. This comprehensive information is essential for many advanced applications. Third, this approach enhances the adaptability of the system, enabling it to handle a variety of different scenarios, from static indoor environments to dynamically changing urban streets.

[0170] In order to effectively improve the accuracy and real-time performance of target detection in open scenes, the present application provides an embodiment of an open scene target detection device for implementing all or part of the open scene target detection method, see Figure 8 , the open scene object detection device specifically includes the following contents:

[0171] The image processing module 10 is used to send the input image into the preset convolutional neural network backbone network to extract the local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation;

[0172] A motion detection module 20 is used to input the multi-level feature representation into a set motion probability propagation model to perform a target detection operation, initialize a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically update the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, mark a low-probability target as a static target according to the updated dynamic probability value, and exclude the static target from the target detection operation;

[0173] The target recognition module 30 is used to output the final position coordinates and category labels corresponding to the remaining objects recognized as dynamic targets as target detection results through the mobile probability propagation model.

[0174] From the above description, it can be seen that the open scene target detection device provided in the embodiment of the present application can extract the local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, perform local perception of the local granularity features through deep convolution, and extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into the set moving probability propagation model to perform target detection operations, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0175] From the hardware level, in order to effectively improve the accuracy and real-time performance of target detection in open scenes, the present application provides an embodiment of an electronic device for implementing all or part of the content of the open scene target detection method, and the electronic device specifically includes the following content:

[0176] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the open scene target detection device and related devices such as the core business system, user terminal and related database; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the open scene target detection method and the embodiment of the open scene target detection device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0177] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0178] In practical applications, part of the open scene object detection method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0179] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0180] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0181] In one embodiment, the open scene object detection method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control:

[0182] Step S101: Send the input image to the preset convolutional neural network backbone network to extract the local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation;

[0183] Step S102: inputting the multi-level feature representation into a set moving probability propagation model to perform a target detection operation, initializing a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically updating the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, marking a low-probability target as a static target according to the updated dynamic probability value, and excluding the static target from the target detection operation;

[0184] Step S103: Outputting the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets through the mobile probability propagation model as target detection results.

[0185] From the above description, it can be seen that the electronic device provided in the embodiment of the present application extracts the local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, performs local perception of the local granularity features through deep convolution, extracts the corresponding local spatial information, and captures the global context dependency between the local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into the set moving probability propagation model to perform target detection operations, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0186] In another embodiment, the open scene object detection device can be configured separately from the central processing unit 9100. For example, the open scene object detection device can be configured as a chip connected to the central processing unit 9100, and the function of the open scene object detection method is implemented through the control of the central processing unit.

[0187] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0188] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0189] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0190] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0191] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0192] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0193] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0194] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0195] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the open scene object detection method in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the open scene object detection method in the above embodiment, where the execution subject is a server or a client, are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0196] Step S101: Send the input image to the preset convolutional neural network backbone network to extract the local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation;

[0197] Step S102: inputting the multi-level feature representation into a set moving probability propagation model to perform a target detection operation, initializing a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically updating the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, marking a low-probability target as a static target according to the updated dynamic probability value, and excluding the static target from the target detection operation;

[0198] Step S103: Outputting the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets through the mobile probability propagation model as target detection results.

[0199] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application extracts the local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, performs local perception of the local granularity features through deep convolution, extracts the corresponding local spatial information, and captures the global context dependency between the local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into the setting moving probability propagation model to perform target detection operations, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as the target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0200] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the open scene object detection method in the above embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the open scene object detection method are implemented. For example, the computer program / instruction implements the following steps:

[0201] Step S101: Send the input image to the preset convolutional neural network backbone network to extract the local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract the corresponding local spatial information, and capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation;

[0202] Step S102: inputting the multi-level feature representation into a set moving probability propagation model to perform a target detection operation, initializing a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically updating the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, marking a low-probability target as a static target according to the updated dynamic probability value, and excluding the static target from the target detection operation;

[0203] Step S103: Outputting the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets through the mobile probability propagation model as target detection results.

[0204] From the above description, it can be seen that the computer program product provided in the embodiment of the present application extracts the local granularity features of the input image by sending the input image into the backbone network of a preset convolutional neural network, performs local perception of the local granularity features through deep convolution, extracts the corresponding local spatial information, and captures the global context dependency between the local spatial information through a multi-head self-attention mechanism, and performs multi-scale feature fusion on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation; the multi-level feature representation is input into the set moving probability propagation model to perform target detection operations, and the moving probability propagation model outputs the final position coordinates and category labels corresponding to the objects identified as dynamic targets as target detection results, thereby effectively improving the accuracy and real-time performance of target detection in open scenes.

[0205] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0209] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting an object in an open scene, characterized in that: The method comprises: The input image is sent to the preset convolutional neural network backbone network to extract the local granularity features of the input image, the local granularity features are locally perceived through deep convolution, the corresponding local spatial information is extracted, and the global context dependency between the local spatial information is captured through a multi-head self-attention mechanism, and the global context dependency and the local granularity features are multi-scale feature fused to obtain the corresponding multi-level feature representation; The multi-level feature representation is input into a moving probability propagation model to perform a target detection operation, a dynamic probability value is initialized for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and the dynamic probability value of the target is dynamically updated in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, and a low-probability target is marked as a static target according to the updated dynamic probability value, and the static target is excluded from the target detection operation; The mobile probability propagation model outputs the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets as target detection results.

2. The open scene target detection method according to claim 1, characterized in that: The step of sending the input image to a preset convolutional neural network backbone network to extract local granularity features of the input image includes: Obtaining an input image to be detected, and inputting the input image into a pre-set convolutional neural network; The convolutional pooling operation is performed on the input image through the backbone network of the convolutional neural network to extract local granularity features for reflecting local granularity information.

3. The open scene target detection method according to claim 1, characterized in that: The local granularity feature is locally sensed by deep convolution to extract corresponding local spatial information, including: Acquire the extracted local granularity features and input the local granularity features into a deep convolutional network structure, wherein the deep convolutional network comprises a plurality of convolutional layers and activation layers; The local granularity features are locally perceived through layer-by-layer deep convolution operations of the deep convolutional network structure to extract corresponding local spatial information.

4. The open scene target detection method according to claim 1, characterized in that: The multi-head self-attention mechanism is used to capture the global context dependency between the local spatial information, and multi-scale feature fusion is performed on the global context dependency and the local granularity features to obtain the corresponding multi-level feature representation, including: The extracted local spatial information is input into the multi-head self-attention mechanism module to calculate the mutual correlation weights, and the correlation weights are concatenated to generate a feature vector representing the global context dependency relationship; The feature vector of the global context dependency and the local granularity feature are extracted on the convolution and pooling layers of different receptive fields through a parallel multi-branch network structure and combined in a cascade or weighted manner to obtain the final multi-level feature representation.

5. The open scene target detection method according to claim 1, characterized in that: The step of inputting the multi-level feature representation into a moving probability propagation model to perform a target detection operation, and initializing a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, includes: Inputting the multi-level feature representation into a mobile probability propagation model based on a probability graph to perform target recognition through a multi-level feature target detector; A dynamic probability value is initialized for each detected target through the motion predictor in the motion probability propagation model to characterize the possibility that the target belongs to a dynamic object.

6. The open scene target detection method according to claim 1, characterized in that: The dynamically updating the dynamic probability value of the target in subsequent consecutive frames according to the degree of conformity of the target with the camera motion model in multiple consecutive frames, marking the low-probability target as a static target according to the updated dynamic probability value, and excluding the static target from the target detection operation, includes: Obtain the dynamic probability value corresponding to each initialized target, and dynamically update the probability value of each target according to the degree of consistency between each target and the camera motion model in multiple consecutive frames when processing consecutive image frames; The dynamic probability value of each target after probability update is obtained. For targets with lower probability values, they are marked as static targets and these static targets are excluded from subsequent target detection operations.

7. The open scene target detection method according to claim 1, characterized in that: Outputting the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets through the mobile probability propagation model as target detection results includes: Obtain the remaining objects identified as dynamic targets after the static targets are excluded by the mobile probability propagation model; The final position coordinate information of the dynamic target and the corresponding category label are extracted from the output of the mobile probability propagation model as the target detection result.

8. An open scene object detection device, characterized in that: The device comprises: An image processing module is used to send an input image into a preset convolutional neural network backbone network to extract local granularity features of the input image, perform local perception on the local granularity features through deep convolution, extract corresponding local spatial information, capture the global context dependency between the local spatial information through a multi-head self-attention mechanism, and perform multi-scale feature fusion on the global context dependency and the local granularity features to obtain corresponding multi-level feature representation; A motion detection module, used to input the multi-level feature representation into a set motion probability propagation model to perform a target detection operation, initialize a dynamic probability value for each target detected during the target detection operation to characterize the possibility that the target belongs to a dynamic object, and dynamically update the dynamic probability value of the target in subsequent continuous frames according to the degree of fit between the target and the camera motion model in multiple continuous frames, mark a low-probability target as a static target according to the updated dynamic probability value, and exclude the static target from the target detection operation; The target recognition module is used to output the final position coordinates and category labels corresponding to the remaining objects identified as dynamic targets as target detection results through the mobile probability propagation model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the open scene object detection method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the open scene object detection method described in any one of claims 1 to 7 are implemented.

Images