Multi-view pedestrian detection method based on multi-stage deformable decoupling

Abstract

The invention discloses a multi-view pedestrian detection method based on multi-stage deformable decoupling. The method comprises: for a target scene, obtaining a plurality of view images, and inputting the view images into a pedestrian detection model, the pedestrian detection model executing: extracting pedestrian features from the plurality of view images; task-related discriminative information is decoupled from pedestrian features, and effective features after first decoupling are obtained; projecting the effective features decoupled for the first time to a unified ground plane coordinate system to generate aerial view features; calculating a dynamic fusion weight based on multi-view feature semantics, and further obtaining a fused ground plane world feature based on the dynamic fusion weight; scene related features are decoupled from the fused ground plane world features, and effective features decoupled for the second time are obtained; and based on the effective features after the second decoupling, generating a ground plane pedestrian position thermodynamic diagram, and obtaining a multi-view pedestrian detection result. According to the method, the pedestrian detection precision in a multi-view complex scene is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a multi-view pedestrian detection method based on multi-stage deformable decoupling. Background Art

[0002] Pedestrian detection is widely used in practical application scenarios such as autonomous driving, video surveillance, and robot navigation. In recent years, multi-view pedestrian detection methods based on deep learning have significantly alleviated the problem of missed detection caused by occlusion in monocular detection by fusing the overlapping view information of multiple synchronously calibrated cameras. Specifically, in a multi-view scene, camera images with multiple viewpoints need to be subjected to feature extraction operations. Then, the features of each viewpoint are projected onto a unified ground plane through homography transformation, and a scene-level bird's-eye view (BEV) is constructed for feature fusion. The detection accuracy is then calculated for the final fusion result. In the process of image feature extraction, complex scenes with multiple views contain a large amount of noise information, such as background texture and lighting effects. In addition, the manually designed heuristic rules or single scale weight map used in the fusion stage are difficult to adapt to dynamic view configurations.

[0003] To address the problem of redundant noise in the feature space, existing solutions based on variational autoencoders (VAEs), such as β-VAE, enhance the decoupling capability of the latent space by introducing a hyperparameter β. However, this approach is limited by the trade-off between reconstruction accuracy and decoupling strength. Some solutions further optimize this through regularization strategies, such as employing a total correlation penalty to reduce redundant information between latent variables. Another example is combining this with the generative adversarial network (GAN) framework to achieve decoupling by maximizing the mutual information between the latent code and the generated data.

[0004] In order to solve the problem of differences in important relationships between different views during multi-view fusion, existing studies have calculated the fusion weights between multiple views based on camera distance, or introduced image confidence to determine the important relationships between views, or used the supervision results of a single view to participate in the fusion weighting between multiple views.

[0005] After analysis, it was found that in existing technologies, recognition tasks based on the single-view field only focus on the channel dimension of data features, ignoring the impact of spatial deformation (such as geometric distortion caused by perspective differences) on feature discriminability. This impact is particularly significant in multi-view crowd detection scenarios. For example, the apparent features of the target will produce non-rigid deformations due to factors such as perspective projection transformation and occlusion pattern changes. At the same time, current multi-view fusion methods usually rely on fixed priors (such as camera distance or projection confidence) to assign weights, ignoring the semantic associations and channel differences between multi-view features, resulting in a lack of scene adaptability of the fusion weights and difficulty in adapting to dynamic perspective configurations.

[0006] In summary, existing technologies still need to be improved to address the problem of decreased detection accuracy in complex multi-view scenes due to the large amount of noise information irrelevant to detection mixed in the feature space; and the problem of the use of manually designed heuristic rules or single scale weight maps in the fusion process, which leads to a heavy reliance on the fixed camera layout assumption and a lack of scene adaptability. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a multi-view pedestrian detection method based on multi-stage deformable decoupling. The method comprises:

[0008] For the target scene, multiple view images are acquired;

[0009] Inputting the multiple view images into a trained pedestrian detection model to obtain a multi-view pedestrian detection result;

[0010] Among them, the pedestrian detection model includes a feature extraction layer, a first deformable decoupling module, a bird's-eye view feature extraction module, a scene joint attention module, a second deformable decoupling module and a decoder. The feature extraction layer is used to extract pedestrian features from the multiple view images; the first deformable decoupling module is used to decouple task-related discriminative information from the pedestrian features to obtain effective features after the first decoupling; the bird's-eye view feature extraction module is used to project the effective features after the first decoupling to a unified ground plane coordinate system through homography transformation to generate multi-view aligned bird's-eye view features; the scene joint attention module is used to calculate the dynamic fusion weight based on the semantics of the multi-view features themselves, and then obtain the fusion features based on the dynamic fusion weights; the second deformable decoupling module is used to decouple scene-related features from the fused features to obtain effective features after the second decoupling; the decoder generates a ground plane pedestrian position heat map based on the effective features after the second decoupling to obtain multi-view pedestrian detection results.

[0011] Compared with the existing technology, the advantage of the present invention is that it provides a multi-view pedestrian detection method based on multi-stage deformable decoupling, especially a multi-view pedestrian detection method based on multi-stage deformable decoupling and scene joint attention fusion weights, which suppresses the interference of irrelevant noise information while retaining spatial deformation information. In the feature fusion process, the view weight does not rely on the fixed camera layout assumption, and can better decouple the features related to and irrelevant to the detection task in a multi-view environment. At the same time, it adaptively models the importance relationship of different perspectives in multi-view fusion, significantly improving the pedestrian detection accuracy in complex multi-view scenes.

[0012] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0014] Figure 1 4 is a flowchart of a multi-view pedestrian detection method based on multi-stage deformable decoupling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0016] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0017] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0018] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0019] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0020] In general, the multi-view pedestrian detection method based on multi-stage deformable decoupling provided by the present invention includes: acquiring multiple view images for a target scene; inputting the multiple view images into a trained pedestrian detection model to obtain the results of multi-view pedestrian detection. The pedestrian detection model performs the following steps: extracting pedestrian features from the multiple view images; decoupling task-related discriminative information from the pedestrian features to obtain effective features after the first decoupling; projecting the effective features after the first decoupling to a unified ground plane coordinate system through a homography transformation to generate multi-view aligned bird's-eye view features; calculating dynamic fusion weights based on the semantics of the multi-view features themselves, and then obtaining fused ground plane world features based on the dynamic fusion weights; decoupling scene-related features from the fused ground plane world features to obtain effective features after the second decoupling; generating a ground plane pedestrian position heat map based on the effective features after the second decoupling, and then obtaining multi-view pedestrian detection results.

[0021] Specifically, the provided multi-view pedestrian detection method based on multi-stage deformable decoupling includes the following steps:

[0022] Step S110 , constructing a pedestrian detection model for multi-view pedestrian detection.

[0023] A pedestrian detection model (or multi-view pedestrian detection model) can be constructed using multiple functional modules or various types of network structures. For example, the pedestrian detection model includes a feature extraction layer, a first deformable decoupling module, a bird's-eye view feature extraction module, a scene joint attention module, a second deformable decoupling module, and a decoder. The feature extraction layer is used to extract pedestrian features from the multiple view images; the first deformable decoupling module is used to decouple task-related discriminative information from the pedestrian features to obtain valid features after the first decoupling; the bird's-eye view feature extraction module is used to project the valid features after the first decoupling to a unified ground plane coordinate system through a homography transformation to generate multi-view aligned bird's-eye view features; the scene joint attention module is used to calculate dynamic fusion weights based on the semantics of the multi-view features themselves, and then obtain fused ground plane world features based on the dynamic fusion weights; the second deformable decoupling module is used to decouple scene-related features from the fused ground plane world features to obtain valid features after the second decoupling; and the decoder generates a ground plane pedestrian position heat map based on the valid features after the second decoupling to obtain multi-view pedestrian detection results.

[0024] In one embodiment, the feature extraction layer uses a ResNet-18 backbone network to extract pedestrian features from multiple views within the target scene. A deformable decoupling module (DDM) is embedded after the feature extraction layer to decouple discriminative information related to the multi-view detection task (such as pedestrian footsteps and spatial positions). The decoupled features are projected to a unified ground plane coordinate system via a homography transformation to generate multi-view aligned bird's-eye view (BEV) features. Furthermore, a scene joint attention module (SJA) is used to calculate dynamic fusion weights, resulting in dynamic weights based on the semantics of the multi-view features. These dynamic weights are scene-adaptive. Next, the fusion weights are used to calibrate the importance of the multi-view features, achieving feature fusion without prior constraints. A second-stage deformable decoupling module (DDM) is then constructed on the fused ground-level world features to further separate scene-level task-related features from irrelevant features, resulting in valid features after the second decoupling. For these valid features after the second decoupling, a decoder network is constructed using deformable convolution to adaptively adjust the receptive field to generate a ground-level pedestrian position heatmap for pedestrian detection.

[0025] It should be noted that the designed pedestrian detection model includes two important modules: a deformable decoupling module and a joint scene attention module. The deformable decoupling module can adaptively adjust the receptive field based on the geometric structure of the input features, accurately modeling local deformation patterns in the spatial dimension and learning high-level decoupled representations related to the detection task in the data latent space from the residual features. The joint scene attention module, without introducing any prior knowledge, can generate a reasonable and effective fusion weight map. This weight map can be completely based on the semantics of the image's own features and dynamically adjusted as the characteristics of the actual application scene change, thereby enhancing the universality of the pedestrian detection model.

[0026] In the following, the embodiments of the deformable decoupling module and the joint scene attention module will be highlighted.

[0027] 1) Deformable decoupling module

[0028] It should be noted that the functions and execution processes of the first deformable decoupling module and the second deformable decoupling module are basically the same, and will be collectively referred to as deformable decoupling modules hereinafter.

[0029] In one embodiment, the deformable disentanglement module includes two key steps: semantically aligning shallow features through the instance normalization layer (IN) to effectively reduce the feature distribution differences between different camera perspectives; and introducing a deformable disentanglement mechanism to achieve channel-space collaborative feature calibration through a dual-path attention mechanism.

[0030] First, the input features Perform instance normalization (IN) to reduce the differences in feature distribution between images captured by different camera perspectives:

[0031]

[0032] Here, μ(·) and σ(·) represent the mean and standard deviation calculated independently for each channel and sample / instance in the spatial dimension, that is, they represent the mean and standard deviation calculated for each independent channel of each instance feature in the spatial dimension, β is the translation parameter, and γ is the scaling parameter.

[0033] Then, the residual feature R is extracted, and the formula is defined as:

[0034]

[0035] The residual feature R retains the discriminative information related to the detection task in the original feature.

[0036] Furthermore, deformable convolution is used to generate spatial attention weights and combined with channel attention to enhance the multi-view detection task-related features in the residual information and achieve feature decoupling.

[0037] First, deformable convolution is used to capture spatial deformation-sensitive features and generate a spatial attention map, which is expressed as:

[0038] Offset=Conv offset (R)(3)

[0039] Rspatial=DeformConv(R,Offset)(4)

[0040] Among them, Conv offset For the offset prediction network, output offset k is the size of the deformable convolution kernel, for example, set to k = 3, DeformConv is a deformable convolution operation, generating a spatial attention map

[0041] Next, the channel attention vector is generated through the squeeze-excitation (SE) module

[0042] a=σ(W2·δ(W1·GAP(R)))(5)

[0043] Among them, σ(·) and δ(·) are Sigmoid function and Relu activation function respectively, GAP is global average pooling, and is the fully connected layer parameter, r is the dimensionality reduction ratio, for example, it is set to 16 to reduce the number of parameters.

[0044] Finally, feature decoupling is performed, spatial attention is integrated with channel attention, and the multi-view detection task-related features R are obtained. + Features R that are irrelevant to the multi-view detection task - , respectively expressed as:

[0045] R + (:,:,c)=a c ·Rspatial⊙R(:,:,c)(6)

[0046] R - (:,:,c)=(1-a c )·Rspatial⊙R(:,:,c)(7)

[0047] Among them, a c represents the attention weight of the c-th channel, R(:,:,c) is the semantic information of the c-th channel, and ⊙ represents element-by-element multiplication. This design emphasizes the complementarity of spatial and channel attention, ensuring that R+ +R - =R.

[0048] R + As a relevant feature of the multi-view detection task, it can be regarded as a subset of representations decoupled from the original feature F, so it needs to be restored to the feature space after instance normalization In order to perform feature reconstruction, it is expressed as:

[0049]

[0050] in, It is an effective representation obtained after reconstruction by the deformable decoupling module. Compared with the original feature F, it significantly enhances the discriminative information related to the multi-view detection task (such as pedestrian foot points and spatial positions). Correspondingly, the task-independent residual feature R - After reconstruction, an invalid representation is obtained, which concentrates interference information such as background texture and lighting noise.

[0051] In order to guide the model to mine high-level decoupled representations related to the multi-view detection task from the underlying feature space, in the subsequent model training, a dual-constraint loss function can be used to drive the dynamic separation of task-related features and irrelevant features through an adversarial optimization mechanism. For example, the loss is composed of L useful and L useless The composition is specifically expressed as:

[0052] L dual =L useful +L useless (9)

[0053] Among them, L dual represents the dual constraint loss, L useful represents the loss of task-related features, L useless represents the loss of task-irrelevant features.

[0054] The core idea of the above adversarial optimization mechanism is to decouple the task-related representation R + The detection performance should be significantly enhanced (i.e., the entropy of the predicted likelihood distribution is reduced), while the task-irrelevant feature R - Then the detection performance needs to be suppressed (i.e. the prediction entropy value increases).

[0055] Specifically, considering that each spatial position in the multi-view detection task corresponds to a class of potential pedestrian position probability distribution, it is necessary to quantify the prediction entropy of the feature space region by region. First, the ground plane features after multi-view aggregation are Implement proportional feature scaling. Compress its spatial dimension to The scaling factor is defined as:

[0056]

[0057] Among them, s h and s w is the scaling factor, H and W represent the height and width of the original feature, H ′ and W′ represent the height and width of the scaled feature.

[0058] This operation preserves the integrity of regional semantics through local feature aggregation, while significantly reducing computational complexity and providing an efficient and robust feature subspace for subsequent entropy calculation.

[0059] After scaling, features The local features of each spatial position (i, j) in Calculate its class probability distribution through the Softmax function And further calculate the normalized entropy value:

[0060]

[0061] in, is the normalized entropy value, represents the class probability distribution of the c-th channel, where C is the number of channels.

[0062] Finally, the average entropy is calculated as the constraint loss:

[0063]

[0064] Among them, L useful represents the average entropy of task-related features, L useless represents the average entropy of task-independent features, Represents the scaled feature The local features of each spatial position (i, j) in Represents task-irrelevant features after scaling The local features of each spatial position (i, j) in Represents the task-related features after scaling The local features of each spatial position (i, j). Softplus(·)=ln(1+exp(·)) is used as a smooth activation function to ensure that the loss value is non-negative. L useful By minimizing the task-related features The prediction entropy of L useless By maximizing task-independent features The prediction entropy of , suppresses noise interference. This adversarial optimization mechanism forces the model to explicitly separate the two types of features, namely R + Focusing on key semantic information such as pedestrian foot points and spatial positions, R -It encodes illumination differences and background redundant information.

[0065] 2) Joint scene attention module

[0066] In one embodiment, the scene joint attention module includes single-view channel importance modeling, cross-view semantic association modeling, and scene-level dynamic weight fusion.

[0067] First, given a multi-view feature set Where N is the number of views, C is the number of channels, H×W is the spatial resolution, and for each view F m Perform single-view channel importance modeling. For example, global average pooling (GAP) is used to compress the spatial dimension and two fully connected layers are used to learn channel weights:

[0068]

[0069] s m =F ex (z,W)=σ(W2·δ(W1·z m ))(15)

[0070] Among them, z m represents the global average pooling weight of the m-th view, s m represents the channel attention weight of the m-th view, F sq (F) represents the use of global average pooling function F for the multi-view feature set F sq , z represents the global average pooling feature set of multiple views. σ(·) is the Sigmoid function, δ(·) is the Relu activation function, and is a learnable parameter. r is the compression ratio, for example, set to r = 16. The weighted features are:

[0071]

[0072] Among them, F scale (s,F) represents the weighting function, s represents the channel attention weight set of multiple views. Compressed to a single channel through 1×1 convolution, the semantic saliency map is obtained Where N is the number of camera views.

[0073] Subsequently, the cross-view attention mechanism is used to model cross-view semantic associations:

[0074]

[0075] Among them, x m represents the global average pooling feature of the m-th view, F sq(S) represents the global average pooling of the multi-view semantic saliency map S, S represents the semantic saliency map, S m (i, j) represents the eigenvalue of the position (i, j) in the m-th view, [x1; x2; ...; x n ] represents the global average pooled feature set of multiple views, N represents the number of camera views, and c is the cross-view weight. To learn the correlation matrix, the Softmax function ensures the weight normalization of the aggregation across views.

[0076] Based on the cross-view weight c, the original features are adaptively adjusted to obtain the joint scene-level multi-view attention fusion weight

[0077]

[0078] Finally, the scene-level dynamic weight fusion is performed by using the joint scene-level multi-view attention fusion weights, and the fusion feature M is obtained by element-by-element weighted summation, which is expressed as:

[0079]

[0080] In summary, the scene joint attention module adopts hierarchical modeling, where weights are driven by feature semantics, and is dynamically adaptable without relying on camera calibration or layout priors. It also enhances the collaborative expression of local details and global relevance.

[0081] Step S120: Use the data set to train the constructed pedestrian detection model until the set loss function standard is met.

[0082] The dataset for training the pedestrian detection model can be a self-constructed dataset or a public dataset, such as MultiViewX and WildTrack.

[0083] Wildtrack is a real-world dataset that captures crowded pedestrian scenes in a 12×36 square meter outdoor area. The dataset consists of 400 frames recorded by seven synchronized and calibrated cameras. Each frame contains an average of approximately 20 pedestrians, with each location observed by 3.74 cameras. The ground is discretized into a 480×1440 grid, with each grid cell representing an area of 2.5 square centimeters. The original image resolution is 1080×1920, and the dataset uses a standard split of 360 training frames and 40 test frames.

[0084] MultiviewX is a synthetic dataset generated using the Unity engine with a high pedestrian density for controlled occlusion analysis. It covers an area of 16×25 square meters captured by 6 virtual cameras. Each frame contains about 40 pedestrians, with an average of 4.41 cameras observing each location. The ground plane is quantized into a 640×1000 grid (each cell is 2.5 square centimeters). Similar to Wildtrack, it also contains 400 frames of images (360 frames for training and 40 frames for testing) with a resolution of 1080×1920.

[0085] During the pedestrian detection model training phase, single-view detection loss and ground plane detection loss are used for training. At the same time, the dual-constraint loss function introduced in step S110 above is added to drive the dynamic separation of task-related features from irrelevant features through an adversarial optimization mechanism.

[0086] For example, the single-view loss is used for single-view pedestrian head and foot point detection and is defined as:

[0087] L img =L img,det +L img,off +L img,box (twenty one)

[0088] Among them, L img,det is the detection loss of a single view, and detection is performed by regressing the pedestrian’s foot occupancy map. img,off is the offset loss of a single view, which is used to compensate for the omitted parts during the downsampling process, L img,nox is the bounding box regression loss.

[0089] In one embodiment, the dual-constraint loss function L useful By minimizing the task-related features Prediction is similar to the calculation of likelihood entropy, L useless Then through task-independent features Calculation, defined as:

[0090] L dual 1 =L useful 1 +L useless 1 (twenty two)

[0091] L dual 2 =L useful 2 +L useless 2 (twenty three)

[0092] Among them, L dual1 is the one-stage dual constraint loss generated in the single-view detection stage, L dual 2 is the two-stage dual constraint loss generated in the ground plane detection stage, L useful 1 represents the task-related feature loss of one stage, L useless 1 represents the task-independent feature loss of one stage, L useful 2 represents the task-related feature loss of the second stage, L useless 2 represents the task-independent feature loss of the second stage.

[0093] In one embodiment, ground plane loss is detected and decoded by decoupled BEV fused plane features, which is defined as:

[0094] L world =L world,det +L world,off (twenty four)

[0095] Among them, L world is the ground plane loss, L world,det represents the detection loss of the ground plane, L world,off Represents the offset loss from the ground plane.

[0096] Finally, the overall loss function L for training the pedestrian detection model is all Expressed as:

[0097] L all =L img +L dual 1 +L dual 2 +L world (25)

[0098] During the training process of the pedestrian detection model, the optimized model parameters can be obtained by minimizing the overall loss function.

[0099] In step S130 , a plurality of view images are acquired for the target scene, and then a multi-view pedestrian detection result is obtained using the trained pedestrian detection model.

[0100] After completing the training of the pedestrian detection model, the trained parameter configuration can be retained for actual pedestrian detection. During the model application process, detection calculations are performed based on the fused features obtained from the deformable decoupling to obtain the multi-view pedestrian detection results.

[0101] To further verify the effectiveness of the present invention, the datasets MultiViewX and WildTrack were used for verification. Experimental results show that the combination of DDM and SJA has achieved consistent and significant improvements in multiple evaluation indicators (such as multi-target detection accuracy, multi-target detection precision, precision and recall). This synergy highlights the complementary advantages of task-related feature decomposition and semantic-driven fusion weights in complex multi-view scenes. In addition, DDM and SJA can be seamlessly integrated into existing detection pipelines, providing a flexible solution for enhancing robustness in complex multi-view scenes.

[0102] In summary, the multi-view pedestrian detection method based on multi-stage deformable decoupling and scene joint attention fusion weights provided by the present invention adaptively adjusts the receptive field according to the geometric structure of the pedestrian features extracted by the neural network, thereby accurately modeling the local deformation pattern in the spatial dimension, realizing feature decoupling through a dual-path attention mechanism, and separating the discriminant row features related to the detection task. At the same time, a hierarchical attention mechanism is used to jointly model the importance of single-view channels and cross-view semantic correlations, generating dynamically adaptive scene-level fusion weights. The weights are based on the semantics of the features themselves and get rid of the dependence on fixed camera layouts and artificial priors. In addition, as a single-stage framework without ROI proposals, the present invention has a lightweight design, uses fewer parameters, and maintains competitive performance. The model adopts an end-to-end training method, does not require staged training, greatly improves training efficiency, and achieves the best balance between stability and accuracy.

[0103] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0104] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0105] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0106] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, Python, and conventional procedural programming languages such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be personalized by utilizing the state information of the computer readable program instructions. The electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0107] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0111] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A multi-view pedestrian detection method based on multi-stage deformable decoupling, comprising: For the target scene, multiple view images are acquired; Inputting the multiple view images into a trained pedestrian detection model to obtain a multi-view pedestrian detection result; Among them, the pedestrian detection model includes a feature extraction layer, a first deformable decoupling module, a bird's-eye view feature extraction module, a scene joint attention module, a second deformable decoupling module and a decoder. The feature extraction layer is used to extract pedestrian features from the multiple view images; the first deformable decoupling module is used to decouple task-related discriminative information from the pedestrian features to obtain effective features after the first decoupling; the bird's-eye view feature extraction module is used to project the effective features after the first decoupling to a unified ground plane coordinate system through homography transformation to generate multi-view aligned bird's-eye view features; the scene joint attention module is used to calculate the dynamic fusion weight based on the semantics of the multi-view features themselves, and then obtain the fusion features based on the dynamic fusion weights; the second deformable decoupling module is used to decouple scene-related features from the fused features to obtain effective features after the second decoupling; the decoder generates a ground plane pedestrian position heat map based on the effective features after the second decoupling to obtain multi-view pedestrian detection results.

2. The method according to claim 1, characterized in that The first deformable decoupling module obtains the effective features after the first decoupling according to the following steps: Perform instance normalization on the input feature F to obtain the normalized feature Expressed as: Where μ(·) and σ(·) represent the mean and standard deviation of each independent channel of each instance feature, respectively, calculated in the spatial dimension, β is the translation parameter, and γ is the scaling parameter; Extract the residual feature R, expressed as: A spatial attention map is generated based on the residual feature R, which is expressed as: Offset=Conv offset (R) Rspatial=DeformConv(R,Offset) Among them, Conv offset is the offset prediction network, Offset is the offset output by the offset prediction network, DeformConv represents the deformable convolution operation, and Rspatial is the generated spatial attention map; Generate the channel attention vector according to the following formula a=σ(W2·δ(W1·GAP(R))) Among them, σ(·) is the Sigmoid function, δ(·) is the Relu activation function, GAP is the global average pooling, and is the corresponding fully connected layer parameter, r is the dimensionality reduction ratio, and C is the number of channels; Based on the spatial attention map Rspatial and the channel attention vector, the task-related feature R is obtained. + , expressed as: R + (:,:,c)=a c ·Rspatial⊙R(:,:,c) Among them, a c represents the attention weight of the c-th channel, R(:,:,c) represents the semantic information of the c-th channel, and ⊙ represents element-by-element multiplication; The task-related feature R + Restore to the feature space after instance normalization Expressed as: in, It is the effective representation of the first decoupling output by the first deformable decoupling module.

3. The method according to claim 2, characterized in that The scene joint attention module obtains the dynamic fusion weight according to the following steps: For a given multi-view feature set View F m Perform single-view channel importance modeling to obtain channel attention weight s m , expressed as: s m =F ex (z,W)=σ(W2 δ(W1 z m )) Where N is the number of views, C is the number of channels, H×W is the spatial resolution, σ(·) is the Sigmoid function, δ(·) is the ReLU activation function, and is a learnable parameter, z m represents the global average pooling weight of the m-th view, s m represents the channel attention weight of the m-th view, F sq (F) represents the use of the global average pooling function for the multi-view feature set F, and z represents the global average pooling feature set of the multi-view; For view F m , using the channel attention weight s m Perform weighting to obtain view F m The weighted features Expressed as: Among them, F scale (s,F) represents the weighting function, s represents the channel attention weight set of multiple views; The weighted features Compress to a single channel through 1×1 convolution to obtain a multi-view semantic saliency map For the semantic saliency map, a cross-view attention mechanism is used to model cross-view semantic associations and obtain the cross-view weight c, which is expressed as: Among them, x m represents the global average pooling feature of the m-th view, F sq (S) represents the global average pooling of the multi-view semantic saliency map S, S m (i, j) represents the eigenvalue of the position (i, j) in the m-th view, [x1; x2; ...; x n ] represents the global average pooling feature set of multiple views, is the learnable association matrix; Based on the cross-view weight c, the dynamic fusion weight of the joint scene-level multi-view is calculated Expressed as: in, represents the dynamic fusion weight.

4. The method according to claim 3, characterized in that The fusion feature is obtained by element-by-element weighted summation, which is expressed as: Wherein, M represents the fusion weight.

5. The method according to claim 4, characterized in that The overall loss function for training the pedestrian detection model is set to: L all =L img +L dual 1 +L dual 2 +L world in: L img =L img,det +L img,off +L img,box L dual 1 =L useful 1 +L useless 1 L dual 2 =L useful 2 +L useless 2 L world =L world,det +L world,off Among them, L all Represents the overall loss value, L img represents the single view loss, L dual 1 is the dual constraint loss generated in the single view detection stage, L dual 2 is the dual constraint loss generated in the ground plane detection stage, L img,det is the single-view detection loss, L img,off is the offset loss of a single view, L img,box is the bounding box regression loss, L useful 1 represents the task-related feature loss in the single-view detection stage, L useless 1 represents the task-independent feature loss in the single-view detection stage, L useful 2 represents the task-related feature loss in the ground plane detection stage, L useless 2 represents the task-independent feature loss in the ground plane detection stage, L world Represents the ground plane loss, L world,det represents the detection loss of the ground plane, L world,off Represents the offset loss from the ground plane.

6. The method according to claim 5, characterized in that For the single-view detection stage and the ground plane detection stage, the task-related feature loss and the task-irrelevant feature loss are calculated using average entropy and are uniformly expressed as: in, is the normalized entropy value, set to: in, represents the class probability distribution of the cth channel, L useful represents the average entropy of task-related features, L useless represents the average entropy of task-independent features, Softplus(·)=ln(1+exp(·)), Yes After scaling, H ′ and W′ represent the height and width of the scaled feature, is the scaled feature The local features of the spatial position (i, j) in Represents the task-related features after scaling The local features of each spatial position (i, j), Represents task-irrelevant features after scaling The local features of each spatial position (i, j) in .

7. The method according to claim 1, characterized in that The feature extraction layer is constructed based on the residual network.

8. The method according to claim 1, characterized in that The datasets used to train the pedestrian detection model include MultiViewX or WildTrack.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images