A multi-view audio assisted crowd counting method

Abstract

The application discloses a kind of crowd counting methods based on multi-directional audio auxiliary, the counting method includes the following steps: VCC module combines the hollow convolution of multiple size convolution kernel, extracts crowd feature from image;By adaptively encoding multi-level context information into the features generated by VCC module, visual features are extracted;ACC module carries out preliminary fusion and feature extraction to original multi-segment multidirectional audio;The multi-channel visual features of VCC module and the multidirectional audio features of ACC module are fused, and then a single convolution operation is performed to generate a density map.The framework proposed by the application solves the perspective problem, low-light scene and the influence of different positions on audio feature extraction, and it is the first time to use multi-segment multidirectional audio to assist crowd counting in the field of crowd counting.

Description

Technical Field

[0001] This invention relates to the field of crowd counting technology, and specifically to a crowd counting method based on multi-directional audio assistance. Background Technology

[0002] Existing auxiliary methods mainly use single audio segments to assist in counting dense crowds. However, the crowd counting accuracy is poor when faced with perspective distortion and low-light scenes. In addition, existing audio-assisted methods only use single audio segments and cannot provide spatial sound feature information from various directions within the space.

[0003] To address the issues of perspective distortion, poor crowd counting accuracy in low-light scenes, and the fact that existing audio-assisted methods only use local audio data and cannot provide spatial sound feature information from various directions within the space, this patent proposes a novel video audio-assisted crowd counting network framework. Summary of the Invention

[0004] The purpose of this invention is to provide a crowd counting method based on multi-directional audio assistance to address the shortcomings of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a crowd counting method based on multi-directional audio assistance, the counting method comprising the following steps:

[0006] S1: The VCC module combines dilated convolution with kernels of various sizes to extract crowd features from images;

[0007] S2: Visual features are extracted by adaptively encoding multi-level contextual information into the features generated by the VCC module;

[0008] S3: The ACC module performs preliminary fusion and feature extraction on the original multi-segment, multi-directional audio;

[0009] S4: Fuse the multi-channel visual features of the VCC module and the multi-directional audio features of the ACC module, and then perform a single convolution operation to generate a density map.

[0010] Preferably, step S1 further includes the following steps:

[0011] S1.1: Feed the initial image into the first 10 convolutional layers of the VGG16 network to create the initial feature map;

[0012] S1.2: The feature map contains detailed information about the initial image.

[0013] Preferably, the VCC module encodes the initial feature map using a three-channel encoding method. Each channel performs convolution operations of varying degrees to obtain image features with different levels of abstraction. The three-channel encoding includes channel 1, channel 2, and channel 3.

[0014] Channel 1 includes convolution operations with a kernel size of 128×128, max pooling operations, and average pooling operations;

[0015] Channel 2 contains two convolution operations with kernels of 256×256 and 128×128, as well as max pooling and average pooling operations;

[0016] Channel 3 contains two convolution operations with kernels of 512×512 and 256×256, as well as max pooling and average pooling operations.

[0017] Preferably, the VCC module performs secondary operations on the generated image features using a context-aware feature generation method, including the following steps:

[0018] S1.3: Perform average pooling on the initial image features;

[0019] S1.4: Perform operations using dilated convolutions with different dilation rates;

[0020] S1.5: Upsample the initial image features to the original feature size to form channel-aware features.

[0021] Preferably, the context-aware feature generation method enhances image features based on a multi-channel approach. Image I is sent to the first ten layers of a pre-trained VGG network, where a second convolution operation is performed, as shown in formula (1):

[0022]

[0023] ψ(·) represents the pooling operation, φ d＝i (·) represents the convolution operation, and d represents the dilation rate, which is a hyperparameter in the convolution kernel. When i is set to 1, it represents the traditional convolution operation. When i>2, the operation is a dilated convolution operation with an interval of i.

[0024] Preferably, in the context-aware feature generation method, for features with information of type i×i, where i is 256 or 512, dilated convolution with an expansion rate of 2 is performed, max pooling is used to reduce the computational complexity of the upper layer, and average pooling is performed to retain more context-aware information of the image. The process is shown in formula (2):

[0025] f FEMC =Avg(φ d＝3 (ψ max (φ d＝2 (fi×i (2)

[0026] Shallow features f FEMC The features are passed to the next layer for further convolution to obtain more abstract features, connecting the features between different channels to generate preliminary features f. θ The feature information obtained after all operations is concatenated, and a convolution operation is performed to retain shallow feature information. The original features are then passed to the next layer for further convolution.

[0027] Preferably, the context-aware feature generation method performs average pooling on the initial image features to preserve the context-aware information of the image, and uses dilated convolutions with different dilation rates to output feature maps. In f θ Based on this, the process is shown in formula (3):

[0028]

[0029] f represents the weight matrix of the j-th layer with an i×i convolution kernel. θ Represents the input feature map, b j Representing weights, three different dilated convolutions φ are used. d＝i (·), where i∈{2,3,4}, and sigmoid(·) represents the ReLU function;

[0030] Output feature f θ The information is sent to the context-aware block to extract multi-scale contextual information from the VGG features, as shown in formula (4):

[0031] F output =sigmoid(concat(upsample(φ)) d＝i ψ(f θ (4)

[0032] θ(·) represents the upsampling operation, and Sigmoid(·) represents the ReLU function for f θ Adaptive average pooling is performed by convolutional operations on image features using different dilation rates.

[0033] ′

[0034] The features obtained after the product operation are concatenated to generate image features f containing more feature information. θ .

[0035] Preferably, the VCC module uses the Speex algorithm for noise reduction processing. The speech noise reduction algorithm adopts a speech enhancement algorithm based on minimum mean square error short-time spectral amplitude estimation, which is expressed by equation (5):

[0036]

[0037] Where Γ(·) is the gamma function, M(a;b;c) is the hypergeometric function, and V k Depend on Obtain, ξ k For the prior signal-to-noise ratio, γ k This represents the posterior signal-to-noise ratio.

[0038] Preferably, the VCC module performs audio fusion processing by linearly fusing the input audio data, as expressed by equation (6):

[0039]

[0040] Alternatively, the average value can be taken after superposition, and the process can be represented by equation (7):

[0041]

[0042] Where m is the number of input audio streams, n is the number of samples in a frame, [i] is the i-th sample in a frame, [j] is the j-th audio stream, output [i] is the i-th sample in the mixed frame, and input [j][i] is the i-th sample in the current frame of the j-th input audio stream. The speech data is 16 bits.

[0043] Preferably, during the VCC module fusion process, an attenuation factor is used to attenuate the audio data, and the data process of mixing n audio segments is represented by equation (8):

[0044]

[0045] Where n is the number of samples in a frame, [i] is the i-th sample in a frame, [j] is the j-th audio stream, and Input[j][i] is the i-th sample in the current frame of the j-th input audio stream.

[0046] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0047] 1. This invention proposes the VCCNet framework, which solves the problems of perspective, low-light scenes, and the impact of different positions on audio feature extraction. At the same time, this is the first time in the field of crowd counting that multiple segments of multi-directional audio have been used to assist in crowd counting.

[0048] 2. To address the perspective problem, this invention proposes a Video Crowd Counting (VCC) module. The VCC module is designed with a multi-channel structure to generate a high-precision density map. To solve the problem that single audio segments have limited ability to represent spatial features and low-light scenes, an Audio Assisted Crowd Counting (ACC) module is proposed. This module mainly fuses audio from multiple directions and extracts their features. Experimental results based on existing benchmarks and PIANO (self-built dataset) show that this method improves accuracy by an average of 14.23% compared to traditional methods. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1 This is a schematic diagram of VCCNet, the framework for crowd counting using video and audio in this invention.

[0051] Figure 2 This is a schematic diagram of the VCC module of the present invention.

[0052] Figure 3 This invention converts ordinary waveforms into Log-Mel spectra.

[0053] Figure 4 This is a diagram illustrating a portion of the dataset from this invention.

[0054] Figure 5 The first row is the feature map generated by VCCNet in this invention; the second row is the ground-truth; the third row is the feature map generated by VCCNet. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] Please see Figure 1As shown in the figure, this embodiment describes a crowd counting method based on multi-directional audio assistance. The counting method includes the following steps: it comprises two sub-modules: a video crowd counting (VCC) module and an audio-assisted crowd counting (ACC) module. The VCC module is a novel convolutional neural network structure. This structure combines dilated convolutions with convolutional kernels of various sizes to better extract crowd features from images. By adaptively encoding multi-level contextual information into the generated features, more accurate visual features are extracted. In the ACC module, the original multi-segment multi-directional audio is initially fused and features are extracted, fusing the multi-channel visual features from the VCC module and the multi-directional audio features from the ACC module. A single convolution operation is then performed to generate the final density map.

[0058] The main principle of VCC is to feed the initial image into the first 10 convolutional layers of the VGG16 network to create an initial feature map. This feature map contains detailed information about the initial image. Within the VCC module, this application proposes a three-channel encoding method based on FEMC (Feature Enhancement Based on Multichannel Method) to encode the initial feature map. Each channel performs convolution operations of varying degrees to obtain image features with different levels of abstraction.

[0059] "Channel 1" contains a convolution operation with a kernel size of 128×128, a max pooling operation, and an average pooling operation; "Channel 2" contains two convolution operations with kernel sizes of 256×256 and 128×128, a max pooling operation, and an average pooling operation; "Channel 3" contains two convolution operations with kernel sizes of 512×512 and 256×256, a max pooling operation, and an average pooling operation.

[0060] In ablation studies, experiments further demonstrate that using three channels improves model performance compared to using only one or two channels. Concatenating image features enhances the initial image features. The CAFG (Context-Aware Feature Generation) method performs a secondary operation on the image features generated by the FEMC method. The initial image features are average pooled and then subjected to dilated convolutions with different dilation rates. They are then upsampled to their original feature size to form channel-aware features. These channel-aware features are also used to learn the weights of scale-aware features. These weights, along with the image features generated by the FEMC method, are fed into the backend network to obtain the final ground truth map.

[0061] The FEMC method primarily enhances image features using a multi-channel approach. To generate initial image features, image I is fed into the first ten layers of a pre-trained VGG network. To address the limitation that it only encodes the same receptive field across the entire image, this work uses multiple convolution operations to enhance the initial image features. To further extract more comprehensive information, a second convolution operation is performed. The calculation process is shown in Equation (1):

[0062]

[0063] ψ(·) represents the pooling operation, φ d＝i (·) represents a convolution operation. d represents the dilation rate, a hyperparameter in the convolution kernel. When i is set to 1, it represents a traditional convolution operation. When i exceeds 2, it means the operation is a dilated convolution with an interval of i.

[0064] For features of size i×i, where i is 256 or 512, dilated convolution with a dilation rate of 2 is performed to maintain resolution and still expand the receptive field. Max pooling is used to reduce the computational complexity of the upper layers. Different dilation rates are set to obtain multi-scale contextual information. Average pooling is performed to retain more context-aware information of the image. The entire process is shown in Equation (2):

[0065] f FEMC =Avg(φ d＝3 (ψ max (φ d＝2 (f i×i (2)

[0066] Shallow features f FEMC The data is then passed to the next layer for further convolution to obtain more abstract features. To obtain more comprehensive features, features from different channels are concatenated to generate preliminary features f. θ .

[0067] Traditional convolution operations preserve channel independence, but this independence limits the network's representational power. Furthermore, the feature information obtained after each operation is different. Therefore, concatenation operations are needed to connect the feature information obtained from all operations. Only one convolution operation is performed to preserve shallow feature information. The original features are then passed to the next layer for further convolution to obtain more abstract features to supplement the shallow features obtained in the first step. Features from different channels are concatenated to generate a more comprehensive initial feature f. θ And then send it to the next convolutional layer.

[0068] The stitching operation combines multiple features of the image with crowd information, resulting in a better representation of image features. This work connects feature maps along the channel dimension. Due to perspective distortion, each channel undergoes convolution operations using dilated convolution and regular convolution. These different convolution operations extract multi-scale information, thus enhancing the features.

[0069] The CAFG method mainly focuses on the image features f generated by the FEMC method. FEMC A second operation is performed. Average pooling is applied to the initial image features to preserve more context-aware information. Then, dilated convolutions with different dilation rates are used to output the feature map f. i θ f i θ In f θ Based on this, the entire process is shown in formula (3):

[0070]

[0071] f represents the weight matrix of the j-th layer with an i×i convolution kernel. θ This represents the input feature map. j This represents the weights. In addition, we used three different dilated convolutions φ. d＝i (·), where i∈{2,3,4}. sigmoid(·) represents the ReLU function.

[0072] Output feature f θ The data is sent to the context-aware block to extract multi-scale contextual information from the VGG features. The entire process is described by formula (4):

[0073] F output =sigmoid(concat(upsample(φ)) d＝i ψ(f θ (4)

[0074] θ(·) represents the upsampling operation. Sigmoid(·) represents the ReLU function.

[0075] To compress the spatial dimensionality, the mean of the corresponding dimension is extracted, discarding some useless features to a certain extent. θ Adaptive average pooling is performed. Since some information is lost during the computation, convolution operations with different dilation rates are applied to the image features. The resulting features are then concatenated to generate an image feature f′ containing more information. θ Because f θ It contains more shallow feature information of the image, while f′ θThe image contains a lot of deep feature information. In order to obtain visual feature information that includes more comprehensive contextual information, f... θ and f′ θ The two features are concatenated.

[0076] Example 2

[0077] Figure 2 The structure of the VCC module is shown. The VCC module contains two sub-modules: a multi-channel feature enhancement module (FEMC) and a context-aware feature generation module (CAFG).

[0078] The ACC module mainly performs preliminary fusion and feature extraction on the original multi-segment, multi-directional audio. For the acquired original audio, the Speex algorithm is used for simple noise reduction. The speech denoising algorithm adopts a speech enhancement algorithm based on minimum mean square error short-time spectral amplitude estimation, called STSA-MMSE. This algorithm takes advantage of the fact that the human ear is not sensitive to speech phase transformation. After estimating the short-time amplitude spectrum, the phase information of the noisy speech is used to synthesize enhanced speech. The observed speech signal can be represented by equation (5):

[0079]

[0080] Where Γ(·) is the gamma function, M(a;b;c) is the hypergeometric function, and V k Depend on Obtain, ξ k For the prior signal-to-noise ratio, γ k This represents the posterior signal-to-noise ratio.

[0081] Existing audiovisual assistance methods only use local single-segment audio data, which cannot accurately count the number of people and ignores the stereo features of sound. Single-segment audio does have some effectiveness in assisting crowd counting methods, but when the sources of audio information in different directions in a scene are significantly different, a single audio clip is difficult to contain audio feature information from all directions in the entire space, and the assistance effect needs to be improved. Therefore, multiple audio clips in the space are collected and fused. For audio fusion processing, the most commonly used algorithm is linear fusion of input audio data, and the fusion process can be represented by equation (6):

[0082]

[0083] Alternatively, the average of the sums can be taken, and the process can be represented by equation (7):

[0084]

[0085] Where m is the number of input audio streams, and n is the number of samples in a frame. [i] is the i-th sample in a frame. [j] is the j-th audio stream. Output [i] is the i-th sample in the mixed frame, and input [j][i] is the i-th sample in the current frame of the j-th input audio stream. Typical speech data is 16 bits. When multiple audio streams are linearly superimposed, they may overflow.

[0086] This application proposes an audio fusion method based on attenuation factors, which solves the overflow problem during the superposition process. During fusion, an attenuation factor is used to attenuate the audio data. To avoid noise caused by speech data overflow, the attenuation factor varies with the data. When overflow occurs, the attenuation factor is relatively small, ensuring that the overflowed audio data remains within a critical value after attenuation. When there is no overflow, the attenuation factor gradually increases to keep the data as smooth as possible. Different attenuation coefficients are used across the entire frame to ensure that the overall sound intensity does not decay too quickly and also to ensure a small degree of distortion. The process of mixing n audio segments can be represented by equation (8):

[0087]

[0088] Where n is the number of samples in a frame. [i] is the i-th sample in a frame. [j] is the j-th audio stream. Input[j][i] is the i-th sample in the current frame of the j-th input audio stream.

[0089] Figure 3 This section demonstrates the conversion of the original audio graph into a Log-Mel spectrogram for audio extraction. Convolutional Neural Networks (CNNs) have proven highly effective in image classification. Audio features derived from LMS have been widely applied to CNN-like sound neural models and have shown remarkable performance. To extract features from fused multi-directional audio, a ResNet-like CNN architecture is applied to compute the Log-Mel spectrogram (LMS).

[0090] The original fused audio signal is processed using a Fast Fourier Transform (FFT). After decomposition by the FFT, the time-domain signal becomes a superposition of different sine wave signals, and the frequencies of these sine waves are transformed from one signal to the frequency domain. Some signals are difficult to identify in the time domain, but their features become easy to see if they are transformed to the frequency domain. Typically, the FFT is used to convert the signal into an energy distribution in the frequency domain for observation. Different energy distributions represent the characteristics of different speech sounds. The FFT equation is expressed by equation (9):

[0091]

[0092] Where S i (p) is a frequency domain sample, s ih(n) is the time-domain sample, h(n) is the N-sample analysis window, and N is the size of the Fast Fourier Transform. The unit of frequency is Hertz (Hz), but the human ear does not have a linear perception relationship with the scale unit of Hertz. Therefore, it is converted to the Mel frequency scale. The Mel frequency scale can be represented by equation (10):

[0093]

[0094] Where mel (frequency) represents the mel frequency of a linear frequency. The Fourier transform spectrum is transformed into a Mel spectrum using a Mel filter bank. The Log-Mel spectrum is generated by logarithmic calculation of the Mel spectrum. The fused multi-directional audio features A feature It can be expressed by equation (11):

[0095] A feature =A cnn (A LMS (11)

[0096] in This indicates the extracted audio.

[0097] A linear fusion method is used to fuse the multi-channel visual features of the VCC module and the multi-directional audio features of the ACC module. After a single-layer convolution, the final density map is obtained. The general framework of the proposed method is as follows: Figure 1 As shown. The characteristics F of the fused audio and video signals. fused As shown in equation (12):

[0098] F fused =Conv(F output ⊙A feature (12)

[0099] F output It is the final video feature, A feature These are the final merged audio features. ⊙ indicates the merging operation.

[0100] The ACC module solves the problem that a single audio segment cannot provide complete audio feature information for a specific scenario, greatly improving the accuracy of dense crowd counting models.

[0101] VACCNet was implemented in a Windows 10 and PyTorch 1.4.0 experimental environment. The hardware environment consisted of an Intel E-21363, 3GHz, and a Quadro P5000. This work used the Adam optimizer with a learning rate of 1×e. -5 L 2 The loss function is defined by formula (13):

[0102]

[0103] Where m is the number of samples selected in one training iteration. The ground-truth density map f(x) is obtained. i The method is as follows: First, the center of each head in the dataset images is labeled, and the coordinates of each head center position are used as label information. Based on the head position, the size of the head at that position is predicted, and the coverage area of ​​the head is obtained. Then, a geometrically adaptive Gaussian kernel is used to transform this area into the probability that a head may appear in that area. Finally, a ground-truth density map is generated.

[0104] This work uses common standard evaluation metrics to evaluate the performance of each model: mean absolute error (MAE) and mean squared error (MSE), which are calculated as shown in equations (14)(15):

[0105]

[0106]

[0107] Where N represents the number of images, y i It represents the actual situation on the ground, while This indicates the predicted number of people. The smaller the MAE and MSE values, the better the experimental results.

[0108] This work compares state-of-the-art methods on three benchmark datasets: DISCO, ShanghaiTech, and multi-PledIrectionAssistancecouNtingnetwOrk (a self-built dataset, PIANO), which includes audiovisual and purely visual benchmark datasets.

[0109] Example 3

[0110] The ShanghaiTech dataset contains 1198 annotated images involving 330,165 people. It consists of two parts: part_A with 482 images and part_B with 716 images. These two parts are used for training and testing data, respectively. The DISCO dataset contains 1935 one-second audio clips and crowd images of 170,270 people.

[0111] To demonstrate the impact of multiple audio segments from multiple directions on crowd counting, this application establishes a new dataset named Multidirectional Auxiliary Counting Network Dataset (PIANO), such as... Figure 4As shown in the figure. To simultaneously capture video image sequences and record audio signals, we used five video recorders. During the acquisition process, we simulated the camera's perspective and recorded crowds at different times in several typical scenes. From these raw data, we selected 575 images and 2875 audio clips from four scenes, containing 32,510 people, to construct our proposed dataset. This dataset includes four scenes: Scene 1 collected 160 audio clips from five directions and 32 images containing 1,053 people; Scene 2 collected 1,505 audio clips from five directions and 301 images containing 16,937 people; Scene 3 collected 455 audio clips from five directions and 91 images containing 2,954 people; and Scene 4 collected 755 audio clips from five directions and 151 images containing 11,566 people.

[0112] Table 1 shows the comparison results of the VCC module with other methods on the ShanghaiTech dataset. In Part A of the ShanghaiTech dataset, the VCC module has MAE values ​​of 60 and 85 and MSE values ​​of 93 and 7. The VCC module achieves the lowest MAE value on Part A of the ShanghaiTech dataset. Since a smaller MAE value indicates better model performance, this means that the VCC module performs best compared to other methods on Part A of the ShanghaiTech dataset. This demonstrates that the proposed VCC module outperforms some existing traditional vision networks in addressing the perspective distortion problem.

[0113]

[0114]

[0115] Table 1

[0116] Table 2 shows the comparison results of VACCNet with other methods on the DISCO dataset, which contains both audio and video signals. VACCNet achieved the lowest MAE (14, 91) and MSE (30, 37) values ​​on the DISCO dataset, outperforming the latest method, AudioCSRNet. Furthermore, VACCNet's MAE and MSE values ​​are the lowest among the four algorithms, indicating that VACCNet performs best on the DISCO dataset. Experimental results demonstrate that this method outperforms traditional visual crowd counting methods and some existing audio-assisted crowd counting methods. This suggests that the introduction of audio information has a certain auxiliary effect on crowd counting methods.

[0117]

[0118] Table 2

[0119] Table 3 shows a comparison of the performance of VCCNet and AudioCANNet on PIANO when using single-segment and fused audio. In all four scenarios, VCCNet, using multi-directional fused audio assistance, has lower MAE and MSE values ​​than the case using only single-segment audio assistance. This indicates that using multi-directional fused audio is better for crowd counting than using single-segment audio assistance. Experimental results show that VCCNet, based on multi-segment multi-directional fused audio, outperforms AudioCANNet in both MAE and MSE values ​​across all four scenarios, demonstrating that VCCNet performs better than AudioCANNet in all four scenarios. Compared to existing audio-assisted networks, the VCCNet algorithm improves crowd counting performance. In scenarios 1-3, VCCNet outperforms AudioCANNet significantly, regardless of whether single or multi-segment audio assistance is used. In the best case, VCCNet's MAE value is 3.06 lower than AudioCANNet, which is a significant improvement. Due to the poor lighting conditions in scenario 4, VCCNet's MAE and MSE values ​​are lower, but the difference is not substantial. Therefore, VCCNet's performance needs improvement in harsh environments.

[0120]

[0121] Table 3

[0122] Experimental results show that VCCNet performs well, such as... Figure 5 As shown, on the DISCO dataset, this method generates density maps that are more accurate than other methods.

[0123] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0124] It should be understood that the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this application generally indicates an "or" relationship between the preceding and following related objects, but it may also indicate an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0125] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0126] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0132] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A crowd counting method based on multi-directional audio assistance, characterized in that: The counting method includes the following steps: S1: The VCC module combines dilated convolution with kernels of various sizes to extract crowd features from images; S2: Visual features are extracted by adaptively encoding multi-level contextual information into the features generated by the VCC module; S3: The ACC module performs preliminary fusion and feature extraction on the original multi-segment, multi-directional audio; S4: Fuse the multi-channel visual features of the VCC module and the multi-directional audio features of the ACC module, and then perform a single convolution operation to generate a density map; Step S1 further includes the following steps: S1.1: Feed the initial image into the first 10 convolutional layers of the VGG16 network to create the initial feature map; S1.2: The feature map contains detailed information about the initial image; The VCC module encodes the initial feature map using a three-channel encoding method. Each channel performs convolution operations of varying degrees to obtain image features with different levels of abstraction. The three-channel encoding includes channel 1, channel 2, and channel 3. Channel 1 contains a convolution kernel of Convolution operations, max pooling operations, and average pooling operations; Channel 2 contains two cores. and Convolution operations, max pooling operations, and average pooling operations; Channel 3 contains two cores. and Convolution operations, max pooling operations, and average pooling operations; The VCC module performs secondary operations on the generated image features using a context-aware feature generation method, including the following steps: S1.3: Perform average pooling on the initial image features; S1.4: Perform operations using dilated convolutions with different dilation rates; S1.5: Upsample the initial image features to the original feature size to form channel-aware features.

2. The crowd counting method based on multi-directional audio assistance according to claim 1, characterized in that: The context-aware feature generation method enhances image features based on a multi-channel approach. The first ten layers of the pre-trained VGG network are sent to each layer for a second convolution operation, as shown in formula (1): (1) Represents pooling operations, Represents convolution operation. The dilation rate represents the hyperparameter in the convolution kernel. Setting it to 1 indicates the traditional convolution operation. When 2, the interval of the operation is... The dilated convolution operation.

3. The crowd counting method based on multi-directional audio assistance according to claim 2, characterized in that: In the context-aware feature generation method, for feature information as ,and For features of 256 or 512, dilated convolution with a dilation rate of 2 is performed. Max pooling is used to reduce the computational complexity of the upper layers, and average pooling is performed to retain more context-aware information of the image. The process is shown in Equation (2): (2) Shallow features The features are passed to the next layer for further convolution to obtain more abstract features, connecting features from different channels to generate preliminary features. The feature information obtained after all operations is concatenated, and a convolution operation is performed to retain shallow feature information. The original features are then passed to the next layer for further convolution.

4. The crowd counting method based on multi-directional audio assistance according to claim 3, characterized in that: The context-aware feature generation method performs average pooling on the initial image features to preserve the context-aware information of the image, and uses dilated convolutions with different dilation rates to output feature maps. , exist Based on this, the process is shown in formula (3): (3) The representative convolution kernel is The The weight matrix of the layer, Represents the input feature map, Representing weights, three different types of dilated convolutions are used. ,in , express function; Output features The information is sent to the context-aware block to extract multi-scale contextual information from the VGG features, as shown in formula (4): (4) This represents an upsampling operation. Represents the ReLU function, for Adaptive average pooling is performed, and convolution operations with different dilation rates are applied to the image features. The resulting convolutional features are then concatenated to generate image features containing more information. .

5. A crowd counting method based on multi-directional audio assistance according to any one of claims 1-4, characterized in that: The VCC module uses the Speex algorithm for noise reduction. The speech noise reduction algorithm adopts a speech enhancement algorithm based on minimum mean square error short-time spectral amplitude estimation, which is expressed by equation (5): (5) in It is the gamma function. It is a hypergeometric function. Depend on get, The prior signal-to-noise ratio. This represents the posterior signal-to-noise ratio.

6. A crowd counting method based on multi-directional audio assistance according to claim 5, characterized in that: The VCC module performs audio fusion processing by linearly fusing the input audio data, as expressed by equation (6): (6) Alternatively, the average value can be taken after superposition, and the process can be represented by equation (7): (7) in It is the number of input audio streams. It is the number of samples in the frame. It is the first in a frame One sample, It is the first One audio stream, output It is the first in the mixed frame One sample, input It is the first The current frame of the input audio stream. One sample, the voice data is Bit.

7. A crowd counting method based on multi-directional audio assistance according to claim 6, characterized in that: During the VCC module fusion process, an attenuation factor is used to attenuate the audio data. The process of mixing n audio segments is represented by equation (8): (8) in It is the number of samples in a frame. It is the first in a frame One sample, It is the first One audio stream, It is the first The current frame of the input audio stream. One sample.

Images