Network training method, target counting method and electronic equipment

Abstract

The invention is applicable to the technical field of counting, and provides a network training method, a target counting method and electronic equipment, which are applied to a density map prediction network, and the method comprises the following steps: carrying out feature extraction processing on a source domain sample to obtain a source domain feature; random disturbance is carried out on the source domain characteristics to obtain disturbed source domain characteristics, and the random disturbance is used for disturbing each characteristic value of the source domain characteristics based on a random disturbance value; performing decoding processing on the source domain features to obtain a source domain density map, and performing decoding processing based on the disturbed source domain features to obtain a style density map; and updating parameters of the density map prediction network according to the difference between the source domain density map and the style density map to obtain a pre-trained density map prediction network. The accuracy of target counting can be improved.

Description

Technical Field

[0001] This application belongs to the field of counting technology, and in particular relates to network training methods, target counting methods, electronic devices and computer program products. Background Technology

[0002] Target counting refers to the technology of automatically counting the number of targets in video or images. Because target counting provides accurate and real-time target counts, it is widely used in fields such as crowd counting and cell counting. For example, in the field of crowd counting, crowd counting can provide reliable data for monitoring crowd flow, managing large events, and assessing public safety, thus contributing to improving the level of intelligence in public safety management.

[0003] Currently, when counting targets, the distribution of targets in an image is usually transformed into a density map using a model, and then the density map is analyzed to estimate the number of targets in the image. However, in practical applications, target clustering scenarios such as crowds are usually diverse, while the scenarios in the training dataset are limited, resulting in weak generalization ability of the model. When encountering new scenarios that differ greatly from the training dataset, frequent model adjustments or retraining are required. Summary of the Invention

[0004] This application provides a network training method, a target counting method, an electronic device, and a storage medium, which can improve the accuracy of target counting.

[0005] In a first aspect, embodiments of this application provide a network training method, including:

[0006] The source domain samples are processed to extract features, thus obtaining source domain features;

[0007] The source domain features are randomly perturbed to obtain the perturbed source domain features. The random perturbation is used to perturb each feature value of the source domain features based on a random perturbation value.

[0008] The source domain features are decoded to obtain a source domain density map, and the perturbed source domain features are decoded to obtain a style density map.

[0009] The parameters of the density map prediction network are updated based on the difference between the source domain density map and the style density map to obtain the pre-trained density map prediction network.

[0010] Secondly, embodiments of this application provide a target counting method, including:

[0011] Obtain the image to be counted;

[0012] The image to be counted is used as input to a pre-trained density map prediction network to obtain a density map output by the density map prediction network, which is trained according to the network training method described in the first aspect above.

[0013] The density map is used as input to a pre-trained counting network to obtain the counting result output by the counting network. The counting network is used to analyze the number of targets in the image to be counted based on the density map to obtain the counting result.

[0014] Thirdly, embodiments of this application provide a network training apparatus, comprising:

[0015] The feature extraction module is used to extract features from source domain samples to obtain source domain features;

[0016] The perturbation module is used to randomly perturb the source domain features to obtain the perturbed source domain features. The random perturbation is used to perturb each feature value of the source domain features based on a random perturbation value.

[0017] The decoding module is used to decode the source domain features to obtain a source domain density map, and to decode the perturbated source domain features to obtain a style density map.

[0018] An update module is used to update the parameters of the density map prediction network based on the difference between the source domain density map and the style density map, so as to obtain the pre-trained density map prediction network.

[0019] Fourthly, embodiments of this application provide a target counting device, comprising:

[0020] The image acquisition module is used to acquire the images to be counted.

[0021] The density map prediction module is used to take the image to be counted as input to a pre-trained density map prediction network to obtain a density map output by the density map prediction network, wherein the density map prediction network is trained according to the network training method described in the first aspect above.

[0022] A counting module is used to take the density map as input to a counting network and obtain the counting result output by the counting network. The counting network is used to analyze the number of targets in the image to be counted based on the density map and obtain the counting result.

[0023] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the network training method described in the first aspect or the target counting method described in the second aspect.

[0024] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the network training method described in the first aspect or the target counting method described in the second aspect.

[0025] In a seventh aspect, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the network training method described in the first aspect or the target counting method described in the second aspect.

[0026] The beneficial effects of the embodiments in this application compared with the prior art are:

[0027] In this embodiment, when training the density map prediction network, the source domain features corresponding to the source domain samples are randomly perturbed to obtain perturbed source domain features. Since the random perturbation of the source domain features affects each feature value based on a random perturbation value—that is, the potential domain transition is represented by the random perturbation value—training the density map prediction network based on the source domain features and the perturbed source domain features can significantly enhance the network's generalization ability to unknown scenarios. This allows the network to handle data that differs significantly from the source domain samples effectively, resulting in a more accurate density map. Furthermore, target counting is performed based on this more accurate density map, improving the accuracy of target counting. Moreover, no fine-tuning or retraining of the density map prediction network is required, improving both model performance and target counting efficiency, which is beneficial for practical applications. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0029] Figure 1 This is a schematic flowchart of a network training method provided in an embodiment of this application;

[0030] Figure 2 This is a flowchart illustrating a target counting method provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of the network training device provided in the embodiments of this application;

[0032] Figure 4 This is a schematic diagram of the structure of the target counting device provided in the embodiments of this application;

[0033] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0035] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0036] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0037] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0039] Example 1:

[0040] Figure 1 A schematic flowchart of a network training method provided in an embodiment of this application is shown, applied to a density map prediction network. The method is described in detail below:

[0041] Step S101: Perform feature extraction processing on the source domain samples to obtain source domain features.

[0042] Optionally, the source domain samples mentioned above may be based on public datasets or collected by users themselves, or they may be automatically generated by generative models such as generative adversarial networks. This application embodiment does not impose specific restrictions on the method of obtaining source domain samples.

[0043] Optionally, when performing feature extraction processing on source domain sample images, the density map prediction network can extract source domain features through methods such as histogram of oriented gradients, contour descriptors, or encoders.

[0044] As an example, density map prediction networks can use pre-trained encoders, such as general encoders, to extract features from source domain samples, resulting in highly accurate source domain features. This allows for subsequent perturbation and decoding based on these highly accurate source domain features. Furthermore, the pre-trained encoder can be adjusted during training to further improve its accuracy in extracting source domain sample images.

[0045] Step S102: Randomly perturb the source domain features to obtain the perturbed source domain features. The random perturbation is used to perturb each feature value of the source domain features based on random perturbation values.

[0046] The perturbation value used in the above-mentioned random perturbation can be a random value generated by a generator such as a pseudo-random number generator or an algorithm such as Lagrange interpolation, or it can be a random value generated based on the characteristics of the source domain. The embodiments of this application do not impose specific restrictions on the method of generating perturbation values.

[0047] In this embodiment, random perturbation values ​​are used to perturb each perturbation value in the source domain features. That is, the potential domain bias is represented by the uncertain perturbation. The perturbed source domain features are used as the features corresponding to the unknown target domain, so that the trained density map prediction network has good performance in the unknown target domain, which is beneficial to improving the accuracy of the predicted density map.

[0048] Step S103: Decode the above source domain features to obtain a source domain density map, and decode the above source domain features based on the perturbation to obtain a style density map.

[0049] The above decoding process is used to gradually restore high-level features into a high-resolution density map.

[0050] The density map described above is used to represent the distribution of targets (such as people or vehicles) in an image. The value of each element (i.e., pixel) in the density map can reflect the target density or the probability of its presence at that location.

[0051] As an example, density map prediction networks can use deconvolutional layers to decode source domain features and perturbed source domain features to obtain the corresponding density map.

[0052] As another example, density map prediction networks can use a decoder to decode both the source domain features and the perturbed source domain features to obtain the corresponding density map. For instance, a density map prediction network can use a transformer-based decoder to decode both the source domain features and the perturbed source domain features. The transformer allows the decoder to consider the information of the entire feature sequence when processing features at each location, which can generate high-resolution density maps effectively, thus improving the prediction performance of the density map prediction network.

[0053] Step S104: Update the parameters of the density map prediction network according to the difference between the source domain density map and the style density map to obtain the pre-trained density map prediction network.

[0054] In some embodiments, a loss value can be calculated based on the source domain density map, style density map, and true density map, and the parameters of the density map prediction network can be updated based on the loss value. Since the true density map is a density map that reflects the true distribution of targets in the source domain samples, calculating the loss value in conjunction with the true density map can constrain the accuracy and authenticity of the density map predicted by the density map prediction network.

[0055] Optionally, to ensure the accuracy of density map prediction, a pre-trained density map prediction network can be determined based on the density map prediction network that meets the training requirements when the density map prediction network meets the training requirements.

[0056] As an example, training can be stopped after the number of iterations of the density map prediction network reaches a threshold (e.g., 100), and then the pre-trained density map prediction network can be determined based on the density map prediction network obtained from the last iteration.

[0057] In this embodiment, the network parameters are updated based on the difference between the source domain density map corresponding to the source domain features and the style density map corresponding to the perturbed source domain features. This enables the density map prediction network to predict similar density maps for different versions of the same image, ensuring that the density map prediction network predicts consistent density maps for different versions of the input image, thereby improving the robustness and generalization ability of the density map prediction network.

[0058] In this embodiment, during training, the density map prediction network randomly perturbs the source domain features corresponding to the source domain samples, obtaining perturbed source domain features. Since the random perturbation affects each feature value of the source domain features based on random perturbation values—that is, using random perturbation values ​​to represent potential domain bias—training the density map prediction network based on the source domain features and the perturbed source domain features significantly enhances its generalization ability to unknown scenarios. This allows the density map prediction network to handle data that differs significantly from the source domain data effectively, resulting in a more accurate density map. Furthermore, target counting is performed based on this more accurate density map, improving the accuracy of target counting. Moreover, no fine-tuning or retraining of the density map prediction network is required, improving both model performance and target counting efficiency. Additionally, training the density map prediction network through the above steps is beneficial for practical applications.

[0059] In some embodiments, prior to step S102 described above, the method further includes:

[0060] A1. Sample the source domain features corresponding to K source domain samples in the same batch to obtain C basis vectors, where K and C are both greater than 1.

[0061] A2. Determine the sampled source domain features based on each of the aforementioned basis vectors.

[0062] Correspondingly, step S102 above includes:

[0063] The sampled source domain features are randomly perturbed to obtain the perturbed source domain features.

[0064] In model training, a large number of training images are usually divided into multiple batches for processing. Each batch includes multiple training images. In this embodiment, a batch may include K source domain samples, which are input into the density map prediction network for processing at once.

[0065] The basis vectors mentioned above refer to the fundamental vectors that constitute the feature space. Any vector in this feature space can be uniquely represented by a linear combination of a set of basis vectors. It can be understood that the sampled basis vectors can capture the common features (such as texture or color distribution) of each source domain sample in K source domain samples.

[0066] Optionally, when sampling the source domain features corresponding to the K source domain samples, the K source domain features can be sampled based on methods such as principal component analysis, independent component analysis, or sparse coding, without specific restrictions here.

[0067] Optionally, the number of basis vectors C can be determined based on the dimension of the source domain features. In this embodiment, the value of C is less than or equal to the dimension of the source domain features.

[0068] In this embodiment, before randomly perturbing the source domain features, C basis vectors are sampled from the source domain features corresponding to K source domain samples from the same batch. Since the sampling process typically filters out variations (i.e., noise) that contribute little to the overall features, retaining only the parts that best represent the source domain sample features, the sampled source domain features are determined based on these basis vectors. This results in purer source domain features, and by approximating the original source domain features with C basis vectors, effective data compression is achieved. This significantly reduces the computational complexity of network training while preserving important information in the source domain samples, thus accelerating training and prediction.

[0069] Correspondingly, when randomly perturbing the source domain features, the perturbation is performed on the sampled source domain features, which can reduce the amount of data processing during the random perturbation process and help to further improve the network training efficiency.

[0070] Optionally, when sampling each source domain feature and obtaining the sampled source domain features based on the combination of each basis vector, the focus (i.e., sampling target) in the sampling process can be determined according to the actual application scenario.

[0071] As an example, to obtain a density map prediction network with good generalization ability on image content, when sampling basis vectors from source domain features, the sampling target can be features related to the content of the source domain samples, such as contours, shapes, or spatial relationships. That is, it is equivalent to sampling basis vectors from the feature space of source domain features. These basis vectors focus on information such as the content and structure of the source domain samples. Furthermore, the sampled source domain features based on the combination of these basis vectors can better reflect the basic patterns of the source domain features and can better help the density map prediction model capture the core structural information of the target in the image. By learning the core structural information shared among source domain samples, the density map prediction network can better predict the density map of the image under different angle changes, thereby improving its performance on new data.

[0072] As another example, to obtain a density map prediction network with good generalization ability on image styles, when sampling basis vectors from source domain features, the sampling target can be the color distribution or texture of the source domain samples and the style-related features of the source domain samples. That is, it is equivalent to sampling basis vectors from the style space of the source domain features. These basis vectors focus on the style information of the source domain samples. Furthermore, the sampled source domain features based on the combination of these basis vectors can better reflect the style elements of the source domain features. This allows the density map prediction model to learn a wider range of style features better during the training process based on the sampled source domain features and the perturbed source domain features, thereby improving the performance of the density map prediction network on new data of different styles.

[0073] In some embodiments, the style space of the source domain features can be represented by the mean and variance of the K source domain features in the channel dimension. Then, C basis vectors are sampled from the style space, and these basis vectors are combined to obtain the sampled source domain features.

[0074] In some embodiments, the density map prediction network may include multiple pooling layers. These pooling layers can reduce the dimensionality of the input source domain features to accelerate computation and reduce the number of parameters. Since shallow networks typically capture more style information and deep networks capture more semantic information, and this embodiment requires capturing global basic style or pattern information from the source domain features, a sampling network can be set between the Nth and N+1th pooling layers of the deep density map prediction network. The value of N can be a small value determined by the number of pooling layers L (e.g., 10) in the density map prediction network; for example, N can be set to 0.2*L. This setting allows sampling of the source domain features containing more style information output from the shallow pooling layers of the density map prediction network, which helps improve the accuracy of the sampled source domain features.

[0075] In some embodiments, step A2 above includes:

[0076] The weights corresponding to each of the above basis vectors are obtained by sampling from the Dirichlet distribution function.

[0077] The sampled source domain features are determined based on each of the aforementioned basis vectors and their corresponding weights.

[0078] The Dirichlet distribution function mentioned above is a multivariate distribution function, often used to describe the probability distribution of multivariate random variables. It can assign different weights to each component in the multivariate variable, thereby achieving fine modeling of the data.

[0079] Since the Dirichlet distribution function can control the shape of the weight vector by adjusting its concentration parameter, the concentration parameter of the Dirichlet distribution can be set according to actual needs before sampling the weights corresponding to each basis vector from the Dirichlet distribution function, so as to generate the required weight vector by adjusting the concentration parameter.

[0080] As an example, the Dirichlet distribution function described above is uniformly distributed, meaning that each concentration parameter of the Dirichlet distribution function is equal. For example, the value of each concentration parameter can be set to 1 / C. Since the concentration parameters are equal, each weight value in the weight vector generated using the Dirichlet process is approximately equal. That is, the weights corresponding to each basis vector sampled from the Dirichlet distribution function are approximately equal. Therefore, based on each basis vector and its corresponding weight, uniformly distributed sampled source domain features can be obtained, making the obtained sampled source domain features more fair and accurate.

[0081] As another example, the Dirichlet distribution function mentioned above can be sparsely distributed, meaning that the concentration parameters can be different, so as to emphasize or suppress certain features with different weights, thereby controlling the overall representation of the source domain features after sampling and the subsequent density map prediction results.

[0082] In some embodiments, prior to step S102 described above, the method further includes:

[0083] The perturbation distribution function is determined based on the Gaussian distribution parameters of the characteristics of each source domain in the same batch.

[0084] For each of the aforementioned source domain features in the aforementioned batch, the perturbation value corresponding to the aforementioned source domain feature is randomly determined according to the aforementioned perturbation distribution function.

[0085] Correspondingly, the above-mentioned random perturbation of the source domain features to obtain the perturbed source domain features includes:

[0086] The source domain features are perturbed according to the corresponding perturbation values ​​to obtain the perturbed source domain features.

[0087] The parameters of the Gaussian distribution mentioned above include the mean and variance, which determine the shape and location of the Gaussian distribution.

[0088] In this embodiment, before perturbing the source domain features, the distribution function of the perturbation value can be determined based on the statistical distribution of the source domain features, thus obtaining the perturbation distribution function. Then, the perturbation value of the source domain features is randomly determined according to this perturbation distribution function. Since this perturbation distribution function is determined based on the true statistical distribution of the source domain features, using the perturbation value determined by this function to perturb the source domain features can control the randomness of the perturbation to a certain extent, avoiding excessive deviation of the perturbed source domain features, and improving the authenticity and reliability of the perturbed source domain features.

[0089] Since the brightness and color features of an image typically exhibit Gaussian distribution characteristics, and the Gaussian distribution has good stability, it can ensure that various operations on the image do not change the distribution characteristics of the data. Therefore, in this embodiment, the perturbation distribution function is determined based on the Gaussian distribution parameters of the source domain features, which helps the density map prediction network maintain consistency when processing source domain features and perturbed source domain features.

[0090] Specifically, the perturbation distribution function can be determined based on the Gaussian distribution of each source domain feature in the same batch. A more stable Gaussian distribution can be obtained through the source domain features in the batch, resulting in a more stable and reliable perturbation distribution function. Furthermore, the overall distribution of the source domain features in the same batch can better capture the common features and patterns in the batch of source domain features, which helps to generate more representative perturbations, rather than just based on the local characteristics of a single image. This allows the perturbation distribution function to cover a wider range of changes, thereby improving the adaptability of the density map prediction network to different changes.

[0091] As an example, the perturbation distribution function can be a Gaussian distribution function, where the Gaussian distribution parameters of the perturbation distribution function can be expressed in the following form:

[0092]

[0093] μ lup σ represents the mean of the distributed perturbation function. lup μ represents the standard deviation of the distributed perturbation function. un This represents the initial mean (e.g., 1), σ un This indicates the initial standard deviation (e.g., 0.01), f s μ(f) represents the source domain features corresponding to the same batch of source domain samples. s σ(f) represents the mean of the Gaussian distribution of the source domain features corresponding to the same batch of source domain samples (let's call it the source domain mean). s σ represents the standard deviation of the Gaussian distribution of the source domain features corresponding to the same batch of source domain samples (let's call it the source domain standard deviation). b (μ(f s)) represents the standard deviation of the Gaussian distribution of the source domain mean, σ b (σ(g s )) represents the standard deviation of the Gaussian distribution of the source domain standard deviation, and ○ represents the Hadamard product operation.

[0094] In this embodiment, the perturbation distribution function is determined based on the Gaussian parameters of each source domain sample in the same batch. Then, the random perturbation value of each source domain feature in the batch is determined based on the perturbation distribution function. That is, the perturbation value of the source domain feature is randomly determined based on the true distribution of the source domain feature. This allows the randomness of the perturbation to be controlled to a certain extent when perturbing the source domain feature, and ensures the authenticity of the perturbed source domain feature while realizing the potential domain transfer.

[0095] In some embodiments, the above steps involve decoding based on the perturbed source domain features to obtain a style density map, including:

[0096] Based on the aforementioned source domain features, the sampled source domain features, and the Gaussian distribution parameters corresponding to the perturbed source domain features, the target Gaussian distribution parameters are determined.

[0097] Based on the aforementioned target Gaussian distribution parameters, the perturbed source domain features are adaptively instantiated to obtain style features.

[0098] The style features described above are decoded in the above manner to obtain the style density map.

[0099] The aforementioned adaptive instance processing, also known as Adaptive Instance Normalization, is a type of instance normalization. Its core idea is to achieve real-time, arbitrary style transfer by aligning the mean and variance of content features with the mean and variance of style features.

[0100] To further improve the realism of the style density map, in the process of decoding the style density map based on the perturbated source domain features, the perturbated source domain features are first mixed with the source domain features to obtain style features with a true distribution containing the perturbation and source domain features. Then, the style density map is obtained by decoding based on the style features.

[0101] Since the sampled source domain features can better reflect the main features in the source domain, when mixing, the perturbed source domain features can be mixed with the source domain features and the sampled source domain features, so as to better obtain the true features of the source domain and improve the authenticity of the obtained style features.

[0102] In this process, when mixing the perturbed source domain features with the original source domain features and the sampled source domain features, a comprehensive target Gaussian distribution parameter can be calculated first based on the Gaussian distribution parameters of the original source domain features, the sampled source domain features, and the perturbed source domain features. Then, an adaptively instantiated parameter is generated based on this comprehensive target Gaussian distribution parameter. This adaptively instantiated parameter is then used to transform the perturbed source domain features; that is, the perturbed source domain features are dynamically adjusted by integrating the data distribution characteristics of the source domain and the unknown target domain, resulting in the adjusted source domain features, i.e., the style features. Finally, the style features are decoded to obtain the style density map. Through this process, while achieving the potential domain shift, the final style density map can be made closer to the true feature distribution of the source domain, improving the realism of the style density map.

[0103] As an example, the parameters of the target Gaussian distribution mentioned above can be expressed in the following form:

[0104] μ m =mean(μ) lup +μ gsee μ x )

[0105] σ m =mean(σ) lup +σ gsee , σ x )

[0106] μ m σ represents the mean of the parameters in the target Gaussian distribution. m The mean represents the standard deviation of the Gaussian distribution parameters, and mean() is the function for calculating the mean. gsee σ represents the mean of the Gaussian distribution of the sampled source domain features. gsee μ represents the standard deviation of the Gaussian distribution of the sampled source domain features. x σ represents the mean of the Gaussian distribution of the source domain characteristics. x The standard deviation of the Gaussian distribution representing the characteristics of the source domain.

[0107] In this embodiment, the target Gaussian distribution parameter is calculated based on the statistical distribution of the true features of the source domain and the statistical distribution after the latent domain shift. Since the differences between these statistical distributions can reflect the data shift or feature change between the source domain and the unknown target domain, that is, the target Gaussian distribution parameter can reflect the data shift or feature change between the source domain and the unknown target domain, the adaptive instantiation of the perturbated source domain features based on the target Gaussian parameter can make the obtained style features retain the latent domain shift increased during random perturbation, while the statistical distribution is closer to the statistical distribution of the source domain, ensuring the authenticity of the final generated style density map. Thus, while ensuring the generalization ability of the density map prediction network, the reliability of the density map prediction network can also be guaranteed.

[0108] In some embodiments, step S104 includes:

[0109] B1. Calculate the target loss value based on the preset loss function, the source domain density map, and the style density map. The preset loss function includes at least the statistical distribution consistency loss function, which is used to measure the difference between the statistical distributions of the source domain density map and the style density map.

[0110] B2. Update the parameters of the density map prediction network according to the target loss value to obtain the pre-trained density map prediction network.

[0111] Optionally, the statistical distribution consistency loss function can be a loss function based on KL divergence (Kullback-Leibler Divergence) or a loss function based on Wasserstein distance. This application does not impose specific restrictions on the specific manifestation of the statistical distribution consistency loss function.

[0112] In this embodiment, considering the diversity of data, the loss value is calculated based on the difference between the statistical distributions of the source domain density map and the style density map. The target loss value is then used to update the parameters of the density map prediction network, thereby limiting the range of the statistical distribution of the generated style density map and making its statistical distribution as close as possible to the statistical distribution of the source domain density map. Instead of directly calculating the loss value based on the difference between the source domain density map and the style density map, the density map prediction network can pay more attention to the overall distribution characteristics of the image, rather than just the differences of individual pixels. This allows it to better capture the essential features of the image and further improve the generalization ability and robustness of the density map prediction network.

[0113] In some embodiments, the preset loss function further includes a loss function based on Gaussian operation, the statistical distribution consistency loss function is a loss function based on JS divergence, and step B1 includes:

[0114] B11. Based on the above source domain density map, the above style density map, and the above true density map corresponding to the source domain samples, perform Gaussian operation to obtain the first loss value.

[0115] B12. Based on the above JS divergence, the above source domain density map, and the above style density map, the second loss value is calculated.

[0116] B13. Determine the target loss value based on the first loss value and the second loss value mentioned above.

[0117] The loss function based on Gaussian operations described above is designed based on the characteristics of the Gaussian distribution, aiming to optimize the performance of the density map prediction network by leveraging these characteristics. The Gaussian operation is a fast algorithm for calculating the sum of an arithmetic sequence, also known as the Gaussian algorithm. In the Gaussian algorithm, the first and last terms of the arithmetic sequence are added together, then multiplied by the number of terms and divided by 2 to obtain the sum of the sequence.

[0118] Optionally, the loss function based on Gaussian operations can be a mean squared error loss function, a negative log-likelihood loss function, or a maximum mean difference loss function, etc.

[0119] The loss function based on Jensen-Shannon Divergence (JSD) mentioned above refers to a loss function used to measure the difference between the probability distributions of the source domain density map and the style density map. JS divergence is defined as the average of the KL divergences of two probability distributions P and Q relative to their mean distribution M.

[0120] In this embodiment of the application, in order to improve the realism of the density map generated by the density map prediction network when processing unknown new data, during the training process, JS divergence is used to evaluate the posterior probability between the source domain density map and the style density map, and then a second loss value is calculated based on the posterior probability. This ensures that the posterior probability between the generated style density map and the source domain density map is as close as possible by minimizing the second loss value, thereby generating a more realistic density map that is more consistent with the source domain.

[0121] In other embodiments, a second loss value can be calculated based on the JS divergence, the source domain density map, the style density map, and the true density map to constrain the statistical distribution of the density map predicted by the density map prediction network to be consistent with the statistical distribution of the true density map.

[0122] As an example, the first loss value mentioned above can be expressed in the following form:

[0123]

[0124] Among them, L mseThis represents the first loss value, Gauss() represents the Gaussian operation, B represents the number of source domain samples in the same batch, and d i Let represent the predicted density map corresponding to the i-th source domain sample, including the source domain density map and the style density map, and gi represent the annotated true density map corresponding to the i-th source domain sample.

[0125] As an example, the second loss value mentioned above can be expressed in the following form:

[0126]

[0127] Among them, L ddc Let x represent the second loss value. s Represents the source domain density map, x^ s This represents the style density plot, KL() represents the KL (Kullback-Leibler) divergence, and p() represents the probability distribution. This represents the mean distribution of the probability distributions of the source domain density map and the style density map.

[0128] As an example, the target loss value mentioned above can be expressed in the following form:

[0129] L = L mse +L ddc

[0130] Where L represents the target loss value.

[0131] As another example, the target loss value mentioned above can be expressed in the following form:

[0132] L=λ1L mse +λ2L ddc

[0133] λ1 is the first weight corresponding to the first loss value, and λ2 is the second weight corresponding to the second loss value. Optionally, the sum of λ1 and λ2 is 1.

[0134] The values ​​of the first and second weights mentioned above can be fixed or dynamically determined according to actual needs.

[0135] As an example, the first and second weights can be set based on the training progress of the network predicted by the density map. For example, suppose the training process is divided into an early stage (e.g., the first 100 iterations), a middle stage (e.g., the 101st to the 200th iteration), and a late stage (e.g., the 201st iteration until the end) based on the required number of iterations (e.g., 300 iterations) for the network predicted by the density map. In the initial training phase, the first weight can be set to a larger value (e.g., 0.7), and the second weight to a smaller value (e.g., 0.3). This allows the density map prediction network to focus more on the features of the source domain and learn the feature distribution of the source domain better. In the intermediate training phase, the second weight can be gradually increased (e.g., from 0.3 to 0.55 over multiple iterations), while the first weight can be gradually decreased (e.g., from 0.7 to 0.45 over multiple iterations). By balancing the weights of content consistency loss and statistical distribution consistency loss, feature alignment between the source domain and the unknown target domain can be promoted. In the later training phase, the value of the second weight can be further increased (e.g., increased to 0.7), while the value of the first weight can be decreased (e.g., decreased to 0.3). This allows the density map prediction network to focus more on the features of the unknown target domain, thereby enhancing its generalization ability on unknown data.

[0136] As another example, the first and second weights can be dynamically adjusted based on the degree of difference between the statistical distributions of the source domain density map and the style density map. For instance, if the degree of difference is greater than a distribution difference threshold (e.g., 0.2), the second weight can be increased and the first weight decreased; if the degree of difference is less than or equal to the distribution difference threshold, the second weight can be decreased and the first weight increased.

[0137] In this embodiment, since the loss function based on Gaussian operation can consider the consistency of content between the source domain density map, style density map and the true density map, and the loss function based on JS divergence can consider the consistency of statistical distribution between the source domain density map and the style density map, determining the final target loss value based on the loss value calculated by these two loss functions can more comprehensively and accurately evaluate the difference between the source domain density map and the style density map. At the same time, it enables the density map prediction network to pay more attention to the statistical distribution and content of the image during the training process, helping the density map prediction network to better capture the essential features of the image and improve the accuracy and quality of the density map generated by the density map prediction network.

[0138] Corresponding to the network training method described in the above embodiments, Figure 2 A flowchart illustrating a target counting method provided in an embodiment of this application is shown below:

[0139] Step S201: Obtain the image to be counted.

[0140] It is understandable that the method of acquiring the image to be counted can be set according to the actual application scenario. For example, in a crowd counting scenario, the image to be counted can be an image captured by a surveillance camera; in a cell counting scenario, the image to be counted can be acquired by a dedicated microscope camera, or it can be acquired by connecting the camera to the microscope eyepiece.

[0141] Step S202: The image to be counted is used as the input to the pre-trained density map prediction network to obtain the density map output by the density map prediction network. The density map prediction network is trained according to the network training method described in the above embodiment.

[0142] In some embodiments, the pre-trained density map prediction network can be trained using the network training method described in the above embodiments based on source domain samples corresponding to the actual application scenario. This enables the pre-trained density map prediction network to better identify targets in the image to be counted, which is beneficial to further improve the accuracy of the obtained density map.

[0143] As an example, in a crowd counting scenario, the source domain samples for training the density map prediction network are images of people taken from the crowd. That is, the target is the human body in the image, so that the density map prediction network can focus on learning the human body features in the image, thereby being able to better identify the human body in the image and predict a crowd distribution density map with high accuracy.

[0144] Step S203: The density map is used as the input to the pre-trained counting network to obtain the counting result output by the counting network. The counting network is used to analyze the number of targets in the image to be counted based on the density map to obtain the counting result.

[0145] Optionally, the above-mentioned counting network can be constructed based on convolutional neural networks such as VGGNet or ResNet, or it can be constructed based on lightweight networks such as MobileNet, ShuffleNet or SqueezeNet. The embodiments of this application do not impose specific restrictions on the network structure of the counting network.

[0146] In this embodiment, during the training of the density map prediction network, the source domain features corresponding to the source domain samples are randomly perturbed. The random perturbation value represents the potential domain transfer. The density map prediction network is then trained based on the source domain features and the perturbed source domain features. Therefore, the pre-trained density map prediction network has good generalization ability. Even when the image to be counted differs significantly from the source domain image, the density map prediction network can still process it well, obtaining a density map with high accuracy. Furthermore, during counting, the counting network can perform target counting based on this high-accuracy density map, improving the accuracy of the counting results. Moreover, when counting images that differ significantly from the source domain image, there is no need to fine-tune or retrain the density map prediction network. This ensures the accuracy of target counting while improving the efficiency of target counting, which is beneficial for practical applications.

[0147] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0148] Example 2:

[0149] Corresponding to the network training method described in the above embodiments, Figure 3 A structural block diagram of the network training device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0150] Reference Figure 3 The device includes: a feature extraction module 31, a perturbation module 32, a decoding module 33, and an update module 34. Among them,

[0151] Feature extraction module 31 is used to perform feature extraction processing on source domain samples to obtain source domain features;

[0152] The perturbation module 32 is used to randomly perturb the source domain features to obtain the perturbed source domain features. The random perturbation is used to perturb each feature value of the source domain features based on a random perturbation value.

[0153] Decoding module 33 is used to decode the above source domain features to obtain a source domain density map, and to decode the above source domain features based on the perturbation to obtain a style density map.

[0154] The update module 34 is used to update the parameters of the density map prediction network according to the difference between the source domain density map and the style density map, so as to obtain the pre-trained density map prediction network.

[0155] In this embodiment, during training, the density map prediction network randomly perturbs the source domain features corresponding to the source domain samples, obtaining perturbed source domain features. Since the random perturbation affects each feature value of the source domain features based on random perturbation values—that is, using random perturbation values ​​to represent potential domain bias—training the density map prediction network based on the source domain features and the perturbed source domain features significantly enhances its generalization ability to unknown scenarios. This allows the density map prediction network to handle data that differs significantly from the source domain data effectively, resulting in a more accurate density map. Furthermore, target counting is performed based on this more accurate density map, improving the accuracy of target counting. Moreover, no fine-tuning or retraining of the density map prediction network is required, improving both model performance and target counting efficiency. Additionally, training the density map prediction network through the above steps is beneficial for practical applications.

[0156] Corresponding to the target counting method described in the above embodiments, Figure 4 A structural block diagram of the target counting device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0157] Reference Figure 4 The device includes: an image acquisition module 41, a density map prediction module 42, and a counting module 43. Among them,

[0158] Image acquisition module 41 is used to acquire the image to be counted.

[0159] The density map prediction module 42 is used to take the image to be counted as the input of the pre-trained density map prediction network to obtain the density map output by the density map prediction network, which is trained according to the network training method described in the above embodiment.

[0160] The counting module 43 is used to take the density map as input to the counting network and obtain the counting result output by the counting network. The counting network is used to analyze the number of targets in the image to be counted based on the density map and obtain the counting result.

[0161] In this embodiment, during the training of the density map prediction network, the source domain features corresponding to the source domain samples are randomly perturbed. The random perturbation value represents the potential domain transfer. The density map prediction network is then trained based on the source domain features and the perturbed source domain features. Therefore, the pre-trained density map prediction network has good generalization ability. Even when the image to be counted differs significantly from the source domain image, the density map prediction network can still process it well, obtaining a density map with high accuracy. Furthermore, during counting, the counting network can perform target counting based on this high-accuracy density map, improving the accuracy of the counting results. Moreover, when counting images that differ significantly from the source domain image, there is no need to fine-tune or retrain the density map prediction network. This ensures the accuracy of target counting while improving the efficiency of target counting, which is beneficial for practical applications.

[0162] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0163] Example 3:

[0164] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5 The diagram shows only one processor, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, which, when executed, performs the steps of any of the above method embodiments.

[0165] The electronic device 5 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This electronic device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0166] The processor 50 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0167] In some embodiments, the memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or memory of the electronic device 5. In other embodiments, the memory 51 may be an external storage device of the electronic device 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 5. Furthermore, the memory 51 may include both internal and external storage units of the electronic device 5. The memory 51 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0169] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0170] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0171] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0172] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A network training method, characterized in that, The method, applied to density map prediction networks, includes: The source domain samples are processed to extract features, thus obtaining source domain features; The source domain features are randomly perturbed to obtain the perturbed source domain features. The random perturbation is used to perturb each feature value of the source domain features based on a random perturbation value. The source domain features are decoded to obtain a source domain density map, and the perturbed source domain features are decoded to obtain a style density map. The parameters of the density map prediction network are updated based on the difference between the source domain density map and the style density map to obtain the pre-trained density map prediction network.

2. The network training method as described in claim 1, characterized in that, Before randomly perturbing the source domain features to obtain the perturbed source domain features, the method further includes: The source domain features corresponding to K source domain samples in the same batch are sampled to obtain C basis vectors, where K and C are both greater than 1; The sampled source domain features are determined based on each of the basis vectors; Correspondingly, the step of randomly perturbing the source domain features to obtain the perturbed source domain features includes: The sampled source domain features are randomly perturbed to obtain the perturbed source domain features.

3. The network training method as described in claim 2, characterized in that, The step of determining the sampled source domain features based on each of the basis vectors includes: The weights corresponding to each basis vector are obtained by sampling from the Dirichlet distribution function; The sampled source domain features are determined based on each of the basis vectors and the corresponding weights.

4. The network training method as described in claim 2, characterized in that, The decoding process based on the perturbated source domain features to obtain a style density map includes: Based on the source domain features, the sampled source domain features, and the Gaussian distribution parameters corresponding to the perturbed source domain features, the target Gaussian distribution parameters are determined. Based on the target Gaussian distribution parameters, the perturbed source domain features are adaptively instantiated to obtain style features; The style features are decoded to obtain the style density map.

5. The network training method as described in claim 1, characterized in that, The step of updating the parameters of the density map prediction network based on the difference between the source domain density map and the style density map to obtain the pre-trained density map prediction network includes: The target loss value is calculated based on the preset loss function, the source domain density map, and the style density map. The preset loss function includes at least the statistical distribution consistency loss function, which is used to measure the difference between the statistical distributions of the source domain density map and the style density map. The parameters of the density map prediction network are updated based on the target loss value to obtain the pre-trained density map prediction network.

6. The network training method as described in claim 5, characterized in that, The preset loss function also includes a Gaussian-based loss function, and the statistical distribution consistency loss function is a JS divergence-based loss function. The target loss value is calculated based on the preset loss function, the source domain density map, and the style density map, including: Gaussian operation is performed based on the source domain density map, the style density map, and the true density map corresponding to the source domain sample to obtain the first loss value; The second loss value is calculated based on the JS divergence, the source domain density map, and the style density map. The target loss value is determined based on the first loss value and the second loss value.

7. The network training method according to any one of claims 1 to 6, characterized in that, Before randomly perturbing the source domain features to obtain the perturbed source domain features, the method further includes: The perturbation distribution function is determined based on the Gaussian distribution parameters of each source domain feature in the same batch; For each source domain feature in the batch, the perturbation value corresponding to the source domain feature is randomly determined according to the perturbation distribution function; Correspondingly, the step of randomly perturbing the source domain features to obtain the perturbed source domain features includes: The source domain features are perturbed according to the corresponding perturbation value to obtain the perturbed source domain features.

8. A target counting method, characterized in that, include: Obtain the image to be counted; The image to be counted is used as input to a pre-trained density map prediction network to obtain a density map output by the density map prediction network, which is trained according to the network training method of any one of claims 1 to 7. The density map is used as input to a pre-trained counting network to obtain the counting result output by the counting network. The counting network is used to analyze the number of targets in the image to be counted based on the density map to obtain the counting result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7 or as described in claim 8.

10. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 7 or as described in claim 8.

Images