Domain adaptation model training method and related method, device, equipment and medium

Abstract

The application provides a domain adaptation model training method and related method, device, equipment and medium, by acquiring an image sample, performing feature extraction to obtain a first image feature; performing convolution processing on the first image feature to obtain a second image feature; performing prediction processing on the second image feature to obtain a predicted virtual class label, training a parameterizer according to the difference between the predicted virtual class label and a target virtual class label; performing information processing on the second image feature to obtain classification information; performing classification processing on the first image feature to obtain a domain adaptation image feature; performing prediction processing on the domain adaptation feature to obtain a predicted density map, training a backbone network according to the difference between the predicted density map and a real density map; returning to the feature extraction step for iteration until the difference converges, obtaining a trained domain adaptation model. Through a single-stage training strategy, the model can classify data of different domains, improving the accuracy of crowd counting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a domain adaptive model training method and related method, device, equipment and medium. BACKGROUND

[0002] Crowd counting has many real-life applications, such as surveillance, public safety, traffic monitoring, urban planning, etc. The method of crowd counting can also be applied to counting in other fields, such as counting cells or bacteria in microscopic images, ecological research on animals, counting vehicles in traffic control, etc. The current model with better performance mainly adopts a supervised learning method, but it faces the problem of time-consuming and laborious annotation data. Moreover, the model trained by the annotation data in a single scene is not applicable in other scenes, i.e. it does not perform well.

[0003] At present, a general model is learned by training with different data from multiple domains, so that the model performs well in multiple domains. However, the neural network model prefers to fit samples from the dominant domain rather than all domains in the learning process, i.e. the learned model only performs well on the data in the dominant domain, and performs poorly on the data in other domains, which is a biased learning problem. In related technologies, a multi-stage training scheme is used to solve the biased learning problem of the model, but this training method has the problems of complex method design and long training time. SUMMARY

[0004] The embodiments of the present application provide a domain adaptive model training method and related method, device, equipment and medium, which use a single-stage training strategy to guide the image features by using the trained parameterizer, so that the domain adaptive model can adaptively learn to distinguish different domain data, modulate the extracted image features, i.e. divide the image features of the same domain together in the feature mapping space, so that the crowd density map predictor can predict the density map according to the division of the domain, output a better predicted density map, improve the accuracy of crowd counting, use a single-stage training strategy to train the modulation domain adaptive network model, improve the performance of the model, solve the domain bias problem in multi-domain crowd counting, and reduce the model training time and the complexity of the model training method.

[0005] To solve the above problems, the embodiments of the present application provide the following technical solutions:

[0006] According to an aspect of the present disclosure, a domain adaptive model training method is provided, comprising:

[0007] obtaining image samples;

[0008] inputting the image samples into a backbone network for feature extraction to obtain first image features;

[0009] The first image feature is input into a parameterizer for convolution processing to obtain a second image feature.

[0010] The second image feature is input into a classifier for first prediction processing to obtain a first predicted virtual class label, and the parameterizer is first trained according to a first difference between the first predicted virtual class label and a target virtual class label.

[0011] The second image feature is processed by the parameterizer to obtain classification information.

[0012] The first image feature is classified according to the classification information to obtain a domain adaptive image feature.

[0013] The domain adaptive image feature is input into a density map predictor for second prediction processing to obtain a predicted density map, and a backbone network is second trained according to a second difference between the predicted density map and a real density map.

[0014] The step of inputting the image sample into the backbone network for feature extraction is returned to be executed for iterative training until the first difference and the second difference converge, thereby obtaining a trained domain adaptive model, wherein the domain adaptive model comprises the trained parameterizer and the trained backbone network.

[0015] According to an aspect of the present disclosure, a crowd counting method is provided, comprising:

[0016] An image sample is obtained.

[0017] The image sample is input into a domain adaptive model trained by the above domain adaptive model training method, and a predicted density map is output.

[0018] The predicted density map is summed to calculate a crowd number corresponding to the predicted density map.

[0019] According to an aspect of the present disclosure, a domain adaptive model training device is provided, comprising:

[0020] A first obtaining unit is configured to obtain an image sample.

[0021] A first generating unit is configured to input the image sample into a backbone network for feature extraction to obtain a first image feature.

[0022] A second generating unit is configured to input the first image feature into a parameterizer for convolution processing to obtain a second image feature.

[0023] a first training unit, configured to input the second image feature into a classifier to perform first prediction processing, to obtain a first predicted virtual class label, and to perform first training on the parameterizer according to a first difference between the first predicted virtual class label and a target virtual class label;

[0024] a third generating unit, configured to perform information processing on the second image feature by the parameterizer to obtain classification information;

[0025] an information classification unit, configured to classify the first image feature according to the classification information to obtain a domain adaptive image feature;

[0026] a second training unit, configured to input the domain adaptive image feature into a density map predictor to perform second prediction processing, to obtain a predicted density map, and to perform second training on a backbone network according to a second difference between the predicted density map and a real density map;

[0027] an iterative training unit, configured to return to perform the step of inputting the image sample into the backbone network to perform feature extraction, to perform iterative training until the first difference and the second difference converge, to obtain a trained domain adaptive model, and the domain adaptive model comprises the trained parameterizer and the trained backbone network.

[0028] In some embodiments, the first training unit is specifically configured to:

[0029] construct a first loss function according to the first difference between the first predicted virtual class label and the target virtual class label;

[0030] perform gradient backpropagation according to the first loss function, and perform the first training on the parameterizer.

[0031] In some embodiments, the information classification unit is specifically configured to:

[0032] perform classification processing on the classification information by a modulator to obtain first sub-classification information and second sub-classification information;

[0033] perform first classification guidance on the first image feature according to the first sub-classification information;

[0034] perform second classification guidance on the first image feature according to the second sub-classification information;

[0035] classify the first image feature according to the first classification guidance and the second classification guidance to obtain a domain adaptive image feature corresponding to the first image feature.

[0036] In some embodiments, the real density map is obtained by preprocessing an image sample corresponding to the domain adaptive image feature, and the second training unit is specifically configured to:

[0037] constructing a second loss function according to a second difference between the predicted density map and the real density map;

[0038] performing gradient back propagation according to the second loss function, and performing the second training on the backbone network.

[0039] In some embodiments, after the first training unit, the training apparatus further comprises:

[0040] a fourth generating unit configured to perform correction processing on the first predicted virtual class label to obtain a second predicted virtual class label;

[0041] a fifth generating unit configured to determine a first iteration number according to a preset training window size;

[0042] a first sub-iteration unit configured to return to performing first prediction processing on the second image feature by a classifier to obtain a first predicted virtual class label, and performing first training on the parameterizer according to a first difference between the first predicted virtual class label and a target virtual class label to complete a first iteration, until the number of first iterations reaches the first iteration number;

[0043] a sixth generating unit configured to take an average of a plurality of second predicted virtual class labels obtained in the first iteration to obtain a third predicted virtual class label, and update the target virtual class label according to the third predicted virtual class label;

[0044] a second sub-iteration unit configured to return to performing the first prediction processing on the second image feature by the classifier to obtain the first predicted virtual class label, and continue performing a second iteration on the step of performing first training on the parameterizer according to a first difference between the first predicted virtual class label and the updated target virtual class label, until a second iteration end condition is met, to obtain an optimized parameterizer.

[0045] In some embodiments, the fourth generating unit is specifically configured to:

[0046] obtain a first maximum vector value of a first part of the first predicted virtual class label;

[0047] obtain a second maximum vector value of a second part of the first predicted virtual class label;

[0048] sum the first maximum vector value and the second maximum vector value to obtain a first target vector value;

[0049] replace the second maximum vector value with the first target vector value to obtain a second predicted virtual class label.

[0050] In some embodiments, the sixth generating unit is specifically configured to:

[0051] obtaining a first vector value of the third predicted virtual class label representation;

[0052] obtaining a second vector value of an initial virtual class label representation, the initial virtual class label being a multi-dimensional vector converted from a first class label, the first class label being derived from manual labeling of an image sample;

[0053] multiplying the first vector value by a reliability to obtain a second target vector value, the reliability being a fixed value for measuring prediction effect of a classifier;

[0054] performing an operation on the second vector value and the reliability to obtain a third target vector value;

[0055] summing the second target vector value and the third target vector value to obtain a fourth predicted virtual class label, and updating the target virtual class label according to the fourth predicted virtual class label.

[0056] According to an aspect of the present disclosure, there is provided a domain adaptation model processing apparatus, comprising:

[0057] a second obtaining unit configured to obtain an image sample;

[0058] a first output unit configured to input the image sample into a domain adaptation model trained according to any one of the above domain adaptation model training methods, and output a predicted density map;

[0059] a calculation unit configured to perform summation calculation on the predicted density map to calculate a number of people corresponding to the predicted density map.

[0060] According to an aspect of the present disclosure, there is provided an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above domain adaptation model training method or people counting method when executing the computer program.

[0061] According to an aspect of the present disclosure, there is provided a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the above domain adaptation model training method or people counting method.

[0062] The embodiment of the present disclosure obtains an image sample; inputs the image sample into a backbone network for feature extraction to obtain first image features; inputs the first image features into a parameterizer for convolution processing to obtain second image features; inputs the second image features into a classifier for first prediction processing to obtain first predicted virtual class labels, and performs first training on the parameterizer according to a first difference between the first predicted virtual class labels and target virtual class labels; performs information processing on the second image features by the parameterizer to obtain classification information; performs classification processing on the first image features according to the classification information to obtain domain self-adaptive image features; inputs the domain self-adaptive image features into a density map predictor for second prediction processing to obtain a predicted density map, and performs second training on the backbone network according to a second difference between the predicted density map and a real density map; returns to perform the step of inputting the image sample into the backbone network for feature extraction, and iteratively trains until the first difference and the second difference converge, to obtain a trained domain self-adaptive model, wherein the domain self-adaptive model comprises the trained parameterizer and the trained backbone network.

[0063] By using a single-stage training strategy, the input image sample is trained, the trained parameterizer is used to guide the image features, the model is adaptively learned to distinguish the differences between different domain data, the image features of the same domain are classified together in the feature mapping space, and the crowd density map predictor can predict the density map of the image features in each classification according to the classification of the domain. The single-stage training strategy can better improve the performance of the model, reduce the training time of the model and the complexity of the model training method, and also enable the crowd density map predictor to output a better predicted density map and improve the accuracy of crowd counting.

[0064] Other features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present disclosure. The objects and other advantages of the present disclosure can be achieved and obtained by the structures specifically pointed out in the specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0065] The accompanying drawings are intended to provide a further understanding of the technical solutions of the present disclosure and constitute a part of the specification, and are used together with the embodiments of the present disclosure to explain the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure.

[0066] Figure 1 FIG. 1 is a flow diagram of a domain self-adaptive model training method in an embodiment of the present disclosure;

[0067] Figure 2 FIG. 2 is a model principle structure diagram of the domain self-adaptive model training method in an embodiment of the present disclosure;

[0068] Figure 3is a flowchart of a domain adaptation model training method in an embodiment of the present application;

[0069] Figure 4 is a flowchart of a domain adaptation model training method in an embodiment of the present application;

[0070] Figure 5 is a flowchart of a correction of a predicted virtual classification label algorithm 1 in a domain adaptation model training method in an embodiment of the present application;

[0071] Figure 6 is a flowchart of a domain adaptation model training method in an embodiment of the present application;

[0072] Figure 7 is a flowchart of a target virtual classification label algorithm 2 in a domain adaptation model training method in an embodiment of the present application;

[0073] Figure 8 is a flowchart of a domain adaptation model training method in an embodiment of the present application;

[0074] Figure 9 is a flowchart of a domain adaptation model training method in an embodiment of the present application;

[0075] Figure 10 is a flowchart of a domain adaptation model training method in an embodiment of the present application;

[0076] Figure 11 is a classification visualization diagram of a domain adaptation model training method in an embodiment of the present application;

[0077] Figure 12 is a test comparison data diagram of a domain adaptation model training method in an embodiment of the present application;

[0078] Figure 13 is a structural principle diagram of a domain adaptation model training device in an embodiment of the present application;

[0079] Figure 14 is a structural principle diagram of a domain adaptation model processing device in an embodiment of the present application;

[0080] Figure 15 is a structural principle diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to enable personnel in the technical field to better understand the schemes of the present application, the technical schemes in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. According to the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0082] It can be understood that in the specific embodiments of the present application, relevant data such as file data corresponding to files and the like are involved, and when the above embodiments of the present application are applied to specific products or technologies, the permission or consent of the user needs to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of countries and regions.

[0083] Before the embodiments of the present disclosure are further described in detail, the terms and phrases involved in the embodiments of the present disclosure are explained, and the terms and phrases involved in the embodiments of the present disclosure are applicable to the following explanations:

[0084] Artificial Intelligence (AI): is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain target results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields. With the research and progress of artificial intelligence technology, artificial intelligence technology is applied in many fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous vehicles, drones, robots, smart medical care, smart customer service and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0085] Domain: A domain is usually defined as a combination of two key elements:

[0086] Feature Space: This is a collection of attributes or features that describe the data. It defines the properties and dimensions of the input data. In neural networks, input data is typically represented as vectors or tensors, and the dimensions of these vectors constitute the feature space.

[0087] Marginal Probability Distribution: This describes the distribution of data in the feature space, i.e., the data distribution.

[0088] Density Map: A density map is a visualization tool used to represent the density of data points in a space. The main purpose of a density map is to show the density distribution of data points through color, grayscale, or other visual attributes, helping observers understand the distribution patterns and trends of data. Density maps have a wide range of applications in various fields, including but not limited to Geographic Information Systems (GIS), social science research, machine learning, data analysis, and image processing. In GIS, density maps can be used to display population density, crime hotspots, etc.; in social science research, they can be used to study human behavior and social phenomena; in machine learning, density maps can be used for anomaly detection; in data analysis, they can be used to find distribution patterns in data sets, such as crowd counting and other applications.

[0089] Density Map Predictor: A technique in computer vision and machine learning used to estimate the distribution of object density in an image. This concept is commonly used in tasks such as crowd counting, vehicle counting, and object detection, where it is necessary to understand the density information of object distribution in the image rather than just the number of objects. The advantage of a density map predictor is that it provides more information than a simple object count, as it can reflect the spatial distribution of objects in the image. This makes it very useful in many applications, especially in situations where real-time monitoring and analysis of crowd or object density is required. The performance of a density map predictor is usually affected by the choice of feature extraction and density map generation model, as well as the quality of the training data.

[0090] In recent years, crowd counting has been applied to more and more different fields, such as surveillance, public safety, traffic monitoring, urban planning, etc. The methods of crowd counting have also been applied in various fields, such as counting cells or bacteria in microscopic images, ecological research on animals, counting vehicles in traffic control, etc. The models currently used with good performance are mostly trained by supervised learning methods, which require time and effort to label data, and the models trained by labeled data in a single scene do not perform well in other scenes.

[0091] Currently, a general model can be learned by training using different data of multiple domains, so that the model performs well in multiple domains. In order to train the general model, the related technology adopts a multi-stage training method, that is, the model is trained by combining domain adaptation and transfer learning technology. This multi-stage training method can effectively solve the biased learning problem of the neural network model, but this multi-stage training method has new problems such as complex design and long training time.

[0092] To solve the above problems, the present application provides a domain adaptation model training method and related method, device, equipment and medium. Through a single-stage training strategy, the trained parameter is used to guide the image features, so that the model can adaptively learn to distinguish different domain data. In the feature mapping space, the image features of the same domain are divided together, so that the crowd density map predictor can predict the density map of the image features according to the domain division, output a better prediction density map, improve the accuracy of crowd counting, use a single-stage training strategy to train the modulation domain adaptation network model, improve the performance of the model, solve the domain bias problem in multi-domain crowd counting, and reduce the model training time and the complexity of the model training method.

[0093] The application content will be further described through the detailed description of the embodiments of the present application combined with the drawings.

[0094] Please refer to Figure 1 , Figure 1 is a flowchart of a domain adaptation model training method provided by an embodiment of the present application. The domain adaptation model training method comprises:

[0095] In step S110, the image samples are obtained.

[0096] In an embodiment of the present application, before the domain adaptation model training starts, image data related to the crowd needs to be collected. This may involve downloading images from the Internet, taking pictures with a camera, or obtaining images from existing data sets. Ensure that the data set is large enough and representative so that the model can learn various situations. At the same time, the image samples need to be labeled to specify the class or domain information of each image sample, which can be done manually or using labeling tools. Repetitive, incomplete or damaged images need to be deleted to ensure that the model will not be affected by these individual special image samples. The data set composed of image samples is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the performance of the final model.

[0097] Further, the image samples used in the embodiments of the present application can be SHA, SHB, QNRF, NWPU data sets, or other data sets related to crowd counting tasks. These data sets can be two or more. Before the model obtains the image samples, the data set where the image samples are located is read. When reading the data set, the class or domain information of each image sample is labeled by manual labeling or using a labeling tool, which represents the true class label of the image sample.

[0098] It should be noted that the model is batch processing of image samples, and the processing of one batch of image samples is performed in parallel, and the same processing steps are performed on each image sample.

[0099] In step S120, the image sample is input into the backbone network for feature extraction to obtain a first image feature.

[0100] In an embodiment of the present application, please refer to Figure 2 , Figure 2 is a network architecture diagram of a domain adaptation model provided by an embodiment of the present application. After receiving an image sample, the domain adaptation network model inputs the image sample into a backbone network (Backbone: HRNet) to perform feature extraction on the image sample. The specific process of feature extraction usually includes a series of convolution layers, activation functions, pooling layers and batch normalization modules. The convolution layer is used to extract features from the original image. The activation function is used to perform nonlinear processing on the features, which helps the backbone network to learn nonlinear features and increase the expression ability of the backbone network. The batch normalization module standardizes the output of each convolution layer, which helps to alleviate the gradient problem. Finally, the feature map of the image sample, i.e. the first image feature, is output, which contains the abstract features extracted by the backbone network from the input image sample.

[0101] It should be noted that in combination with subsequent steps, as shown in Figure 2 , the first image feature has two information flow directions. One information flow direction is input into the domain guided power batch normalization parameterizer, and the other information flow direction is output to the feature mapping space. The two information flow directions are performed synchronously and there is no sequence.

[0102] Further, the first image feature output by the backbone network exists in the form of a feature matrix in the feature mapping space. The feature mapping space stores the corresponding first image features generated by all image samples in the process of batch processing of the model. All first image features in the feature mapping space are disordered and belong to domain-unidentifiable image features.

[0103] In step S130, the first image feature is input into the parameterizer for convolution processing to obtain a second image feature.​

[0104] In an embodiment of the present application, referring to Figure 2 , the domain-guided instance batch normalization parameterizer in the domain adaptation model receives the first image features output by the backbone network, and performs multiple convolution operations on the first image features in the domain-guided instance batch normalization parameterizer, including a convolutional layer (Conv), an adaptive average pooling layer (Adapative Avg Pool), a batch normalization module (Batch Normalization, BN), an activation function layer (Rectified Linear Unit, ReLu), and a fully connected layer (Fully Connected Layer, FC). After the convolution operation, the second image features are obtained.

[0105] It should be noted that, in combination with subsequent steps, as shown in Figure 2 , the second image features have two information flow directions, one of which is input into the domain classifier, and the other of which continues to flow to the FC layer. The two information flows are performed synchronously, and there is no sequence.

[0106] In step S140, the second image features are input into the classifier for first prediction processing to obtain a first predicted virtual class label, and the parameterizer is first trained according to a first difference between the first predicted virtual class label and the target virtual class label.

[0107] In an embodiment of the present application, referring to Figure 2 , after the domain classifier receives the second image features, the domain classifier performs first prediction processing on the second image features. The prediction processing is a predicted virtual class label vector generated after the second image features pass through the FC layer. After the predicted virtual class label vector is processed by a Sigmoid activation function, a first predicted virtual class label is obtained. Then, the domain-guided instance batch normalization parameterizer is first trained according to a first difference between the first predicted virtual class label and the target virtual class label.

[0108] The domain-guided instance batch normalization parameterizer is first trained by the constraint of the first difference, and the parameters of the domain-guided instance batch normalization parameterizer are optimized so that the domain-guided instance batch normalization parameterizer can better capture the difference between the explicit data sets as a latent domain-specific feature information classification space (domain separable latent space), and divide the data distribution of the domain overlap (category overlap) so that the domain-guided instance batch normalization parameterizer can output better classification information, realize the classification of the domain category of the image features extracted by the backbone network, obtain domain-classifiable image features, and enable the subsequent crowd density map predictor to perform more accurate crowd prediction based on the classified domain category information, and improve the accuracy of crowd counting.

[0109] It should be noted that the domain classifier is used to learn the information of the domain, which is used for subsequent feature learning of the guided model. In order to accurately modulate the domain-specific information, a domain classifier is introduced to learn the domain separable latent space (i.e. the domain-guided instance batch normalization parameterizer). The data is labeled with the corresponding domain class label according to the domain (data set) it comes from, and the virtual class is generated and gradually optimized in the training process according to the subsequent algorithm 1 and algorithm 2, and the template virtual class with virtual class and real class is obtained, which is used as the learning target of the domain classifier. Through this training method, the domain classifier learns to distinguish the domain knowledge of the input image, so that the domain-guided instance batch normalization parameterizer outputs the guidance information of the domain to the IsBN instance batch normalization module, modulates the image feature information (first image feature) transmitted in the backbone network, and refines the feature information according to different domain knowledge. Finally, the feature with domain knowledge discrimination information is input into the "crowd density map predictor", generates a density map for different domains, and improves the accuracy of crowd counting.

[0110] Further, the real class target in the domain classifier is defined as follows:

[0111]

[0112] The definition of the virtual class target extended on this basis is as follows:

[0113]

[0114] Where M is the number of data sets, m is the value of the real class label, which can be exemplified as follows: four public data sets in the field of crowd counting are used, and different domain class labels are labeled for different data sets to become the real class label of the domain. The value range of the class label here is [0, 1, 2, 3], and the value of m is [0, 1, 2, 3].

[0115] In some embodiments, please refer to Figure 3 as shown, Figure 3is a flowchart about the subsequent step of step S140, after step S140, further comprising:

[0116] Step S310, the first predicted virtual class label is corrected to obtain a second predicted virtual class label;

[0117] Step S320, the first iteration number is determined according to the preset training window size;

[0118] Step S330, the second image feature is input into the classifier for first prediction processing to obtain a first predicted virtual class label, and the first iteration is completed on the parameterizer according to the first difference between the first predicted virtual class label and the target virtual class label until the number of first iterations reaches the first iteration number;

[0119] Step S340, the average value of the plurality of second predicted virtual class labels obtained in the first iteration is obtained to obtain a third predicted virtual class label, and the target virtual class label is updated according to the third predicted virtual class label;

[0120] Step S350, the first prediction processing is performed again according to the second image feature by the classifier to obtain the first predicted virtual class label, and the step of first training the parameterizer according to the first difference between the first predicted virtual class label and the updated target virtual class label is continued for second iteration until the end condition of the second iteration is met, and the optimized parameterizer is obtained.

[0121] In an embodiment of the present application, step S310 corrects the first predicted virtual class label to obtain a second predicted virtual class label, and the weight of the virtual class can be increased through correction processing. Please refer to Figure 2 The target virtual class label located in the virtual class library is calculated through the second predicted virtual class label, the update of the target virtual class label can give the domain classifier a more accurate label to guide the domain classifier to output more accurate first predicted virtual class label, and thus the accuracy of crowd counting is improved.

[0122] It should be noted that the target virtual class label is stored in a space, and the virtual class library is only a substitute name and does not exist in the actual sense of virtual class library. The target virtual class label can be updated through first training, and each new target virtual class label will be used to guide the domain classifier to predict the output of the first predicted virtual class label in the next round of prediction.

[0123] In some embodiments, please refer to Figure 4 The first predicted virtual class label is corrected to obtain a second predicted virtual class label. Figure 4 is a flowchart about the detailed step of step S310, and step S310 further comprises:

[0124] Step S410, obtaining a first maximum vector value of a first part in the first predicted virtual class label;

[0125] Step S420, obtaining a second maximum vector value of a second part in the first predicted virtual class label;

[0126] Step S430, summing the first maximum vector value and the second maximum vector value to obtain a first target vector value;

[0127] Step S440, replacing the second maximum vector value with the first target vector value to obtain a second predicted virtual class label.

[0128] The maximum vector value of the real class label part in the first predicted virtual class label is obtained, the maximum vector value of the virtual class label part in the first predicted virtual class label is obtained, the maximum vector values of the two parts are accumulated and summed to obtain a target vector value, and the maximum vector value of the virtual class label part is replaced with the target vector value to obtain a second predicted virtual class label whose maximum vector value of the virtual class label part in the first predicted virtual class label is updated. The first predicted virtual class label contains multi-dimensional vectors, and each vector value represents a probability value of the first image feature being predicted as a class represented by the vector.

[0129] In an embodiment of the present application, the first predicted virtual class label is represented by a ten-dimensional vector, and the dimension of the multi-dimensional vector is calculated according to the number of domain classes. Each vector represents a different class label, which is divided into a real class label (the number of domain classes) and a virtual class label (two-by-two combination of domain classes). The value of each vector represents a probability value of the first image feature being predicted as the class label represented by the vector. The addition operation on the first predicted virtual class label can increase the probability value of the virtual class label vector, so that the model focuses on training the class label represented by the virtual class label vector, thereby learning the knowledge of the cross-domain (overlapping domain) image feature, facilitating subsequent domain classification, and generating better domain adaptive image features.

[0130] It should be noted that, please refer to Figure 2As shown, the specific numerical values are exemplified: the first predicted virtual class label vector is [0.6, 0.3, 0.01, 0.01, 0.04, 0.01, 0.02, 0.01, 0.02], and the corresponding classes are [first class, second class, third class, fourth class, one-two cross class, one-three cross class, one-four cross class, two-three cross class, two-four cross class], then 0.6 represents the probability of the domain classifier dividing the image feature into the first class is 0.6, 0.3 represents the probability of the domain classifier dividing the image feature into the second class is 0.3, and the remaining class probability values are small and can be ignored. As can be seen, when the domain classifier predicts the image feature, it predicts two classes, i.e. the image feature has a probability of 0.6 of being the first class and a probability of 0.3 of being the second class, so the image feature can be regarded as cross-domain data and needs to be stored in the virtual class label one-two cross class, so the vector value 0.6 of the first class is the maximum vector value of the real class label part of the first predicted virtual class label, and the vector value 0.04 of the one-two cross class is the maximum vector value of the virtual class label part of the first predicted virtual class label. Then, by adding the probability of the first class to the one-two cross class, the second modified virtual class label [0.6, 0.3, 0.01, 0.01, 0.64, …] is obtained, so that the model can focus on the learning of the one-two cross class feature and perform domain classification.

[0131] Please refer to Figure 5 As shown, Figure 5 The algorithm 1 flow chart for generating the second predicted virtual class label is as follows:

[0132] (1) Loop through the data set, and assign the class label value of the data set to t;

[0133] (2) If the class label value of the class label of the data set to which the current image belongs is s, and s≠t (i.e. s and t are not the same class label), then the virtual class label value of t class predicted as s class = the virtual class label value of t class predicted as s class + the predicted class label value of s class;

[0134] (3) Continue the loop until all class labels of the data set are looped through;

[0135] (4) The loop ends, and the modified virtual class label value is obtained.

[0136] In an embodiment of the present application, step S320 is to determine the number of first iterations according to the size of the preset training window. The first iteration specifically refers to that the domain classifier will perform multiple first prediction processes on the second image feature, and the first predicted virtual class label obtained each time will be compared with the target virtual class label for first difference comparison, and the instance batch normalization parameterizer of the domain guide will be first trained.

[0137] In an embodiment of the present application, step S330 is to complete one first iteration by returning to step S140 until the number of completed first iterations reaches the first iteration number determined in step 320, and then the first iteration is ended and the subsequent steps are continued. The multiple rounds of iteration prediction training of the preset training window size is to obtain more robust prediction results, so that the generated parameters are more reliable.

[0138] In an embodiment of the present application, step S340 is to average the multiple second prediction virtual class labels obtained in the first iteration. Specifically, the first prediction virtual class label is generated in each round of prediction in the first iteration, and the first prediction virtual class label is optimized by the domain-guided instance batch normalized parameterizer through the first loss function under the guidance of the target virtual class label. The first prediction virtual class label generated in the synchronous first iteration is corrected by algorithm 1 as shown in Figure 5 to obtain multiple second prediction virtual class labels. In order to obtain more robust second prediction virtual class labels, the multiple second prediction virtual class labels are averaged to obtain a more robust third prediction virtual class label, and then the target virtual class label is updated according to the third prediction virtual class label. The target virtual class label is updated to better guide the domain classifier to make prediction.

[0139] In some embodiments, referring to Figure 6 , the flowchart of a detailed sub-step of step S340 is shown in Figure 6 , which includes:

[0140] Step S610: obtaining a first vector value represented by the third prediction virtual class label;

[0141] Step S620: obtaining a second vector value represented by an initial virtual class label, the initial virtual class label being a multi-dimensional vector converted from a first class label, the first class label being derived from manual annotation of an image sample;

[0142] Step S630: multiplying the first vector value by a reliability to obtain a second target vector value, the reliability being a fixed value for measuring the prediction effect of a classifier;

[0143] Step S640: performing an operation on the second vector value and the reliability to obtain a third target vector value;

[0144] Step S650: summing the second target vector value and the third target vector value to obtain a fourth prediction virtual class label, and updating the target virtual class label according to the fourth prediction virtual class label.

[0145] In an embodiment of the present application, the first type of label is the real label, and the real label is converted to obtain an initial virtual label in the form of a multi-dimensional vector. Please refer to Figure 2 , for example: the real label can be expressed in the form of a vector as [0, 1, 0, 0], which represents that the image feature is divided into the second type of image feature during manual labeling, and the initial virtual label can be expressed in the form of a vector as [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], that is, the real label vector is directly converted into a 10-dimensional initial virtual label vector.

[0146] wherein the first vector value is the vector form of the third predicted virtual label, the second vector value is the vector form of the initial virtual label, the first vector value is multiplied by a fixed reliability to obtain a second target vector value, and the reliability is a constant for measuring the prediction effect of the domain classifier. Then, the second vector value and the reliability are operated to obtain a third target vector value, the second target vector value and the third target vector value are summed to obtain a fourth predicted virtual label, and the target virtual label is updated according to the fourth predicted virtual label.

[0147] Regarding Figure 6 the specific calculation process of the steps shown in Figure 7 , Figure 7 is a flowchart of an algorithm 2 for generating a target virtual label, and the specific steps are as follows:

[0148] (1) Obtain the parameters of the iteration round e, the starting combination round S, and the training window size T;

[0149] (2) Determine whether the current round is less than the preset round. If it is less than the preset round, the reliability a will not be enabled, which is equivalent to setting a to 0. Please refer to Figure 2 When a does not work, the target virtual label obtained by combining the third predicted virtual label and the initial virtual label each time is the initial virtual label;

[0150] (3) When the preset round is exceeded, the following steps are performed:

[0151] (4) Define the window index k = (current round - preset round + 1) / window size, and k represents the kth time of updating the target virtual label;

[0152] (5) Define the current kth window wink = {e | e ∈ [(window index - 1) * window size + preset round, window index * window size + preset round]}, that is, the range of rounds calculated according to the current window index all belong to the kth window;

[0153] (6) End determination;

[0154] (7) Set the confidence ac to 0;

[0155] (8) Loop for target round belongs to the kth window wink:

[0156] (9) Take out the corrected second prediction virtual class generated by algorithm 1,

[0157] (10) Accumulate the second prediction virtual class into the confidence ac;

[0158] (11) End the loop (that is, end when the target round exceeds the round range of the current window, and the effect is to accumulate all virtual class labels of all rounds in the round range of the window);

[0159] (12) According to the window size, the confidence ac is averaged (equivalent to obtaining a reliable third prediction virtual class after averaging);

[0160] (13) Based on the reliability a, the third prediction virtual class is combined with the initial virtual class for calculation;

[0161] (14) The fourth virtual class after the combined calculation is processed by the Softmax activation function to obtain the target virtual class, which is input to the virtual class library for updating.

[0162] It should be noted that the preset round in algorithm 2 is generally 200 rounds, that is, before 200 rounds, the effect of reliability a is ignored, the purpose is to let the model learn the first 200 rounds of training with reliable real class labels as target virtual class labels, when the model training number of image features exceeds 200 rounds, it is considered that the model can correctly identify which domain category the image feature belongs to, at this time, the reliability a is valued, and the vector value of the virtual cross class is accumulated and weighted, so that the model focuses on training the features of the cross class. Finally, after meeting the convergence of the first loss function, the model has learned the knowledge of the domain category (including cross class and non-cross class) represented by the image feature.

[0163] In some embodiments, referring to Figure 8 , Figure 8 is a flowchart of the refinement steps of step S140, including:

[0164] Step S810, constructing a first loss function according to the first difference between the first prediction virtual class and the target virtual class;

[0165] Step S820, gradient back propagation according to the first loss function, and first training of the parameterizer.

[0166] In an embodiment of the present application, a first difference is obtained by comparing the first predicted virtual category with the target virtual category, a first loss function is constructed according to the first difference, the comparison is performed by the first loss function, and then the result after the comparison according to the first loss function is used for gradient back propagation to perform first training on the parameterizer. The parameters of the parameterizer are optimized in the manner of gradient back propagation, so that the parameterizer can better perform a series of convolution processing to generate a second image feature with better effect, so as to obtain more accurate classification information by performing information processing according to the second image feature, and then modulate and classify the first image feature in the backbone network according to the classification information to obtain a domain-classifiable image feature, which is conducive to outputting a more accurate crowd prediction density map by the crowd density map predictor according to the domain-classifiable image feature, thereby improving the accuracy of crowd counting.

[0167] It should be noted that the first difference is a comparison between the multi-dimensional vector value represented by the first predicted virtual category and the multi-dimensional vector value represented by the target virtual category. In the present embodiment, the first loss function is specifically a mean square error (MSE) function, which is defined as follows:

[0168]

[0169] where N is the total number of image samples for training, and V represents the total number of categories (including virtual categories). When the number of domains is M, it is defined as follows:

[0170]

[0171] where φ(x i ) is the output of the domain-guided instance batch normalization parameterizer (see Figure 2 ), w c is the cth column vector of the weight matrix w of the domain classifier, that is, the center vector of the cth category. v i,c is the cth item in the virtual category of the ith sample, that is, the probability value of predicting the sample as the cth category.

[0172] It should be noted that the classification represents a real class and a virtual class, wherein the real class is obtained when reading the data set, specifically, a domain class annotated by a person for each image sample, and is an initial classification division. The virtual class is generated in the process of the first training. The domain-guided instance batch normalization parameterizer gradually learns the domain knowledge in the process of the first training. For the domain data that exists in the cross and overlap between the domains, specifically, the first predicted virtual class generated by multiple times of predicting the second image feature in the process of the first training, if the first predicted virtual class label is indefinite between two domain classes, that is, there is a probability value on the two real class label vectors of the first predicted virtual class label, when the image feature of the image sample is predicted by the domain classifier, the image feature of the image sample is predicted as two different classes, that is, the class label of each dynamic prediction is indefinite between two domain classes, a virtual class label is redefined to store the cross and overlap domain data (image sample) of the two domain classes.

[0173] In step S150, the second image feature is information-processed by the parameterizer to obtain classification information.

[0174] In an embodiment of the present application, referring to Figure 2 After the domain-guided instance batch normalization parameterizer outputs the second image feature, the second image feature flows to the FC layer for further information processing to obtain classification information. The classification information is a classification information vector with a size of 1024. The classification information vector flows to the instance batch normalization module IsBN (modulator) and is split into two sub-classification information vectors with a size of 512, which are used for subsequent classification of the first image feature. Through accurate modulation of the image feature (first image feature) transmitted in the backbone network under the guidance of unbiased domain-specific guide information, the classification of the extracted image feature is realized, which facilitates the subsequent crowd density map predictor to perform more accurate crowd prediction according to the classified features, and improves the accuracy of crowd counting.

[0175] In step S160, the first image feature is classified according to the classification information to obtain a domain-adaptive image feature.

[0176] In an embodiment of the present application, the classification information is obtained by processing the second image feature by the IsBN. The classification of the first image feature according to the classification information is to apply the classification information output by the IsBN to the first image feature in the feature mapping space, so that the first image feature originally without domain discrimination is changed in feature information through displacement and mapping after classification processing by the classification information, and the first image feature without domain discrimination is classified, so that the subsequent crowd density map predictor can perform more accurate crowd prediction according to the classified features, and the accuracy of crowd counting is improved.

[0177] In some embodiments, please refer to Figure 9 as shown, Figure 9 is a flowchart of the step S160 refinement step, including:

[0178] Step S910, the classification information is classified by the modulator to obtain the first sub-classification information and the second sub-classification information;

[0179] Step S920, according to the first sub-classification information, the first image feature is guided for the first classification;

[0180] Step S930, according to the second sub-classification information, the first image feature is guided for the second classification;

[0181] Step S940, according to the first classification guide and the second classification guide, the first image feature is classified to obtain the domain adaptive image feature corresponding to the first image feature.

[0182] In an embodiment of the present application, the instance batch normalization module IsBN can also be regarded as a modulator IsBN, which mainly modulates the domain non-discriminative image feature in the feature mapping space to obtain the domain classifiable image feature, please continue to refer to Figure 2 , the modulator IsBN classifies the classification information output by the instance batch normalization parameterer guided by the domain, and outputs the first sub-classification information Scale and the second sub-classification information Shift, wherein the classification information is a classification information vector with a size of 1024, the instance batch normalization module IsBN divides the 1024 vector into two parts to obtain Scale and Shift, which are defined as follows:

[0183]

[0184] The input image sample x is obtained after batch normalization Its definition is as follows:

[0185]

[0186] Wherein, μ is the mean of batch data, σ is the variance of batch data, Scale refers to γ in the formula, and Shift refers to β in the formula. The image features in the feature mapping space are multiplied by the Scale information, and then the image features in the feature mapping space are added by the Shift information, to obtain a classified image feature, that is, a domain classifiable image feature, or a domain adaptive image feature. Through the modulator IsBN, the mixed distribution of multiple data sets can be well handled. Under the action of the modulator IsBN, the feature mapping space in the backbone network is more discriminative, so that the mixed distribution of multiple domains can be well captured and learned, the feature information of the image is refined according to different domain knowledge, and finally a feature input "crowd density map predictor" with domain knowledge discriminative information is generated. The density map is generated for different domains, which can improve the accuracy of crowd counting.

[0187] It should be noted that the modulator IsBN is actually also a BN module, only the Alpha and Beta parameters in the BN are replaced by the Shift and Scale of the domain separable information, that is, the IsBN is obtained. Because the Alpha and Beta in the BN are processed by using the fixed values provided by themselves for different pictures. Now the IsBN has corresponding Shift and Scale for different pictures, and the Alpha and Beta are replaced to realize the domain classification of the image features in the backbone network, and obtain the domain classifiable image features.

[0188] In step S170, the domain adaptive image features are input into the density map predictor for second prediction processing, to obtain a predicted density map, and the backbone network is second trained according to a second difference between the predicted density map and the real density map.

[0189] Specifically, the φ(x i ) output by the instance batch normalization parameterizer is constrained by the domain classifier, so as to learn a domain separable latent space, and the space information is used as domain guide information to modulate the image features transmitted in the backbone network through the IsBN, so that the mixed distribution of multiple data sets can be well handled.

[0190] In an embodiment of the present application, after the density map predictor receives the domain adaptive image features (domain classifiable image features), the second prediction processing is performed, that is, the prediction of the density map is performed according to the domain adaptive image features, and the predicted density map needs to be compared with the real density map, and a second difference is obtained according to the comparison, and the second training is performed under the constraint of the second difference.

[0191] It should be noted that the real density map is a real density map of an image sample corresponding to the domain adaptive image feature, and the real density map is obtained by preprocessing the image, that is, generating by manually labeling or using other labeling methods to label the image samples in the data set to obtain a real density map corresponding to each image sample.

[0192] In some embodiments, referring to Figure 10 Figure 10 is a flowchart of the step S170 refinement step, including:

[0193] Step S1010, constructing a second loss function according to the second difference between the predicted density map and the real density map;

[0194] Step S1020, gradient backpropagation according to the second loss function, and second training of the backbone network.

[0195] In an embodiment of the present application, the second difference is obtained by comparing the predicted density map with the real density map, the second loss function is constructed according to the second difference, the comparison is performed by the second loss function, and the second training of the backbone network is performed according to the result of the comparison of the second loss function. The parameters of the backbone network are optimized by gradient backpropagation, so that the backbone network can better perform a series of feature extraction processing to generate better first image features, so that the domain classifiable image features are more accurate, which is beneficial to the crowd density map predictor to output more accurate crowd predicted density map according to the domain classifiable image features, thereby improving the accuracy of crowd counting.

[0196] It should be noted that the real density map is obtained by preprocessing the image sample corresponding to the domain adaptive image feature, that is, converting the real point labeling into a density map as the learning target of the density map predictor, and the second loss function is defined as mean square error (MSE) as follows:

[0197]

[0198] Wherein, N is the number of training samples, is the predicted density map, is the real density map.

[0199] In step S180, the step of inputting the image sample into the backbone network for feature extraction is returned to perform iterative training until the first difference and the second difference converge, and a trained domain adaptive model is obtained, wherein the domain adaptive model includes a trained parameterizer and a trained backbone network.

[0200] ​In an embodiment of the present application, after steps S110 to S170 are performed, step S120, i.e., the step of inputting the image sample into the backbone network for feature extraction, needs to be returned to continue the iterative training until the convergence requirements of the first difference and the second difference are met, the model iterative training is ended, and the trained domain adaptive model is obtained. The convergence of the first difference represents that the domain-guided instance batch normalization parameterizer in the domain adaptive model has been trained with the assistance of the domain classifier, and the convergence of the second difference represents that the backbone network has been trained with the assistance of the crowd density map predictor. The second image feature output by the parameterizer is optimized by training the domain-guided instance batch normalization parameterizer, so as to output better classification information to guide the classification of the image features existing in the feature mapping space. The first image feature output by the backbone network is optimized by training the backbone network, so as to output better image features to enable the crowd density map predictor to better predict the density map, output a more accurate predicted density map, and improve the accuracy of crowd counting.

[0201] Further, the convergence condition of the first difference is actually the convergence condition of the first loss function, and the convergence condition of the second difference is actually the convergence condition of the second loss function. The model is actually end-to-end trained, so the first loss function and the second loss function can be integrated. The integrated training loss function is defined as follows:

[0202]

[0203] where λ c is a regularization factor (used to balance the two losses) defined by experiments.

[0204] In summary, the domain classifier learns the domain knowledge of distinguishing the input data. The domain knowledge is input into the instance batch normalization module IsBN as domain guidance information by the domain-guided instance batch normalization parameterizer to modulate the feature information output by the backbone network. The feature information is refined according to different domain knowledge, and finally the feature with domain knowledge discrimination information is input into the crowd density map predictor to generate a density map for different domains, effectively solving the biased learning of the deep network model to the domain. And using a single-stage training strategy to train the modulation domain adaptive network model improves the performance of the model, reduces the model training time and the complexity of the model training method, and improves the accuracy of crowd counting.

[0205] In an embodiment of the present application, the domain adaptive model trained is also tested, and the performance of the test results is evaluated after the test. The related specific steps are as follows:

[0206] Input the image sample into the domain adaptive model;

[0207] The backbone network extracts features from the image sample and outputs first image features;

[0208] The instance batch normalization parameterizer of the domain guidance performs processing on the first image features to obtain classification information;

[0209] The instance batch normalization module processes the classification information to obtain domain guidance information, and applies the domain guidance information to the image features transmitted in the backbone network to perform domain classification on the image features, thereby obtaining domain adaptive image features;

[0210] The domain adaptive image features are input into the density map predictor, and the density map predictor performs prediction of a density map on the image features according to domain information of the image features, thereby outputting a predicted density map;

[0211] The predicted density map is subjected to summation calculation to obtain a final predicted number of people;

[0212] The final predicted number of people and the actual number of people are subjected to verification calculation by using mean absolute error (MAE) and root mean squared error (RMSE), and if the calculation result satisfies preset values of the mean absolute error and the root mean squared error, it is considered that the domain adaptive model has been trained and completed, and has reliable crowd prediction counting effect.

[0213] In an embodiment of the present application, for an input image sample The density map prediction given by the regressor in the crowd density map predictor is The definition of the final predicted number of people is as follows:

[0214]

[0215] Wherein, is the predicted density map for the input image sample is the summation of all pixels of the matrix, is the predicted number of people for the input image sample .

[0216] It should be noted that the performance test evaluation standard of the domain adaptive model is mean absolute error (MAE) and root mean squared error (RMSE).

[0217] The definition of MAE is as follows:

[0218]

[0219] The definition of RMSE is as follows:

[0220]

[0221] Wherein, N is the number of image samples, C iFor input image samples The corresponding real number of people.

[0222] Based on the above embodiments of the present application, a trained image regressor can be obtained, which can be used for crowd counting on actual input images or image sets, and output crowd number counting results.

[0223] Based on the above domain adaptive model training method, in an embodiment of the present application, a crowd counting method is also proposed, comprising:

[0224] Obtaining image samples;

[0225] Inputting the image samples into the domain adaptive model trained by the above domain adaptive model training method, and outputting a predicted density map;

[0226] Summation calculation is performed on the predicted density map to calculate the number of people corresponding to the predicted density map.

[0227] The crowd counting method in this embodiment refers to processing the input image samples using the trained domain adaptive model, and outputting the predicted density map corresponding to the image samples. The crowd counting is performed according to the predicted density map, and the number of people corresponding to the predicted density map is calculated through summation calculation, and the crowd number counting result is output. For details, please refer to Figure 11 , Figure 11 is a classification visualization diagram, which represents the process of domain classification in an abstract way. The specific process is as follows:

[0228] A batch of image samples are input into the backbone network, and feature extraction is performed through the backbone network to obtain domain-universal image features. These image features are located in a feature mapping space. A domain classifier is used to generate a virtual class of the domain according to the artificial labeled class of the domain, i.e. the class label in the image sample. The virtual class of the domain is input into the modulator as the guide information of the domain. The modulator modulates the domain-universal image features, and outputs domain adaptive image features. The domain adaptive image features include cross-domain features and non-cross-domain features. The density map predictor can predict the image features according to the cross-domain feature information and the non-cross-domain feature information. In this way, a better density map can be predicted, thereby realizing more accurate crowd counting.

[0229] In an embodiment of the present application, a large number of experiments are carried out on public data sets, including training using a single scene data set (SHA / SHB / QNRF / NWPU) and training using multiple scene data sets (SHA+SHB+QNRF+NWPU). It is proved that the domain adaptation model trained by the single-stage strategy of the present application has very significant effect on multi-domain learning, especially on crowd counting. Through the training method of the present application, the limited labeled data can be fully utilized, and a general domain adaptation model that performs well in multiple scenes is trained through a single-stage training strategy.

[0230] Referring to Figure 12 , as shown in Figure 12 , the performance table of the method of the present application can be seen. In the same experiment of the public data set, the performance and effect of the domain adaptation model trained by the method of the present application are obviously better than those of the existing method. Figure 12 The optimal performance in the table is indicated by bold, and the suboptimal performance is indicated by underlining. It can be seen that in the test performance of the public data set, the method of the present application can achieve optimal performance compared with the existing method for most data sets. It can be seen that the method of the present application is suitable for multi-domain learning problems, especially for crowd counting problems.

[0231] It can be understood that although each step in each of the above flowcharts is displayed in sequence according to the arrow representation, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated in this embodiment, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowcharts can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0232] It should be noted that in each specific embodiment of the present application, when it is necessary to process data related to the characteristics of the target object according to the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards of the country and region. In addition, when the embodiment of the present application needs to obtain target object attribute information, it will obtain the separate permission or separate consent of the target object through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the target object, the necessary target object related data for enabling the embodiment of the present application to function normally will be obtained.

[0233] In an embodiment of the present application, referring to Figure 13 , Figure 13 FIG. 13 is a structural schematic diagram of a domain adaptation model training apparatus 1300. The domain adaptation model training apparatus 1300 comprises:

[0234] A first acquisition unit 1310 is configured to acquire image samples.

[0235] A first generation unit 1320 is configured to input the image samples into a backbone network for feature extraction to obtain first image features.

[0236] A second generation unit 1330 is configured to input the first image features into a parameterizer for convolution processing to obtain second image features.

[0237] A first training unit 1340 is configured to input the second image features into a classifier for first prediction processing to obtain first predicted virtual class labels, and perform first training on the parameterizer according to a first difference between the first predicted virtual class labels and target virtual class labels.

[0238] A third generation unit 1350 is configured to perform information processing on the second image features by the parameterizer to obtain classification information.

[0239] An information classification unit 1360 is configured to classify the first image features according to the classification information to obtain domain adaptation image features.

[0240] A second training unit 1370 is configured to input the domain adaptation image features into a density map predictor for second prediction processing to obtain a predicted density map, and perform second training on the backbone network according to a second difference between the predicted density map and a real density map.

[0241] An iterative training unit 1380 is configured to return to the step of inputting the image samples into the backbone network for feature extraction, and perform iterative training until the first difference and the second difference converge, to obtain a trained domain adaptation model, wherein the domain adaptation model comprises a trained parameterizer and a trained backbone network.

[0242] In an embodiment of the present application, referring to Figure 14 , Figure 14 FIG. 14 is a structural schematic diagram of a domain adaptation model processing apparatus 1400. The domain adaptation model processing apparatus 1400 comprises:

[0243] A second acquisition unit 1410 is configured to acquire image samples.

[0244] The first output unit 1420 is used to input the image sample into the domain adaptive model trained by any one of the domain adaptive model training methods according to claims 1-7, and output a prediction density map.

[0245] The calculation unit 1430 is used to perform summation calculation on the predicted density map to calculate the number of people corresponding to the predicted density map.

[0246] In one embodiment of this application, based on the above-described domain adaptive model training method, an electronic device is also provided, such as... Figure 15 As shown, it includes at least one processor 1510, a display screen 1520, and a memory 1530, and may also include a communications interface 1540 and a bus 1550. The processor 1510, display screen 1520, memory 1530, and communications interface 1540 can communicate with each other via the bus 1550. The display screen 1520 is configured to display a preset user guide interface in the initial setup mode. The communications interface 1540 can transmit information. The processor 1510 can call logical instructions in the memory 1530 to execute the methods described in the above embodiments.

[0247] Furthermore, the logic instructions in the aforementioned memory 1530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0248] In one embodiment of this application, the memory 1530 can serve as a computer-readable storage medium, configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 1510 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 1530, i.e., implementing the steps in the image regressor training method as described in the above embodiments, or implementing the steps in the crowd counting method as described in the above embodiments.

[0249] The memory 1530 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 1530 may include high-speed random access memory (RAM) and may also include non-volatile memory. For example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, may be used, or they may be transient storage media.

[0250] In addition, the specific procedures of the storage medium and the plurality of instruction processors in the terminal device are loaded and executed in the above method, which will not be described one by one here.

[0251] The terms "first", "second", "third", "fourth" and the like in the description of the present disclosure and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "contain" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0252] It should be understood that in the present disclosure, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including single or multiple combinations of items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0253] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is two or more, greater than, less than, more than, etc. are not included in the number, above, below, etc. are understood to include the number.

[0254] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0255] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0256] It should also be understood that various embodiments provided by the embodiments of the present disclosure can be combined in any manner to achieve different technical effects.

[0257] The above is a specific description of the embodiments of the present disclosure, but the present disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present disclosure, and these equivalent modifications or replacements are included in the scope defined by the claims of the present disclosure.

Claims

1. A training method for a domain adaptive model, characterized in that, The method comprises: acquiring an image sample; inputting the image sample into a backbone network for feature extraction to obtain first image features; inputting the first image features into a parameterizer for convolution processing to obtain second image features, the parameterizer comprising a convolution layer, a mean pooling layer, a batch normalization module, an activation function layer, and a full connection layer; inputting the second image features into a classifier for first prediction processing to obtain first predicted virtual class labels, and performing first training on the parameterizer according to a first difference between the first predicted virtual class labels and target virtual class labels, wherein the classifier comprises a full connection layer and a preset activation function for classification, the full connection layer is used to process the second image features to obtain a predicted virtual class label vector, and the preset activation function is used to process the predicted virtual class label vector to obtain the first predicted virtual class labels; performing information processing on the second image features by the parameterizer to obtain classification information; performing classification processing on the first image features according to the classification information to obtain domain adaptive image features; inputting the domain adaptive image features into a density map predictor for second prediction processing to obtain a predicted density map, and performing second training on the backbone network according to a second difference between the predicted density map and a real density map, the backbone network being an HRnet network; returning to the step of inputting the image sample into the backbone network for feature extraction, and performing iterative training until the first difference and the second difference converge, to obtain a trained domain adaptive model, the domain adaptive model comprising a trained parameterizer and a trained backbone network.

2. The method of claim 1, wherein, After the step of inputting the second image features into the classifier for first prediction processing to obtain first predicted virtual class labels, and performing first training on the parameterizer according to a first difference between the first predicted virtual class labels and target virtual class labels, the method further comprises: performing correction processing on the first predicted virtual class labels to obtain second predicted virtual class labels; determining a first iteration number according to a preset training window size; returning to the step of inputting the second image features into the classifier for first prediction processing to obtain first predicted virtual class labels, and performing first training on the parameterizer according to a first difference between the first predicted virtual class labels and target virtual class labels, to complete a first iteration, until the number of the first iteration reaches the first iteration number; averaging a plurality of second predicted virtual class labels obtained in the first iteration to obtain third predicted virtual class labels, and updating the target virtual class labels according to the third predicted virtual class labels; re-performing the first prediction processing on the second image features by the classifier to obtain the first predicted virtual class labels, and returning to the step of performing first training on the parameterizer according to a first difference between the first predicted virtual class labels and updated target virtual class labels to continue a second iteration, until an end condition of the second iteration is met, to obtain an optimized parameterizer.

3. The method of claim 2, wherein, The first predicted virtual class labels comprise a plurality of vector values, and the correction processing on the first predicted virtual class labels to obtain second predicted virtual class labels comprises: obtaining a first maximum vector value of a first part in the first predicted virtual class label; obtaining a second maximum vector value of a second part in the first predicted virtual class label; summing the first maximum vector value and the second maximum vector value to obtain a first target vector value; replacing the second maximum vector value with the first target vector value to obtain a second predicted virtual class label.

4. The method of claim 2, wherein, The updating of the target virtual class label according to the third predicted virtual class label comprises: obtaining a first vector value represented by the third predicted virtual class label; obtaining a second vector value represented by an initial virtual class label, the initial virtual class label being a multi-dimensional vector converted from a first class label, the first class label being derived from artificial labeling of an image sample; multiplying the first vector value by a reliability to obtain a second target vector value, the reliability being a fixed value for measuring the prediction effect of a classifier; operating the second vector value with the reliability to obtain a third target vector value; summing the second target vector value and the third target vector value to obtain a fourth predicted virtual class label, and updating the target virtual class label according to the fourth predicted virtual class label.

5. The method of claim 2, wherein, The first training comprises: constructing a first loss function according to a first difference between the first predicted virtual class label and the target virtual class label; performing gradient backpropagation according to the first loss function to perform the first training on the parameterizer.

6. The method of claim 1, wherein, The classification processing of the first image feature according to the classification information to obtain a domain adaptive image feature comprises: classifying the classification information by a modulator to obtain first sub-classification information and second sub-classification information; performing first classification guidance on the first image feature according to the first sub-classification information; performing second classification guidance on the first image feature according to the second sub-classification information; classifying the first image feature according to the first classification guidance and the second classification guidance to obtain a domain adaptive image feature corresponding to the first image feature.

7. The method of claim 1, wherein, The second training of the backbone network according to a second difference between the predicted density map and the real density map comprises: constructing a second loss function according to the second difference between the predicted density map and the real density map; performing gradient backpropagation according to the second loss function to perform the second training on the backbone network.

8. A method of crowd counting, characterized by, comprises: obtaining an image sample; inputting the image sample into a domain adaptive model trained by the domain adaptive model training method according to any one of claims 1-7 to output a predicted density map; performing summation calculation on the predicted density map to calculate a population quantity corresponding to the predicted density map. 9.A domain adaptation model training apparatus, characterized in that, The training device comprises: a first obtaining unit configured to obtain an image sample; a first generating unit configured to input the image sample into a backbone network for feature extraction to obtain a first image feature; a second generating unit configured to input the first image feature into a parameterizer for convolution processing to obtain a second image feature, the parameterizer comprising a convolution layer, a mean pooling layer, a batch normalization module, an activation function layer and a full connection layer; The first training unit is configured to input the second image feature into a classifier for first prediction processing to obtain a first predicted virtual class label, and perform first training on the parameterizer according to a first difference between the first predicted virtual class label and a target virtual class label, wherein the classifier comprises a full connection layer and a preset activation function for classification, the full connection layer is configured to process the second image feature to obtain a predicted virtual class label vector, and the preset activation function is configured to process the predicted virtual class label vector to obtain the first predicted virtual class label; The third generation unit is configured to perform information processing on the second image feature by the parameterizer to obtain classification information; The information classification unit is configured to classify the first image feature according to the classification information to obtain a domain adaptive image feature; The second training unit is configured to input the domain adaptive image feature into a density map predictor for second prediction processing to obtain a predicted density map, and perform second training on a backbone network according to a second difference between the predicted density map and a real density map, wherein the backbone network is an HRnet network; The iterative training unit is configured to return to the step of inputting the image sample into the backbone network for feature extraction to perform iterative training until the first difference and the second difference converge, and obtain a trained domain adaptive model, wherein the domain adaptive model comprises the trained parameterizer and the trained backbone network.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the domain adaptive model training method according to any one of claims 1-7, or the crowd counting method according to claim 8.

11. A computer readable storage medium, characterized in that, The computer program is executed by the processor to implement the domain adaptive model training method according to any one of claims 1-7, or the crowd counting method according to claim 8.

Images