Training method and device of object counting model, equipment and storage medium

Abstract

The application discloses a training method and device of an object counting model, equipment and a storage medium, which can be applied to artificial intelligence and vehicle-mounted scenes in computer technology. A first sample image used in last round of model training is processed to obtain a second sample image, and the obtained second sample image includes a reference bounding box on the first sample image and a predicted bounding box obtained by mining. The second sample image is input into an object counting model, the second sample image is processed by the object counting model to obtain a predicted density image of the second sample image, and the object counting model is trained based on first difference information between the predicted density image and a reference density image of the second sample image. The training process does not completely depend on labeling, the existing labeling can be expanded in a self-mining form, so that the object counting model learns the features of multiple objects, and the final counting effect of the object counting model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a training method and device of object counting model, equipment and storage medium. BACKGROUND

[0002] With the development of computer technology, the application scenarios of machine learning technology are more and more extensive. For example, small sample counting is an important research direction in the field of machine learning. Here, the small sample refers to a small number of samples, and counting refers to determining the number of objects in an image. The purpose of small sample counting is to train the model using a small number of sample objects, so that the model can determine the number of sample objects in the image.

[0003] In related technologies, only the features of the required sample objects can be obtained on a given number of sample objects, and the model is trained based on the obtained features. The counting effect of the model depends heavily on the annotation. If the number of given sample objects is very small (such as only one sample object is given), the model can only learn the features of this sample object, and when the model is used for counting, it is easy to have a large deviation due to the lack of sample objects, that is, the accuracy of the prediction density map output by the model is low, thereby resulting in inaccurate counting results. SUMMARY

[0004] The embodiments of the present application provide a training method, device and equipment of object counting model and storage medium, which can improve the counting accuracy of the object counting model in the case of a small number of sample objects.

[0005] The technical solution is as follows:

[0006] On the one hand, a training method of an object counting model is provided, and the method comprises:

[0007] inputting a first sample image into an object counting model, the first sample image being a sample image used in the last round of model training, the first sample image comprising a plurality of sample objects and at least one reference bounding box surrounding the sample objects;

[0008] generating a second sample image based on the first sample image through the object counting model, the second sample image comprising the at least one reference bounding box and at least one predicted bounding box, the area surrounded by the predicted bounding box being an area with a confidence meeting a first confidence condition;

[0009] inputting the second sample image into the object counting model, and generating, by the object counting model, a predicted density image of the second sample image based on the second sample image, the predicted density image of the second sample image being used to represent a predicted distribution of the sample objects in the second sample image;

[0010] training the object counting model based on first difference information between the predicted density image of the second sample image and a reference density image, the reference density image being used to represent an actual distribution of the sample objects in the second sample image.

[0011] In a possible implementation, the processing of the second sample image to obtain the predicted density image of the second sample image includes:

[0012] performing feature extraction on the second sample image to obtain a second feature map of the second sample image;

[0013] generating the predicted density image of the second sample image based on similarity between a second target feature region and other feature regions on the second feature map, the second target feature region being a region where the reference bounding box and the predicted bounding box are located.

[0014] In a possible implementation, the method of generating the reference density image includes:

[0015] generating a position image of an initial sample image based on annotation information of the initial sample image, the initial sample image being a sample image used in first round of model training, and the annotation information including a position of the sample object on the initial sample image;

[0016] performing convolution on the position image of the initial sample image to obtain the reference density image.

[0017] In a possible implementation, after the object counting model is trained based on the first difference information between the predicted density image of the second sample image and the reference density image, the method further includes:

[0018] inputting a verification image into the object counting model, the verification image including a plurality of verification objects and at least one verification bounding box surrounding the verification objects;

[0019] processing, by the object counting model, the verification image to obtain a density image of the verification image;

[0020] outputting, by the object counting model, a predicted number of the verification objects in the verification image based on the density image of the verification image through multiple rounds of iteration;

[0021] adjust a hyper parameter of training the object counting model based on second difference information between a predicted number of the verification objects in the verification image and an actual number of the verification objects in the verification image.

[0022] In an aspect, a device for training an object counting model is provided, the device comprising:

[0023] an input module configured to input a first sample image into the object counting model, the first sample image being a sample image used in a previous round of model training, the first sample image comprising a plurality of sample objects and at least one reference bounding box enclosing the sample objects;

[0024] a sample image generation module configured to generate, by the object counting model, a second sample image based on the first sample image, the second sample image comprising the at least one reference bounding box and at least one predicted bounding box, the predicted bounding box enclosing an area having a confidence that meets a first confidence condition;

[0025] a density image generation module configured to input the second sample image into the object counting model, and generate, by the object counting model, a predicted density image of the second sample image based on the second sample image, the predicted density image of the second sample image being used to represent a predicted distribution of the sample objects in the second sample image;

[0026] a training module configured to train the object counting model based on first difference information between the predicted density image of the second sample image and a reference density image, the reference density image being used to represent an actual distribution of the sample objects in the second sample image.

[0027] In a possible implementation, the sample image generation module is configured to process the first sample image to obtain a predicted density image of the first sample image, the predicted density image of the first sample image being used to represent a predicted distribution of the sample objects in the first sample image; and generate the second sample image based on the predicted density image of the first sample image.

[0028] In a possible implementation, the sample image generation module is configured to perform feature extraction on the first sample image to obtain a first feature map of the first sample image; and generate a predicted density image of the first sample image based on a similarity between a first target feature region on the first feature map and other feature regions, the first target feature region being a region where the reference bounding box is located.

[0029] In a possible implementation, the sample image generation module is configured to perform mean filtering on the predicted density image of the first sample image to obtain a first filtered image; perform local maximum value extraction on the first filtered image to obtain at least one candidate position in the first filtered image, the candidate position being a position with a confidence degree that meets a second confidence degree condition and at which the sample object exists; and generate the second sample image based on the at least one candidate position in the first filtered image and the at least one reference bounding box.

[0030] In a possible implementation, the sample image generation module is configured to generate at least one candidate bounding box in the first filtered image based on the at least one candidate position, the size of the candidate bounding box being positively correlated with the size of the at least one reference bounding box; fuse the at least one candidate bounding box to obtain the at least one predicted bounding box; and synthesize the at least one predicted bounding box to the first sample image to obtain the second sample image.

[0031] In a possible implementation, the density image generation module is configured to perform feature extraction on the second sample image to obtain a second feature map of the second sample image; and generate a predicted density image of the second sample image based on a similarity between a second target feature region and other feature regions on the second feature map, the second target feature region being a region in which the reference bounding box and the predicted bounding box are located.

[0032] In a possible implementation, the apparatus further includes:

[0033] The reference density image generation module is configured to generate a position image of an initial sample image based on annotation information of the initial sample image, the initial sample image being a sample image used in first round model training, and the annotation information including a position of the sample object on the initial sample image; and perform convolution on the position image of the initial sample image to obtain the reference density image.

[0034] In a possible implementation, the apparatus further includes:

[0035] The hyperparameter adjustment module is configured to input a verification image into the object counting model, the verification image comprising a plurality of verification objects and at least one verification bounding box enclosing the verification objects; process the verification image by the object counting model to obtain a density image of the verification image; perform a plurality of iterations based on the density image of the verification image by the object counting model, and output a predicted number of the verification objects in the verification image; and adjust a hyperparameter of the object counting model based on second difference information between the predicted number of the verification objects in the verification image and an actual number of the verification objects in the verification image.

[0036] In a possible implementation, the apparatus further comprises:

[0037] The counting module is configured to input a target image into the object counting model, the target image comprising a plurality of target objects and at least one target bounding box enclosing the target objects; process the target image by the object counting model to obtain a density image of the target image; perform a plurality of iterations based on the density image of the target image by the object counting model, and output a predicted number of the target objects in the target image.

[0038] In an aspect, a computer device is provided, comprising one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the training method of the object counting model.

[0039] In an aspect, a computer readable storage medium is provided, the computer readable storage medium storing at least one computer program, the computer program being loaded and executed by a processor to implement the training method of the object counting model.

[0040] In an aspect, a computer program product or computer program is provided, the computer program product or computer program comprising program code stored in a computer readable storage medium, the program code being read by a processor of a computer device from the computer readable storage medium, the processor executing the program code to cause the computer device to perform the training method of the object counting model.

[0041] By means of the technical solutions provided in the embodiments of the present application, when the object counting model is trained, the first sample image used in the last round of model training is processed to obtain a second sample image. This process is equivalent to further mining the information on the first sample image, and the obtained second sample image includes both the reference bounding box on the first sample image and the predicted bounding box obtained by mining. The second sample image is input into the object counting model, and the object counting model is used to process the second sample image to obtain a predicted density image of the second sample image. Subsequently, the object counting model is trained based on the first difference information between the predicted density image and the reference density image of the second sample image. This process is equivalent to training the object counting model by using the second sample image obtained by further mining the first sample image. The training process does not completely depend on labeling, and the existing labeling can be expanded in a self-mining manner, so that the object counting model learns the features of multiple objects and improves the final counting effect of the object counting model. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a schematic diagram of an implementation environment of an object counting model training method provided by an embodiment of the present application;

[0044] Figure 2 is a flowchart of an object counting model training method provided by an embodiment of the present application;

[0045] Figure 3 is another flowchart of an object counting model training method provided by an embodiment of the present application;

[0046] Figure 4 is a structural schematic diagram of an object counting model provided by an embodiment of the present application;

[0047] Figure 5 is another flowchart of an object counting model training method provided by an embodiment of the present application;

[0048] Figure 6 is another flowchart of an object counting model training method provided by an embodiment of the present application;

[0049] Figure 7 is another flowchart of an object counting model training method provided by an embodiment of the present application;

[0050] Figure 8 is a method flowchart for counting using an object counting model provided by an embodiment of the present application;

[0051] Figure 9 is a structural schematic diagram of a training device of an object counting model provided by an embodiment of the present application;

[0052] Figure 10 is a structural schematic diagram of a terminal provided by an embodiment of the present application;

[0053] Figure 11 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0055] In the present application, the terms "first", "second", and the like are used to distinguish the same or similar items with substantially the same function and action, and it should be understood that there is no logical or time sequence relationship between "first", "second", and "nth", and the number and execution order are not limited.

[0056] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0057] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0058] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized field of study that focuses on how computers can simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge sub-models, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to enabling computers to have intelligence. It is applied in various fields of artificial intelligence. Machine learning and deep learning typically include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0059] Semantic features: features used to represent the semantics expressed by the text. Different texts can correspond to the same semantic features, such as the text "Today's weather" and the text "Today's weather" can correspond to the same semantic feature. The computer device can map the characters in the text to a character vector, combine and operate the character vector according to the relationship between the characters, and obtain the semantic features of the text. For example, the computer device can use Bidirectional Encoder Representations from Transformers (BERT).

[0060] Normalization: mapping a number series with different value ranges to the interval (0, 1) for easy data processing. In some cases, the normalized value can be directly implemented as a probability.

[0061] Learning rate: used to control the learning progress of the model. The learning rate can guide the model to adjust the network weights using the gradient of the loss function in the gradient descent method. If the learning rate is too large, the loss function may directly pass through the global optimum, resulting in a large loss. If the learning rate is too small, the loss function changes slowly, greatly increasing the complexity of network convergence, and easily getting stuck in local minimum or saddle point.

[0062] Embedded coding: Embedded coding represents a corresponding relationship in mathematics, i.e. mapping data in X space to Y space through a function F, where the function F is a one-to-one function, and the mapping result is structure-preserving. The one-to-one function means that the mapped data is uniquely corresponding to the pre-mapped data, and the structure-preserving means that the size relationship of the pre-mapped data is the same as that of the post-mapped data, e.g. there are data X1 and X2 before mapping, and Y1 corresponding to X1 and Y2 corresponding to X2 after mapping. If the data X1 > X2 before mapping, then the data Y1 > Y2 after mapping accordingly. For words, it is to map words to another space for subsequent machine learning and processing.

[0063] Attention weight: can represent the importance of certain data in the training or prediction process, the importance represents the size of the influence of the input data on the output data. The data with high importance has a higher value of the corresponding attention weight, and the data with low importance has a lower value of the corresponding attention weight. In different scenarios, the importance of data is not the same, and the process of training attention weight of the model is also the process of determining the importance of data.

[0064] Small sample learning: in the training stage, the object counting model can obtain a sample image, the position label of all sample objects in the sample image, and a plurality of bounding boxes surrounding the sample objects in the sample image, so as to perform counting learning.

[0065] In the test stage, the object counting model can only obtain a test image and a plurality of bounding boxes of target objects corresponding to the test image, so as to determine the number of target objects in the test image.

[0066] It should be noted that the target objects for counting in the test stage are completely new categories, that is, categories that the object counting model has not learned at all in the training stage, that is, the target objects and the sample objects are different types of objects. Therefore, for the object counting model trained under this task definition, it can predict the total number of all objects of a new category on various images of the category only by giving a plurality of bounding boxes of the required category objects, which has a very wide practical application value.

[0067] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0068] Figure 1 is an implementation environment schematic diagram of a training method of an object counting model provided by an embodiment of the present application, referring to Figure 1 The implementation environment can include a terminal 110 and a server 140.

[0069] The terminal 110 is connected to the server 140 through a wireless network or a wired network. Optionally, the terminal 110 is a vehicle-mounted terminal, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 110 installs and runs an application program supporting object counting.

[0070] The server 140 is a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server 140 provides background services for the application program running on the terminal 110, for example, the server 140 can train an object counting model used by the application program running on the terminal 110.

[0071] Optionally, the number of terminals 110 and servers 140 is not limited.

[0072] After introducing the implementation environment of the embodiments of the present application, the application scenarios of the embodiments of the present application will be introduced in combination with the above implementation environment. In the following description process, the terminal is the terminal 110 in the above implementation environment, and the server is the server 140 in the above implementation environment.

[0073] The training method of the object counting model provided by the embodiments of the present application can be applied in scenarios of counting multiple types of objects. For example, it can be applied in scenarios of counting a certain defect on an industrial product, or scenarios of counting a certain animal on an ecological environment image, or scenarios of counting a certain cell in a biological tissue image, or scenarios of counting a certain vehicle on a traffic environment image.

[0074] Taking the case that the counting scheme provided by the embodiments of the present application is applied in scenarios of counting a certain defect on an industrial product, the server trains an object counting model by using the training method of the object counting model provided by the embodiments of the present application. When it is necessary to count a certain defect on an industrial product, a technician generates an image of the industrial product through a terminal, and the image of the industrial product includes a small number of bounding boxes surrounding the defect. In some embodiments, the small number of bounding boxes refers to one bounding box. The terminal uploads the image of the industrial product to the server, and the server inputs the image of the industrial product into the trained object counting model. The object counting model can output the number of the defect in the image of the industrial product. In some embodiments, the industrial product is a wrench, and the defect is a crack on the wrench.

[0075] Take the counting scheme provided in the embodiments of the present application as an example in the scenario of counting a certain animal on an ecological environment image, a server trains an object counting model by using the training method of the object counting model provided in the embodiments of the present application, when it is needed to count a certain animal on an ecological environment image, a technician generates an image of the ecological environment image through a terminal, the image of the ecological environment image includes a small number of bounding boxes surrounding the certain animal, in some embodiments, the small number of bounding boxes refers to one bounding box. The terminal uploads the image of the ecological environment image to the server, the server inputs the image of the ecological environment image into the trained object counting model, and the object counting model can output the number of the certain animal in the image of the ecological environment image. In some embodiments, the certain animal is a giant panda.

[0076] Take the counting scheme provided in the embodiments of the present application as an example in the scenario of counting a certain cell on a biological tissue image, a server trains an object counting model by using the training method of the object counting model provided in the embodiments of the present application, when it is needed to count a certain cell on a biological tissue image, a technician generates an image of the biological tissue image through a terminal, the image of the biological tissue image includes a small number of bounding boxes surrounding the certain cell, in some embodiments, the small number of bounding boxes refers to one bounding box. The terminal uploads the image of the biological tissue image to the server, the server inputs the image of the biological tissue image into the trained object counting model, and the object counting model can output the number of the certain cell in the image of the biological tissue image.

[0077] Take the counting scheme provided in the embodiments of the present application as an example in the scenario of counting a certain vehicle on a traffic environment image, a server trains an object counting model by using the training method of the object counting model provided in the embodiments of the present application, when it is needed to count a certain vehicle on a traffic environment image, a technician generates an image of the traffic environment image through a terminal, the image of the traffic environment image includes a small number of bounding boxes surrounding the certain vehicle, in some embodiments, the small number of bounding boxes refers to one bounding box. The terminal uploads the image of the traffic environment image to the server, the server inputs the image of the traffic environment image into the trained object counting model, and the object counting model can output the number of the certain vehicle in the image of the traffic environment image.

[0078] It should be noted that, in the above process of explaining the application scenarios, the server trains the object counting model. The terminal obtains the image, uploads the image to the server, and the server counts the objects in the image, for example, in other possible implementations, the terminal can directly count the objects in the image after obtaining the image, without uploading the target image to the server, and the embodiments of the present application do not limit this.

[0079] In addition, the object counting model training method provided by the embodiments of the present application can be applied to other counting scenarios in addition to the above-mentioned several application scenarios, and the embodiments of the present application do not limit this.

[0080] After introducing the implementation environment and application scenarios of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are introduced. Referring to Figure 2 The technical solutions provided by the embodiments of the present application can be executed by a terminal or a server, or by a terminal and a server together, and in the embodiments of the present application, the execution subject is taken as an example to be a server, and the method includes the following steps.

[0081] 201. The server inputs a first sample image into the object counting model, the first sample image being a sample image used in the last round of model training, the first sample image including a plurality of sample objects and at least one reference bounding box surrounding the sample objects.

[0082] The first sample image includes a plurality of sample objects, and the reference bounding box surrounds the sample objects, that is, the reference bounding box can represent the positions of the sample objects on the sample image, wherein the reference bounding box surrounding the sample objects means that one reference bounding box surrounds one sample object, and when the object counting model is used for counting, the number of bounding boxes can be counted. The number of reference bounding boxes is less than or equal to a number threshold, and the number threshold is set by a technician according to actual conditions. In an extreme case, the number of reference bounding boxes is 1, and the embodiments of the present application do not limit this. In some embodiments, the sample image is also referred to as a support image (Support Image) or a reference image (Reference Image).

[0083] 202. The server generates a second sample image based on the first sample image through the object counting model, the second sample image including the at least one reference bounding box and at least one predicted bounding box, the area surrounded by the predicted bounding box being an area in which the confidence of the existence of the sample object meets a first confidence condition.

[0084] The predicted bounding box is obtained by the server based on the first sample image through the object counting model, that is, it is predicted based on the reference bounding box on the first sample image. The predicted bounding box in the second sample image is obtained by the object counting model without supervision, which is equivalent to further mining the reference bounding box on the first sample image by the server through the object counting model. The area surrounded by the reference bounding box is an area in which the sample object may exist, that is, an area in which the confidence of the existence of the sample object meets the first confidence condition.

[0085] 203. The server inputs the second sample image into the object counting model, processes the second sample image through the object counting model, and obtains the predicted density image of the second sample image. The predicted density image of the second sample image is used to represent the predicted distribution of the sample object in the second sample image.

[0086] The density image is used to represent the predicted distribution of the sample object in the second sample image. The predicted distribution is the distribution of the sample object predicted by the object counting model based on the reference bounding box.

[0087] 204. Based on the first difference information between the predicted density image and the reference density image of the second sample image, the server performs one round of training on the object counting model. The reference density image is used to represent the actual distribution of the sample object in the second sample image.

[0088] The reference density image can be considered as a label for the second sample image, reflecting the actual distribution of the sample objects within it. Since the predicted density image of the second sample image is obtained by the server through the object counting model, training the object counting model based on the first difference information allows the density image output by the model to be as close as possible to the reference density image, thereby improving the accuracy of the object counting model during counting.

[0089] The technical solution provided in this application, when training the object counting model, processes the first sample image used in the previous training round to obtain a second sample image. This process is equivalent to further mining the information on the first sample image. The resulting second sample image includes both the reference bounding box on the first sample image and the mined predicted bounding box. The second sample image is then input into the object counting model, which processes it to obtain a predicted density image. Subsequently, the object counting model is trained based on the first difference information between the predicted density image and the reference density image. This process essentially uses the second sample image obtained through further mining of the first sample image to train the object counting model. The training process does not rely entirely on annotations; it expands existing annotations through self-mining, enabling the object counting model to learn the features of multiple objects and improving the final counting performance.

[0090] It should be noted that steps 201-204 above are a simplified description of the technical solutions provided in the embodiments of this application. The technical solutions provided in the embodiments of this application will be described in more detail below with reference to some examples. See [link to relevant documentation]. Figure 3The technical scheme provided by the embodiments of the present application can be executed by a terminal or a server, or executed by a terminal and a server together. In the following description, the technical scheme is executed by a server as an example. In order to more clearly describe the technical scheme provided by the embodiments of the present application, the first round of model training and other rounds of model training will be described respectively. Steps 301-304 are used to describe the first round of model training, and steps 601-605 are used to describe other rounds of model training.

[0091] 301. The server acquires a sample image set, the sample image set including a plurality of initial sample images.

[0092] The sample objects included in the plurality of initial sample images can be sample objects of the same type or sample objects of different types, which is not limited in the embodiments of the present application. When training the object counting model, one initial sample image can be used for multiple rounds of training, that is, one initial sample image can be regarded as one batch, and another initial sample image can be regarded as another batch.

[0093] In a possible implementation, in response to the operation on the sample image set, the terminal sends the sample image set to the server, and the server acquires the sample image set. In this implementation, the video producer can control the terminal to send the sample image set to the server by operating the sample image set, and the video producer can select the sample image set by himself / herself, so that the efficiency of human-computer interaction is higher.

[0094] For example, the terminal displays an image set selection page, and the image set selection page includes a plurality of candidate image sets. In response to a click operation on a sample image set in the plurality of candidate image sets, the terminal sends the sample image set to the server, and the server acquires the sample image set. In this case, the plurality of candidate image sets are image sets stored on the terminal. In the case where the plurality of candidate image sets are image sets stored on the server, in response to a click operation on a sample image set on the image set selection page, the terminal sends an image set selection instruction to the server, and the image set selection instruction carries an identifier of the sample image set. After receiving the image set selection instruction, the server acquires the identifier of the sample image set from the image set selection instruction. The server queries based on the identifier of the sample image set and acquires the sample image set.

[0095] In some embodiments, after obtaining the sample image set, the server can further obtain a verification image set and a test image set. The verification image set includes a plurality of verification images, each of which includes a plurality of sample objects and at least one reference bounding box enclosing the sample objects. The test image set includes a plurality of test images, each of which includes a plurality of sample objects and at least one reference bounding box enclosing the sample objects. The sample image set is used to train the object counting model, the verification image set is used to determine the hyperparameters of the object counting model, and the test image set is used to test the effect of the object counting model.

[0096] 302. The server obtains any initial sample image from the sample image set and inputs the initial sample image into the object counting model, the initial sample image including a plurality of sample objects and at least one reference bounding box enclosing the sample objects.

[0097] The object counting model is used to determine the number of objects in an image, where the objects in the image refer to specified objects in the image, and the specification of the objects is performed by a technician according to actual needs. Taking the counting scheme provided in the embodiments of the present application as an example applied to the scene of counting a certain defect on an industrial product, the image is the image of the industrial product, and the object is the specified defect, such as a crack on the industrial product. The way to specify the object on the image is to add a bounding box enclosing the object on the image. After inputting the image with the added bounding box into the trained object counting model, the object counting model can output the number of the object in the image. In some embodiments, the sample object is also referred to as a sample, and the reference bounding box enclosing the sample object is also referred to as a sample target bounding box. In some embodiments, the object counting model is also referred to as a small sample counting model, where the small sample refers to a small number of reference bounding boxes labeled in the sample image, and the small number refers to a number of reference bounding boxes less than or equal to a number threshold, which is set by a technician according to actual conditions, such as 1 or 2, etc.

[0098] The initial sample image is used to start training the object counting model, that is, K rounds of training of the object counting model can be performed based on the initial sample image, and the sample images used in each round of training are obtained based on the initial sample image, where K is a positive integer.

[0099] In some embodiments, the server stores the sample images, the positions of the sample objects in the sample images, and the positions of the reference bounding boxes in the sample images in the form of a sample image set, a sample object position set, and a reference bounding box set. For example, the sample image set is X train = {X1, X2,... X n}, the reference bounding box set and the sample object position set are and wherein, denotes the i-th sample image in the sample image set, H i and W i are the height and width of the spatial dimension of the sample image, respectively, H i and W i may or may not be the same, and embodiments of the present application do not limit this. and 0 in the above formulae denotes the reference bounding box set and the sample object position set when no iteration is performed, wherein the reference bounding box set stores the positions of a plurality of reference bounding boxes on the sample image, and the sample object position set stores the positions of a plurality of sample objects on the sample image, and 0 increases with the increase of the number of iterations. denotes the i-th sample image X i corresponding to the a i reference bounding boxes, denotes the i-th sample image X i corresponding to the b i sample object positions (a i <<b i ).

[0100] In some embodiments, the verification image set and the test image set are denoted as X val and X test respectively, the corresponding reference bounding box set and the sample object position set of the verification image set X val are denoted as and the corresponding reference bounding box set and the sample object position set of the test image set X test are denoted as and

[0101] 303. The server generates, by using the object counting model, a predicted density image of the initial sample image based on the initial sample image, the predicted density image of the initial sample image being used to represent a predicted distribution of the sample object in the initial sample image.

[0102] In some embodiments, the pixel value of each pixel point in the predicted density image of the initial sample image is a confidence degree of the existence of the sample object corresponding to the pixel point, and the predicted distribution of the sample object in the initial sample image can be obtained by using the pixel values of the plurality of pixel points in the predicted density image.

[0103] In a possible implementation, the server extracts features of the initial sample image based on the object counting model to obtain a feature map of the initial sample image.

[0104] In the above implementation, the other feature regions refer to regions on the feature map of the initial sample image other than the target feature region.

[0105] In this implementation, the features in the target feature region are used when generating the predicted density image of the initial sample image. Since the target feature region corresponds to the region where the reference bounding box is located, and the reference bounding box encloses the sample object, the target feature region also corresponds to the features of the sample object. Determining the similarity between the target feature region and the other feature regions can determine the confidence that the sample object exists in the other feature regions, and thus the predicted density image of the initial sample image is generated based on the confidence that the sample object exists in the other feature regions.

[0106] To make the above implementation clearer, the above implementation will be described in two parts.

[0107] In the first part, the server extracts features of the initial sample image based on the object counting model to obtain a feature map of the initial sample image.

[0108] In a possible implementation, the server extracts features of the initial sample image based on the convolution layer of the object counting model to obtain a feature map of the initial sample image.

[0109] In this implementation, the server can extract features of the initial sample image based on the convolution layer of the object counting model. Since the convolution operation is fast, the efficiency of obtaining the feature map of the initial sample image is high.

[0110] For example, the object counting model comprises a feature extraction unit, and the server inputs the initial sample image into the feature extraction unit of the object counting model. The server convolves the initial sample image through the convolutional layer of the feature extraction unit, that is, the convolution kernel on the convolutional layer of the feature extraction unit slides in the initial sample image, and the convolution kernel convolves the region covered by the convolution kernel in the initial sample image in the sliding process to obtain a feature map of the initial sample image. The number of convolution kernels on the convolutional layer of the feature extraction unit is one or more, which is not limited in the embodiments of the present application. In the case where the initial sample image comprises a plurality of color channels, the number of convolution kernels on the convolutional layer of the feature extraction unit is also a plurality, and the plurality of convolution kernels are used to extract features of the plurality of color channels of the initial sample image to obtain color feature maps of the respective color channels. The server fuses the color feature maps of the plurality of color channels through the feature extraction unit to obtain the feature map of the initial sample image. In some embodiments, when the server fuses the color feature maps of the plurality of color channels through the feature extraction unit to obtain the feature map of the initial sample image, a weighted summation method can be used. In some embodiments, the feature extraction unit is a feature extractor based on a convolutional neural network (CNN), such as a neural network ResNet-50 pre-trained on a large-scale open source dataset ImageNet.

[0111] In a possible implementation, the server encodes the initial sample image based on an attention mechanism through the object counting model to obtain the feature map of the initial sample image.

[0112] In this implementation, the server can perform feature extraction based on an attention mechanism through the object counting model. The introduction of the attention mechanism enables the object counting model to focus on regions with higher importance in the initial sample image, and the obtained feature map can more accurately reflect the features of the initial sample image.

[0113] For example, the object counting model includes a feature extraction unit, and the server inputs the initial sample image into the feature extraction unit of the object counting model. The server encodes multiple parts of the initial sample image based on an attention mechanism through the feature extraction unit to obtain a feature map of the initial sample image. For example, the server inputs the initial sample image into the feature extraction unit, embeds and encodes multiple parts of the initial sample image through the feature extraction unit to obtain multiple embedding vectors, one embedding vector corresponding to one part of the initial sample image, and the embedding vectors being used to represent the positions of the parts in the initial sample image and the contents of the parts. The server inputs the multiple embedding vectors into the feature extraction unit, linearly transforms the multiple embedding vectors through three linear transformation matrices of the feature extraction unit to obtain a query vector, a key vector, and a value vector corresponding to each part of the initial sample image. The server obtains attention weights of the multiple parts of the initial sample image based on the query vector and the key vector corresponding to the multiple parts of the initial sample image through the feature extraction unit. The server obtains attention encoding vectors of the multiple parts of the initial sample image based on the attention weights of the multiple parts of the initial sample image and the value vectors of the multiple parts of the initial sample image through the feature extraction unit. The attention encoding vectors of the multiple parts of the initial sample image are combined according to the positions of the parts in the initial sample image to obtain the feature map of the initial sample image.

[0114] In a possible implementation, the server fully connects the initial sample image through a fully connected layer of the object counting model to obtain the feature map of the initial sample image. In this implementation, the server can obtain the feature map of the initial sample image by fully connecting the initial sample image through the object counting model, and the fully connection can extract the features of the initial sample image as a whole to obtain the feature map that can reflect the features of the initial sample image as a whole.

[0115] For example, the object counting model includes a feature extraction unit, and the server inputs the initial sample image into the feature extraction unit of the object counting model. The server fully connects the initial sample image through a fully connected layer of the feature extraction unit to obtain a feature map of the initial sample image. In some embodiments, the feature extraction model is a feature extractor based on a deep neural network (DNN).

[0116] It should be noted that the server can obtain the feature map of the initial sample image in any of the above manners, or use a ViT (Vision Transformer) or other conventional image feature extractor to extract features of the initial sample image. Of course, as science and technology develop, the server can also obtain the feature map in other manners, which are not limited by the embodiments of the application.

[0117] The second part, the server generates a predicted density image of the initial sample image based on the similarity between the target feature region and other feature regions on the feature map of the initial sample image through the object counting model.

[0118] The similarity between the target feature region and other feature regions represents the confidence that other feature regions exist sample objects.

[0119] In one possible implementation, the server obtains a target sub-feature map from the feature map of the initial sample image through the object counting model, the target sub-feature map being a part of the feature map of the initial sample image corresponding to the target feature region, i.e., a part of the initial sample image corresponding to the reference bounding box. The server determines the similarity between the target feature region and other feature regions based on the target sub-feature map through the object counting model. For example, the server determines the similarity between the target feature region and other feature regions by sliding the target sub-feature map on the feature map of the initial sample image through the object counting model, other feature regions being regions covered by the target sub-feature map during the sliding process. The server generates density map prediction features of the initial sample image based on the similarity between the target feature region and other feature regions. The server convolves the density map prediction features of the initial sample image through the object counting model to obtain a predicted density image of the initial sample image. When determining the similarity between the target feature region and other feature regions, i.e., performing correlation operation on the target feature region and other feature regions to obtain the similarity between the target feature region and other feature regions, the server can perform correlation operation on the target feature region and other feature regions by merging and connecting.

[0120] For example, the object counting model includes a local feature acquisition unit, a matching unit, and a density map generation unit. The server obtains an initial sub-feature map from a feature map of the initial sample image by the object counting model, the initial sub-feature map being a part of the feature map of the initial sample image corresponding to the initial feature region, that is, a part of the initial sample image corresponding to the reference bounding box. The server obtains a target sub-feature map of a target size by pooling the initial sub-feature map by the local feature acquisition unit, wherein the purpose of pooling the initial sub-feature map is to adjust the initial sub-feature map to the target size, which is set by a technician according to actual conditions, and the embodiments of the present application do not limit this. The server inputs the target sub-feature map and the feature map of the initial sample image into the matching unit, determines the similarity between the target feature region and other feature regions by the matching unit, that is, controls the target sub-feature map to slide on the feature map of the initial sample image, determines the similarity between the target feature region and other feature regions, and outputs a density map prediction feature of the initial sample image, which is a tensor. The server inputs the density map prediction feature of the initial sample image into the density map generation unit, convolves the density map prediction feature of the initial sample image by the convolution layer of the density map generation unit, and outputs a predicted density image of the initial sample image. In some embodiments, the above process of obtaining the predicted density image of the initial sample image can be represented by the following formula (1).

[0121]

[0122] wherein, is the predicted density image of the initial sample image, M0 is the object counting model in the first model training, X train is a sample image set, is a reference bounding box set in the first model training.

[0123] In some embodiments, referring to Figure 4 , a structural diagram of an object counting model is provided, and the object counting model 400 includes a feature extraction unit 401, a local feature acquisition unit 402, a matching unit 403, and a density map generation unit 404. It should be noted that, Figure 4 The structure of the provided object counting model is only an example, and in other possible implementations, the object counting model can have other structures, and the embodiments of the present application do not limit this.

[0124] 304, the server performs first round training on the object counting model based on the difference information between the predicted density image of the initial sample image and the reference density image, and the reference density image is used to represent the actual distribution of the sample object in the initial sample image.

[0125] The reference density image can be regarded as supervision for training the object counting model, and the object counting model is trained to make the predicted density image output by the object counting model as close as possible to the reference density image, that is, the difference information between the predicted density image and the reference density image is as small as possible.

[0126] To make the above-mentioned embodiments clearer, the method for generating the reference density image by the server is described below. It should be noted that the server generates the reference density image before step 304, such as before step 301, and the embodiments of the present application do not limit this.

[0127] In a possible implementation, the server generates a position image of the initial sample image based on the annotation information of the initial sample image, the initial sample image being a sample image used in the first round of model training, and the annotation information including the position of the sample object on the initial sample image. The server convolves the position image of the initial sample image to obtain the reference density image.

[0128] The annotation information of the initial sample image includes the position of the sample object on the initial sample image. In some embodiments, the position of the sample object on the initial sample image is represented by the coordinates of the sample object on the initial sample image.

[0129] For example, the server binarizes the initial sample image based on the annotation information of the initial sample image to obtain a position image of the initial sample image, wherein the purpose of binarization is to assign a first value as a pixel value to the pixel points in the initial sample image corresponding to the sample object; and assign a second value as a pixel value to the pixel points not corresponding to the sample object. The position image thus obtained can represent the actual distribution of the sample object on the initial sample image through the numerical value of the pixel points. The server convolves the position image to obtain the reference density image of the initial sample image.

[0130] For example, the server binarizes the initial sample image based on the annotation information of the initial sample image based on the following formula (2) to obtain a position image of the initial sample image.

[0131]

[0132] wherein, is the pixel value of the pixel point on the position image, p is the coordinates of the pixel point on the initial sample image i, C i is the annotation information of the initial sample image i.

[0133] The server convolves the position image by a Gaussian kernel according to formula (3) to obtain a reference density image of the initial sample image.

[0134]

[0135] wherein D i is the reference density image of the initial sample image i, F i is the position image of the initial sample image i, is a convolution operation, G σ is a Gaussian kernel.

[0136] It should be noted that the above is an example of the server obtaining a reference density image of an initial sample image. The method for the server to obtain other initial sample images and the method for the server to obtain the reference density image of the initial sample image belong to the same inventive concept, and the implementation process will not be described again. For example, the sample image set is X train ={X1, X2,... X n}, then the server can obtain a reference density image set D train ={D1, D2,... D n}, and the reference density image set D train can be used to calculate the model loss when training the object counting model.

[0137] After introducing the method for the server to generate the reference density image, the above step 304 will be described.

[0138] In one possible implementation, the server substitutes the difference information between the predicted density image of the initial sample image and the reference density image into a target loss function, and uses gradient descent method to adjust the model parameters of the object counting model, thereby completing the first round of training of the object counting model.

[0139] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described again.

[0140] The above steps 301-304 are an explanation of the first round of model training, and the steps 501-505 below are an explanation of other rounds of model training. Referring to Figure 5 , still taking the server as an example of the execution subject, the method comprises the following steps.

[0141] 501. The server inputs a first sample image into the object counting model, the first sample image being a sample image used in the last round of model training, the first sample image comprising a plurality of sample objects and at least one reference bounding box surrounding the sample objects.

[0142] The first sample image is the initial sample image in steps 301-304 or a sample image obtained by performing multiple rounds of model training on the object counting model based on the initial sample image, which is not limited in the embodiments of the present application. In the case where the first sample image is the initial sample image in steps 301-304, the reference bounding box on the first sample image is the reference bounding box of the initial sample image, that is, the bounding box labeled on the initial sample image. In the case where the first sample image is a sample image obtained by performing multiple rounds of model training on the object counting model based on the initial sample image, the reference bounding box of the first sample image includes the reference bounding box of the initial sample image and a predicted bounding box obtained in the model training process. For the generation method of the predicted bounding box, refer to the description of step 502 below.

[0143] For ease of understanding, the following description takes the first sample image as the initial sample image in steps 301-304 as an example.

[0144] 502. The server generates a second sample image based on the first sample image by using the object counting model, the second sample image including the at least one reference bounding box and at least one predicted bounding box, the predicted bounding box enclosing an area with a confidence meeting a first confidence condition.

[0145] The confidence meeting the first confidence condition means that the confidence is greater than or equal to a first confidence threshold, or means that the confidence is the confidence of a bounding box with greater confidence among two bounding boxes that overlap each other, wherein whether the two bounding boxes overlap is determined based on the overlap degree of the two bounding boxes. In some embodiments, in the case where the overlap degree of the two bounding boxes is greater than or equal to an overlap degree threshold, it is determined that the two bounding boxes overlap; in the case where the overlap degree of the two bounding boxes is less than the overlap degree threshold, it is determined that the two bounding boxes do not overlap, wherein the overlap degree threshold is set by a technician according to actual conditions, which is not limited in the embodiments of the present application.

[0146] In a possible implementation, the server processes the first sample image by using the object counting model to obtain a predicted density image of the first sample image, the predicted density image of the first sample image being used to represent the predicted distribution of the sample object in the first sample image. The server generates the second sample image based on the predicted density image of the first sample image by using the object counting model.

[0147] In a case where the first sample image is the initial sample image, the predicted density image of the first sample image can also be different from the predicted density image of the initial sample image in the steps 301-304, because the object counting model has been trained for one round of model training in the steps 301-304, and the model parameters of the object counting model have been changed, and the predicted density image can be different even in a case where the first sample image is the initial sample image. In some embodiments, in order to distinguish the object counting model, the object counting model in the steps 301-304 can be referred to as a first object counting model, and the object counting model in the steps 501-505 can be referred to as a second object counting model, the first object counting model and the second object counting model have the same model structure, and are object counting models obtained based on the initial sample image after different rounds of model training.

[0148] In order to make the above-mentioned embodiments clearer, the above-mentioned embodiments will be described in two parts.

[0149] The server processes the first sample image by using the object counting model to obtain a predicted density image of the first sample image.

[0150] In a possible implementation, the server extracts features of the first sample image by using the object counting model to obtain a first feature map of the first sample image. The server generates the predicted density image of the first sample image based on similarities between a first target feature region and other feature regions on the first feature map by using the object counting model, the first target feature region being a region where the reference bounding box is located.

[0151] In order to make the above-mentioned embodiments clearer, the above-mentioned embodiments will be described in two parts.

[0152] A, the server extracts features of the first sample image by using the object counting model to obtain a first feature map of the first sample image.

[0153] In a possible implementation, the server convolves the first sample image by using a convolution layer of the object counting model to obtain a feature map of the first sample image.

[0154] In this implementation, the server can extract features of the first sample image by using the convolution layer of the object counting model, and the efficiency of obtaining the feature map of the first sample image is high because of the high speed of the convolution operation.

[0155] For example, the object counting model comprises a feature extraction unit, and the server inputs the first sample image into the feature extraction unit of the object counting model. The server convolves the first sample image through the convolutional layer of the feature extraction unit, that is, the convolution kernel on the convolutional layer of the feature extraction unit slides in the first sample image, and the convolution kernel convolves the region covered by the convolution kernel in the first sample image during the sliding process to obtain the feature map of the first sample image. The number of convolution kernels on the convolutional layer of the feature extraction unit is one or more, which is not limited in the embodiments of the present application. In the case where the first sample image comprises a plurality of color channels, the number of convolution kernels on the convolutional layer of the feature extraction unit is also a plurality, and the plurality of convolution kernels are used to extract features of the plurality of color channels of the first sample image to obtain the color feature map of each color channel. The server fuses the color feature maps of the plurality of color channels through the feature extraction unit to obtain the feature map of the first sample image. In some embodiments, when the server fuses the color feature maps of the plurality of color channels through the feature extraction unit to obtain the feature map of the first sample image, a weighted summation method can be used. In some embodiments, the feature extraction unit is a feature extractor based on a convolutional neural network (CNN).

[0156] In a possible implementation, the server encodes the first sample image based on an attention mechanism through the object counting model to obtain the feature map of the first sample image.

[0157] In this implementation, the server can extract features based on the attention mechanism through the object counting model. The introduction of the attention mechanism enables the object counting model to focus on the regions with higher importance in the first sample image, and the obtained feature map can more accurately reflect the features of the first sample image.

[0158] For example, the object counting model includes a feature extraction unit, and the server inputs the first sample image into the feature extraction unit of the object counting model. The server encodes multiple parts of the first sample image based on an attention mechanism through the feature extraction unit to obtain a feature map of the first sample image. For example, the server inputs the first sample image into the feature extraction unit, embeds and encodes multiple parts of the first sample image through the feature extraction unit to obtain multiple embedding vectors, one embedding vector corresponding to one part of the first sample image, and the embedding vectors being used to represent the positions of the parts in the first sample image and the contents of the parts. The server inputs the multiple embedding vectors into the feature extraction unit, linearly transforms the multiple embedding vectors through three linear transformation matrices of the feature extraction unit to obtain a query vector, a key vector, and a value vector corresponding to each part of the first sample image. The server obtains attention weights of the multiple parts of the first sample image based on the query vector and the key vector corresponding to the multiple parts of the first sample image through the feature extraction unit. The server obtains attention encoding vectors of the multiple parts of the first sample image based on the attention weights of the multiple parts of the first sample image and the value vectors of the multiple parts of the first sample image through the feature extraction unit. The attention encoding vectors of the multiple parts of the first sample image are combined according to the positions of the parts in the first sample image to obtain the feature map of the first sample image.

[0159] In a possible implementation, the server fully connects the first sample image through a fully connected layer of the object counting model to obtain the feature map of the first sample image. In this implementation, the server can obtain the feature map of the first sample image by fully connecting the first sample image through the object counting model, and the fully connection can extract the features of the first sample image as a whole to obtain the feature map that can reflect the features of the first sample image as a whole.

[0160] For example, the object counting model includes a feature extraction unit, and the server inputs the first sample image into the feature extraction unit of the object counting model. The server fully connects the first sample image through a fully connected layer of the feature extraction unit to obtain a feature map of the first sample image. In some embodiments, the feature extraction model is a feature extractor based on a deep neural network (DNN).

[0161] It should be noted that the server can obtain the feature map of the first sample image in any of the above manners, or use a ViT (Vision Transformer) or other conventional image feature extractor to perform feature extraction on the first sample image. Of course, as science and technology develop, the server can also obtain the feature map in other manners, which are not limited by the embodiments of the application.

[0162] B. The server generates a predicted density image of the first sample image based on the similarity between the first target feature region and other feature regions on the first feature map through the object counting model.

[0163] The size of the feature map of the first sample image is the same as that of the first sample image, and the similarity between the first target feature region and other feature regions represents the confidence of the existence of sample objects in other feature regions.

[0164] In a possible implementation, the server obtains a target sub-feature map from the feature map of the first sample image through the object counting model, the target sub-feature map being a part of the feature map of the first sample image corresponding to the first target feature region, i.e., a part of the first sample image corresponding to the reference bounding box. The server determines the similarity between the first target feature region and other feature regions based on the target sub-feature map through the object counting model. For example, the server determines the similarity between the first target feature region and other feature regions by sliding the target sub-feature map on the feature map of the first sample image through the object counting model, other feature regions being regions covered by the target sub-feature map during sliding. The server generates a density map prediction feature of the first sample image based on the similarity between the first target feature region and other feature regions. The server convolves the density map prediction feature of the first sample image through the object counting model to obtain a predicted density image of the first sample image. When determining the similarity between the first target feature region and other feature regions, i.e., performing correlation operation on the first target feature region and other feature regions to obtain the similarity between the first target feature region and other feature regions, the server can perform correlation operation on the first target feature region and other feature regions through merging connection.

[0165] For example, the object counting model comprises a local feature acquisition unit, a matching unit, and a density map generation unit. The server acquires, through the object counting model, an initial sub-feature map from a feature map of the first sample image, the initial sub-feature map being a part of the feature map of the first sample image corresponding to the initial feature region, i.e., a part of the first sample image corresponding to the reference bounding box. The server performs pooling on the initial sub-feature map through the local feature acquisition unit to obtain a target sub-feature map of a target size, wherein the purpose of performing pooling on the initial sub-feature map is to adjust the initial sub-feature map to the target size, which is set by a technician according to actual conditions, and embodiments of the present application do not limit this. The server inputs the target sub-feature map and the feature map of the first sample image into the matching unit, determines the similarity between the first target feature region and other feature regions through the matching unit, i.e., controls the target sub-feature map to slide on the feature map of the first sample image, determines the similarity between the first target feature region and other feature regions, and outputs a density map prediction feature of the first sample image, the density map prediction feature being a tensor. The server inputs the density map prediction feature of the first sample image into the density map generation unit, performs convolution on the density map prediction feature of the first sample image through the convolution layer of the density map generation unit, and outputs a predicted density image of the first sample image.

[0166] In some embodiments, the above process of acquiring the predicted density image of the first sample image can be represented by the following formula (4).

[0167]

[0168] Wherein, is the predicted density image of the first sample image, M1 is the object counting model in this model training, X train is a set of sample images, is a set of reference bounding boxes in this model training, in the case of the first sample image being an initial sample image, is a set of reference bounding boxes in the first model training.

[0169] The second part, the server generates the second sample image based on the predicted density image of the first sample image through the object counting model.

[0170] Wherein, the confidence meets the second confidence condition means that the confidence is greater than or equal to a second confidence threshold, the second confidence threshold being set by a technician according to actual conditions, such as 0.8 or 0.85, and embodiments of the present application do not limit this.

[0171] In a possible implementation, the server performs mean filtering on the predicted density image of the first sample image based on the object counting model to obtain a first filtered image. The server performs local maximum value extraction on the first filtered image based on the object counting model to obtain at least one candidate position in the first filtered image, where the candidate position is a position with a confidence value meeting a second confidence condition. The server generates the second sample image based on the at least one candidate position and the at least one reference bounding box based on the object counting model. In some embodiments, the confidence value of the candidate position is less than a target value, such as 0.9, which is set by a technician according to actual conditions, and the embodiments of the present application do not make any limitation. Accordingly, in the embodiments of the present application, the confidence value of the reference bounding box in the initial sample image is 1.0, and the confidence value of the candidate position is less than the target value, which aims to ensure that the reference bounding box in the initial sample image can be retained after non-maximum suppression. In some embodiments, the above two values can be replaced by other values, as long as the confidence value of the reference bounding box is higher than the maximum confidence value of the candidate position.

[0172] In this implementation, the server determines the candidate position by mean filtering and local maximum value extraction, which can improve the accuracy of the determined candidate position, that is, the position where the sample object may exist.

[0173] For example, the server filters the pixel values of the plurality of pixels in the predicted density image of the first sample image based on the average pixel value of the plurality of pixels in the predicted density image of the first sample image through the object counting model, that is, subtracts the average pixel value of the plurality of pixels from the pixel values of the plurality of pixels in the predicted density image of the first sample image, so as to filter out some noise in the predicted density image of the first sample image, and obtains the first filtered image. The server divides the first filtered image into a plurality of regions through the object counting model, and in some embodiments, the plurality of regions have no overlapping parts, wherein the number of the plurality of regions is set by a technician according to actual conditions, and the embodiments of the present application do not limit this. Each region on the first filtered image can be regarded as a "local" of the first filtered image. The server determines the candidate positions by performing local maximum value extraction on the plurality of regions, that is, determines the pixel point with the maximum pixel value in each region as a candidate position, and obtains the at least one candidate position through the plurality of regions. The server generates at least one candidate bounding box in the first filtered image based on the at least one candidate position through the object counting model, and the size of the candidate bounding box is positively correlated with the size of the at least one reference bounding box. The server fuses the at least one candidate bounding box through the object counting model to obtain the at least one predicted bounding box. The server synthesizes the at least one predicted bounding box to the first sample image through the object counting model to obtain the second sample image.

[0174] The size of the candidate bounding box is positively correlated with the size of the at least one reference bounding box includes two cases. The first case is that when there is only one reference bounding box, the size of the candidate bounding box is the same as the size of the reference bounding box, that is, the larger the size of the reference bounding box, the larger the size of the candidate bounding box. The second case is that when there are a plurality of (more than one) reference bounding boxes, the size of the candidate bounding box is positively correlated with the average size of the plurality of reference bounding boxes, that is, the larger the average size of the plurality of reference bounding boxes, the larger the size of the candidate bounding box. When the candidate bounding box is generated based on the candidate position, the candidate bounding box is generated with the candidate position as the center, and the center of the candidate bounding box is the corresponding candidate position. Synthesizing the predicted bounding box to the first sample image means adding the predicted bounding box to the first sample image, or binding and storing the position information of the predicted bounding box with the first sample image.

[0175] For example, the server performs mean filtering on the pixel values of the plurality of pixel points in the predicted density image of the first sample image based on the average pixel value of the plurality of pixel points in the predicted density image of the first sample image by using the following formula (5), to obtain the first filtered image. The server divides the first filtered image into a plurality of regions by using the object counting model, and performs local maximum extraction in the plurality of regions to determine a set of candidate positions. The server splices the at least one candidate bounding box and the at least one reference bounding box to obtain a set of spliced bounding boxes, which is a process of obtaining a union set, that is, a process of fusing the at least one reference bounding box. The server performs non-maximum suppression on the set of spliced bounding boxes to obtain at least one predicted bounding box, which is shown in the following formula (7).

[0176]

[0177] wherein, is the pixel value of a pixel point in the predicted density image of the first sample image i, is the pixel value of a pixel point in the first filtered image of the first sample image i, mean() is a function of obtaining a mean value.

[0178]

[0179] wherein, is the set of spliced bounding boxes obtained by the current model training, is the set of candidate bounding boxes in the current model training, is the set of reference bounding boxes in the current model training, and union() is a function of obtaining a union set.

[0180]

[0181] wherein, is the set of predicted bounding boxes, and nms() is a function of non-maximum suppression, wherein the principle of non-maximum suppression is to compare the confidence of two mutually overlapping bounding boxes, and in the case that the overlap degree of the two bounding boxes meets the target overlap degree condition, the bounding box with a lower confidence is deleted, wherein the confidence of the bounding box is the confidence of the candidate position corresponding to the bounding box, and the overlap degree meets the target overlap degree means that the overlap degree is greater than or equal to an overlap degree threshold, and the overlap degree threshold is set by a technician according to actual conditions, and embodiments of the present application do not limit this. It should be noted that the confidence of the reference bounding box in the initial sample image is 1.

[0182] 503、The server inputs the second sample image into the object counting model, and generates, by the object counting model, a predicted density image of the second sample image based on the second sample image, the predicted density image of the second sample image being used to represent a predicted distribution of the sample object in the second sample image.

[0183] In a possible implementation, the server inputs the second sample image into the object counting model, and extracts, by the object counting model, features of the second sample image to obtain a second feature map of the second sample image. The server generates, by the object counting model, the predicted density image of the second sample image based on similarities between a second target feature region and other feature regions on the second feature map, the second target feature region being a region where the reference bounding box and the predicted bounding box are located.

[0184] To make the above implementation clearer, the above implementation will be described in two parts.

[0185] In the first part, the server extracts, by the object counting model, features of the second sample image to obtain a first feature map of the second sample image.

[0186] In a possible implementation, the server convolves, by a convolution layer of the object counting model, the second sample image to obtain a feature map of the second sample image.

[0187] In this implementation, the server can extract features of the second sample image by the convolution layer of the object counting model. Since the convolution operation is fast, the efficiency of obtaining the feature map of the second sample image by using this way is high.

[0188] For example, the object counting model comprises a feature extraction unit, and the server inputs the second sample image into the feature extraction unit of the object counting model. The server convolves the second sample image through the convolutional layer of the feature extraction unit, that is, the convolution kernel on the convolutional layer of the feature extraction unit slides in the second sample image, and the convolution kernel convolves the region covered by the convolution kernel in the second sample image during the sliding process to obtain the feature map of the second sample image. The number of convolution kernels on the convolutional layer of the feature extraction unit is one or more, which is not limited in the embodiments of the present application. In the case where the second sample image comprises a plurality of color channels, the number of convolution kernels on the convolutional layer of the feature extraction unit is also a plurality, and the plurality of convolution kernels are used to extract features of the plurality of color channels of the second sample image to obtain the color feature map of each color channel. The server fuses the color feature maps of the plurality of color channels through the feature extraction unit to obtain the feature map of the second sample image. In some embodiments, when the server fuses the color feature maps of the plurality of color channels through the feature extraction unit to obtain the feature map of the second sample image, a weighted summation method can be used. In some embodiments, the feature extraction unit is a feature extractor based on a convolutional neural network (CNN).

[0189] In a possible implementation, the server encodes the second sample image based on an attention mechanism through the object counting model to obtain the feature map of the second sample image.

[0190] In this implementation, the server can extract features based on an attention mechanism through the object counting model. The introduction of the attention mechanism enables the object counting model to focus on regions with higher importance in the second sample image, and the obtained feature map can more accurately reflect the features of the second sample image.

[0191] For example, the object counting model comprises a feature extraction unit, and the server inputs the second sample image into the feature extraction unit of the object counting model. The server encodes multiple parts of the second sample image based on an attention mechanism through the feature extraction unit to obtain a feature map of the second sample image. For example, the server inputs the second sample image into the feature extraction unit, embeds and encodes multiple parts of the second sample image through the feature extraction unit to obtain multiple embedding vectors, one embedding vector corresponding to one part of the second sample image, and the embedding vectors being used to represent the positions of the parts in the second sample image and the contents of the parts. The server inputs the multiple embedding vectors into the feature extraction unit, linearly transforms the multiple embedding vectors through three linear transformation matrices of the feature extraction unit to obtain a query vector, a key vector, and a value vector corresponding to each part of the second sample image. The server obtains attention weights of the multiple parts of the second sample image based on the query vector and the key vector corresponding to the multiple parts of the second sample image through the feature extraction unit. The server obtains attention encoding vectors of the multiple parts of the second sample image based on the attention weights of the multiple parts of the second sample image and the value vectors of the multiple parts of the second sample image through the feature extraction unit. The attention encoding vectors of the multiple parts of the second sample image are combined according to the positions of the parts in the second sample image to obtain the feature map of the second sample image.

[0192] In a possible implementation, the server fully connects the second sample image through a fully connected layer of the object counting model to obtain the feature map of the second sample image. In this implementation, the server can obtain the feature map of the second sample image by fully connecting the second sample image through the object counting model, and the fully connection can extract the features of the second sample image as a whole to obtain the feature map that can reflect the features of the second sample image as a whole.

[0193] For example, the object counting model comprises a feature extraction unit, and the server inputs the second sample image into the feature extraction unit of the object counting model. The server encodes multiple parts of the second sample image based on an attention mechanism through the feature extraction unit to obtain a feature map of the second sample image. For example, the server inputs the second sample image into the feature extraction unit, embeds and encodes multiple parts of the second sample image through the feature extraction unit to obtain multiple embedding vectors, one embedding vector corresponding to one part of the second sample image, and the embedding vectors being used to represent the positions of the parts in the second sample image and the contents of the parts. The server inputs the multiple embedding vectors into the feature extraction unit, linearly transforms the multiple embedding vectors through three linear transformation matrices of the feature extraction unit to obtain a query vector, a key vector, and a value vector corresponding to each part of the second sample image. The server obtains attention weights of the multiple parts of the second sample image based on the query vector and the key vector corresponding to the multiple parts of the second sample image through the feature extraction unit. The server obtains attention encoding vectors of the multiple parts of the second sample image based on the attention weights of the multiple parts of the second sample image and the value vectors of the multiple parts of the second sample image through the feature extraction unit. The attention encoding vectors of the multiple parts of the second sample image are combined according to the positions of the parts in the second sample image to obtain the feature map of the second sample image.

[0194] It should be noted that the server can obtain the feature map of the second sample image in any of the above manners, or use a ViT (Vision Transformer) or other conventional image feature extractor to perform feature extraction on the second sample image. Of course, as science and technology develop, the server can also obtain the feature map in other manners, which are not limited by the embodiments of the application.

[0195] The second part, the server, generates a predicted density image of the second sample image based on the similarity between the first target feature region and other feature regions on the first feature map through the object counting model.

[0196] The feature map of the second sample image has the same size as the second sample image, and the similarity between the first target feature region and other feature regions represents the confidence of the existence of sample objects in other feature regions.

[0197] In a possible implementation, the server obtains a target sub-feature map from the feature map of the second sample image through the object counting model, where the target sub-feature map is a part of the feature map of the second sample image corresponding to the first target feature region, i.e., a part of the second sample image corresponding to the reference bounding box. The server determines the similarity between the first target feature region and other feature regions based on the target sub-feature map through the object counting model. For example, the server determines the similarity between the first target feature region and other feature regions by sliding the target sub-feature map on the feature map of the second sample image through the object counting model, where other feature regions are regions covered by the target sub-feature map during the sliding process. The server generates a density map prediction feature of the second sample image based on the similarity between the first target feature region and other feature regions. The server convolves the density map prediction feature of the second sample image through the object counting model to obtain a predicted density image of the second sample image. When determining the similarity between the first target feature region and other feature regions, i.e., performing correlation operation on the first target feature region and other feature regions to obtain the similarity between the first target feature region and other feature regions, the server can perform correlation operation on the first target feature region and other feature regions through merging and connection.

[0198] For example, the object counting model comprises a local feature acquisition unit, a matching unit, and a density map generation unit. The server acquires, from the feature map of the second sample image, an initial sub-feature map corresponding to the initial feature region on the feature map of the second sample image, i.e., a part corresponding to the reference bounding box on the second sample image, by using the object counting model. The server obtains a target sub-feature map of a target size by performing pooling on the initial sub-feature map by using the local feature acquisition unit, where the purpose of performing pooling on the initial sub-feature map is to adjust the initial sub-feature map to the target size, which is set by a technician according to actual conditions, and embodiments of the present application do not limit this. The server inputs the target sub-feature map and the feature map of the second sample image into the matching unit, determines the similarity between the first target feature region and other feature regions by using the matching unit, i.e., controls the target sub-feature map to slide on the feature map of the second sample image, determines the similarity between the first target feature region and other feature regions, and outputs a density map prediction feature of the second sample image, which is a tensor. The server inputs the density map prediction feature of the second sample image into the density map generation unit, performs convolution on the density map prediction feature of the second sample image by using the convolution layer of the density map generation unit, and outputs a predicted density image of the second sample image.

[0199] 504. The server trains the object counting model based on the first difference information between the predicted density image of the second sample image and the reference density image, where the reference density image is used to represent the actual distribution of the sample object in the second sample image.

[0200] In a possible implementation, the server brings the difference information between the predicted density image of the second sample image and the reference density image into a target loss function, adjusts the model parameters of the object counting model by using a gradient descent method, and thus completes a round of training of the object counting model.

[0201] The technical solutions provided in the embodiments of the present application will be described below in combination with Figure 6 , steps 301-304 and 501-504 described above.

[0202] For example, the object counting model comprises a local feature acquisition unit, a matching unit, and a density map generation unit. The server acquires, from the feature map of the second sample image, an initial sub-feature map corresponding to the initial feature region on the feature map of the second sample image, i.e., a part corresponding to the reference bounding box on the second sample image, by using the object counting model. The server obtains a target sub-feature map of a target size by performing pooling on the initial sub-feature map by using the local feature acquisition unit, where the purpose of performing pooling on the initial sub-feature map is to adjust the initial sub-feature map to the target size, which is set by a technician according to actual conditions, and embodiments of the present application do not limit this. The server inputs the target sub-feature map and the feature map of the second sample image into the matching unit, determines the similarity between the first target feature region and other feature regions by using the matching unit, i.e., controls the target sub-feature map to slide on the feature map of the second sample image, determines the similarity between the first target feature region and other feature regions, and outputs a density map prediction feature of the second sample image, which is a tensor. The server inputs the density map prediction feature of the second sample image into the density map generation unit, performs convolution on the density map prediction feature of the second sample image by using the convolution layer of the density map generation unit, and outputs a predicted density image of the second sample image. Figure 6The server obtains an initial sample image 601 including at least one reference bounding box 6011, the initial sample image 601 belonging to a sample image set 602. The server inputs the initial sample image 601 into the object counting model 603, and outputs a predicted density image 604 of the initial sample image 601 by the object counting model 603. The server performs mean filtering and local maximum extraction on the predicted density image 604 of the initial sample image 601 to generate at least one candidate bounding box. Non-maximum suppression is performed on the at least one candidate bounding box to obtain at least one predicted bounding box. The at least one predicted bounding box 605 is synthesized on a sample image 606, which is a sample image used in the next round of model training.

[0203] Optionally, after step 504, the server can also adjust the hyperparameters of the object counting model through the following step 505.

[0204] 505. The server adjusts the hyperparameters of the object counting model based on the verification image.

[0205] The hyperparameters of the object counting model include learning rate, model parameters, model node number, and model iteration number of the object counting model, etc., which affect the training effect of the object counting model.

[0206] In a possible implementation, the server inputs a verification image into the object counting model, the verification image including a plurality of verification objects and at least one verification bounding box surrounding the verification objects. The server processes the verification image by the object counting model to obtain a density image of the verification image. The server performs a plurality of iterations based on the density image of the verification image by the object counting model, and outputs a predicted number of the verification objects in the verification image. The server adjusts the hyperparameters of the object counting model based on a second difference information between the predicted number of the verification objects in the verification image and an actual number of the verification objects in the verification image.

[0207] In some embodiments, when adjusting the hyperparameters of the object counting model based on the second difference information, the adjustment can be performed in a grid search or random search manner, wherein the grid search is to train the object counting model by using different hyperparameter combinations, and to select appropriate hyperparameters from the plurality of hyperparameter combinations according to the second difference information. The random search is to randomly generate hyperparameter combinations, and to select the best hyperparameter combination based on the second difference information. Alternatively, the skilled person can also adjust the hyperparameters according to experience, which is not limited in the embodiments of the present application.

[0208] It should be noted that the above steps 501-505 are described by taking the object counting model as an example for one round of model training. In training the object counting model, K rounds of training are often required. The above steps 301-304 are the first round of training, and the steps 501-505 are the kth round of training, 1 < k ≤ K, k and K are positive integers, and K is set by the technician according to the actual situation, which is not limited by the embodiments of the present application.

[0209] The technical solutions provided by the above steps 301-304 are summarized as follows: in the first round, the initial sample image, the real labeled sample object, and the real labeled reference bounding box are used to train the object counting model, and the trained object counting model is used for inference to expand the bounding box, thereby obtaining a sample image including a predicted bounding box, and the sample image including the predicted bounding box is used as a supplementary input in the next round.

[0210] The technical solutions provided by the above steps 501-505 are summarized as follows: in other rounds, the real labeled sample object and the predicted bounding box obtained in the last round are used to iteratively train and infer the object counting model. Through this iterative training-inference mode, the final performance of the object counting model is improved.

[0211] Optionally, after K rounds of training of the object counting model, the server can also verify the effect of the object counting model through a test image, as follows:

[0212] In one possible implementation, the server inputs a test image into the object counting model, the test image including a plurality of test objects and at least one test bounding box surrounding the test objects. The server processes the test image through the object counting model to obtain a density image of the test image. The server iterates through the object counting model based on the density image of the test image for multiple rounds to output a predicted number of the test objects in the test image. It should be noted that when the object counting model is tested based on the test image, the network parameters of the object counting model do not need to be updated.

[0213] In some embodiments, the test objects and the sample objects are different types of objects. That is, the test objects counted in the test stage are completely new types of objects that the object counting model has not learned in the training stage.

[0214] For example, the server inputs a test image into the object counting model. The server extracts features of the test image through the object counting model to obtain a feature map of the test image. The server generates a set of predicted density images of the test image through the object counting model through a multi-round iteration process. The server outputs the number of test objects in the test image based on the set of predicted density images of the test image through the object counting model.

[0215] For example, the server generates the set of predicted density images of the test image through Formula (8) below and obtains the number of test objects through Formula (9) below.

[0216]

[0217] wherein, is the predicted density image of the test image, M K is the object counting model obtained through K rounds of training, X test is the set of test images, is the set of reference bounding boxes obtained through the Kth round of iteration.

[0218]

[0219] wherein, is the number of test objects, sum() is a summation operation, and the summation process is to add the numerical values of each element corresponding to the element.

[0220] In some embodiments, the server can only perform training of the object counting model on the sample images in the first round (steps 301-304) and only perform iterations on the test images in the subsequent iterations, so as to improve the counting performance on the test images.

[0221] The technical solutions provided by the embodiments of the present application will be described below in combination with Figure 7 and the various optional embodiments described above.

[0222] Referring to Figure 7 , the method includes two parts of training and testing. In the training process, the data input into the object counting model 701 includes a sample image 702, a position of a sample object in the sample image 703, and a position of a reference bounding box in the sample image 704. In the testing process, the data input into the trained object counting model 705 includes a test image 706 and a position of a reference bounding box in the test image 707, and the trained object counting model 705 outputs a counting result, i.e., the number of test objects in the test image.

[0223] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and will not be described again here.​

[0224] During the experiment, the technical solution provided by the embodiments of the present application is verified on the Few-Shot Counting-147 (FSC-147) dataset. During the experiment, the technical solution provided by the embodiments of the present application sets an extremely few-shot counting condition, that is, only one reference bounding box is given on the sample image and the test image (in contrast, the original FSC-147 dataset gives three reference bounding boxes on the sample image and the test image), and other conditions are the same as the task setting conditions in the FSC-147 dataset. It is found through experimental testing that the technical solution provided by the embodiments of the present application can achieve an almost equivalent counting MAE (Mean Absolute Error) index under the condition of giving only one reference bounding box compared with not using the technical solution provided by the embodiments of the present application and giving three reference bounding boxes. It can be seen that the technical solution provided by the embodiments of the present application can effectively solve the problem of extremely few-shot counting (in this experiment, it refers to giving only one reference bounding box), and can further reduce the reference bounding box labeling cost in the few-shot counting task to a certain extent.

[0225] Through the technical solution provided by the embodiments of the present application, when the object counting model is trained, the first sample image used in the last round of model training is processed to obtain a second sample image. This process is equivalent to further mining the information on the first sample image, and the obtained second sample image includes the reference bounding box on the first sample image and the predicted bounding box obtained by mining. The second sample image is input into the object counting model, and the object counting model processes the second sample image to obtain a predicted density image of the second sample image. Subsequently, the object counting model is trained based on the first difference information between the predicted density image and the reference density image of the second sample image. This process is to train the object counting model by using the second sample image obtained by further mining the first sample image. The training process does not completely depend on labeling, and the existing labeling (reference bounding box) can be expanded through self-mining, so that the object counting model learns the features of multiple objects and improves the final counting effect of the object counting model.

[0226] Through the technical solution provided by the embodiments of the present application, based on the given few bounding boxes of target objects, automatic prediction and supplement of more objects can be performed, so as to realize automatic mining and expansion of object features, obtain more feature information about objects, and finally realize more accurate total quantity prediction.

[0227] The embodiments of the present application also provide a technical solution for counting using the object counting model, which is described with reference to Figure 8Taking the server as an example, the method comprises the following steps.

[0228] 801、The server inputs a target image into the object counting model, the target image comprising a plurality of target objects and at least one target bounding box surrounding the target objects.

[0229] The object counting model in step 801 is a trained object counting model, such as the object counting model obtained after K rounds of iteration training.

[0230] In some embodiments, the target objects and the sample objects are different types of objects. In some embodiments, the target objects and the sample objects are different types of objects. That is, the target objects counted using the stage are completely new types of objects, and the object counting model has not learned about these objects at all during the training stage. Therefore, for the object counting model trained under this task definition, it can predict the total number of objects of all categories on various images with completely new categories under the condition that only the target bounding boxes of a few required categories of target objects are given, which has a very wide practical application value.

[0231] In some embodiments, the target image is also referred to as a query image.

[0232] 802、The server processes the target image through the object counting model to obtain a density image of the target image.

[0233] The method used by the server to obtain the density image of the target image is the same as the method used to obtain the density image of the initial sample image in step 303, and the implementation process is described in detail in the description of step 303, which will not be repeated here.

[0234] 803、The server outputs the predicted number of target objects in the target image through the object counting model based on the density image of the target image.

[0235] In one possible implementation, the server generates a set of predicted density images of the target image through the object counting model through a plurality of rounds of iteration. The server outputs the number of target objects in the target image based on the set of predicted density images of the target image through the object counting model.

[0236] Through steps 801-803, the server can output the number of objects in an image through the object counting model, which is efficient.

[0237] Figure 9 is a structural schematic diagram of an object counting model training device provided by an embodiment of the present application, which is described in detail in the description of the object counting model training device. Figure 9The apparatus comprises an input module 901, a sample image generation module 902, a density image generation module 903, and a training module 904.

[0238] The input module 901 is configured to input a first sample image into an object counting model, the first sample image being a sample image used in a previous round of model training, the first sample image comprising a plurality of sample objects and at least one reference bounding box surrounding the sample objects.

[0239] The sample image generation module 902 is configured to generate, by the object counting model, a second sample image based on the first sample image, the second sample image comprising the at least one reference bounding box and at least one predicted bounding box, the area surrounded by the predicted bounding box being an area in which the sample objects exist with a confidence meeting a first confidence condition.

[0240] The density image generation module 903 is configured to input the second sample image into the object counting model, and generate, by the object counting model, a predicted density image of the second sample image based on the second sample image, the predicted density image of the second sample image being used to represent a predicted distribution of the sample objects in the second sample image.

[0241] The training module 904 is configured to train the object counting model based on first difference information between the predicted density image of the second sample image and a reference density image, the reference density image being used to represent an actual distribution of the sample objects in the second sample image.

[0242] In a possible implementation, the sample image generation module 902 is configured to process the first sample image to obtain a predicted density image of the first sample image, the predicted density image of the first sample image being used to represent a predicted distribution of the sample objects in the first sample image. The second sample image is generated based on the predicted density image of the first sample image.

[0243] In a possible implementation, the sample image generation module 902 is configured to perform feature extraction on the first sample image to obtain a first feature map of the first sample image. The predicted density image of the first sample image is generated based on a similarity between a first target feature region on the first feature map and other feature regions, the first target feature region being a region in which the reference bounding box is located.

[0244] In a possible implementation, the sample image generation module 902 is configured to perform mean filtering on the predicted density image of the first sample image to obtain a first filtered image. Local maximum value extraction is performed on the first filtered image to obtain at least one candidate position in the first filtered image, the candidate position being a position with a confidence that meets a second confidence condition for the presence of the sample object. The second sample image is generated based on the at least one candidate position and the at least one reference bounding box in the first filtered image.

[0245] In a possible implementation, the sample image generation module 902 is configured to generate at least one candidate bounding box in the first filtered image based on the at least one candidate position, the size of the candidate bounding box being positively correlated with the size of the at least one reference bounding box. The at least one predicted bounding box is obtained by fusing the at least one candidate bounding box. The second sample image is obtained by synthesizing the at least one predicted bounding box to the first sample image.

[0246] In a possible implementation, the density image generation module 903 is configured to perform feature extraction on the second sample image to obtain a second feature map of the second sample image. The predicted density image of the second sample image is generated based on the similarity between a second target feature region and other feature regions on the second feature map, the second target feature region being a region where the reference bounding box and the predicted bounding box are located.

[0247] In a possible implementation, the apparatus further includes:

[0248] The reference density image generation module 903 is configured to generate a position image of an initial sample image based on annotation information of the initial sample image, the initial sample image being a sample image used in first-round model training, and the annotation information including a position of the sample object on the initial sample image. Convolution is performed on the position image of the initial sample image to obtain the reference density image.

[0249] In a possible implementation, the apparatus further includes:

[0250] The hyperparameter adjustment module is configured to input a verification image into the object counting model, the verification image including a plurality of verification objects and at least one verification bounding box surrounding the verification objects. The verification image is processed by the object counting model to obtain a density image of the verification image. The object counting model is iterated based on the density image of the verification image for a plurality of rounds to output a predicted number of the verification objects in the verification image. A second difference information between the predicted number of the verification objects in the verification image and an actual number of the verification objects in the verification image is used to adjust hyperparameters for training the object counting model.

[0251] In a possible implementation, the apparatus further includes:

[0252] The counting module is configured to input a target image into the object counting model, the target image including a plurality of target objects and at least one target bounding box surrounding the target objects. The target image is processed by the object counting model to obtain a density image of the target image. Based on the density image of the target image, a plurality of iterations are performed by the object counting model to output a predicted number of the target objects in the target image.

[0253] It should be noted that the training apparatus of the object counting model provided in the above embodiments is only used as an example to illustrate the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above-described functions. In addition, the training apparatus of the object counting model and the training method of the object counting model provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0254] Through the technical solutions provided in the embodiments of the present application, when the object counting model is trained, the first sample image used in the last round of model training is processed to obtain a second sample image. This process is equivalent to further mining the information on the first sample image, and the obtained second sample image includes the reference bounding box on the first sample image and the predicted bounding box obtained by mining. The second sample image is input into the object counting model, and the second sample image is processed by the object counting model to obtain a predicted density image of the second sample image. The object counting model is subsequently trained based on the first difference information between the predicted density image and the reference density image of the second sample image. This process is equivalent to training the object counting model using the second sample image obtained by further mining the first sample image. The training process does not completely rely on labeling, and the existing labeling can be expanded through self-mining, so that the object counting model learns the features of multiple objects and improves the final counting effect of the object counting model.

[0255] The embodiments of the present application provide a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal will be introduced first as follows:

[0256] Figure 10 FIG. 1 is a structural schematic diagram of a terminal provided in an embodiment of the present application.

[0257] Generally, the terminal 1000 includes one or more processors 1001 and one or more memories 1002.

[0258] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1001 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1001 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing of content to be displayed by the display screen. In some embodiments, the processor 1001 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0259] The memory 1002 can include one or more computer-readable storage media that can be non-transitory. The memory 1002 can also include high-speed random access memory and nonvolatile, computer-readable storage media such as one or more magnetic disk storage devices, flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one computer program for being executed by the processor 1001 to implement the training method of the object counting model provided by the method embodiments in the present application.

[0260] In some embodiments, the terminal 1000 can also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002, and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0261] The peripheral interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002 and the peripheral interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002 and the peripheral interface 1003 can be implemented on a separate chip or circuit board, and the present embodiments are not limited to this.

[0262] The radio frequency circuit 1004 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.

[0263] The display screen 1005 is used to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on or above the surface of the display screen 1005. The touch signals can be input as control signals to the processor 1001 for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0264] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal.

[0265] The audio circuit 1007 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals input to the processor 1001 for processing, or input to the radio frequency circuit 1004 to realize voice communication.

[0266] The power supply 1008 is used to supply power to each component in the terminal 1000. The power supply 1008 can be alternating current, direct current, disposable batteries or rechargeable batteries.

[0267] In some embodiments, the terminal 1000 further comprises one or more sensors 1009. The one or more sensors 1009 include, but are not limited to, an acceleration sensor 1100, a gyroscope sensor 1011, a pressure sensor 1012, an optical sensor 1013, and a proximity sensor 1014.

[0268] The acceleration sensor 1100 can detect the acceleration magnitude in three coordinate axes of a coordinate system established by the terminal 1000.

[0269] The gyroscope sensor 1011 can detect the body direction and rotation angle of the terminal 1000. The gyroscope sensor 1011 can cooperate with the acceleration sensor 1100 to collect the 3D action of the user on the terminal 1000.

[0270] The pressure sensor 1012 can be arranged on the side frame of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1012 is arranged on the side frame of the terminal 1000, the holding signal of the user on the terminal 1000 can be detected, and the left-hand or right-hand recognition or shortcut operation can be performed by the processor 1001 according to the holding signal collected by the pressure sensor 1012. When the pressure sensor 1012 is arranged on the lower layer of the display screen 1005, the controllable control on the UI interface can be controlled by the processor 1001 according to the pressure operation of the user on the display screen 1005.

[0271] The optical sensor 1013 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1013.

[0272] The proximity sensor 1014 is used to collect the distance between the user and the front of the terminal 1000.

[0273] Those skilled in the art can understand that, Figure 10 The structure shown in the above description does not constitute a limitation on the terminal 1000, and can include more or less components than the drawings, or combine certain components, or use different component arrangements.

[0274] The above computer device can also be implemented as a server, and the structure of the server will be introduced as follows:

[0275] Figure 11is a structural schematic diagram of a server provided by an embodiment of the present application. The server 1100 can have great differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1101 and one or more memories 1102, where the one or more memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the method provided by each of the above methods. Of course, the server 1100 can also include other components for implementing device functions, which are not described here.

[0276] In an example embodiment, a computer-readable storage medium, for example, a memory including a computer program, is also provided, where the computer program can be executed by a processor to complete the training method of the object counting model in the above embodiments. For example, the computer-readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0277] In an example embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium, where the processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code to make the computer device execute the training method of the object counting model.

[0278] In some embodiments, the computer program related to the embodiments of the present application can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple places and interconnected through a communication network, which can constitute a blockchain system.

[0279] Those of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing relevant hardware, which can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a Read-Only Memory, a magnetic disk or an optical disk, etc.

[0280] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for an object counting model, characterized in that, The method includes: The first sample image used in the previous round of model training is input into the object counting model. The first sample image includes multiple sample objects and at least one reference bounding box. One reference bounding box surrounds one sample object. The at least one reference bounding box includes a reference bounding box that is ground truth labeled on the initial sample image and a predicted bounding box that is predicted based on the reference bounding box on the initial sample image during model training. The first round of training is based on the initial sample image, the ground truth labeled sample objects, and the ground truth labeled reference bounding boxes. The object counting model is used to obtain a first predicted density image representing the predicted distribution of sample objects in the first sample image; The first predicted density image is subjected to mean filtering to obtain the first filtered image; The first filtered image is divided into multiple regions with no overlap. In the multiple regions, at least one candidate location is determined where the confidence of the sample object meets the confidence condition. The confidence value of the reference bounding box is higher than the confidence value of the candidate location. Based on at least one candidate location, the predicted bounding box obtained from this round of training is obtained; The predicted bounding boxes are composited into the first sample image to obtain the second sample image, where the region where the predicted bounding boxes are located is a region other than the region where the reference bounding boxes are located. The second sample image is input into the object counting model, and the object counting model generates a second prediction density image based on the second sample image to represent the predicted distribution of sample objects in the second sample image. The object counting model is trained based on the first difference information between the second predicted density image and the reference density image, wherein the reference density image is used to represent the actual distribution of the sample objects in the second sample image.

2. The method according to claim 1, characterized in that, The first predicted density image obtained to represent the predicted distribution of sample objects in the first sample image includes: The first sample image is processed to obtain the first predicted density image of the first sample image.

3. The method according to claim 2, characterized in that, The step of processing the first sample image to obtain the first predicted density image of the first sample image includes: Feature extraction is performed on the first sample image to obtain the first feature map of the first sample image; Based on the similarity between the first target feature region and other feature regions on the first feature map, a first predicted density image of the first sample image is generated, wherein the first target feature region is the region where the reference bounding box is located.

4. The method according to claim 2, characterized in that, The method further includes: Local maximum extraction is performed on the first filtered image to obtain at least one candidate position in the first filtered image, wherein the candidate position is the position where the confidence of the sample object meets the second confidence condition.

5. The method according to claim 4, characterized in that, The step of obtaining the predicted bounding box based on at least one candidate location in this round of training includes: Based on the at least one candidate location, at least one candidate bounding box is generated in the first filtered image, the size of the candidate bounding box being positively correlated with the size of the at least one reference bounding box; The at least one candidate bounding box is fused to obtain at least one predicted bounding box; The step of synthesizing the predicted bounding box into the first sample image to obtain the second sample image includes: The at least one predicted bounding box is synthesized into the first sample image to obtain the second sample image.

6. The method according to claim 1, characterized in that, The method further includes: The target image is input into the object counting model, the target image including a plurality of target objects and at least one target bounding box surrounding the target objects; The target image is processed using the object counting model to obtain a density image of the target image; The object counting model iterates through multiple rounds based on the density image of the target image to output the predicted number of target objects in the target image.

7. The method according to claim 1, characterized in that, The step of generating a second prediction density image based on the second sample image to represent the predicted distribution of sample objects in the second sample image includes: Feature extraction is performed on the second sample image to obtain the second feature map of the second sample image; Based on the similarity between the second target feature region and other feature regions on the second feature map, a second prediction density image of the second sample image is generated, wherein the second target feature region is the region where the reference bounding box and the predicted bounding box are located.

8. The method according to claim 1, characterized in that, The method for generating the reference density image includes: Based on the annotation information of the initial sample image, a position image of the initial sample image is generated, wherein the annotation information includes the position of the sample object on the initial sample image; The reference density image is obtained by convolving the position image of the initial sample image.

9. The method according to claim 1, characterized in that, Based on the first difference information between the second predicted density image and the reference density image, after training the object counting model for one round, the method further includes: The verification image is input into the object counting model, the verification image including a plurality of verification objects and at least one verification bounding box surrounding the verification objects; The verification image is processed using the object counting model to obtain a density image of the verification image; The object counting model is used to perform multiple iterations based on the density image of the verification image to output the predicted number of verification objects in the verification image. The hyperparameters of the object counting model are adjusted based on the second difference information between the predicted number of the verification objects in the verification image and the actual number of the verification objects in the verification image.

10. A training device for an object counting model, characterized in that, The device includes: The input module is used to input the first sample image used in the previous round of model training into the object counting model. The first sample image includes multiple sample objects and at least one reference bounding box. One reference bounding box surrounds one sample object. The at least one reference bounding box includes a reference bounding box that is ground truth labeled on the initial sample image and a predicted bounding box that is predicted based on the reference bounding box on the initial sample image during model training. The first round of training is based on the initial sample image, the ground truth labeled sample objects, and the ground truth labeled reference bounding boxes. The sample image generation module is used to obtain a first prediction density image representing the predicted distribution of sample objects in the first sample image using the object counting model; to perform mean filtering on the first prediction density image to obtain a first filtered image; to divide the first filtered image into multiple regions without overlapping parts, and to determine at least one candidate location in the multiple regions where the confidence of a sample object meets the confidence condition, wherein the confidence value of the reference bounding box is higher than the confidence value of the candidate location; to obtain the predicted bounding box obtained from the current training prediction based on at least one candidate location; and to synthesize the predicted bounding box into the first sample image to obtain a second sample image, wherein the region where the predicted bounding box is located is a region other than the region where the reference bounding box is located. The density image generation module is used to input the second sample image into the object counting model, and generate a second predicted density image based on the second sample image to represent the predicted distribution of sample objects in the second sample image; The training module is used to train the object counting model based on the first difference information between the second predicted density image and the reference density image, wherein the reference density image is used to represent the actual distribution of the sample objects in the second sample image.

11. The apparatus according to claim 10, characterized in that, The sample image generation module is used for: The first sample image is processed to obtain the first predicted density image of the first sample image.

12. The apparatus according to claim 11, characterized in that, The sample image generation module is used for: Feature extraction is performed on the first sample image to obtain the first feature map of the first sample image; Based on the similarity between the first target feature region and other feature regions on the first feature map, a first predicted density image of the first sample image is generated, wherein the first target feature region is the region where the reference bounding box is located.

13. The apparatus according to claim 11, characterized in that, The sample image generation module is used for: Local maximum extraction is performed on the first filtered image to obtain at least one candidate position in the first filtered image, wherein the candidate position is the position where the confidence of the sample object meets the second confidence condition.

14. The apparatus according to claim 13, characterized in that, The sample image generation module is used for: Based on the at least one candidate location, at least one candidate bounding box is generated in the first filtered image, the size of the candidate bounding box being positively correlated with the size of the at least one reference bounding box; The at least one candidate bounding box is fused to obtain at least one predicted bounding box; The at least one predicted bounding box is synthesized into the first sample image to obtain the second sample image.

15. The apparatus according to claim 10, characterized in that, The device further includes a counting module for: The target image is input into the object counting model, the target image including a plurality of target objects and at least one target bounding box surrounding the target objects; The target image is processed using the object counting model to obtain a density image of the target image; The object counting model iterates through multiple rounds based on the density image of the target image to output the predicted number of target objects in the target image.

16. The apparatus according to claim 10, characterized in that, The density image generation module is used for: Feature extraction is performed on the second sample image to obtain the second feature map of the second sample image; Based on the similarity between the second target feature region and other feature regions on the second feature map, a second prediction density image of the second sample image is generated, wherein the second target feature region is the region where the reference bounding box and the predicted bounding box are located.

17. The apparatus according to claim 10, characterized in that, The device further includes a reference density image generation module for: Based on the annotation information of the initial sample image, a position image of the initial sample image is generated, wherein the annotation information includes the position of the sample object on the initial sample image; The reference density image is obtained by convolving the position image of the initial sample image.

18. The apparatus according to claim 10, characterized in that, The device also includes a hyperparameter adjustment module for: The verification image is input into the object counting model, the verification image including a plurality of verification objects and at least one verification bounding box surrounding the verification objects; The verification image is processed using the object counting model to obtain a density image of the verification image; The object counting model is used to perform multiple iterations based on the density image of the verification image to output the predicted number of verification objects in the verification image. The hyperparameters of the object counting model are adjusted based on the second difference information between the predicted number of the verification objects in the verification image and the actual number of the verification objects in the verification image.

19. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the training method of the object counting model as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the training method for the object counting model as described in any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the training method for the object counting model according to any one of claims 1 to 9.

Images