A counting method and device, electronic equipment and storage medium
By performing feature extraction and distribution position information processing on the input image, a general counting of objects of different categories is achieved, and the problem of low efficiency in statistics of various categories of objects in the prior art is solved, and the universality and accuracy of counting are improved.
Patent Information
- Application Number
- CN202210115053.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-01-29
AI Technical Summary
Existing visual counting technology requires collecting a large amount of data when counting multiple categories of objects, which consumes a lot of human resources, is inefficient, and the model relies on the characteristics of objects of a specific category and is not universal.
By extracting the input image features, the first feature map and the second feature map are obtained, and the object features are extracted using the marked position information of the object to be counted, the distribution position and feature distribution map are determined, and the total number of objects to be counted is determined based on these images. Implement a general count of objects of different categories.
It improves the versatility and efficiency of counting, reduces the dependence on specific category models, and can accurately count the number of objects of different categories.
Smart Images

Figure CN114445778B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a counting method and device, an electronic device and a storage medium. Background Art
[0002] With the development of artificial intelligence technology, visual counting has been widely used in counting statistics. Visual counting can count the number and distribution of objects to be counted in an image.
[0003] In related technologies, visual counting work is mainly focused on some specific categories, such as pedestrians, cars, cells, animals, etc. In these technical works, most of them train a counting model for one category of objects, and the obtained counting model is not universal. If the application scenario needs to count multiple categories of objects, it is necessary to collect a large amount of data to train the model, which consumes a lot of human resources and has low efficiency. Summary of the invention
[0004] The present disclosure proposes a visual counting solution.
[0005] According to one aspect of the present disclosure, there is provided a counting method, comprising:
[0006] Performing feature extraction on the input image to obtain a first feature map and a second feature map;
[0007] extracting object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image;
[0008] Determining, in the first feature map, distribution position information of at least two objects to be counted according to the object features;
[0009] Determine a feature distribution map of the object to be counted from the second feature map using the distribution position information;
[0010] The total number of objects to be counted is determined based on the characteristic distribution map.
[0011] In a possible implementation manner, extracting object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image includes:
[0012] When there are at least two marked objects to be counted, extracting first features of each marked object to be counted from the first feature map according to position information of the at least two marked objects to be counted;
[0013] The extracted first features of each object to be counted are fused to obtain the object features of the object to be counted.
[0014] In a possible implementation manner, determining, in the first feature map according to the object features, distribution position information of at least two objects to be counted includes:
[0015] Calculating the similarity between the object feature and the feature at each position in the second feature map;
[0016] Based on the similarity of the features at each position, the distribution position information of the at least two objects to be counted in the first feature map is obtained.
[0017] In a possible implementation manner, the using the distribution position information to determine a feature distribution map of the to-be-counted object from the second feature map includes:
[0018] The distribution position information is multiplied by the second feature map to obtain a feature distribution map of the objects to be counted.
[0019] In a possible implementation manner, determining the total number of objects to be counted based on the characteristic distribution graph includes:
[0020] Upsampling the feature distribution map to obtain a density map of objects to be counted;
[0021] The total number of objects to be counted is determined based on the density map.
[0022] In a possible implementation, extracting features from the input image to obtain a first feature map and a second feature map includes:
[0023] Perform initial feature extraction on the input image to obtain an initial feature map;
[0024] Performing a first feature extraction on the initial feature map to obtain a first feature map;
[0025] Perform a second feature extraction on the initial feature map to obtain a second feature map.
[0026] In a possible implementation, the input image is a sample image pre-labeled with the distribution positions of the at least two objects to be counted, the counting method is implemented based on a neural network, and the parameter updating process of the neural network includes:
[0027] Determine a first loss based on the distribution position information and the distribution positions of at least two objects to be counted in the sample image that are pre-marked;
[0028] Determining a second loss based on the density map of the objects to be counted and the pre-labeled density map of the objects to be counted;
[0029] Based on the first loss and / or the second loss, parameters in the neural network are updated.
[0030] In a possible implementation, the counting method is implemented based on a neural network, and a method for constructing a training sample and / or a test sample of the neural network includes:
[0031] Obtaining at least one target sub-image of the target object according to the annotation information of the first sample image;
[0032] Paste the target sub-image into the second sample image to obtain a composite image and pasting position information of the target sub-image in the composite image;
[0033] The pasting position information is used as annotation information in a synthetic image to generate a synthetic sample image.
[0034] In a possible implementation manner, obtaining at least one target sub-image of the target object according to the annotation information of the first sample image includes:
[0035] extracting an image in the target area corresponding to the annotation information as a target sub-image; and / or,
[0036] Perform at least one image transformation on each image in the target area corresponding to the annotation information, and use the image after the image transformation as the target sub-image.
[0037] In a possible implementation manner, before obtaining at least one target sub-image of the target object according to the annotation information of the first sample image, the method further includes:
[0038] According to the size information of the target area and the size information of the second sample image, a first number of target sub-images to be generated is determined; wherein the first number of target sub-images is positively correlated with the size information of the second sample image and negatively correlated with the size information of the target area.
[0039] In a possible implementation manner, after determining the number of target sub-images to be generated, performing at least one image transformation on each image in the target area corresponding to the annotation information includes:
[0040] According to the first number, performing at least one image transformation on each of the images in the target area to obtain the first number of target sub-images;
[0041] The image transformation includes at least one of the following:
[0042] Image stretching, image shrinking, image selection, image symmetry transformation, adding noise to images.
[0043] According to one aspect of the present disclosure, there is provided a counting device, comprising:
[0044] An input image feature extraction module is used to extract features from the input image to obtain a first feature map and a second feature map;
[0045] an object feature extraction module, configured to extract object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image;
[0046] a distribution position information determination module, configured to determine distribution position information of at least two objects to be counted in the first feature map according to the object features;
[0047] A feature distribution map determining module, used to determine a feature distribution map of the object to be counted from the second feature map using the distribution position information;
[0048] A total number determination module is used to determine the total number of objects to be counted based on the characteristic distribution graph.
[0049] In a possible implementation, the object feature extraction module includes:
[0050] an object feature extraction submodule, configured to extract first features of the marked objects to be counted from the first feature map according to position information of the at least two marked objects to be counted when there are at least two marked objects to be counted;
[0051] The fusion module is used to fuse the extracted first features of the objects to be counted to obtain the object features of the objects to be counted.
[0052] In a possible implementation, the distribution location information determination module includes:
[0053] A similarity determination module, used to calculate the similarity between the object feature and the feature at each position in the second feature map;
[0054] The distribution position information determination submodule is used to obtain the distribution position information of the at least two objects to be counted in the first feature map based on the similarity of the features at each position.
[0055] In a possible implementation manner, the feature distribution map determining module is used to multiply the distribution position information by the second feature map to obtain a feature distribution map of the object to be counted.
[0056] In a possible implementation, the total quantity determination module includes:
[0057] A density map acquisition module, used for upsampling the feature distribution map to obtain a density map of the object to be counted;
[0058] The total number determination submodule is used to determine the total number of objects to be counted based on the density map.
[0059] In a possible implementation, the input image feature extraction module includes:
[0060] An initial feature extraction module is used to extract initial features from an input image to obtain an initial feature map;
[0061] A first feature map extraction module, used to extract a first feature from the initial feature map to obtain a first feature map;
[0062] The second feature map extraction module is used to perform second feature extraction on the initial feature map to obtain a second feature map.
[0063] In a possible implementation, the input image is a sample image pre-labeled with the distribution positions of the at least two objects to be counted, the counting device is implemented based on a neural network, and the neural network is updated based on a parameter updating module, and the parameter updating module includes:
[0064] A first loss determination module, configured to determine a first loss based on the distribution position information and the distribution positions of at least two objects to be counted in the sample image that are pre-marked;
[0065] A second loss determination module, configured to determine a second loss based on the density map of the objects to be counted and the density map of the objects to be counted that are marked in advance;
[0066] A parameter updating submodule is used to update the parameters in the neural network based on the first loss and / or the second loss.
[0067] In a possible implementation, the counting method is implemented based on a neural network, and the device further includes: a sample construction module, which is used to construct a training sample and / or a test sample of the neural network;
[0068] The sample building blocks include:
[0069] A target sub-image determination module, configured to obtain at least one target sub-image of the target object according to the annotation information of the first sample image;
[0070] A pasting module, used for pasting the target sub-image into the second sample image to obtain a composite image and pasting position information of the target sub-image in the composite image;
[0071] The synthesis module is used to use the pasting position information as annotation information in a synthetic image to generate a synthetic sample image.
[0072] In a possible implementation, the target subgraph determination module includes:
[0073] A first target sub-image determination module is configured to extract an image in a target area corresponding to the annotation information as a target sub-image; and / or,
[0074] The second target sub-image determination module is used to perform at least one image transformation on the image in the target area corresponding to the annotation information, and use the image after the image transformation as the target sub-image.
[0075] In a possible implementation, the sample construction module further includes:
[0076] A quantity determination module is used to determine a first quantity of target sub-images to be generated according to the size information of the target area and the size information of the second sample image; wherein the first quantity of the target sub-images is positively correlated with the size information of the second sample image and negatively correlated with the size information of the target area.
[0077] In a possible implementation, the second target sub-image determination module is configured to perform at least one image transformation on each of the images in the target area according to the first number, to obtain the first number of target sub-images;
[0078] The image transformation includes at least one of the following:
[0079] Image stretching, image shrinking, image selection, image symmetry transformation, adding noise to images.
[0080] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.
[0081] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.
[0082] In the disclosed embodiment, a first feature map and a second feature map are obtained by extracting features from an input image, extracting object features of the object to be counted from the first feature map based on the position information of at least one object to be counted marked in the input image, determining the distribution position information of at least two objects to be counted in the first feature map based on the object features; determining a feature distribution map of the objects to be counted from the second feature map using the distribution position information; and determining the total number of objects to be counted based on the feature distribution map. Thus, when counting, the user is required to first mark (e.g., box select) the objects to be counted in the input image, i.e., obtain the position information of the objects to be counted, and then use the position information to extract the object features of the objects to be counted, i.e., obtain the feature information of the unclassified objects to be counted from the image to be counted, rather than counting through a network of a specific category. Therefore, the method has good versatility and can be applied to counting objects of different categories. In addition, in the embodiment of the present disclosure, two feature maps are extracted. For the two extracted feature maps, the object features are used to determine the distribution position information of each object to be counted from the first feature map, and then the distribution position information is used to determine the feature distribution map of the object to be counted from the second feature map. That is, of the two extracted feature maps, one is used to determine the distribution position, and the other is used to determine the feature distribution. The two feature maps each use two sets of parameters to determine the distribution position and the feature distribution, so the number of objects to be counted can be accurately determined.
[0083] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0085] Figure 1 A flowchart of a counting method according to an embodiment of the present disclosure is shown.
[0086] Figure 2 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.
[0087] Figure 3 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.
[0088] Figure 4 A flow chart of a feature extraction method according to an embodiment of the present disclosure is shown.
[0089] Figure 5A schematic diagram of the location distribution of convolution kernel sampling points according to an embodiment of the present disclosure is shown.
[0090] Figure 6 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.
[0091] Figure 7 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.
[0092] Figure 8 A block diagram of a counting device according to an embodiment of the present disclosure is shown.
[0093] Fig. 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.
[0094] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0095] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0096] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0097] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0098] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0099] In related technologies, visual counting mainly focuses on some specific categories. For example, a pedestrian counting model is trained for pedestrians by using samples that are labeled with the location and number of pedestrians. In applications, the number of pedestrians can be counted by inputting the image into the trained pedestrian counting model. However, the pedestrian counting model cannot count the number of vehicles because the model is trained based on pedestrian samples. If you want to count the number of vehicles, you need to use samples that are labeled with the location and number of vehicles to train new model parameters in order to count the number of vehicles. Therefore, this visual counting method requires the collection of a large amount of data to train the model, which consumes a lot of human resources and is inefficient.
[0100] In addition, in these visual counting tasks, the counting algorithm model relies on the features of the specific category of objects to be identified to generate convolution operation parameters. However, the network model designed in this way has a large number of parameters, slow model operation speed, and low efficiency.
[0101] In the disclosed embodiment, a first feature map and a second feature map are obtained by extracting features from an input image, extracting object features of the object to be counted from the first feature map based on the position information of at least one object to be counted marked in the input image, determining the distribution position information of at least two objects to be counted in the first feature map based on the object features; determining a feature distribution map of the objects to be counted from the second feature map using the distribution position information; and determining the total number of objects to be counted based on the feature distribution map. Thus, when counting, the user is required to first mark (e.g., box select) the objects to be counted in the input image, i.e., obtain the position information of the objects to be counted, and then use the position information to extract the object features of the objects to be counted, i.e., obtain the feature information of the unclassified objects to be counted from the image to be counted, rather than counting through a network of a specific category. Therefore, the method has good versatility and can be applied to counting objects of different categories. In addition, in the embodiment of the present disclosure, two feature maps are extracted. For the two extracted feature maps, the distribution position information of each object to be counted is determined from the first feature map using the object features, and then the feature distribution map of the object to be counted is determined from the second feature map using the distribution position information. That is, of the two extracted feature maps, one is used to determine the distribution position, and the other is used to determine the feature distribution. The two feature maps each use two sets of parameters to determine the distribution position and the feature distribution, so the number of objects to be counted can be accurately determined.
[0102] In a possible implementation, the counting method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in a memory.
[0103] For ease of description, in one or more embodiments of this specification, the execution subject of the counting method may be a terminal device. The following text takes the execution subject as an example to introduce the implementation of the method. It can be understood that the execution subject of the method as a terminal device is only an exemplary description and should not be understood as a limitation on the method.
[0104] Figure 1 A flowchart of a counting method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the counting method includes:
[0105] In step S11, feature extraction is performed on the input image to obtain a first feature map and a second feature map;
[0106] The input image is the image to be counted, which may be a single image captured by an image acquisition device, for example, a photo captured by the camera mode of the image acquisition device; or, the image may be a video frame in a video captured by the image acquisition device, for example, a video frame in a video captured by the video recording mode of the image acquisition device. The embodiments of the present disclosure do not limit the specific form of the image. The input image may be an image stored locally on the device, or may be a cloud image obtained from the network, and the embodiments of the present disclosure do not limit the storage form.
[0107] In one example, the input image may be an image of a road captured by an image acquisition device, which is used to count the number of people and vehicles on the road, etc.; in another example, the input image may be a cell image under a microscope captured by an image acquisition device, which is used to count the number of cells, etc. In addition, the embodiments of the present disclosure may also be applied to counting the number of other types of objects, such as animals, etc., which are not specifically limited here.
[0108] Feature extraction of the input image can be achieved through a convolutional network layer. In the disclosed embodiment, the first feature map and the second feature map can be obtained by extracting features through two different convolutional network layers. The structures of the two convolutional network layers can be exactly the same, or completely different, that is, the first feature map and the second feature map can be the same or different. In one example, the two convolutional network layers can be network layers with different parameters obtained after training two identical initial convolutional networks. Since the first feature map and the second feature map have different functions, one is used to determine the distribution position and the other is used to determine the feature distribution, therefore, as the network is iteratively trained based on the first loss determined by the distribution position and the second loss determined by the feature distribution, the parameters of the two convolutional networks will become different, so as to better realize the respective functions of the two feature maps.
[0109] For the specific method of determining the first characteristic graph and the second characteristic graph, please refer to the possible implementation method provided in the present disclosure, which will not be described in detail here. It can be understood that the "first" and "second" in the embodiments of the present disclosure are used to distinguish the objects described, and should not be understood as other limitations such as the order of the objects described, indication or implication of relative importance.
[0110] In step S12, according to the position information of at least one object to be counted marked in the input image, the object feature of the at least one object to be counted is extracted from the first feature map;
[0111] In the input image, the user can pre-mark the location information of the objects to be counted. It should be noted that the user does not need to mark the location information of all the objects to be counted. Since there are often many objects to be counted, the user only needs to mark some of the objects to be counted. In one example, the user can randomly select three objects to be counted in the input image, and then select them with a rectangular frame, or can also select them with a polygon or a circle as the annotation of the objects to be counted; in another example, the user can select any one object to be counted from the input image.
[0112] After the user marks the position information of the object to be counted, the object features of the object to be counted can be extracted from the first feature map based on the position information. In one example, when the position information is the coordinates of a rectangular box, the area corresponding to the coordinates of the rectangular box in the first feature map (i.e., the area selected by the rectangular box) can be extracted according to the coordinates of the rectangular box as the object features of the object to be counted.
[0113] In step S13, according to the object features, distribution position information of at least two objects to be counted is determined in the first feature map;
[0114] After determining the object features of the objects to be counted, the object features can be used to find the positions of at least two objects to be counted in the first feature map to obtain the distribution position information of the objects to be counted, where the at least two objects to be counted can be all the objects to be counted. Specifically, the positions where the objects to be counted may be distributed can be found by calculating the similarity between the object features and the features at each position in the first feature map. For details, please refer to the possible implementation methods provided in the present disclosure, which will not be described in detail here.
[0115] The distribution position information is used to characterize the position of the object to be counted in the input image. The distribution position information can be specifically implemented through a segmentation map or segmentation mask of the object to be counted, wherein the scale of the segmentation map or segmentation mask is the same as the scale of the second feature map. The segmentation map or segmentation mask can be used to determine the feature distribution map of the object to be counted from the second feature map.
[0116] Exemplarily, the segmentation map, the segmentation mask and the second feature map can be represented in a computer in the form of a matrix, then the specific representation of the scale here can be the number of parameters of the matrix in each dimension, for example, the number of rows and columns of the matrix, etc. When the segmentation map or the segmentation mask has the same scale as the second feature map, it indicates that the parameters in the segmentation map or the segmentation mask correspond to the distribution position of the object to be counted in the second feature map. Among them, each parameter in the matrix of the segmentation map or the segmentation mask represents whether there is an object to be counted at the position, and each parameter in the matrix of the second feature map represents the image features at the position.
[0117] In step S14, using the distribution position information, a feature distribution map of the object to be counted is determined from the second feature map;
[0118] The first feature map after feature extraction is used to determine the location information of the object to be counted. Therefore, the network parameters for extracting the first feature map are more inclined to extract accurate location information, and the obtained first feature map will contain more location features of the object to be counted. The second feature map after extraction is used to determine the feature distribution. Therefore, the network parameters for extracting the second feature map are more inclined to extract accurate feature representations of the object to be counted. Therefore, in the embodiment of the present disclosure, the feature distribution map of the object to be counted is obtained from the second feature map in combination with the location features of the object to be counted, so as to more accurately calculate the number of objects to be counted.
[0119] In step S15, the total number of objects to be counted is determined based on the characteristic distribution graph.
[0120] The feature distribution map can represent the density feature information of the objects to be counted marked by the user, that is, the features and distribution of the objects to be counted. The distribution is not related to the specific category, but is related to the objects to be counted selected by the user.
[0121] In a possible implementation manner, determining the total number of objects to be counted based on the feature distribution map includes: upsampling the feature distribution map to obtain a density map of the objects to be counted; and determining the total number of objects to be counted based on the density map.
[0122] The density map of the objects to be counted is used to characterize the density of the objects to be counted. In one example, the specific implementation form of the density map can be a matrix. Each element in the matrix represents whether there is an object to be counted at that position. Specifically, it can be obtained by upsampling the feature distribution map. The feature distribution map is enlarged to the same scale as the input image through the upsampling operation, and then the enlarged feature distribution map is convolved through a convolution operation to obtain the density map. The parameters of the upsampling operation and the convolution operation can be obtained by training the network. In the density map, the probability of the existence of a crowd at a single pixel point is represented by a value between 0 and 1, and then all the values of the matrix are accumulated (integrated), and the obtained value is the total number of objects to be counted.
[0123] In the disclosed embodiment, a first feature map and a second feature map are obtained by extracting features from an input image, extracting object features of the object to be counted from the first feature map based on the position information of at least one object to be counted marked in the input image, determining the distribution position information of at least two objects to be counted in the first feature map based on the object features; determining a feature distribution map of the objects to be counted from the second feature map using the distribution position information; and determining the total number of objects to be counted based on the feature distribution map. Thus, when counting, the user is required to first mark (e.g., box select) the objects to be counted in the input image, i.e., obtain the position information of the objects to be counted, and then use the position information to extract the object features of the objects to be counted, i.e., obtain the feature information of the unclassified objects to be counted from the image to be counted, rather than counting through a network of a specific category. Therefore, the method has good versatility and can be applied to counting objects of different categories. In addition, in the embodiment of the present disclosure, two feature maps are extracted. For the two extracted feature maps, the object features are used to determine the distribution position information of each object to be counted from the first feature map, and then the distribution position information is used to determine the feature distribution map of the object to be counted from the second feature map. That is, of the two extracted feature maps, one is used to determine the distribution position, and the other is used to determine the feature distribution. The two feature maps each use two sets of parameters to determine the distribution position and the feature distribution, so the number of objects to be counted can be accurately determined.
[0124] In a possible implementation, the extracting object features of the object to be counted from the first feature map based on the position information of at least one object to be counted marked in the input image includes: when there are at least two marked objects to be counted, respectively extracting features of each marked object to be counted based on the position information of the at least two marked objects to be counted; and fusing the extracted first features of each object to be counted to obtain the object features of the object to be counted.
[0125] In the input image, the user may pre-mark the position information of the object to be counted. When the user marks an object to be counted, for example, the user only selects one object to be counted, the features of the object to be counted may be directly used as the object features.
[0126] When the user has marked at least two objects to be counted, the first features in each marked frame can be extracted respectively, and then the average value of each object to be counted can be extracted as the object feature of the object to be counted. Specifically, the first features extracted from each marked frame can be unified to the same size, and then the first features can be fused. The specific fusion method can be to obtain the average value of the first features, that is, to sum up the first features and then divide them by the number of annotations. For example, if the user selects the positions of three objects to be counted, the features at these three positions can be extracted from the first feature map respectively to obtain three features v1, v2 and v3, and then these three features are unified in scale and the average value is taken to obtain the feature That is, the object characteristics of the objects to be counted.
[0127] In this implementation, when there are at least two marked objects to be counted, the features of the marked objects to be counted are extracted respectively according to the position information of the at least two marked objects to be counted; the extracted first features of the objects to be counted are fused to obtain the object features of the objects to be counted. Thus, when the features of the marked objects to be counted in the image are different, more accurate object features can be obtained by fusion of the first features of at least two objects to be counted to obtain the object features of the objects to be counted, thereby improving the accuracy of counting the objects to be counted.
[0128] In a possible implementation, determining the distribution position information of at least two objects to be counted in the first feature map based on the object features includes: calculating the similarity between the object features and the features at each position in the second feature map; and obtaining the distribution position information of the at least two objects to be counted in the first feature map based on the similarity of the features at each position.
[0129] The distribution position information of at least two objects to be counted in the first feature map obtained here may be the distribution position information of all objects to be counted in the first feature map. By using the object features, the positions of all objects to be counted can be found in the first feature map to obtain the distribution position information of the objects to be counted. In this implementation, it can be achieved by calculating the similarity between the object features and the features at each position in the second feature map. The higher the similarity, the greater the possibility that the feature at the position is the object to be counted, and the lower the similarity, the less likely the feature at the position is the object to be counted.
[0130] In this implementation, the distribution position information of the objects to be counted can be calculated by cosine similarity. Specifically, the object features of the objects to be counted can be represented by vectors in the computer, and each position in the first feature map can also be an image feature represented by a vector. Then, the cosine similarity of the vector representing the object features and the vector representing the features at each position in the first feature map can be calculated to obtain the cosine similarity value at each position. Here, the cosine similarity value at each position can be directly used as the distribution position information of the objects to be counted.
[0131] In one example, the cosine similarity value may be directly assigned to each position to obtain a feature matrix representing the distribution positions of the objects to be counted, that is, a segmentation map or segmentation mask of the objects to be counted in the input image is obtained.
[0132] In this implementation, the similarity between the object feature and the feature at each position in the second feature map is calculated; based on the similarity of the features at each position, the distribution position information of all objects to be counted in the first feature map is obtained. Thus, the segmentation map or segmentation mask of the object to be counted can be obtained by calculating the similarity. Compared with the method of extracting features from the feature map through convolution operation to obtain the segmentation map, it can save the network operation parameters, improve the efficiency of obtaining the distribution position information, and thus improve the counting efficiency.
[0133] In a possible implementation, using the distribution position information to determine a feature distribution map of the object to be counted from the second feature map includes: multiplying the distribution position information by the second feature map to obtain the feature distribution map of the object to be counted.
[0134] Since the extracted second feature map is used to determine the feature distribution, the network parameters for extracting the second feature map are more inclined to extract accurate feature representation, and thus, the features of the objects to be counted can be accurately represented in the second feature map.
[0135] In this implementation, since the value of each position in the distribution position information is the value of the cosine similarity, that is, the probability of each position being the object to be counted, the value of the feature at the lower probability can be suppressed and the value of the feature at the higher probability can be enhanced by directly multiplying the distribution position information with the second feature map. Then the result obtained after the multiplication can be used as the feature distribution map of the object to be counted.
[0136] In the disclosed embodiment, the characteristic distribution map of the object to be counted is obtained by multiplying the distribution position information with the second characteristic map, thereby enabling the characteristic distribution map of the object to be counted to be obtained quickly, thereby improving the counting efficiency.
[0137] In a possible implementation, the extracting features of the input image to obtain the first feature map and the second feature map includes: performing initial feature extraction on the input image to obtain an initial feature map; performing first feature extraction on the initial feature map to obtain the first feature map; and performing second feature extraction on the initial feature map to obtain the second feature map.
[0138] The input image is subjected to initial feature extraction to extract some information common to the first feature map and the second feature map. In one example, the first feature map and the second feature map can be extracted by a convolutional neural network designed based on the VGG-16 convolutional neural network. Specifically, the first 10 convolutional layers and 3 pooling layers of the VGG-16 can be used to extract the initial feature map in the input image. Of course, the initial feature map can also be extracted by other convolutional neural networks, which is not limited in the present disclosure.
[0139] In the disclosed embodiment, an initial feature extraction is performed on an input image to obtain an initial feature map; a first feature extraction is performed on the initial feature map to obtain a first feature map; and a second feature extraction is performed on the initial feature map to obtain a second feature map. Thus, the first feature map and the second feature map are extracted through two different operations, and the first feature map and the second feature map respectively use two sets of parameters to determine the distribution position and the feature distribution, so that the distribution position and features of the objects to be counted can be accurately determined, and the number of objects to be counted can be accurately determined. In addition, the first feature map and the second feature map are both extracted from the initial feature map, that is, the initial feature map is obtained by performing initial feature extraction on the input image, so as to extract some common information in the first feature map and the second feature map, which can reduce the parameters of the network operation, and it is not necessary to extract the first feature map and the second feature map from the input image separately, thereby improving the efficiency of extracting the first feature map and the second feature map.
[0140] In a possible implementation, the input image is a sample image with the distribution positions of the at least two objects to be counted pre-labeled, the counting method is implemented based on a neural network, and the parameter updating process of the neural network includes: determining a first loss based on the distribution position information and the pre-labeled distribution positions of at least two objects to be counted in the sample image; determining a second loss based on a density map of the objects to be counted and a pre-labeled density map of the objects to be counted, and updating the parameters in the neural network based on the first loss and / or the second loss.
[0141] In this implementation, the neural network may be a convolutional neural network, and the training of the neural network may be achieved by updating the parameters of the neural network. The distribution positions of all objects to be counted may be pre-marked in the sample image, and the distribution positions are used to calculate the first loss with the distribution positions predicted by the neural network, thereby training the neural network. The position information of at least one object to be counted marked in step S11 is used to select the object to be counted. The user only needs to select a small number (for example, 3) of objects to be counted, and the neural network can predict the total number of all objects to be counted.
[0142] In the process of updating the parameters in the neural network based on the first loss and / or the second loss, the parameters in the neural network may be updated based on the first loss, or based on the second loss, or the total loss may be determined based on the first loss and the second loss, and the parameters in the neural network may be updated based on the total loss. The total loss may be determined based on the first loss and the second loss, and specifically the first loss and the second loss may be summed to obtain the total loss. In addition, the first loss and the second loss may be weighted by assigning corresponding weights to each other, and then summed to obtain the total loss. The present disclosure does not limit the specific process of determining the total loss.
[0143] During the updating process, a first loss between the distribution position information and the distribution position of the object to be counted in the pre-annotated input image can be calculated, and the parameters in the neural network are updated based on the first loss. The pre-annotated distribution position can be the position of all objects to be counted in the input image pre-annotated by the user. The position can be specifically selected by a rectangular box or a polygonal box or other selection tools. After the selection is completed, a mask label representing the distribution position of the object to be counted in the input image can be generated, which can be used as the distribution position information of the object to be counted in the pre-annotated input image.
[0144] In the process of updating the parameters in the neural network based on the first loss, the parameters in the network can be updated so that the distribution position information output by the updated neural network after receiving the input image is as consistent as possible with the pre-labeled distribution position. In one example, the first loss can be determined by a binary cross entropy loss function (BCE loss), and then the parameters of the neural network are updated.
[0145] During the updating process, a second loss between the density map of the objects to be counted and the density map of the objects to be counted that are pre-labeled can also be calculated, and the parameters in the neural network are updated based on the second loss. The density map of the objects to be counted that are pre-labeled can be generated based on the distribution position information of the objects to be counted in the pre-labeled input image.
[0146] In the process of updating the parameters in the neural network based on the second loss, the parameters in the network can be updated so that the density map of the objects to be counted output by the updated neural network after receiving the input image is as consistent as possible with the pre-labeled density map. In one example, the second loss can be determined by a mean square error loss function (L2 loss), and then the parameters of the neural network are updated.
[0147] In the disclosed embodiment, a first loss is determined based on the distribution position information and the distribution position of the objects to be counted in the pre-labeled input image; a second loss is determined based on the density map of the objects to be counted and the pre-labeled density map of the objects to be counted, and the parameters in the neural network are updated based on the first loss and / or the second loss. Since the first feature map and the second feature map have different functions, one is used to determine the distribution position and the other is used to determine the feature distribution, the loss is calculated based on the distribution position and the density function respectively, then as the network training proceeds, the parameters of the two initial convolutional networks will be different, so as to better realize the respective functions of the two feature maps and more accurately determine the number of objects to be counted.
[0148] The following is an application scenario of the embodiment of the present disclosure. Figure 2 , is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. In this application scenario, the counting method provided by the present disclosure is implemented by a convolutional neural network. The user pre-labels three objects to be counted in the input image. The input image is feature extracted by using the first 10 convolutional layers and 3 pooling layers of VGG-16 to obtain the initial feature F; then the first feature F is extracted from the initial feature F. c and the second feature F d ; From the first feature F c Extract the features v1, v2 and v3 of the three labeled objects to be counted, and average these three features to get calculate With F c The cosine similarity of the features at each position on the input image is used to obtain the distribution position information of the objects to be counted on the input image, that is, the segmentation map of the objects to be counted in the input image. Will With F d Multiply to get the density feature information without category. At this time, the feature information containing only the category objects to be counted has been obtained. The density feature information is decoded (upsampled and enlarged to the same scale as the input image) to get the density map of the objects to be counted.
[0149] During the training of the neural network, the binary cross entropy loss function (BCE loss) is used to calculate The first loss between the label information S, and then update the parameters of the neural network; determine it through the mean square error loss function (L2loss) The second loss between the label information D is then used to update the parameters of the neural network.
[0150] In one possible implementation, the counting method is implemented based on a neural network, and a method for constructing training samples and / or test samples of the neural network includes: obtaining at least one target sub-image of the target object based on the annotation information of the first sample image; pasting the target sub-image into the second sample image to obtain a composite image and the pasting position information of the target sub-image in the composite image; using the pasting position information as the annotation information in the composite image to generate a composite sample image.
[0151] The first sample image may be an image with annotation information, and the annotation information is used to indicate the location of the target object annotated in the first sample image. The target object may be any object annotated by a user, and the present disclosure does not limit the specific type of the target object. Figure 3 The image in the upper left corner shows that the locations of three glass balls are selected by rectangular boxes in this sample image.
[0152] The annotation information is used to indicate the location of the target object annotated in the first sample image. The annotation information may be the coordinates of the target object, and the annotated target area may be determined based on the coordinates. In one example, when the location information is the coordinates of a rectangular box, the annotated target area in the first sample image may be determined based on the coordinates of the rectangular box.
[0153] After determining the target area, at least one target sub-image can be obtained based on the image in the target area. Specifically, the image in the target area corresponding to the annotation information can be extracted as the target sub-image; and / or, at least one image transformation can be performed on the image in the target area corresponding to the annotation information, and the image after the image transformation is used as the target sub-image.
[0154] When there are at least two target areas, images in the target areas can be extracted respectively to obtain at least two target sub-images, that is, the image in a single target area is taken as a target sub-image. For example, assuming that the positions of three target objects are marked in advance, then three target sub-images can be obtained.
[0155] In addition, the image in the target area can be transformed, and the transformed image can be used as the target sub-image. The image transformation includes at least one of the following: image stretching, image shrinking, image selection, image symmetric transformation, and adding noise to the image. In one example, the image transformation here can be stretching the image, for example, stretching it horizontally and / or vertically; in another example, the image transformation here can also be shrinking the image, for example, stretching it horizontally and / or vertically; in another example, the image transformation here can also be rotating the image, for example, rotating it 90° to the left; in another example, the image transformation here can also be symmetric transformation of the image; in another example, the image transformation here can also be adding noise to the image, and the noise can be, for example, Gaussian noise.
[0156] In this implementation, the extracted image and the image obtained by image transformation can be used together as the target sub-image.
[0157] After obtaining the target sub-image, the target sub-image can be pasted into the second sample image to obtain a composite image. The second sample image can be any image, and the second sample image is used as a background. When pasting the target sub-image into the second sample image, it can be pasted randomly at any position in the second image, and the pasting position information is recorded at the same time.
[0158] The pasting position information is used as annotation information in a synthetic image to generate a synthetic sample image.
[0159] In the disclosed embodiment, at least one target region annotated in the first sample image is determined based on the annotation information of the first sample image; at least one target sub-image is obtained based on the image in the target region; the target sub-image is pasted into the second sample image to obtain a composite image and the pasting position information of the target sub-image in the composite image; the pasting position information is used as the annotation information in the composite image to generate a composite sample image. Thus, a large number of composite sample images can be generated by copying and pasting. Compared with manual annotation, the speed of constructing sample images is faster, and the accuracy of the annotation information is higher when the pasting position information is used as the annotation information of the composite sample image.
[0160] In a possible implementation, before obtaining at least one target sub-image of the target object according to the annotation information of the first sample image, the method further includes: determining a first number of target sub-images to be generated according to the size information of the target area and the size information of the second sample image; wherein the first number of target sub-images is positively correlated with the size information of the second sample image and negatively correlated with the size information of the target area.
[0161] The size information of the target area may be the area of the target area, or the length, width and other information of the target area. When the target area is a rectangle, the area of the target area may be obtained by multiplying the length of the target area by the width. When the target area includes multiple target areas, the size information of the target area may be the average value of the sizes of the multiple target areas, such as the average value of the areas of the multiple target areas. The larger the target size, the fewer target sub-images can be placed in the second sample image. Therefore, the first number of target sub-images is positively correlated with the size information of the second sample image.
[0162] The size information of the second sample image may be the area of the second sample image, or the length, width and other information of the second sample image. When the second sample image is a rectangle, the area of the second sample image may be obtained by multiplying the length by the width of the second sample image. The larger the size of the second sample image, the more target sub-images may be placed in the second sample image. Therefore, the first number of target sub-images is positively correlated with the size information of the second sample image.
[0163] In order to facilitate a clearer understanding of the process of determining the first number N, the process of determining the first number N is explained below through a specific mathematical expression. It should be noted that the specific mathematical expression provided in the present disclosure is a possible implementation method of the embodiment of the present disclosure during specific implementation, and should not be understood as a limitation on the protection scope of the embodiment of the present disclosure.
[0164]
[0165] Wherein, α is a weight parameter, which can be a random number between 0.25 and 0.75, for example. is the height of the second sample image, is the width of the second sample image, e i represents the i-th target subgraph, i is a positive integer, for e i The set composed of is the height of the target subgraph, is the width of the target sub-image, Characterize the first sample image I f Zhonge i The number of .
[0166] In the disclosed embodiment, the first number of target sub-images to be generated is determined according to the size information of the target area and the size information of the second sample image; and the first number of target sub-images is positively correlated with the size information of the second sample image, and the first number of target sub-images is negatively correlated with the size information of the target area. Thus, the number of maps calculated satisfies the distribution law of actual objects, and the obtained synthetic sample images are closer to the real captured images, thereby improving the accuracy of neural network training.
[0167] In one possible implementation, after determining the number of target sub-graphs to be generated, performing at least one image transformation on the images in the target area corresponding to the annotation information, including: performing at least one image transformation on the images in the target area according to the first number, so as to obtain a total number of target sub-graphs that meets the first number.
[0168] After determining the first number, to obtain the first number of target sub-images, the first number of target sub-images may be obtained by image transformation. The specific transformation method can be found in the possible implementation method provided in the present disclosure, which will not be described in detail here.
[0169] The following is an application scenario of the embodiment of the present disclosure. Figure 3 , is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. In this application scenario, a first sample image I is selected from a training set. f According to the annotation information provided by the training set, the image corresponds to three rectangular boxes e1, e2, and e3, which represent the objects to be counted. Another second sample image I is randomly selected from the training set. b As background image. f Copy the images in the three object boxes and perform several random image transformations on them. Finally, a total of N target sub-images are obtained. The target sub-images are randomly pasted onto the background image I b On, such as Figure 3As shown, according to the pasted image position, the label information corresponding to the object on the synthetic sample image is obtained, so that the density map can be generated as a annotation (ground truth) and the neural network can be pre-trained.
[0170] After pre-training is completed, the neural network can also be trained using manually annotated label information, which can further optimize the neural network and improve its ability to learn object feature information.
[0171] During testing, the trained neural network can be tested using the label information manually annotated in the test image set, or the test image can be synthesized by copying and pasting the embodiment of the present disclosure to optimize the neural network.
[0172] In the embodiment of the present disclosure, when initial feature extraction is performed on the input image, the feature extraction can be performed based on a convolutional neural network. Figure 4 A flow chart of a feature extraction method according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the initial feature extraction is performed on the input image to obtain the initial feature map, including:
[0173] In step S21, an input feature is obtained, where the input feature includes an input image or a feature obtained by performing at least one convolution operation on the input image;
[0174] The input feature here can be the original input image, that is, the convolution operation can be performed directly with the pixel value of the image as input, or the input feature can also be a feature obtained after performing at least one convolution operation on the input image. The convolution operation here can be a convolution operation with a fixed convolution kernel size, or it can be a convolution operation with a variable convolution kernel size provided in an embodiment of the present disclosure. The present disclosure does not limit this.
[0175] In step S22, the size information of the target object in the input image is obtained;
[0176] The target object here may be the object to be counted mentioned above, and the size information of the target object may be the information of the size occupied by the target object in the input image, for example, it may be the area occupied by the target object in the input image, or the length and width of the area occupied by the target object in the input image.
[0177] The size information of the target object can be determined based on the annotation information of the target object annotated in the input image. For example, when the position of the target object is annotated by a rectangular frame, the size information of the rectangular frame can be directly used as the size information of the target object. For example, the area of the target object can be determined based on the length and width of the rectangular frame as the size information of the target object. In addition, the size information of the target object can also be input by the user. The present disclosure does not limit the specific method for obtaining the size information of the target object.
[0178] In step S23, based on the size information and the input feature, a position offset of a sampling point when the convolution kernel samples the input feature is determined;
[0179] When the convolution kernel samples the input features, there will be at least one sampling point. If the size of the convolution kernel is 1×1, the convolution kernel has 1 sampling point. If the size of the convolution kernel is 2×2, the convolution kernel has 4 sampling points. If the size of the convolution kernel is 3×3, the convolution kernel has 9 sampling points. The corresponding relationship between the sizes of more convolution kernels and sampling points is not described here. When sampling, each sampling point in the convolution kernel will be sampled in the input features according to the predetermined position, such as Figure 5 As shown, it is a schematic diagram of the location distribution of convolution kernel sampling points provided by the present disclosure. Figure 5 The arrows in the figure indicate the direction and distance of the position offset of the sampling point. The hollow circle represents the position of the sampling point before the position offset, and the solid circle represents the position of the sampling point after the position offset. Figure 5 The two sampling points after offset are shown. It can be seen that the position of the sampling points after offset is more flexible, which can realize the change of the receptive field during sampling. Figure 5 After the sampling points of the convolution kernel on the right are offset, it can be clearly seen that the range of the receptive field becomes larger, which is suitable for determining large-sized target objects.
[0180] In the embodiment of the present disclosure, the sampling point position offset is determined based on the size information and input features of the target object. Exemplarily, a convolution operation can be performed on the size information and the input features to obtain the sampling point position offset. The parameters of the convolution operation can be obtained by training with samples. The convolution operation can be a part of the feature extraction process of the counting method provided by the present disclosure. The parameters of the convolution operation can be trained through the network training process. The specific training method can be found in the relevant description in the previous text and will not be repeated here.
[0181] In step S24, a convolution operation is performed on the input feature based on the convolution kernel after the position offset is adjusted to obtain an output feature.
[0182] The obtained output features can be used as initial features, or the output features can be convolved at least once again to obtain initial features.
[0183] Each sampling point in the convolution kernel can get a position offset, for example, Figure 5 The 9 sampling points in can all be offset. Exemplarily, the position offset of each sampling point can include two directions, the x-axis and the y-axis. Based on the x-axis offset and the y-axis offset, the initial position of the sampling point is offset to obtain the convolution kernel after the position offset is adjusted.
[0184] Based on the sampling point adjusted by the position offset, the image features at the corresponding position are collected, and then the convolution operation is performed to obtain the output features. Therefore, the output features obtained are sampled based on the size of the target object, which can better characterize the target object in the input image. Then the target object in the input image determined based on the output features will be more accurate.
[0185] In a possible implementation, when there is size information of at least two target objects, obtaining the size information of the target object in the input image includes: taking an average value of the size information of the at least two target objects obtained as the size information of the target object in the input image.
[0186] In one example, the size information of at least two target objects here may be the size information of two or more target objects pre-marked in the input image. In this way, when the size information of two or more target objects exists, the average value of the size information of at least two target objects obtained can be used as the size information of the target object in the input image.
[0187] Exemplarily, when multiple target objects in the input image are marked by multiple rectangular boxes, the area of the target object can be calculated for each rectangular box, and then the average value of the areas is taken as the size information of the target object.
[0188] In the embodiments of the present disclosure, since there may be multiple target objects in the input image, especially when counting target objects, the number of target objects is often large and the sizes may vary. Therefore, based on the average value of the size information of the multiple target objects, all target objects in the input image can be more accurately characterized, thereby improving the accuracy of determining all target objects in the input image and improving the accuracy of counting target objects in the input image.
[0189] In a possible implementation, determining the position offset of the sampling point when the convolution kernel samples the input feature based on the size information and the input feature includes: fusing the size information with the input feature to obtain a fused feature; and determining the position offset of the sampling point when the convolution kernel samples the input feature based on the fused feature.
[0190] The sampling point position offset is determined based on the size information and input features of the target object. Specifically, the size information and the input features are fused, and the sampling point position offset is determined based on the fused features. The fusion here can be, for example, splicing the size information and the input features. The size information and the input features can be represented in a computer in the form of a matrix, so the splicing of the size information and the input features here can be splicing two matrices representing the size information and the input features.
[0191] Exemplarily, when determining the position offset of the convolution kernel sampling point based on the fused feature, a convolution operation can be performed on the fused feature to obtain the sampling point position offset. The parameters of the convolution operation can be obtained by training with samples. The convolution operation can be part of the feature extraction process of the counting method provided in the present disclosure. The parameters of the convolution operation can be trained through the network training process. The specific training method can be found in the relevant description in the previous text and will not be repeated here.
[0192] In a possible implementation, the fusing the size information with the input feature to obtain a fused feature includes: performing a nonlinear transformation operation on the size information to obtain a size feature that characterizes the size information; performing a convolution operation on the input feature to obtain a convolution input feature; and fusing the size feature with the convolution input feature to obtain a fused feature.
[0193] As mentioned above, in the embodiments of the present disclosure, the size information may be information such as area, length and width. Here, a nonlinear transformation operation may be used to convert it into size features that represent the size information. The size features here may be in vector form to facilitate subsequent convolution operations, or they may be features of the vector after size transformation. The specific process of performing the nonlinear transformation operation may be referred to in the relevant technology and will not be described in detail here.
[0194] The scale of the vector representing the size information is often different from the scale of the input feature. Therefore, the vector can be scaled here to transform the scale of the vector to the same scale as the input feature, so as to facilitate subsequent fusion with the input feature.
[0195] For the input features, a convolution operation can be performed to obtain the convolution input features, and then the convolution input features are concatenated with the size features to obtain the fusion features.
[0196] In the disclosed embodiment, a nonlinear transformation operation is performed on the size information to obtain a vector representing the size information; a scale transformation is performed on the vector to obtain a size feature with the same scale as the input feature; a convolution operation is performed on the input feature to obtain a convolution input feature; and the size feature is concatenated with the convolution input feature to obtain a fusion feature. Thus, the obtained fusion feature can more accurately represent the size information and fusion feature of the target object, so as to accurately obtain the position offset of the sampling point.
[0197] The following is an application scenario of the embodiment of the present disclosure. Figure 6 , is a schematic diagram of an application scenario provided by an embodiment of the present disclosure, in which the size information of the target object in the upper right corner is subjected to a nonlinear transformation operation to obtain a vector, and then the scale of the vector is expanded in the spatial dimension to obtain a size feature g that is consistent with the scale of the input feature; the input feature in the lower left corner is subjected to a convolution operation to obtain a convolution input feature c; the input feature c and the size feature g are concatenated in the channel dimension to obtain a fused feature r, and the fused feature r is subjected to a convolution operation to obtain a position offset of the sampling point, and a convolution operation is performed on the input feature based on the convolution kernel adjusted by the position offset to obtain an output feature in the lower right corner.
[0198] The following is an application scenario of the embodiment of the present disclosure. Figure 7 , is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. In this application scenario, the initial feature extraction provided by the present disclosure is implemented by a convolutional neural network. Figure 7 In the above example, the first 10 convolutional layers and 3 pooling layers of VGG-16 are used to extract features from the input image. Here, the last convolution operation of each stage is replaced with a convolutional layer with a variable convolution kernel sampling point to construct a skeleton network to extract object feature information. The specific implementation process of the convolutional layer with a variable convolution kernel sampling point can be found in the relevant description of steps S21-24 of the present disclosure. Then, the output features of the last convolutional layer with a variable convolution kernel are used as the initial feature map, and the number of target objects in the input image is further determined based on the initial feature map.
[0199] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.
[0200] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.
[0201] In addition, the present disclosure also provides a counting device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any counting method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.
[0202] Figure 8 A block diagram of a counting device according to an embodiment of the present disclosure is shown. Figure 8 As shown, the device 30 comprises:
[0203] An input image feature extraction module 31 is used to extract features from the input image to obtain a first feature map and a second feature map;
[0204] An object feature extraction module 32, configured to extract object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image;
[0205] A distribution position information determination module 33, configured to determine distribution position information of at least two objects to be counted in the first feature map according to the object features;
[0206] A feature distribution map determining module 34, configured to determine a feature distribution map of the object to be counted from the second feature map using the distribution position information;
[0207] The total number determination module 35 is used to determine the total number of objects to be counted based on the characteristic distribution graph.
[0208] In a possible implementation, the object feature extraction module includes:
[0209] an object feature extraction submodule, configured to extract first features of the marked objects to be counted from the first feature map according to position information of the at least two marked objects to be counted when there are at least two marked objects to be counted;
[0210] The fusion module is used to fuse the extracted first features of the objects to be counted to obtain the object features of the objects to be counted.
[0211] In a possible implementation, the distribution location information determination module includes:
[0212] A similarity determination module, used to calculate the similarity between the object feature and the feature at each position in the second feature map;
[0213] The distribution position information determination submodule is used to obtain the distribution position information of the at least two objects to be counted in the first feature map based on the similarity of the features at each position.
[0214] In a possible implementation manner, the feature distribution map determining module is used to multiply the distribution position information by the second feature map to obtain a feature distribution map of the object to be counted.
[0215] In a possible implementation, the total quantity determination module includes:
[0216] A density map acquisition module, used for upsampling the feature distribution map to obtain a density map of the object to be counted;
[0217] The total number determination submodule is used to determine the total number of objects to be counted based on the density map.
[0218] In a possible implementation, the input image feature extraction module includes:
[0219] An initial feature extraction module is used to extract initial features from an input image to obtain an initial feature map;
[0220] A first feature map extraction module, used to extract a first feature from the initial feature map to obtain a first feature map;
[0221] The second feature map extraction module is used to perform second feature extraction on the initial feature map to obtain a second feature map.
[0222] In a possible implementation, the input image is a sample image pre-labeled with the distribution positions of the at least two objects to be counted, the counting device is implemented based on a neural network, and the neural network is updated based on a parameter updating module, and the parameter updating module includes:
[0223] A first loss determination module, configured to determine a first loss based on the distribution position information and the distribution positions of at least two objects to be counted in the sample image that are pre-marked;
[0224] A second loss determination module, configured to determine a second loss based on the density map of the objects to be counted and the density map of the objects to be counted that are marked in advance;
[0225] A parameter updating submodule is used to update the parameters in the neural network based on the first loss and / or the second loss.
[0226] In a possible implementation, the counting method is implemented based on a neural network, and the device further includes: a sample construction module, which is used to construct a training sample and / or a test sample of the neural network;
[0227] The sample building blocks include:
[0228] A target sub-image determination module, configured to obtain at least one target sub-image of the target object according to the annotation information of the first sample image;
[0229] A pasting module, used for pasting the target sub-image into the second sample image to obtain a composite image and pasting position information of the target sub-image in the composite image;
[0230] The synthesis module is used to use the pasting position information as annotation information in a synthetic image to generate a synthetic sample image.
[0231] In a possible implementation, the target subgraph determination module includes:
[0232] A first target sub-image determination module is configured to extract an image in a target area corresponding to the annotation information as a target sub-image; and / or,
[0233] The second target sub-image determination module is used to perform at least one image transformation on the image in the target area corresponding to the annotation information, and use the image after the image transformation as the target sub-image.
[0234] In a possible implementation, the sample construction module further includes:
[0235] A quantity determination module is used to determine a first quantity of target sub-images to be generated according to the size information of the target area and the size information of the second sample image; wherein the first quantity of the target sub-images is positively correlated with the size information of the second sample image and negatively correlated with the size information of the target area.
[0236] In a possible implementation, the second target sub-image determination module is configured to perform at least one image transformation on each of the images in the target area according to the first number, to obtain the first number of target sub-images;
[0237] The image transformation includes at least one of the following:
[0238] Image stretching, image shrinking, image selection, image symmetry transformation, adding noise to images.
[0239] In a possible implementation, the initial feature extraction module includes:
[0240] An input feature acquisition module, used to acquire input features, wherein the input features include an input image or features obtained by performing at least one convolution operation on the input image;
[0241] A size information acquisition module, used to acquire size information of a target object in the input image;
[0242] A position offset determination module, used to determine the position offset of a sampling point when a convolution kernel samples the input feature based on the size information and the input feature;
[0243] A convolution operation module is used to perform a convolution operation on the input feature based on the convolution kernel adjusted by the position offset to obtain an output feature, wherein the output feature is used to determine the target object in the input image.
[0244] In a possible implementation manner, the size information acquisition module is configured to use an average value of the acquired size information of the at least two target objects as the size information of the target object in the input image.
[0245] In a possible implementation, the position offset determination module includes:
[0246] A fusion feature determination module, used for fusing the size information with the input feature to obtain a fusion feature;
[0247] The position offset determination submodule is used to determine the position offset of the sampling point when the convolution kernel samples the input feature based on the fused feature.
[0248] In a possible implementation, the fusion feature determination module includes:
[0249] A nonlinear transformation operation module, used for performing a nonlinear transformation operation on the size information to obtain a size feature representing the size information;
[0250] A convolution submodule, used to perform a convolution operation on the input feature to obtain a convolution input feature;
[0251] The size feature is fused with the convolution input feature to obtain a fused feature.
[0252] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0253] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.
[0254] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.
[0255] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0256] The electronic device may be provided as a terminal, a server, or a device in other forms.
[0257] Fig. 9 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or other terminal device.
[0258] Reference Fig. 9 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0259] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0260] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0261] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.
[0262] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0263] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0264] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0265] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0266] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G), a long-term evolution (LTE) of a universal mobile communication technology, a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0267] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0268] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.
[0269] The present disclosure relates to the field of augmented reality. By acquiring the image information of the target object in the real environment, the relevant features, states and attributes of the target object are detected or identified by various visual related algorithms, so as to obtain an AR effect that combines virtuality and reality and matches the specific application. Exemplarily, the target object may involve the face, limbs, gestures, actions, etc. related to the human body, or markers, markers related to objects, or sand tables, display areas or display items related to venues or places. Visual related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and position or depth detection of objects. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstruction, virtual effect superposition and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, limb beautification, special effects display, virtual model display and other interactive scenes. The relevant features, states and attributes of the target object can be detected or identified by a convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by model training based on a deep learning framework.
[0270] Fig.10 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.10 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions that can be executed by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.
[0271] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.
[0272] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.
[0273] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0274] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0275] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0276] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0277] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0278] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0279] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0280] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0281] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.
[0282] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0283] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0284] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0285] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A counting method, It is characterized in that include: Performing feature extraction on the input image to obtain a first feature map and a second feature map; extracting object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image; Determining, in the first feature map, distribution position information of at least two objects to be counted according to the object features; Determine a feature distribution map of the object to be counted from the second feature map using the distribution position information; The total number of objects to be counted is determined based on the characteristic distribution map.
2. The method according to claim 1, It is characterized in that The step of extracting object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image includes: When there are at least two marked objects to be counted, extracting first features of each marked object to be counted from the first feature map according to position information of the at least two marked objects to be counted; The extracted first features of each object to be counted are fused to obtain the object features of the object to be counted.
3. The method according to claim 1, It is characterized in that The determining, in the first feature map according to the object features, distribution position information of at least two objects to be counted includes: Calculating the similarity between the object feature and the feature at each position in the first feature map; Based on the similarity of the features at each position, the distribution position information of the at least two objects to be counted in the first feature map is obtained.
4. The method according to any one of claims 1 to 3, It is characterized in that The step of using the distribution position information to determine a feature distribution map of the object to be counted from the second feature map includes: The distribution position information is multiplied by the second feature map to obtain a feature distribution map of the objects to be counted.
5. The method according to claim 1, It is characterized in that Determining the total number of objects to be counted based on the characteristic distribution graph includes: Upsampling the feature distribution map to obtain a density map of objects to be counted; The total number of objects to be counted is determined based on the density map.
6. The method according to claim 1, It is characterized in that The step of extracting features from the input image to obtain a first feature map and a second feature map includes: Perform initial feature extraction on the input image to obtain an initial feature map; Performing a first feature extraction on the initial feature map to obtain a first feature map; Perform a second feature extraction on the initial feature map to obtain a second feature map.
7. The method according to any one of claims 1 to 3, It is characterized in that The input image is a sample image pre-labeled with the distribution positions of the at least two objects to be counted. The counting method is implemented based on a neural network. The parameter updating process of the neural network includes: Determine a first loss based on the distribution position information and the distribution positions of at least two objects to be counted in the sample image that are pre-marked; Determining a second loss based on the density map of the objects to be counted and the pre-labeled density map of the objects to be counted; Based on the first loss and / or the second loss, parameters in the neural network are updated.
8. The method according to any one of claims 1 to 3, It is characterized in that The counting method is implemented based on a neural network, and a method for constructing a training sample and / or a test sample of the neural network includes: Obtaining at least one target sub-image of the target object according to the annotation information of the first sample image; Paste the target sub-image into the second sample image to obtain a composite image and pasting position information of the target sub-image in the composite image; The pasting position information is used as annotation information in a synthetic image to generate a synthetic sample image.
9. The method according to claim 8, It is characterized in that The step of obtaining at least one target sub-image of the target object according to the annotation information of the first sample image includes: extracting an image in the target area corresponding to the annotation information as a target sub-image; and / or, Perform at least one image transformation on each image in the target area corresponding to the annotation information, and use the image after the image transformation as the target sub-image.
10. The method according to claim 9, It is characterized in that Before obtaining at least one target sub-image of the target object according to the annotation information of the first sample image, the method further includes: According to the size information of the target area and the size information of the second sample image, a first number of target sub-images to be generated is determined; wherein the first number of target sub-images is positively correlated with the size information of the second sample image and negatively correlated with the size information of the target area.
11. The method according to claim 10, It is characterized in that After determining the number of target sub-images to be generated, performing at least one image transformation on each image in the target area corresponding to the annotation information includes: According to the first number, performing at least one image transformation on each of the images in the target area to obtain the first number of target sub-images; The image transformation includes at least one of the following: Image stretching, image shrinking, image selection, image symmetry transformation, adding noise to images.
12. A counting device, It is characterized in that include: An input image feature extraction module is used to extract features from the input image to obtain a first feature map and a second feature map; an object feature extraction module, configured to extract object features of the object to be counted from the first feature map according to position information of at least one object to be counted marked in the input image; a distribution position information determination module, configured to determine distribution position information of at least two objects to be counted in the first feature map according to the object features; A feature distribution map determining module, used to determine a feature distribution map of the object to be counted from the second feature map using the distribution position information; A total number determination module is used to determine the total number of objects to be counted based on the characteristic distribution graph.
13. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 11.
14. A computer readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
Context-aware embedded crowd counting method and system, medium and electronic equipment
CN111062274A
Guide area dense crowd counting method based on telescopic convolutional neural network
CN111985381A
