Method, device, equipment, medium and program product for determining the number of objects
The self-activated processing of the correlation feature representation between the target image and the labeled object solves the problem of feature information loss in small sample counts, and achieves more accurate target counting and more efficient model applications.
Patent Information
- Application Number
- CN202210207245.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-04
AI Technical Summary
In the small sample counting scenario, the density map regression is performed directly from the correlation results of low resolution and low channel count, resulting in a large amount of loss of feature information of the original image, increasing the counting error of the objects to be counted.
By obtaining the correlation feature representation between the target image and the labeled object, self-activated processing is performed to obtain the activation feature representation, and density data is obtained through feature regression to determine the number of objects to be counted, retain the complete feature information of the image, and reduce the counting error .
It effectively reduces the counting error of the target number, improves the retention rate of image feature information, simplifies the application process of the model, and improves the application scope of the model and the training and testing efficiency.
Smart Images

Figure CN114612402B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, equipment, medium and program product for determining the number of objects. Background Art
[0002] In the field of artificial intelligence, Few-Shot Learning is a machine learning paradigm for a small number of data sets, which is applied in computer vision technology, natural language processing and other fields. In the field of image processing, Few-Shot Learning can be applied to the task of counting objects in images, that is, Few-Shot Counting.
[0003] In the related art, the Few-Shot Adaptation & Matching Network (FamNet) proposes a small sample counting matching network and an adaptive update strategy in the test phase to achieve the above-mentioned small sample counting task. In the FamNet model, the features of the image to be identified are first extracted, and then the supporting sample features in the image features are intercepted, and the correlation operation is performed on them with the complete image features. The correlation results are sent to the regression network to obtain the density map of the objects to be counted, and the number of objects to be counted in the image can be determined based on the density map.
[0004] However, in the application process of the above model, since the density map is directly regressed from the correlation results of low resolution and low number of channels, a large amount of feature information of the original image is lost, which increases the counting error of the model for the counting objects. Summary of the invention
[0005] The embodiments of the present application provide a method, apparatus, device, medium and program product for determining the number of objects, which can reduce the target counting error in a small sample counting scenario. The technical solution is as follows:
[0006] In one aspect, a method for determining the number of objects is provided, the method comprising:
[0007] Acquire a target image including objects to be counted, wherein the objects to be counted include at least one annotated object having annotated information;
[0008] Based on the feature correlation between the target image and the annotated object, obtaining a correlation feature representation;
[0009] Using the correlation feature representation as an activation weight, self-activating the image features of the target image to obtain an activated feature representation;
[0010] Performing feature regression processing on the activated feature representation to obtain density data of the object to be counted in the target image;
[0011] The objects to be counted in the target image are accumulated based on the density data to determine the number of targets corresponding to the objects to be counted.
[0012] In another aspect, a device for determining the number of objects is provided, the device comprising:
[0013] An acquisition module, used for acquiring a target image including objects to be counted, wherein the objects to be counted include at least one annotated object with annotation information;
[0014] A prediction module, configured to obtain a correlation feature representation based on a feature correlation between the target image and the annotated object;
[0015] The prediction module is further configured to use the correlation feature representation as an activation weight to self-activate the image features of the target image to obtain an activated feature representation;
[0016] The prediction module is further used to perform feature regression processing on the activated feature representation to obtain density data of the object to be counted in the target image;
[0017] A determination module is used to accumulate the objects to be counted in the target image based on the density data to determine the target quantity corresponding to the objects to be counted.
[0018] On the other hand, a computer device is provided, wherein the terminal includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for determining the number of objects described in any one of the embodiments of the present application.
[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by a processor to implement any method for determining the number of objects described in the embodiments of the present application.
[0020] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for determining the number of objects described in any of the above embodiments.
[0021] The technical solution provided by this application includes at least the following beneficial effects:
[0022] When it is necessary to identify and count the targets in the image, a part of the targets to be counted is annotated to obtain the annotated objects, and the image features of the image itself are self-activated by the correlation feature representation between the image and the annotated object to obtain the activated feature representation, and the density of the objects to be counted in the image is determined by the feature regression of the activated feature representation, so that the number of targets corresponding to the objects to be counted can be obtained. That is, the whole image is self-activated by the correlation between the whole image and the annotated object, so that the feature information of the original whole image can be better retained while capturing the correlation information between the complete features of the image and the features of the annotated object, so as to obtain better density data and reduce the error of the determined number of targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0024] Figure 1 is a schematic diagram of the definition of a small sample counting task provided by an exemplary embodiment of the present application;
[0025] Figure 2 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;
[0026] Figure 3 is a flow chart of a method for determining the number of objects provided by an exemplary embodiment of the present application;
[0027] Figure 4 It is a functional module schematic diagram provided by an exemplary embodiment of the present application;
[0028] Figure 5 is a flow chart of a feature self-activation method provided by an exemplary embodiment of the present application;
[0029] Figure 6 is a schematic diagram of a self-activation module provided by an exemplary embodiment of the present application;
[0030] Figure 7 is a flow chart of a method for obtaining density data provided by an exemplary embodiment of the present application;
[0031] Figure 8 is a flow chart for determining a target quantity provided by an exemplary embodiment of the present application;
[0032] Fig. 9 is a flow chart of a training method of a model provided by an exemplary embodiment of the present application;
[0033] Fig.10 is a structural block diagram of a device for determining the number of objects provided by an exemplary embodiment of the present application;
[0034] Fig.11 is a structural block diagram of a device for determining the number of objects provided by another exemplary embodiment of the present application;
[0035] Fig.12 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0037] First, the nouns involved in the embodiments of the present application are briefly introduced:
[0038] Artificial intelligence: It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0039] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.
[0040] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0041] Computer Vision (CV): It is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, autonomous driving, smart transportation and other technologies, as well as common feature recognition technologies such as facial recognition and texture recognition.
[0042] Small sample learning: It is a machine learning paradigm that usually requires the model to be trained on base class samples and then given only a small number of samples on new classes that it has not learned before, so that it can quickly adapt to new classes.
[0043] Small sample counting: It is an application of small sample learning in the field of image object counting. The definition of the small sample counting task is schematically explained: Specifically, in the training phase, the model can obtain training images and two types of annotations corresponding to the training images: (a) center point annotations of all sample objects in the base category; (b) bounding box annotations of some (a small number) sample objects in the base category. After the model is trained with the above training images and the corresponding two annotations, in the testing phase, the model can only obtain bounding box annotations of the test image and several test objects of the new category, and then it is necessary to predict the specific number of all objects corresponding to the new category in the test image.
[0044] In one example, if Figure 1As shown, it shows a schematic diagram of the definition of a small sample counting task provided by an exemplary embodiment of the present application. The model learns the training target corresponding to the category of "triangle" through the training image 111 in the training stage 110. After the bounding boxes of the test targets A122, B123, and C124 in the test image 121 are annotated in the test stage 120, the model needs to predict the total number of all "diamond" test targets (new categories) in the above test image 121.
[0045] It should be emphasized that the base categories of the model during the training phase and the new categories during the test phase are completely different categories. Figure 1 In the training phase, the model learns the category of "triangle", but in the test phase, it needs to count the target of "diamond", which is a category that has not been learned. Therefore, the small sample counting task requires the model to have good and fast adaptability to new categories.
[0046] In the related art, FamNet proposes a small sample count matching network and an adaptive update strategy in the test phase to achieve the above-mentioned small sample counting task. In the FamNet model, the features of the image to be identified are first extracted, and then the supporting sample features in the image features are intercepted, and the correlation operation is performed with the complete image features. The low-resolution, low-channel correlation results obtained by the operation are sent to the regression network to obtain the density map of the objects to be counted, and the number of objects to be counted in the image can be determined based on the density map. Among them, when identifying small samples, the model as a whole includes a training phase and a testing phase. In the training phase, the model is trained by sample targets with a large amount of data. In the testing phase, the trained network model is further adapted to the test target of the test image through the minimum-number loss (Min-Count Loss) and the perturbation loss (Perturbation Loss), that is, the parameters of the network model are further iteratively trained through the minimum-number loss and the perturbation loss to obtain the network model parameters adapted to the recognition and counting of the test targets in the test image.
[0047] However, in the application of the above scheme, the following problems exist: (1) Directly regressing the target count density map from the low-resolution, low-channel number correlation map will cause information loss of the original image features, which is not conducive to the accurate prediction of the density map. (2) During the test reasoning process, the learning and updating of the network model parameters are required on the test image, which consumes a lot of test time, making the network model more limited in practical application.
[0048] The method for determining the number of objects provided in the embodiment of the present application is to obtain the correlation feature representation between the overall image and the labeled object according to the feature correlation between the target image and the partially labeled object in the target image that needs to be counted, and the image feature of the target image is self-activated through the correlation feature representation, thereby obtaining the activated feature representation, and the density data of the object to be counted is obtained by performing feature regression processing on the activated feature representation, and the number of targets in the image can be obtained according to the density data. That is, the overall image feature is self-activated through the correlation between the overall image and the labeled object, so that the feature information of the original complete image can be better retained while capturing the correlation information between the complete image feature and the labeled object feature, thereby obtaining better density data and reducing the error of the determined number of targets. At the same time, the method for determining the number of objects provided in the embodiment of the present application can be obtained by training with sample data of sample targets with a large sample size in the training stage. In the testing and application process, even if the target to be identified is different from the sample target, the model can be directly used without adaptively retraining the parameters of the model, thereby improving the scope of application of the model and the training and testing efficiency of the model.
[0049] Secondly, the application scenarios of the embodiments of the present application are schematically described:
[0050] First, it can be applied to the intelligent product quality inspection system in industrial AI. Illustratively, through the method for determining the number of objects provided in the embodiment of the present application, product defects can be identified and counted, thereby providing certain quantitative support for downstream product quality assessment. In practical applications, after the target counting model is obtained by training defect targets of known defect categories as sample data, new defect categories that have not been learned by the model may appear on the production line. In the embodiment of the present application, there is no need to adaptively train the model for new defect categories. Instead, the defect targets corresponding to the new defect categories can be partially labeled, and the defect targets of the new defect categories can be counted directly through the target counting model.
[0051] Second, it can be applied to the ecological environment protection visual AI monitoring system. Illustratively, through the method for determining the number of objects provided in the embodiment of the present application, it is possible to identify images of the intelligent image acquisition device containing dense targets to be monitored, and to predict the total number of the dense target group to be monitored, thereby providing quantitative data support for the downstream analysis of the ecological environment protection intelligent monitoring system. The above-mentioned targets to be monitored may be flocks of birds, flora, etc. that need to be monitored and protected. During the application process, when the target counting model obtained through training identifies and counts the targets to be monitored, it is only necessary to mark some of the targets to be monitored in the image at a very low cost.
[0052] Third, it can be applied to the agricultural production visual AI monitoring system. In an illustrative manner, the method for determining the number of objects provided in the embodiment of the present application can identify and count images of various categories of agricultural products of the intelligent image acquisition device, and predict the number of agricultural products to be counted, thereby providing quantitative data support for agricultural production situation analysis.
[0053] Illustratively, the above only uses three specific application scenarios as examples for illustrative explanation. The method for determining the number of objects can also be applied to other scenarios that require monitoring and counting of targets in images, such as vehicle flow monitoring systems in smart transportation systems, passenger flow thermal monitoring systems in shopping malls / attractions, etc. The application scenarios are not specifically limited here.
[0054] Combined with the above explanations of terms and descriptions of application scenarios, the implementation environment of the embodiments of the present application is described. Figure 2 As shown, the computer system of the implementation environment includes: a terminal device 210, a server 220 and a communication network 230.
[0055] The terminal device 210 includes various types of devices such as mobile phones, tablet computers, desktop computers, portable notebook computers, smart home appliances, and vehicle-mounted terminals.
[0056] The server 220 is used to provide a function of counting objects in an image, that is, the server 220 can call a corresponding operation module to count objects in a specified image to be identified according to a request of the terminal device 210 .
[0057] Illustratively, the server 220 is configured with a pre-trained target counting model, and the terminal device 210 transmits the target image and the annotation information of several corresponding annotation objects that need to be queried for the target quantity to the server 220. The server 220 inputs the target image and the annotation information corresponding to the above-mentioned annotation objects into the target counting model, and outputs the target quantity corresponding to the above-mentioned target. The server 220 transmits the above-mentioned target quantity to the terminal device 210.
[0058] In some embodiments, if the computing capability of the terminal device 210 satisfies the application process of the above target counting model, the above process of counting targets in the image to be identified can also be implemented by the terminal device 210 alone.
[0059] It is worth noting that the above-mentioned server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.
[0060] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.
[0061] In some embodiments, the server 220 may also be implemented as a node in a blockchain system.
[0062] Illustratively, the terminal device 210 and the server 220 are connected via a communication network 230 , wherein the communication network 230 may be a wired network or a wireless network, which is not limited herein.
[0063] Please refer to Figure 3 , which shows a method for determining the number of objects shown in an embodiment of the present application. In the embodiment of the present application, the method is applied to Figure 2 The server shown in the figure is used as an example for schematic illustration. In other embodiments, when the computing resources of the terminal device are sufficient, the method can also be applied to the terminal device, which is not limited here. The method includes:
[0064] Step 301: Acquire a target image including objects to be counted.
[0065] Illustratively, the objects to be counted include at least one annotated object with annotation information. In some embodiments, the target image includes a first number of objects to be counted, wherein the first number of objects to be counted includes a second number of annotated objects with annotation information, and the first number is greater than the second number. Optionally, the second number may be system-specified or user-defined, which is not limited here.
[0066] Optionally, the annotation information of the annotated object may be obtained by manual annotation. The annotation information may also be obtained by annotation through an automatic annotation module, wherein the automatic annotation module is a functional module obtained by training the target samples corresponding to the objects to be counted, and can identify and annotate the objects to be counted in the image. In this case, the automatic annotation module has a lower recognition accuracy than the overall counting model, and is used for pre-identification before the identification and counting process of the objects to be counted.
[0067] In some embodiments, the above-mentioned annotation information can be a bounding box used to select the annotated object. Schematically, the shape corresponding to the above-mentioned bounding box can be a frame of a regular shape (for example, a rectangular frame) sufficient to select the annotated object, or it can be a segmentation box for performing pixel segmentation according to the boundary of the annotated object, that is, the annotation information is a segmentation annotation of the annotated object at the pixel level, wherein the corresponding annotation boundary of the above-mentioned annotation information is the same as the boundary between the annotated object and the image background, and is not specifically limited here.
[0068] Optionally, the target image may be uploaded by a terminal device, obtained from a database, or obtained from the Internet, which is not limited here. In some embodiments, the target image may be data represented by a three-dimensional matrix, for example, the target image X∈R H×W×3 , where R represents a real number, H represents the matrix height of the three-dimensional matrix, W represents the matrix width of the three-dimensional matrix, and 3 represents the matrix length of the three-dimensional matrix (i.e., the number of channels), where the elements in the three-dimensional matrix can be the grayscale values or pixel values corresponding to the pixel points in the target image.
[0069] In some embodiments, after acquiring the target image, the target image can be feature extracted to obtain image features, and the image features are used in the subsequent data processing process. Schematically, the target image is input into a feature extraction network for feature extraction to obtain image features. Optionally, the feature extraction network can be a neural network such as a residual network (Residual Network, ResNet), a visual geometry group network (Visual Geometry Group Network, VGGNet), or an Alex Network (AlexNet), or some network structures based on transformers, such as a vision transformer (Vision Transformer, ViT), a pyramid vision transformer (Pyramid Vision Transformer, PVT), or a Swin Transformer, or some traditional manual feature extraction methods, which are not limited here.
[0070] In some embodiments, the target image is subjected to feature extraction to obtain at least one layer of image features, that is, feature extraction is performed on the target image to obtain image features of a target number of layers. Optionally, the target number of layers may be preset by the system or indicated by the terminal device, which is not limited here.
[0071] Step 302: Obtain correlation feature representation based on feature correlation between the target image and the annotated object.
[0072] Illustratively, the above-mentioned feature correlation is used to indicate the image semantic correlation between the target image and the annotated object, that is, the correlation is calculated for the semantic meanings respectively corresponding to the target image and the annotated object.
[0073] In some embodiments, the correlation feature representation between the target image and the annotated object can be obtained through a correlation calculation unit. Schematically, the correlation calculation unit performs correlation calculation on the feature level for the input target image and the annotated object with annotation information to output the correlation feature representation.
[0074] According to the target information corresponding to the annotated object, the target features corresponding to the annotated object can be obtained. By performing a convolution operation on the target features and the image features, the acquisition of the correlation feature representation can be realized. Schematically, after the target image undergoes a feature extraction operation, the image features are obtained. The target features corresponding to the annotated object can be obtained from the image features through the annotation information of the annotated object.
[0075] In some embodiments, when the annotation information is a rectangular bounding box, the correlation feature representation can be directly obtained through the convolution operation of the image feature and the target feature; in other embodiments, when the annotation information is the above-mentioned segmentation box, in order to facilitate the convolution operation process, the intercepted target feature is filled with a rectangular frame that is sufficient to select the above-mentioned segmentation box, wherein the feature data inside the rectangular frame and inside the segmentation box (i.e., the part including the annotated object) is set to the feature data of the annotated object intercepted from the image feature, and the feature data outside the segmentation box inside the rectangular frame is set to 0, and the feature data corresponding to the above-mentioned rectangular frame is used as the target feature to perform a convolution operation with the image feature.
[0076] Step 303, using the correlation feature representation as the activation weight, self-activating the image features of the target image to obtain the activated feature representation.
[0077] In the embodiment of the present application, after the correlation feature representation is obtained from the overall features of the target image and the target features of the labeled object, the correlation feature representation can be regarded as a weight multiplied back to the complete semantic features of the target image to achieve the process of feature self-activation, thereby obtaining the activated feature representation. Schematically, the above-mentioned activated feature representation is obtained by multiplying the image features of the target image and the correlation feature representation pixel by pixel.
[0078] Illustratively, in the process of self-activating the correlation feature representation and the image features of the target image, the correlation feature representation may also be preprocessed.
[0079] In some embodiments, the preprocessing process includes non-negative processing of the correlation feature representation, that is, the correlation feature representation is non-negatively processed to obtain a non-negative correlation feature representation. In other embodiments, the correlation feature representation does not need to be non-negatively processed, which can be specifically set according to actual data processing requirements and is not limited here.
[0080] In other embodiments, since the data size between the obtained correlation feature representation and the image features of the target image is different, before performing self-activation, the preprocessing process may also include copying and expanding the similarity data to align it with the image features of the target image in the channel dimension to obtain an expanded feature representation, and performing a correlation self-activation operation on the expanded feature representation and the image features of the target image. Schematically, the correlation feature representation is copied and expanded in the channel dimension to obtain an expanded feature representation, and the image features are self-activated for semantic feature correlation using the expanded feature representation as the activation weight to obtain an activated feature representation.
[0081] Step 304: perform feature regression processing on the activated feature representation to obtain density data of the objects to be counted in the target image.
[0082] After the activation feature representation is obtained, the activation feature representation can be input into the regression network, and the regression network is used for prediction to obtain density data corresponding to the object to be counted. Optionally, the above regression network can be any network that can achieve spatial resolution upsampling, and its regression process can also include various interpolation algorithms, which are not limited here.
[0083] Optionally, the output density data may be data in the form of an array, a matrix, or an image, which is not limited here.
[0084] Step 305 , accumulating the objects to be counted in the target image based on the density data, and determining the number of targets corresponding to the objects to be counted.
[0085] Illustratively, after obtaining the density data corresponding to the objects to be counted, the number of the objects to be counted in the target image can be determined by the density data. In some embodiments, the target number corresponding to the objects to be counted can be obtained by integrating the density data.
[0086] In one example, if Figure 4 As shown, it shows a schematic diagram of a functional module 400 provided by an exemplary embodiment of the present application, wherein the functional module 400 for determining the number of targets includes a feature extraction module 410, a self-activation module 420, a regression module 430, and a statistical module 440, wherein the feature extraction module 410 is used to perform feature extraction on an input target image to obtain image features corresponding to the target image, the self-activation module 420 is used to extract feature correlations between the target image and the labeled object to obtain correlation feature representations, and then perform feature correlation self-activation on the image features according to the correlation feature representations to obtain activated feature representations, the regression module 430 is used to perform feature regression processing on the activated feature representations to obtain density data, and the statistical module 440 is used to perform statistical integration on the density data to output the number of targets corresponding to the objects to be counted.
[0087] In summary, the method for determining the number of objects provided by the embodiment of the present application, when it is necessary to identify and count the targets in the image, marks a part of the targets in the objects to be counted to obtain the marked objects, and self-activates the image features of the image itself through the correlation feature representation between the image and the marked object to obtain the activated feature representation, and determines the density of the objects to be counted in the image through the feature regression of the activated feature representation, so as to obtain the number of targets corresponding to the objects to be counted. That is, the overall image features are self-activated through the correlation between the overall image and the marked object, so as to better capture the correlation information between the complete image features and the marked object features, and better retain the feature information of the original complete image, so as to obtain better density data and reduce the error of the determined number of targets.
[0088] Please refer to Figure 5 , which shows a feature self-activation method shown in an embodiment of the present application. In the embodiment of the present application, the self-activation module in the overall functional module is schematically described, wherein the self-activation module includes a process for obtaining the correlation feature representation and a process for processing the self-activation operation. The method includes:
[0089] Step 501, obtaining image features of the i-th layer in the image features.
[0090] Schematically, after the target image is subjected to feature extraction, the image features of the target layer are obtained. In the embodiment of the present application, at least one layer of image features in the target layer is used to perform a feature self-activation process to finally obtain the number of objects to be counted in the target image. i The processing process of is illustrated as an example, where h i Indicates the height of the three-dimensional matrix representing the image features in the i-th layer image features, w i represents the matrix width of the above three-dimensional matrix, c i Represents the number of channels corresponding to the image features of the i-th layer, where i is a positive integer.
[0091] Step 502 : Map the annotation information corresponding to the annotation object to the image features corresponding to the i-th layer to obtain the annotation information at the feature level.
[0092] In some embodiments, the annotation information for the annotated object is a annotation box for the annotated object based on the target image. Schematically, since the annotation information carried by the annotated object in the target image corresponds to the image size of the target image, that is, the size of the annotation information corresponds to the size of the annotated object in the target image, and after the target image is subjected to feature extraction, the sizes of the image features and the target image are different, and the spatial resolution corresponding to the image features decreases with the increase in the number of feature layers, that is, the spatial resolution is negatively correlated with the number of feature layers. If the annotation information of the annotated object is directly corresponded to the image features of the i-th layer, errors will occur due to the size difference. Therefore, the annotation information corresponding to the annotated object needs to be mapped in the spatial dimension to correspond to the image features of the i-th layer.
[0093] Indicatively, for the annotation information box corresponding to the jth annotation object among the k annotation objects j , map it to the size dimension corresponding to the image feature of the i-th layer, and obtain the annotation information box at the feature level ij , where k is a positive integer and j is a positive integer less than or equal to k.
[0094] In some embodiments, the mapping process of the above-mentioned annotation information can be implemented according to the scaling ratio between the image features of the i-th layer and the corresponding target image. For example, the first size information corresponding to the target image (such as the height or width of the spatial dimension) is obtained, and the second size information corresponding to the image features of the i-th layer is obtained. Based on the ratio between the first size information and the second size information, the annotation information is scaled to obtain the annotation information box at the feature level. ij .
[0095] Step 503, based on the feature correlation between the annotation information at the feature level and the image features at the i-th layer, obtain a correlation feature representation.
[0096] After mapping the annotation information to the image features corresponding to the i-th layer, the correlation between the annotation information and the image features can be calculated at the same feature level.
[0097] In some embodiments, the process of obtaining the correlation feature representation through the above-mentioned annotation information and image features includes:
[0098] S1: Based on the annotation information at the feature level, the image features of the i-th layer are intercepted to obtain the target features of the annotated object.
[0099] Schematically, marked information box ij When the image features of the i-th layer are at the same feature level, the information box is marked ij By intercepting the image features of the i-th layer, the target feature box_feat corresponding to the labeled object can be obtained.ij ,in, Indicates that the height, width, and number of channels are c i A three-dimensional matrix.
[0100] In some other embodiments, the target feature corresponding to the labeled object may also be a target feature corresponding to the labeled information box. j The target image is cropped to obtain the labeled object image, and the labeled object image is subjected to feature extraction through a feature extraction network to obtain the target features of the labeled object whose feature level is the same as the image features of the i-th layer.
[0101] S2: According to the box ij Perform a filling operation on the image features of the i-th layer to obtain the image filling features corresponding to the j-th labeled object in the i-th layer.
[0102] Schematically, it is necessary to perform a padding operation on the image features of the i-th layer. In one example, as shown in Formula 1, padded_feat ij According to the box ij The image filling feature of the jth labeled object in the i-th layer obtained after the filling operation, feat i is the image feature of the i-th layer, where Indicates that the height, width, and number of channels are c i The purpose of the pad operation is to ensure that the size of the feature does not decrease during the subsequent convolution operation, so as to achieve deeper convolution and retain more information in the original image during the acquisition of the correlation feature representation.
[0103] Formula 1: padded_feat ij =pad(feat i ,box ij )
[0104] S3: Perform a convolution operation on the target features of the labeled object and the image filling features of the jth labeled object in the i-th layer to obtain a correlation feature representation.
[0105] Schematically, the target feature box_feat of the labeled object ij and the image filling feature padded_feat of the jth labeled object in the i-th layer ij The correlation is calculated by convolution operation, and the above correlation feature representation is obtained. ij , as shown in Formula 2, where Indicates that the height, width and number of channels are h respectively i 、w i and 1, conv() represents the convolution operation.
[0106] Formula 2: reat ij =conv(padded_feat ij ,box_feat ij )
[0107] In other embodiments, the filling operation may be omitted, and the image features of the i-th layer may be directly convolved with the labeled object features, and then the spatial resolution may be restored to the same as the spatial resolution of the image features of the i-th layer before the convolution operation through an interpolation algorithm.
[0108] Step 504, performing non-negative processing on the correlation feature representation to obtain non-negative correlation data.
[0109] Illustratively, the preprocessing process here includes non-negative processing of the correlation feature representation, that is, performing a non-negative operation on the correlation feature representation to obtain a non-negative correlation feature representation.
[0110] In one example, as shown in Formula 3, the non-negative correlation data reat′ ij By reat ij After non-negative processing, we get: Indicates that the height, width and number of channels are h respectively i 、w i and 1, min() is the operation to get the minimum number.
[0111] Formula 3: react′ ij =reat ij -min(reat ij )
[0112] Step 505, perform data replication and expansion on the channel dimension for the non-negatively correlated data to obtain an expanded feature representation.
[0113] Since the data size between the obtained correlation feature representation and the image features of the target image is different, before self-activation, the preprocessing process also needs to include a copy and expansion operation on the non-negative correlation data to align it with the image features of the target image in terms of channel size to obtain an expanded feature representation, and a correlation self-activation operation is performed through the expanded feature representation and the image features of the above-mentioned target image.
[0114] In one example, as shown in Formula 4, the non-negatively correlated data reat′ ij Perform a copy and expansion operation on the channel dimension to make the number of channels ci , get the expanded feature representation expanded_reat ij , Indicates that the height, width and number of channels are h respectively i 、w i 、c i A three-dimensional matrix.
[0115] Formula 4: expanded_reat ij =expend(reat′ ij )
[0116] Step 506: self-activate the semantic feature correlation of the image features of the i-th layer based on the expanded feature representation to obtain an activated feature representation.
[0117] In the embodiment of the present application, the expanded feature representation is regarded as a weight multiplied back to the complete semantic features of the target image to realize the process of feature self-activation, thereby obtaining the activated feature representation. Schematically, the expanded feature representation and the image feature are multiplied pixel by pixel to obtain the activated feature representation. As shown in Formula 5, the expanded feature representation expanded_reat ij and the image feature feat of the i-th layer i Perform pixel-by-pixel multiplication This achieves the self-activation of semantic feature correlation and obtains the activated feature representation act_feat ij , Indicates that the height, width and number of channels are h respectively i 、w i 、c i A three-dimensional matrix.
[0118] Formula 5:
[0119] In one example, if Figure 6As shown, it shows a schematic diagram of a self-activation module 600 provided by an exemplary embodiment of the present application, wherein the annotation information 601 of the j-th annotation object is mapped to the image feature 602 corresponding to the i-th layer to obtain the annotation information 603 at the feature level, the image feature 602 of the i-th layer is intercepted based on the above-mentioned feature level annotation information 603 to obtain the target feature 604 corresponding to the j-th annotation object, the image feature 602 of the i-th layer is filled to obtain the image filling feature 605 of the i-th layer, the target feature 604 and the image filling feature 605 of the i-th layer are convolved to obtain the correlation feature representation 606, the correlation feature representation 606 is non-negatively processed to obtain non-negative correlation data 607, the non-negative correlation data 607 is data expanded to obtain the expanded feature representation 608, the expanded feature representation 608 and the image feature 602 of the i-th layer are self-activated to obtain the activation feature representation 609.
[0120] In summary, the feature self-activation method provided in the embodiment of the present application performs explicit correlation calculation on the semantic features of the target image and the semantic features of the annotated objects serving as supporting examples, thereby simplifying the difficulty of the network in learning and mining their correlation. At the same time, the calculated correlation feature representation is non-negatively operated and then used as a weight to act on the complete semantic features of the original target image, thereby performing self-activation of the correlation, so that the features after self-activation contain both the feature correlation between the annotated object and the target image and the rich semantic feature information about the target image. In the process of processing the features of the annotated objects, the pooling operation in the related technology is abandoned, and more feature information of the annotated objects is retained, thereby reducing the error in the determined number of targets.
[0121] Please refer to Figure 7 , which shows a method for acquiring density data according to an embodiment of the present application. In the embodiment of the present application, a process for determining the target quantity by using multiple layers of image features is schematically described, wherein the following steps 701 to 704 are executed after step 506. The method includes:
[0122] Step 701, obtaining the first activated feature representation corresponding to the image feature of the i-th layer.
[0123] Schematically, after feature extraction, the target image obtains n layers of image features, wherein the n layers of image features include the image features of the i-th layer, n is a positive integer greater than or equal to 2, and i is a positive integer. The image features of the i-th layer are input into the self-activation module, and the corresponding first activated feature representation is output, wherein the processing process of the self-activation module is shown in steps 501 to 506, which will not be described in detail here.
[0124] Step 702: Obtain a second activated feature representation corresponding to the image feature of the i+1th layer.
[0125] Schematically, the image features of the i+1th layer are the image features of the next layer of the image features of the ith layer, wherein the spatial resolution of the image features of the i+1th layer is smaller than the spatial resolution of the image features of the ith layer. The image features of the i+1th layer are input into the self-activation module, and the corresponding second activated feature representation is output.
[0126] Step 703: upsample the second activation feature representation to obtain a third activation feature representation aligned with the first activation feature representation.
[0127] In an example, taking n as 5 and i as 3, the second activation feature represents act_feat 4j The activation feature representation corresponding to the image feature of the 4th layer, the third activation feature representation act_feat′ 4j It is obtained from Formula 6, where upsample() represents the upsampling operation. Represents a three-dimensional matrix with height, width, and number of channels h3, w3, and c4 respectively.
[0128] Formula 6: act_feat′ 4j = upsample(act_feat 4j )
[0129] Step 704: Perform feature regression processing based on the first activated feature representation and the third activated feature representation to obtain density data.
[0130] Illustratively, after obtaining the first activation feature representation and the third activation feature representation, feature regression processing can be performed through the above activation feature representation to obtain density data. In some embodiments, the first activation feature representation and the third activation feature representation are merged and connected to obtain a fourth activation feature representation, and the fourth activation feature representation is input into a regression network to obtain density data, and the regression network is used to predict the density of the object to be counted for the fourth activation feature representation.
[0131] The process of merging and connecting features is shown in Formula 7, where concate() represents the operation of connecting and merging features, act_feat 3j Represents the first activated feature representation, act_feat′ 4j Represents the third activated feature representation, cat_feat j Represents the fourth activated feature obtained by concatenation and merging, cat_feat j , Represents a three-dimensional matrix with height, width, and number of channels h3, w3, and (c3+c4) respectively.
[0132] Formula 7: cat_feat j =concate(act_feat 3j ,act_feat′ 4j )
[0133] The regression processing of the feature is shown in Formula 8, where regress() represents the neural network for feature regression, and the fourth activated feature represents cat_feat j Input to the regress network and output the density data den_map j , den_map j ∈R H×W×1 , R H×W×1 Represents a matrix with height, width, and number of channels being H, W, and 1 respectively.
[0134] Formula 8: den_map j =regress(cat_feat j )
[0135] It is worth noting that the image features of the above-mentioned i+1th layer can also be the image features of the i+mth layer, wherein i+m≤n, and m is an integer greater than 1. In the above case, when upsampling is performed on the activation feature representation corresponding to the image features of the i+m layer, the number of upsampling is m times, that is, the method for obtaining density data provided in the embodiment of the present application can also be extended to be implemented by non-adjacent image feature layers. Here, only the image feature implementation of the i-th layer and the i+1-th layer is used as an example for illustration, and no specific limitation is imposed on it.
[0136] In some embodiments, when the objects to be counted include at least two labeled objects, corresponding density data is obtained for each labeled object, and the density data corresponding to each labeled object is averaged to obtain density data characterizing the density of the objects to be counted. Schematically, first density data corresponding to at least two labeled objects are obtained, and the density average between the first density data is determined as second density data. The second density data is integrated to obtain the target number corresponding to the objects to be counted.
[0137] In one example, there are k labeled objects in the target image, and the label information box of the jth labeled object j Can get the corresponding first density data den_map j , j∈{1,2,…,k}, then the second density data den_map pred The determination process is shown in Formula 9, where mean() represents the mean operation of the data, and den_map pred ∈R H×W×1, R H×W×1 Represents a matrix with height, width, and number of channels being H, W, and 1 respectively.
[0138] Formula 9: den_map pred =mean(den_map1,den_map2,…den_map k )
[0139] When the second density data den_map is obtained pred After that, the integration operation can be performed on it to obtain the target number corresponding to the objects to be counted in the target image.
[0140] Indicatively, Figure 8 As shown, it shows a target quantity determination flow chart provided by an exemplary embodiment of the present application, wherein the target quantity determination process is realized by using the image features of the third layer and the fourth layer, the target image 801 is extracted by the feature extraction module 810 to obtain four layers of image features, wherein the four layers of image features include the image features 811 of the third layer and the image features 812 of the fourth layer, the image features 811 of the third layer and the image features 812 of the fourth layer are respectively input into the self-activation module 820 for processing, and the first activation feature representation 821 corresponding to the image features 811 of the third layer and the second activation feature representation 822 corresponding to the image features 812 of the fourth layer are output. The first activation feature representation 821 and the second activation feature representation 822 are input to the intermediate processing module 830 together, wherein the second activation feature representation 822 is upsampled by the upsampling unit 831 in the intermediate processing module 830 to obtain the third activation feature representation 823, the third activation feature representation 823 and the first activation feature representation 821 are merged and connected by the connection unit 832 in the intermediate processing module 830 to obtain the fourth activation feature representation 824, the fourth activation feature representation 824 is input to the regression module 840 for regression processing to obtain density data 841, the density data 841 is input to the statistical module 850 for integration to obtain the target quantity 851.
[0141] In summary, the embodiment of the present application provides a method for obtaining density data, which obtains activation feature representation through multi-layer image features, integrates the semantic feature expression between multi-layer image features, and further reduces the error of the overall model when predicting the target quantity.
[0142] Please refer to Fig. 9 , which shows a model training method provided by an exemplary embodiment of the present application. In the embodiment of the present application, the training process of the target counting model is schematically described. The method includes:
[0143] Step 901, obtaining sample data.
[0144] The first sample target in the sample data is annotated with first annotation information, and the first sample target includes at least one second sample target having second annotation information.
[0145] Optionally, the above annotation information is a bounding box used to select the annotated object. Schematically, the shape corresponding to the above bounding box can be a rectangular frame sufficient to select the annotated object, or a frame of a shape corresponding to the annotated object, which is not specifically limited here.
[0146] In an example, the first labeling information may be labeling information used to label a center point corresponding to the first sample target, and the second labeling information may be labeling information used to label a boundary of the second sample target.
[0147] Optionally, the sample data may be uploaded by a terminal device, read from a database, or obtained from the Internet, which is not limited here.
[0148] Step 902: input the sample data and the second annotation information into the counting model to be trained, and predict the predicted density data corresponding to the first sample target.
[0149] The model structure of the counting model to be trained includes a feature extraction module, a self-activation module, an intermediate processing module, a regression module and a statistical module. The data processing process of each of the above modules is as follows: Figure 3 , 5 , 7 corresponding to the embodiments described above, will not be repeated here.
[0150] Illustratively, before training, the model parameters of the counting model to be trained are first initialized. After the parameters are initialized, the sample data is input into the counting model to be trained to predict the predicted density data corresponding to the first sample target. The predicted density data is the predicted density data obtained based on the current model parameters of the counting model to be trained.
[0151] Step 903: Based on the difference between the predicted density data and the first annotation information, the counting model to be trained is iteratively trained to obtain a target counting model.
[0152] Illustratively, the standard density data corresponding to the first sample target in the sample image can be obtained through the first annotation information corresponding to the sample data, that is, the standard density data corresponding to the sample data is obtained based on the first annotation information corresponding to the first sample target. In one example, the first annotation information is the center point of the first sample target in the sample image, and the standard density data can be obtained according to the distribution of the above center point in the sample image.
[0153] After determining the standard density data corresponding to the sample data, the overall model can be trained by comparing the error between the predicted density data and the standard density data. That is, based on the error loss value between the standard density data and the predicted density data, the counting model to be trained is iteratively trained to obtain the target counting model.
[0154] Illustratively, the above error loss value is calculated by a preset loss function. Optionally, the above preset loss function can be a regression loss function such as a mean square error (MSE) loss function, a mean absolute error (MAE) loss function, or a quantile loss (Quantile Loss) function, which is not limited here.
[0155] In an example, taking the preset loss function as a mean square error loss function as an example, the loss function is shown in Formula 10, where: The sth element representing the standard density data of the sample image, Represents the sth element of the predicted density data of the sample image, wherein the above density data are all two-dimensional matrices, H represents the matrix height of the density data, and W represents the matrix width of the density data.
[0156] Formula 10:
[0157] When the above error loss value reaches the training requirement, it is determined that the target counting model is obtained. Schematically, after the model training is completed, the target counting model can be tested by test data, wherein the test data includes a third number of test targets, the third number of test targets includes a fourth number of labeled objects, the labeled objects are labeled with labeling information, the fourth number is less than the third number, and the test data is input into the target counting model for prediction, and the corresponding test density data is output. According to the test density data, the test number of the predicted test target can be obtained, and the error of the model in prediction is determined according to the difference between the test number and the first number.
[0158] In some embodiments, the sample targets in the sample data are of the first category, and the test targets in the test data are of the second category. Optionally, the first category and the second category may be the same or different, which is not limited here.
[0159] To sum up, the model training method provided in the embodiment of the present application, in which the target counting model obtained by training with sample data does not need to perform time-consuming and resource-consuming test-time adaptive training operations even when identifying test targets that are different from training sample targets in the test phase, simplifies the test process, saves a lot of test time and computing resources, and makes the target counting model more convenient and quick in actual application, and has better practical application value.
[0160] In one example, when the target counting model provided in an embodiment of the present application is applied to the FSC-147 (Few-Shot Counting-147) dataset proposed by FamNet for experiments, without the need for additional network updates during the test phase, it is still possible to achieve MAE indicator performance that is significantly better than FamNet on both its validation set and test set.
[0161] Please refer to Fig.10 , which shows a structural block diagram of a device for determining the number of objects provided by an exemplary embodiment of the present application, the device includes the following modules:
[0162] An acquisition module 1010 is used to acquire a target image including objects to be counted, wherein the objects to be counted include at least one annotated object with annotation information;
[0163] A prediction module 1020, configured to obtain a correlation feature representation based on a feature correlation between the target image and the annotated object;
[0164] The prediction module 1020 is further configured to use the correlation feature representation as an activation weight to self-activate the image features of the target image to obtain an activated feature representation;
[0165] The prediction module 1020 is further used to perform feature regression processing on the activated feature representation to obtain density data of the object to be counted in the target image;
[0166] The determination module 1030 is configured to accumulate the objects to be counted in the target image based on the density data to determine the number of targets corresponding to the objects to be counted.
[0167] In some optional embodiments, such as Fig.11 As shown, the prediction module 1020 includes an activation submodule 1040, and the activation submodule 1040 includes:
[0168] An expansion unit 1041 is used to perform data replication and expansion on the correlation feature representation in the channel dimension to obtain an expanded feature representation;
[0169] The activation unit 1042 is used to perform self-activation of semantic feature relevance on the image feature by using the expanded feature representation as the activation weight to obtain the activated feature representation.
[0170] In some optional embodiments, the activation unit 1042 is further configured to perform a pixel-by-pixel multiplication operation on the expanded feature representation and the image feature to obtain the activated feature representation.
[0171] In some optional embodiments, the activation submodule 1040 further includes:
[0172] A non-negative processing unit 1043 is used to perform non-negative processing on the correlation feature representation to obtain non-negative correlation data;
[0173] The expansion unit 1041 is further used to perform data replication and expansion on the non-negative correlation data in the channel dimension to obtain the expanded feature representation.
[0174] In some optional embodiments, the prediction module 1020 further includes:
[0175] A feature extraction submodule 1050 is used to extract features from the target image to obtain image features of a target number of layers, where the image features of the target number of layers include image features of the i-th layer, where i is a positive integer;
[0176] The activation submodule 1040 further includes:
[0177] A mapping unit 1044 is used to map the annotation information corresponding to the annotated object to the image features corresponding to the i-th layer to obtain annotation information at the feature level;
[0178] The correlation processing unit 1045 is used to obtain the correlation feature representation based on the feature correlation between the annotation information at the feature level and the image features at the i-th layer.
[0179] In some optional embodiments, the feature extraction submodule 1050 is further used to intercept the image features of the i-th layer based on the annotation information of the feature layer to obtain the target features of the annotated object;
[0180] The activation submodule 1040 further includes:
[0181] A filling unit 1046, configured to perform a filling operation on the image features of the i-th layer to obtain image filling features of the i-th layer;
[0182] The correlation processing unit 1045 is further configured to perform a convolution operation on the target feature of the labeled object and the image filling feature of the i-th layer to obtain the correlation feature representation.
[0183] In some optional embodiments, the prediction module 1020 further includes a regression submodule 1060, and the regression submodule 1060 further includes:
[0184] An acquisition unit 1061 is used to acquire a first activated feature representation corresponding to the image feature of the i-th layer;
[0185] The acquisition unit 1061 is further used to acquire a second activated feature representation corresponding to the image feature of the i+1th layer;
[0186] an upsampling unit 1062, configured to upsample the second activation feature representation to obtain a third activation feature representation aligned with the first activation feature representation;
[0187] The regression unit 1063 is used to perform feature regression processing based on the first activated feature representation and the third activated feature representation to obtain the density data.
[0188] In some optional embodiments, the regression submodule 1060 further includes:
[0189] A merging unit 1064 is used to merge and connect the first activation feature representation and the third activation feature representation to obtain a fourth activation feature representation;
[0190] The regression unit 1063 is further used to input the fourth activation feature representation into a regression network to obtain the density data, and the regression network is used to predict the density of the object to be counted based on the fourth activation feature representation.
[0191] In some optional embodiments, the objects to be counted include at least two marked objects;
[0192] The acquisition unit 1061 is further configured to acquire first density data respectively corresponding to the at least two labeled objects;
[0193] The determining module 1030 further includes:
[0194] A determining unit 1031 is configured to determine a density mean value between the first density data as second density data;
[0195] The integration unit 1032 is used to perform an integration operation on the second density data to obtain the target number corresponding to the objects to be counted.
[0196] In some optional embodiments, the apparatus further includes a training module 1070, and the training module 1070 further includes:
[0197] An acquiring unit 1071 is configured to acquire the sample data, wherein a first sample target in the sample data is annotated with first annotation information, and the first sample targets include at least one second sample target having second annotation information;
[0198] A prediction unit 1072, configured to input the sample data and the second annotation information into the counting model to be trained, and predict the predicted density data corresponding to the first sample target;
[0199] The training unit 1073 is used to iteratively train the counting model to be trained based on the difference between the predicted density data and the first annotation information to obtain the target counting model.
[0200] In some optional embodiments, the acquisition unit 1071 is further configured to acquire standard density data corresponding to the sample data based on the first annotation information corresponding to the first sample target;
[0201] The training unit 1073 is further used to iteratively train the counting model to be trained based on the error loss value between the standard density data and the predicted density data to obtain the target counting model.
[0202] In summary, the device for determining the number of objects provided by the embodiment of the present application, when it is necessary to identify and count the targets in the image, marks a part of the targets to be counted to obtain the marked objects, and self-activates the image features of the image itself through the correlation feature representation between the image and the marked object to obtain the activated feature representation, and determines the density of the target to be counted in the image through the feature regression of the activated feature representation, so as to obtain the number of targets corresponding to the target to be counted. That is, the whole image is self-activated through the correlation between the whole image and the marked object, so as to better capture the correlation information between the complete features of the image and the features of the marked object, and better retain the complete feature information of the original image, so as to obtain better density data and ensure the accuracy of the determined number of targets.
[0203] It should be noted that the device for determining the number of objects provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining the number of objects provided in the above embodiment and the method for determining the number of objects belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0204] Fig.12The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown, which specifically includes the following structure.
[0205] The server 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The server 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.
[0206] The mass storage device 1206 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1206 and its associated computer-readable media provide non-volatile storage for the server 1200. That is, the mass storage device 1206 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.
[0207] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1204 and mass storage device 1206 can be collectively referred to as memory.
[0208] According to various embodiments of the present application, the server 1200 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1200 can be connected to the network 1212 through the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).
[0209] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.
[0210] The embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the method for determining the number of objects provided in the above-mentioned method embodiments. Optionally, the computer device can be a terminal or a server.
[0211] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method for determining the number of objects provided in the above-mentioned method embodiments.
[0212] The embodiments of the present application also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for determining the number of objects described in any of the above embodiments.
[0213] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.
[0214] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0215] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for determining the number of objects, characterized in that: The method comprises: Acquire a target image including objects to be counted, wherein the objects to be counted include at least one annotated object having annotated information; Based on the feature correlation between the target image and the annotated object, a correlation feature representation is obtained, where the correlation feature representation is used to indicate the image semantic correlation between the target image and the annotated object; Multiplying the correlation feature representation as a weight and the image feature of the target image to achieve self-activation, thereby obtaining an activation feature representation; Performing feature regression processing on the activated feature representation through a regression network to obtain density data of the objects to be counted in the target image, wherein the regression network is used to achieve spatial resolution upsampling; The objects to be counted in the target image are accumulated based on the density data to determine the number of targets corresponding to the objects to be counted.
2. The method according to claim 1, characterized in that The step of multiplying the correlation feature representation as a weight and the image feature of the target image to achieve self-activation and obtain an activation feature representation includes: Performing data replication and expansion on the correlation feature representation in the channel dimension to obtain an expanded feature representation; The expanded feature representation is multiplied by the weight and the image feature of the target image to achieve self-activation, thereby obtaining the activated feature representation.
3. The method according to claim 2, characterized in that The step of multiplying the expanded feature representation as the weight and the image feature of the target image to achieve self-activation to obtain the activated feature representation includes: The expanded feature representation and the image feature are multiplied pixel by pixel to obtain the activated feature representation.
4. The method according to claim 2, characterized in that: The step of performing data replication and expansion on the correlation feature representation in the channel dimension to obtain the expanded feature representation includes: Performing non-negative processing on the correlation feature representation to obtain non-negative correlation data; The non-negatively correlated data is replicated and expanded in the channel dimension to obtain the expanded feature representation.
5. The method according to any one of claims 1 to 4, characterized in that: The obtaining of correlation feature representation based on the feature correlation between the target image and the annotated object includes: Performing feature extraction on the target image to obtain image features of a target number of layers, wherein the image features of the target number of layers include image features of the i-th layer, where i is a positive integer; Mapping the annotation information corresponding to the annotated object to the image features corresponding to the i-th layer to obtain annotation information at the feature level; Based on the feature correlation between the annotation information at the feature level and the image features at the i-th layer, the correlation feature representation is obtained.
6. The method according to claim 5, characterized in that The obtaining the correlation feature representation based on the feature correlation between the annotation information at the feature level and the image features at the i-th layer includes: Based on the annotation information of the feature level, the image features of the i-th layer are intercepted to obtain the target features of the annotated object; Performing a filling operation on the image features of the i-th layer to obtain the image filling features of the i-th layer; A convolution operation is performed on the target feature of the labeled object and the image filling feature of the i-th layer to obtain the correlation feature representation.
7. The method according to claim 4, characterized in that The performing feature regression processing on the activated feature representation to obtain density data of the object to be counted in the target image includes: Obtain the first activated feature representation corresponding to the image feature of the i-th layer; Obtain the second activated feature representation corresponding to the image feature of the i+1th layer; Upsampling the second activation feature representation to obtain a third activation feature representation aligned with the first activation feature representation; Feature regression processing is performed based on the first activated feature representation and the third activated feature representation to obtain the density data.
8. The method according to claim 7, characterized in that The performing feature regression processing based on the first activated feature representation and the third activated feature representation to obtain the density data includes: Merging and connecting the first activation feature representation and the third activation feature representation to obtain a fourth activation feature representation; The fourth activated feature representation is input into a regression network to obtain the density data, and the regression network is used to predict the density of the object to be counted based on the fourth activated feature representation.
9. The method according to any one of claims 1 to 4, characterized in that: The objects to be counted include at least two annotated objects, and the objects to be counted in the target image are accumulated based on the density data to determine the number of targets corresponding to the objects to be counted, including: Acquire first density data respectively corresponding to the at least two labeled objects; Determine a density mean value between the first density data as second density data; An integration operation is performed on the second density data to obtain a target number corresponding to the objects to be counted.
10. The method according to any one of claims 1 to 4, characterized in that: The method is applied to a target counting model, wherein the target counting model is obtained by training a counting model to be trained through sample data, and the training process of the target counting model includes: Acquire the sample data, wherein a first sample target in the sample data is annotated with first annotation information, and the first sample targets include at least one second sample target having second annotation information; Inputting the sample data and the second annotation information into the counting model to be trained, and predicting and obtaining predicted density data corresponding to the first sample target; Based on the difference between the predicted density data and the first annotation information, the counting model to be trained is iteratively trained to obtain the target counting model.
11. The method according to claim 10, characterized in that The iterative training of the to-be-trained counting model based on the difference between the predicted density data and the first annotation information to obtain the target counting model includes: Acquire standard density data corresponding to the sample data based on the first annotation information corresponding to the first sample target; Based on the error loss value between the standard density data and the predicted density data, the counting model to be trained is iteratively trained to obtain the target counting model.
12. A device for determining the number of objects, characterized in that: The device comprises: An acquisition module, used for acquiring a target image including objects to be counted, wherein the objects to be counted include at least one annotated object with annotation information; A prediction module, configured to obtain a correlation feature representation based on a feature correlation between the target image and the annotated object, wherein the correlation feature representation is used to indicate an image semantic correlation between the target image and the annotated object; The prediction module is further configured to use the correlation feature representation as a weight and multiply the image feature of the target image to achieve self-activation, thereby obtaining an activation feature representation; The prediction module is further used to perform feature regression processing on the activated feature representation through a regression network to obtain density data of the object to be counted in the target image, and the regression network is used to achieve spatial resolution upsampling; A determination module is used to accumulate the objects to be counted in the target image based on the density data to determine the target quantity corresponding to the objects to be counted.
13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for determining the number of objects as described in any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the method for determining the number of objects as described in any one of claims 1 to 11.
15. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the method for determining the number of objects as claimed in any one of claims 1 to 11.
