Method and device for training interesting object counting model of image
By extracting features in the image and adjusting the target counting model, the problem of poor timeliness of new object categories in the prior art is solved, efficient identification and counting of targets of interest in any category is achieved, and the robustness of the model is improved. Accuracy.
Patent Information
- Application Number
- CN202411260770.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-10
AI Technical Summary
Existing target counting methods usually focus on objects of a specific category. When facing new object categories, they require repeated data collection and data annotation. The counting timeliness is poor, making it difficult to effectively identify and count any category of interest targets in the image.
A training method for the counting model of the object of interest in image is proposed. By obtaining the reference density distribution map and reference segmentation image in the sample image, the characteristics related and unrelated to the target are extracted, and the initial target counting model is trained, and the predicted density distribution map is output. and foreground segmentation images, adjust the model until the end of training, and obtain more discriminant features for the target of interest.
It improves the learning efficiency of the model and the accuracy of the target counting, ensures accurate identification and counting of targets of interest of any category, and enhances the robustness of the model.
Smart Images

Figure CN118799560B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of image processing, computer vision, and artificial intelligence technology, and in particular to a method and device for training an object of interest counting model for an image. Background Art
[0002] Object counting is an important research field in computer vision. Its main purpose is to infer the total number of any objects of interest from an image. It has important application value in security monitoring, bioinformatics, smart transportation and other fields. Existing object counting methods usually focus on counting the number of objects of a specific category, such as pedestrians, vehicles, cells, etc. When faced with new object categories, it is often necessary to repeat the steps of data collection, data labeling, and training, and the timeliness of counting is poor. Summary of the invention
[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.
[0004] To this end, the first purpose of the present application is to propose a training method for an image object of interest counting model and an image processing method to achieve counting of objects of interest of any category in the image.
[0005] The second objective of the present application is to provide a training device for an object of interest counting model of an image and an image processing device.
[0006] The third objective of the present application is to provide an electronic device.
[0007] A fourth objective of the present application is to provide a computer-readable storage medium.
[0008] A fifth object of the present application is to provide a computer program product.
[0009] To achieve the above-mentioned purpose, the first aspect of the present application proposes a training method for an image target counting model, comprising: obtaining a sample image, a reference density distribution map and a reference segmentation image of the target of interest in the sample image, and intercepting a marked image of the target of interest from the sample image; based on the marked image, performing feature extraction on the sample image to obtain a first feature set and a second feature set, wherein the first feature set includes a first initial feature related to the target of interest, and the second feature set includes a second initial feature unrelated to the target of interest; inputting the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and outputting a predicted density distribution map and a foreground segmentation image of the sample image; based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map and the reference density distribution map, adjusting the first target counting model and continuing training until the training is completed to obtain a trained second target counting model.
[0010] To achieve the above-mentioned objectives, the first aspect of the present application proposes an image processing method, comprising: obtaining an image to be counted and one or more labeled images of targets of interest in the image to be counted; inputting the image to be counted and the labeled image into a pre-trained second target counting model to obtain a density distribution map of the targets of interest in the image to be counted.
[0011] To achieve the above-mentioned purpose, the second aspect of the present application proposes a training device for an image target counting model of interest, comprising: an acquisition module, used to acquire a sample image, a reference density distribution map and a reference segmentation image of the target of interest in the sample image, and to intercept a marked image of the target of interest from the sample image; an extraction module, used to perform feature extraction on the sample image based on the marked image, to obtain a first feature set and a second feature set, wherein the first feature set includes a first initial feature related to the target of interest, and the second feature set includes a second initial feature unrelated to the target of interest; a prediction module, used to input the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and output a predicted density distribution map and a foreground segmentation image of the sample image; an adjustment module, used to adjust the first target counting model based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map and the reference density distribution map, and continue training until the training is completed to obtain a trained second target counting model.
[0012] To achieve the above-mentioned purpose, the second aspect embodiment of the present application proposes an image processing device, including: an acquisition module, used to acquire an image to be counted and one or more marked images of targets of interest in the image to be counted; a processing module, used to input the image to be counted and the marked image into a pre-trained second target counting model to obtain a density distribution map of the targets of interest in the image to be counted.
[0013] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor; and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor can execute the training method of the image object counting model of interest and the image processing method described in the first aspect embodiment.
[0014] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored, and the computer instructions are used to enable the computer to execute the training method of the image object counting model of interest and the image processing method described in the first aspect embodiment.
[0015] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the training method of the image object counting model of interest and the image processing method described in the first aspect of the above-mentioned embodiment.
[0016] The training method and device of the target counting model of an image provided by the present application can obtain a sample image and intercept a labeled image of the target of interest from the sample image, and then extract a first initial feature related to the target of interest and a second initial feature unrelated to the target of interest from the sample image based on the labeled image. Further, based on the labeled image, the sample image, the first feature set and the second feature set, the initial first target counting model is trained, and under the prompts of the first feature set and the second feature set, the model is guided to recognize background noise and directly learn the image features related to the target of interest, thereby improving the learning efficiency of the model. According to the predicted density distribution map, foreground segmentation image, reference density distribution map and reference segmentation image output by the first target counting model, the first target counting model is adjusted and continued to be trained until a trained second target counting model is obtained, so that the model learns more discriminative image features for the target of interest, improves the accuracy of matching image features and sample features, and thus ensures the accuracy of target counting and the robustness of the model.
[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0019] Figure 1 A flowchart of a method for training an object of interest counting model for an image provided in an embodiment of the present application;
[0020] Figure 2 A schematic flow chart of a process for obtaining a predicted density distribution map in a training method for an object of interest counting model of an image provided in an embodiment of the present application;
[0021] Figure 3 A flowchart of another method for training an object of interest counting model for an image provided in an embodiment of the present application;
[0022] Figure 4 A schematic diagram of the structure of the target counting model provided in the embodiment of the present application;
[0023] Figure 5 A schematic diagram of a process for training a target counting model provided in an embodiment of the present application;
[0024] Figure 6 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;
[0025] Figure 7 A schematic diagram of a process for counting objects of interest in an image provided by an embodiment of the present application;
[0026] Figure 8 A schematic diagram of the structure of a training device for an interesting object counting model of an image provided by an embodiment of the present application;
[0027] Fig. 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0029] The following describes a training method for an object of interest counting model for an image, an image processing method and a device according to an embodiment of the present application with reference to the accompanying drawings.
[0030] Figure 1 is a flowchart of a method for training an object of interest counting model of an image according to an exemplary embodiment. Figure 1 As shown, the training method of the object of interest counting model of the image in the embodiment of the present application includes but is not limited to the following steps:
[0031] S101, obtaining a sample image, a reference density distribution map of an object of interest in the sample image, and a reference segmentation image, and intercepting a marked image of the object of interest from the sample image.
[0032] It should be noted that the execution subject of the training method of the object of interest counting model of the image provided in the embodiment of the present application is a server, which has a certain computing power support and a certain graphics processing unit (GPU) memory. Optionally, the server can be a cloud server, a server of a distributed system, or a server combined with a blockchain, etc. The embodiment of the present application is not specifically limited.
[0033] It is understood that the target of interest refers to a specific area or object in the image that needs to be analyzed and processed. These targets can be any identifiable elements in the image, including but not limited to faces, vehicles, buildings, animals, objects of specific shapes or colors, etc.
[0034] In some implementations, multiple sample images can be selected from a data set, where the images in the data set are images with the center point coordinates of the objects of interest labeled, and the data set has a certain scale. In other words, the number of images in the data set reaches a certain order of magnitude, which is sufficient to support effective data analysis, model training or verification. For example, images can be selected as sample images from the Few-Shot Counting Dataset with 147 Categories, FSC147. The FSC147 data set provides the center point coordinates of each object for the same category of objects of interest in the image, as well as the random box coordinates of multiple objects of interest.
[0035] Optionally, based on the coordinates of the center point of the object of interest marked in the sample image, the density distribution map and the segmented image of the object of interest may be inferred to obtain a reference density distribution map and a reference segmented image of the sample image.
[0036] In some implementations, by obtaining random box coordinates of the objects of interest in the sample image, a labeled image of the object of interest can be captured from the sample image.
[0037] S102, based on the marked image, extract features from the sample image to obtain a first feature set and a second feature set, wherein the first feature set includes first initial features related to the object of interest, and the second feature set includes second initial features unrelated to the object of interest.
[0038] In some implementations, the sample image is preprocessed and, based on the labeled image, the preprocessed sample image is segmented into a foreground area containing the object of interest and a background area not containing the object of interest.
[0039] Furthermore, by extracting feature data such as color features, texture features, and shape features from the foreground area, a first initial feature related to the target of interest can be obtained, and a first feature set can be obtained based on the first initial feature.
[0040] Optionally, by extracting feature data such as global features and local features from the background area, a second initial feature that is unrelated to the target of interest is obtained, and a second feature set is obtained based on the second initial feature.
[0041] S103, inputting the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and outputting a predicted density distribution map of the sample image and a foreground segmentation image.
[0042] In some implementations, the first target counting model can obtain a predicted density distribution map of the sample image based on a similarity matrix of the target of interest. Optionally, the similarity matrix of the target of interest can be calculated based on the marker features corresponding to the marker image and the image features corresponding to the sample image, and then the density of the similarity matrix can be estimated to obtain a predicted density distribution map.
[0043] In some implementations, the sample image is segmented to obtain a foreground segmented image of the sample image. Alternatively, the sample image may be segmented based on the marker features to obtain a foreground segmented image related to the object of interest.
[0044] Optionally, by inputting the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, the first target counting model can perform feature extraction on the marked image and the sample image based on the first initial features and the second initial features in the first feature set and the second feature set to obtain the marked features corresponding to the marked image and the image features corresponding to the sample image.
[0045] S104, based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map and the reference density distribution map, the first target counting model is adjusted and training is continued until the training is completed to obtain a trained second target counting model.
[0046] In some implementations, the loss function of the first target counting model can be calculated, and based on the loss function, the model parameters of the first target counting model can be adjusted, and the adjusted first target counting model can be continuously trained until the training is completed to obtain a trained second target counting model.
[0047] Optionally, a loss value between the predicted density distribution map and the reference density distribution map is calculated by a loss function, and the loss value is used as the first loss function of the first target counting model. The model parameters are optimized by a back propagation algorithm to adjust the first target counting model.
[0048] Optionally, the loss value between the foreground segmentation image and the reference segmentation image of the sample image can be calculated through a loss function, and the loss value can be used as the second loss function of the first target counting model. The model parameters can be optimized through a back propagation algorithm to adjust the first target counting model.
[0049] That is to say, the first loss function and the second loss function can be used to optimize and update the model parameters, and the optimized model can be continuously trained until the training is completed to obtain a trained second target counting model.
[0050] Optionally, the first target counting model may meet the training end condition, which is regarded as the end of training. Optionally, the training end condition may be that the number of training times reaches a set number of times; the training end condition may also be that the accuracy of the predicted density distribution map is greater than an accuracy threshold.
[0051] In the training method of the target counting model of an image provided by the embodiment of the present application, by acquiring a sample image and intercepting a labeled image of the target of interest from the sample image, the first initial feature related to the target of interest and the second initial feature unrelated to the target of interest can be extracted from the sample image based on the labeled image. Further, based on the labeled image, the sample image, the first feature set and the second feature set, the initial first target counting model is trained, and under the prompts of the first feature set and the second feature set, the model is guided to recognize background noise and directly learn the image features related to the target of interest, thereby improving the learning efficiency of the model. According to the predicted density distribution map, foreground segmentation image, reference density distribution map and reference segmentation image output by the first target counting model, the first target counting model is adjusted and continued to be trained until a trained second target counting model is obtained, so that the model learns more discriminative image features for the target of interest, improves the accuracy of matching image features and sample features, and thus ensures the accuracy of target counting and the robustness of the model.
[0052] Based on the above embodiments, the present application embodiment can explain the process of obtaining the predicted density distribution map, such as Figure 2 As shown, the process of obtaining the predicted density distribution map in the embodiment of the present application includes but is not limited to the following steps:
[0053] S201, using a first target counting model to perform word segmentation on a sample image and a labeled image to obtain sample word segmentations of the sample image and labeled word segmentations of the labeled image.
[0054] Optionally, the first target counting model can segment the image based on a convolutional neural network (CNN), by extracting a feature map of the image and using a specific part of the feature map, such as a certain area, a block, or the entire feature map, as the image segmentation.
[0055] That is to say, the first target counting model can perform word segmentation on the sample image and the labeled image respectively to obtain sample word segmentation and labeled word segmentation.
[0056] S202, input the first initial feature and the second initial feature, as well as the sample word segmentation and the tag word segmentation into the feature extraction part of the first target counting model, and the self-attention mechanism of the feature extraction part processes the first initial feature and the second initial feature, as well as the sample word segmentation and the tag word segmentation to obtain an updated first target feature and an updated second target feature, as well as an image feature corresponding to the sample image and a tag feature corresponding to the tag image.
[0057] In some implementations, the feature extraction part of the first target counting model can use the first initial feature and the second initial feature as prompt information based on the self-attention mechanism to update the sample segmentation and the tag segmentation, and obtain the image features corresponding to the sample image, the tag features corresponding to the tag image, the updated first target features and the updated second target features.
[0058] It is understandable that the self-attention mechanism is a technique widely used in deep learning models, especially when processing sequence data. The core idea is to allow the model to pay attention to other elements in the sequence when processing each element, so as to learn the dependencies between elements. In the field of images, the self-attention mechanism is used to capture the dependencies between different regions in the image, thereby improving the model's ability to understand image features. In this case, the self-attention mechanism is used to capture the dependencies between the sample image, the example image, and the first and second initial features.
[0059] For example, Figure 4 As shown, the structure of the first target counting model includes: a converter module, a linear similarity calculation module and a density estimation module, wherein the converter module corresponds to the feature extraction part in the above embodiment.
[0060] The first initial feature, the second initial feature, the sample segmentation and the tag segmentation can be input into the converter module. The converter module uses the first initial feature and the second initial feature as prompts based on the self-attention mechanism to guide the converter module to distinguish between foreground targets and background noise, and then update the sample segmentation and the tag segmentation to obtain more discriminative image features and tag features for the target of interest, as well as the first target feature and the second target feature.
[0061] S203: Acquire a first predicted density distribution map of the sample image based on the image features and the mark features.
[0062] In some implementations, the first target counting model can perform similarity calculation on the image features and the mark features to obtain a first similarity matrix of the target of interest, and then perform density estimation on the first similarity matrix to obtain a first predicted density distribution map.
[0063] Optionally, a linear similarity calculation method may be used to perform linear similarity calculation on image features and marker features to obtain a first similarity matrix of the object of interest.
[0064] like Figure 4 As shown, the image features and mark features output by the converter module are input into the linear similarity calculation module, the first similarity matrix of the target of interest is calculated, and the first similarity matrix is input into the density estimation module to obtain the first predicted density distribution map of the sample image.
[0065] Optionally, the formula for calculating the first similarity matrix is as follows:
[0066] (1)
[0067] in, R k represents the first similarity matrix, K Indicates the number of marker features, F Represents image features, Indicates k A marking feature.
[0068] S204: Acquire a second predicted density distribution map of the sample image based on the image feature and the first target feature.
[0069] In some implementations, since the first target feature is a feature related to the target of interest, it can be regarded as another representation of the marking feature. Therefore, the first target counting model can calculate the similarity between the image feature and the first target feature to obtain a second similarity matrix of the target of interest, and then perform density estimation on the second similarity matrix to obtain a second predicted density distribution map.
[0070] Optionally, the second similarity matrix may be calculated based on the above formula (1). The second similarity matrix may be calculated by replacing the marker features in formula (1) with the corresponding first target features.
[0071] In the training method of the target counting model of interest in an image provided in an embodiment of the present application, a first target counting model can obtain image features and marking features based on a marked image, a sample image, a first feature set and a second feature set, and obtain a first predicted density distribution map based on the image features and the marking features, and obtain a second predicted density distribution map based on the image features and the first target features. By training the initial first target counting model with the predicted density distribution map and the reference density distribution map, features related to the target of interest can be directly learned, thereby improving the learning efficiency of the model, thereby ensuring the accuracy of target recognition and counting and the robustness of the model.
[0072] Figure 3 is a flowchart of a method for training an object of interest counting model of an image according to an exemplary embodiment. Figure 3 As shown, the training method of the object of interest counting model of the image in the embodiment of the present application includes but is not limited to the following steps:
[0073] S301, obtaining a sample image, a reference density distribution map of an object of interest in the sample image, and a reference segmentation image, and intercepting a marked image of the object of interest from the sample image.
[0074] In the embodiment of the present application, the implementation method of step S301 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.
[0075] S302, based on the marked image, extract features from the sample image to obtain a first feature set and a second feature set, wherein the first feature set includes first initial features related to the object of interest, and the second feature set includes second initial features unrelated to the object of interest.
[0076] In the embodiment of the present application, the implementation method of step S302 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.
[0077] S303, inputting the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and outputting a predicted density distribution map of the sample image and a foreground segmentation image.
[0078] In the embodiment of the present application, the implementation method of step S303 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.
[0079] In some implementations, the first target counting model can determine, based on the labeled image, the sample image, the first feature set, and the second feature set, a first target feature that is updated after the first initial feature in the first feature set, a second target feature that is updated after the second initial feature in the second feature set, and image features corresponding to the sample image and labeled features corresponding to the labeled image.
[0080] Optionally, the sample image may be segmented based on the marker feature, the second target feature and the image feature to obtain a first foreground segmented image. The sample image may also be segmented based on the first target feature, the second target feature and the image feature to obtain a second foreground segmented image. For example, Figure 4 The foreground segmentation module in can perform segmentation based on the marking feature, the second target feature and the image feature to obtain a first foreground segmentation image.
[0081] S304: Obtain a first loss function of a first target counting model based on the predicted density distribution map and the reference density distribution map.
[0082] In some implementations, the first target counting model can output two predicted density distribution maps, namely a first predicted density distribution map and a second predicted density distribution map. For any predicted density distribution map in the first predicted density distribution map and the second predicted density distribution map, based on any predicted density distribution map and the reference density distribution map, a first loss function of the first target counting model is obtained, and then the model parameters of the first target counting model can be adjusted based on the first loss function.
[0083] That is, the first loss function may be determined according to the reference density distribution map of the sample image and the first predicted density distribution map. The first loss function may also be determined according to the reference density distribution map of the sample image and the second predicted density distribution map.
[0084] S305 , determining a second loss function according to the reference segmentation image and the foreground segmentation image of the sample image.
[0085] In some implementations, the first target counting model can output two foreground segmentation images, namely a first foreground segmentation image and a second foreground segmentation image. For any foreground segmentation image of the first foreground segmentation image and the second foreground segmentation image, a second loss function of the first target counting model is determined based on any foreground segmentation image and a reference segmentation image.
[0086] That is, the second loss function may be determined based on the reference segmented image of the sample image and the first foreground segmented image. The second loss function may also be determined based on the reference segmented image of the sample image and the second foreground segmented image.
[0087] S306: Based on the first loss function and the second loss function, adjust the model parameters of the first target counting model, and continue training until the training is completed to obtain a trained second target counting model.
[0088] Optionally, by calculating the loss value between any predicted density distribution map and the reference density distribution map, and using the loss value as the first loss function of the first target counting model, by calculating the loss value between the reference segmentation image and any foreground segmentation image, and using the loss value as the second loss function, and based on the first loss function and the second loss function, the model parameters are optimized by the back propagation algorithm to adjust the first target counting model.
[0089] Optionally, the first target counting model may meet the training end condition, which is regarded as the end of training. Optionally, the training end condition may be that the number of training times reaches a set number of times; the training end condition may also be that the accuracy of the predicted density distribution map is greater than an accuracy threshold.
[0090] In the training method of the target counting model of interest in an image provided in an embodiment of the present application, a first loss function can be determined based on any one of the first predicted density distribution map and the second predicted density distribution map, and a reference density distribution map, and a second loss function can be determined based on any one of the first foreground segmentation image and the second foreground segmentation image, and a reference segmentation image, so as to adjust the model parameters of the first target counting model according to the first loss function and the second loss function and continue training until a trained second target counting model is obtained, so that the model learns more discriminative features for the target of interest, improves the accuracy of feature matching, and thereby ensures the accuracy of target counting and the robustness of the model.
[0091] For example, Figure 4 Schematic diagram of the structure of the target counting model shown. Figure 4 In the embodiment, the model structure includes a converter module and an output prediction module, wherein the output prediction module includes a foreground segmentation module, a similarity calculation module and a density estimation module. The converter module corresponds to the feature extraction part in the above embodiment.
[0092] By inputting the first initial feature, the second initial feature, the sample image and the labeled image into the converter module in the target counting model, the converter module obtains the sample segmentation of the sample image and the labeled segmentation of the labeled image, and based on the self-attention mechanism, converts the first initial feature and the second initial feature into prompt information, that is, foreground prompt and background prompt, updates the sample segmentation and the labeled segmentation, and obtains image features and label features, as well as first target features and second target features.
[0093] Furthermore, in the output prediction module, the foreground segmentation module segments the sample image based on the marker feature, the second target feature and the image feature to obtain a first foreground segmentation image, and the sample image can also be segmented based on the first target feature, the second target feature and the image feature to obtain a second foreground segmentation image. Then, the image features and the marker features can be input into the similarity calculation module to perform linear similarity calculation to obtain a first similarity matrix, and the density estimation module performs density estimation on the first similarity matrix to obtain a first predicted density distribution map. The image features and the first target features can also be input into the similarity calculation module to perform similarity calculation to obtain a second similarity matrix, and the density estimation module performs density estimation on the second similarity matrix to obtain a second predicted density distribution map. Among them, the first foreground segmentation image, the second foreground segmentation image, the first predicted density distribution map and the second predicted density distribution map can determine the loss function of the model, and the model can be returned and corrected based on the loss function and continued to be trained until a trained target counting model is obtained.
[0094] Figure 5The figure shows a flow chart of training a target counting model. The sample image, the labeled image, and the first feature set and the second feature set extracted based on the labeled image are input into the first target counting model, and the model updates the sample word segmentation of the sample image and the labeled word segmentation of the labeled image based on the first initial feature in the first feature set and the second initial feature in the second feature set, thereby extracting features and obtaining image features corresponding to the sample image and labeled features corresponding to the labeled image, as well as the first target feature and the second target feature.
[0095] Optionally, a linear similarity calculation is performed on the image features and the mark features to obtain a first similarity matrix, and density estimation is performed on the first similarity matrix to obtain a first predicted density distribution map of the sample image; a linear similarity calculation is performed on the image features and the first target features to obtain a second similarity matrix, and density estimation is performed on the second similarity matrix to obtain a second predicted density distribution map of the sample image.
[0096] Optionally, the sample image is segmented based on the marker feature, the second target feature and the image feature to obtain a first foreground segmented image, and the sample image is segmented based on the first target feature, the second target feature and the image feature to obtain a second foreground segmented image.
[0097] Furthermore, for any one of the first predicted density distribution map and the second predicted density distribution map, and based on any one of the predicted density distribution map and the reference density distribution map, a first loss function of the model is determined, and for any one of the first foreground segmentation image and the second foreground segmentation image, and based on any one of the foreground segmentation image and the reference segmentation image, a second loss function of the model is determined. Then, the model can be adjusted based on the first loss function and the second loss function and training can be continued until a trained second target counting model is obtained.
[0098] Figure 6 is a flowchart of a method for processing an image according to an exemplary embodiment. Figure 6 As shown, the image processing method of the embodiment of the present application includes but is not limited to the following steps:
[0099] S601, obtaining an image to be counted and one or more marked images of objects of interest in the image to be counted.
[0100] In some implementations, any image can be selected from an image library as the image to be counted, and / or an image can be downloaded from the Internet as the image to be counted, and / or an image can be captured by an image acquisition device as the image to be counted, and / or an image created by a user through software can be used as the image to be counted.
[0101] Furthermore, the target of interest in the image to be counted is determined, and the target of interest is marked to obtain one or more marked images of the target of interest. For example, suppose the image to be counted contains fruits such as bananas, apples, and peaches, and the target of interest is determined to be apple, and then one or more apples are marked to obtain a marked image.
[0102] S602, inputting the image to be counted and the marked image into a pre-trained second target counting model to obtain a density distribution map of the target of interest in the image to be counted.
[0103] It should be noted that the second target counting model can be used Figures 1 to 5 The training method of the target counting model for the image shown is obtained and will not be repeated here.
[0104] Optionally, the second target counting model performs feature extraction on the image to be counted and the marked image to obtain image features corresponding to the image to be counted and marking features corresponding to the marked image. Optionally, feature extraction can be performed on the image to be counted and the marked image according to the feature extraction part in the first target counting model.
[0105] Furthermore, the second target counting model can perform similarity calculation on the image features and the mark features to obtain a density distribution map. In other words, the similarity calculation can be performed on the image features and the mark features to obtain a similarity matrix, and the density of the similarity matrix can be estimated to obtain a density distribution map of the image to be counted.
[0106] Further, the density distribution map is summed to obtain the number of objects of interest in the image to be counted. Optionally, the number of objects of interest in the density distribution map may be summed based on a summation algorithm.
[0107] In the image processing method provided in the embodiment of the present application, the image to be counted and the marked image of the target of interest are input into a trained second target counting model, and the second target counting model is used to identify the target of interest in the image to be counted to obtain a density distribution map of the target of interest, and then the number of targets of interest can be calculated, thereby accurately identifying the target of interest and reducing the possibility of misidentification and missed identification.
[0108] Figure 7 The figure shows a schematic diagram of a process for counting objects of interest in an image. Figure 7The second target counting model in the present application is used for counting. By inputting the image to be counted and the marked image of the target of interest into the second target counting model, the feature extraction part in the second target counting model extracts features of the image to be counted and the marked image, obtains the image features corresponding to the image to be counted and the marked features corresponding to the marked image, and then calculates the similarity based on the image features and the marked features to obtain a similarity matrix, so as to obtain a density distribution map according to the similarity matrix. Furthermore, after obtaining the density distribution map, the density distribution map is summed to obtain the number of targets of interest in the image to be counted.
[0109] In order to implement the above embodiment, the present application also proposes a training device for an object of interest counting model of an image.
[0110] Figure 8 A schematic diagram of the structure of a training device for an object of interest counting model for an image provided in an embodiment of the present application.
[0111] like Figure 8 As shown, the training device 800 of the object counting model of interest in the image includes:
[0112] An acquisition module 801 is used to acquire a sample image, a reference density distribution map and a reference segmentation image of an object of interest in the sample image, and to intercept a marked image of the object of interest from the sample image;
[0113] An extraction module 802 is used to perform feature extraction on the sample image based on the marked image to obtain a first feature set and a second feature set, wherein the first feature set includes a first initial feature related to the object of interest, and the second feature set includes a second initial feature unrelated to the object of interest;
[0114] A prediction module 803, configured to input the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and output a predicted density distribution map and a foreground segmentation image of the sample image;
[0115] The adjustment module 804 is used to adjust the first target counting model based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map and the reference density distribution map and continue training until the training is completed to obtain a trained second target counting model.
[0116] In a possible implementation manner of the embodiment of the present application, the prediction module 803 is further used to: perform word segmentation on the sample image and the labeled image by the first target counting model to obtain sample word segmentation of the sample image and labeled word segmentation of the labeled image; input the first initial feature and the second initial feature, as well as the sample word segmentation and the labeled word segmentation into the feature extraction part of the first target counting model, and process the first initial feature and the second initial feature, as well as the sample word segmentation and the labeled word segmentation by the self-attention mechanism of the feature extraction part to obtain an updated first target feature and an updated second target feature, as well as an image feature corresponding to the sample image and a labeled feature corresponding to the labeled image; obtain a first predicted density distribution map of the sample image based on the image feature and the labeled feature; and obtain a second predicted density distribution map of the sample image based on the image feature and the first target feature.
[0117] In a possible implementation of the embodiment of the present application, the prediction module 803 is further used to: calculate the similarity between the image features and the mark features to obtain a first similarity matrix of the target of interest; and perform density estimation on the first similarity matrix to obtain the first predicted density distribution map.
[0118] In a possible implementation of the embodiment of the present application, the prediction module 803 is further used to: calculate the similarity between the image features and the first target features to obtain a second similarity matrix of the target of interest; and perform density estimation on the second similarity matrix to obtain the second predicted density distribution map.
[0119] In a possible implementation manner of the embodiment of the present application, the prediction module 803 is further used to: segment the sample image based on the marking feature, the second target feature and the image feature to obtain a first foreground segmented image; segment the sample image based on the first target feature, the second target feature and the image feature to obtain a second foreground segmented image.
[0120] In a possible implementation of an embodiment of the present application, the adjustment module 804 is also used to: for any predicted density distribution map among the first predicted density distribution map and the second predicted density distribution map, obtain a first loss function of the first target counting model based on the any predicted density distribution map and the reference density distribution map; for any foreground segmentation image among the first foreground segmentation image and the second foreground segmentation image, determine a second loss function of the first target counting model based on the any foreground segmentation image and the reference segmentation image; and adjust model parameters of the first target counting model based on the first loss function and the second loss function.
[0121] In the training device of the target counting model of an image provided by the embodiment of the present application, by acquiring a sample image and intercepting a labeled image of the target of interest from the sample image, the first initial feature related to the target of interest and the second initial feature unrelated to the target of interest can be extracted from the sample image based on the labeled image. Further, based on the labeled image, the sample image, the first feature set and the second feature set, the initial first target counting model is trained, and under the prompts of the first feature set and the second feature set, the model is guided to recognize background noise and directly learn the image features related to the target of interest, thereby improving the learning efficiency of the model. According to the predicted density distribution map, foreground segmentation image, reference density distribution map and reference segmentation image output by the first target counting model, the first target counting model is adjusted and continued to be trained until a trained second target counting model is obtained, so that the model learns more discriminative image features for the target of interest, improves the accuracy of matching image features and sample features, and thus ensures the accuracy of target counting and the robustness of the model.
[0122] It should be noted that the explanation of the aforementioned embodiment of the method for training an object of interest counting model for an image is also applicable to the training device for the object of interest counting model for an image of this embodiment, and will not be repeated here.
[0123] In order to implement the above embodiment, the present application also proposes an image processing device.
[0124] Fig. 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.
[0125] like Fig. 9 As shown, the image processing device 900 includes:
[0126] An acquisition module 901 is used to acquire an image to be counted and one or more marked images of an object of interest in the image to be counted;
[0127] The processing module 902 is used to input the image to be counted and the marked image into a pre-trained second target counting model to obtain a density distribution map of the target of interest in the image to be counted.
[0128] In a possible implementation of an embodiment of the present application, the processing module 902 is also used to: perform feature extraction on the image to be counted and the marked image by the second target counting model to obtain image features corresponding to the image to be counted and marking features corresponding to the marked image; perform similarity calculation on the image features and the marking features to obtain the density distribution map.
[0129] In a possible implementation manner of the embodiment of the present application, the processing module 902 is further configured to: sum the density distribution map to obtain the number of objects of interest in the image to be counted.
[0130] In the image processing device provided in the embodiment of the present application, the image to be counted and the target marked image of the target of interest are input into a trained second target counting model, and the second target counting model is used to identify the target of interest in the image to be counted to obtain a density distribution map of the target of interest, and then the number of targets of interest can be calculated, thereby accurately identifying the target of interest and reducing the possibility of misidentification and missed identification.
[0131] It should be noted that the above explanation of the embodiment of the image processing method is also applicable to the image processing device of this embodiment, and will not be repeated here.
[0132] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.
[0133] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.
[0134] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.
[0135] The collection, storage, use, processing, transmission, provision and application of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.
[0136] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.
[0137] This application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, this application is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.
[0138] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0139] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0140] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0142] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0143] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0144] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0145] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for training an interesting object counting model of an image, characterized in that: The method comprises: Acquire a sample image, a reference density distribution map and a reference segmentation image of an object of interest in the sample image, and intercept a marked image of the object of interest from the sample image; Based on the marked image, feature extraction is performed on the sample image to obtain a first feature set and a second feature set, wherein the first feature set includes first initial features related to the object of interest, and the second feature set includes second initial features unrelated to the object of interest; Inputting the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and outputting a predicted density distribution map and a foreground segmentation image of the sample image; Based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map, and the reference density distribution map, adjusting the first target counting model and continuing training until the training is completed to obtain a trained second target counting model; The process of obtaining the predicted density distribution map of the sample image includes: The first target counting model performs word segmentation on the sample image and the labeled image to obtain sample word segmentation of the sample image and labeled word segmentation of the labeled image; Inputting the first initial feature and the second initial feature, as well as the sample segmentation and the marked segmentation into the feature extraction part of the first target counting model, using the self-attention mechanism of the feature extraction part, taking the first initial feature and the second initial feature as prompt information, updating the sample segmentation and the marked segmentation, and obtaining image features corresponding to the sample image and marked features corresponding to the marked image, as well as updated first target features and updated second target features, wherein the first target feature is a feature related to the target of interest; Based on the image feature and the marking feature, obtaining a first predicted density distribution map of the sample image; Based on the image feature and the first target feature, obtaining a second predicted density distribution map of the sample image; The step of acquiring a first predicted density distribution map of the sample image based on the image feature and the mark feature includes: Calculating similarity between the image features and the marking features to obtain a first similarity matrix of the object of interest; Performing density estimation on the first similarity matrix to obtain the first predicted density distribution map; The acquiring, based on the image feature and the first target feature, a second predicted density distribution map of the sample image comprises: Calculating similarity between the image features and the first target features to obtain a second similarity matrix of the target of interest; Perform density estimation on the second similarity matrix to obtain the second predicted density distribution map.
2. The method according to claim 1, characterized in that The process of acquiring the foreground segmentation image includes: Segmenting the sample image based on the marking feature, the second target feature and the image feature to obtain a first foreground segmentation image; The sample image is segmented based on the first target feature, the second target feature and the image feature to obtain a second foreground segmentation image.
3. The method according to claim 2, characterized in that The first target counting model is adjusted and trained continuously based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map and the reference density distribution map until the training is completed to obtain a trained second target counting model, including: For any one of the first predicted density distribution map and the second predicted density distribution map, based on the any one predicted density distribution map and the reference density distribution map, obtaining a first loss function of the first target counting model; For any foreground segmentation image of the first foreground segmentation image and the second foreground segmentation image, determine a second loss function of the first object counting model based on the any foreground segmentation image and the reference segmentation image; Based on the first loss function and the second loss function, model parameters of the first object counting model are adjusted.
4. A method for processing an image, characterized in that: The method comprises: Acquire an image to be counted and one or more marked images of objects of interest in the image to be counted; Inputting the image to be counted and the marked image into a pre-trained second target counting model to obtain a density distribution map of the target of interest in the image to be counted; The second target counting model is a model obtained based on the training method according to any one of claims 1 to 3.
5. The method according to claim 4, characterized in that The step of inputting the image to be counted and the marked image into a pre-trained second target counting model to obtain a density distribution map of the target of interest in the image to be counted includes: The second object counting model performs feature extraction on the image to be counted and the marked image to obtain image features corresponding to the image to be counted and marked features corresponding to the marked image; A similarity calculation is performed on the image feature and the mark feature to obtain the density distribution map.
6. The method according to claim 4, characterized in that The method further comprises: The density distribution graphs are summed to obtain the number of objects of interest in the image to be counted.
7. A training device for an interesting object counting model of an image, characterized in that: The device comprises: An acquisition module, used to acquire a sample image, a reference density distribution map and a reference segmentation image of an object of interest in the sample image, and intercept a marked image of the object of interest from the sample image; an extraction module, configured to perform feature extraction on the sample image based on the marked image to obtain a first feature set and a second feature set, wherein the first feature set includes first initial features related to the object of interest, and the second feature set includes second initial features unrelated to the object of interest; A prediction module, used for inputting the marked image, the sample image, the first feature set and the second feature set into an initial first target counting model, and outputting a predicted density distribution map and a foreground segmentation image of the sample image; An adjustment module, configured to adjust the first target counting model based on the foreground segmentation image, the reference segmentation image, the predicted density distribution map, and the reference density distribution map, and continue training until the training is completed to obtain a trained second target counting model; The process of obtaining the predicted density distribution map of the sample image includes: The first target counting model performs word segmentation on the sample image and the labeled image to obtain sample word segmentation of the sample image and labeled word segmentation of the labeled image; Inputting the first initial feature and the second initial feature, as well as the sample segmentation and the marked segmentation into the feature extraction part of the first target counting model, using the self-attention mechanism of the feature extraction part, taking the first initial feature and the second initial feature as prompt information, updating the sample segmentation and the marked segmentation, and obtaining image features corresponding to the sample image and marked features corresponding to the marked image, as well as updated first target features and updated second target features, wherein the first target feature is a feature related to the target of interest; Based on the image feature and the marking feature, obtaining a first predicted density distribution map of the sample image; Based on the image feature and the first target feature, obtaining a second predicted density distribution map of the sample image; The step of acquiring a first predicted density distribution map of the sample image based on the image feature and the mark feature includes: Calculating similarity between the image features and the marking features to obtain a first similarity matrix of the object of interest; Performing density estimation on the first similarity matrix to obtain the first predicted density distribution map; The acquiring, based on the image feature and the first target feature, a second predicted density distribution map of the sample image comprises: Calculating similarity between the image features and the first target features to obtain a second similarity matrix of the target of interest; Perform density estimation on the second similarity matrix to obtain the second predicted density distribution map.
8. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be counted and one or more marked images of an object of interest in the image to be counted; A processing module, used for inputting the image to be counted and the marked image into a pre-trained second target counting model to obtain a density distribution map of the target of interest in the image to be counted; The second target counting model is a model obtained based on the training method according to any one of claims 1 to 3.
9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.
11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.
