Method and apparatus for processing data, and storage medium

Abstract

A method and apparatus for processing data, and a storage medium are disclosed. The present invention provides a method for performing object counting using a neural network comprising a count error prediction module and a density map estimation module. The density map estimation module comprises a first branch, a second branch and a third branch capable of generating a density map based on an original image, a depth map and a segmentation image respectively. The method comprises: inputting at least one of an original image visual example and a text prompt, and an original image containing one or more objects, to the neural network; predicting, by the count error prediction module, a first error, a second error or a third error with respect to a ground truth density map in the case that a density map is generated by the first branch, the second branch or the third branch based on the original image; determining a minimum error among them; generating a density map by one branch in the density map estimation module corresponding to the minimum error; and counting objects based on the generated density map.

Description

Technical Field

[0001] This disclosure generally relates to the field of information services, and more specifically to methods and apparatus for processing data using large language models, as well as storage media for storing related programs. Background Technology

[0002] With the development of artificial intelligence technology, especially the emergence of large language models (LLM) and multimodal large language models (MLLM), machines' understanding capabilities can reach or even surpass those of ordinary humans in some aspects. However, the counting ability of existing LLMs or MLLMs is still somewhat lacking, and they often cannot accurately determine the number of objects of interest from a given image.

[0003] For example, businesses need to count inventory, and retailers need to count the items on shelves. If machine learning models are used to count items in order to reduce manual labor, the machine learning models must be able to accurately count the number of items in the captured images. Existing LLM or MLLM models are less accurate in this regard.

[0004] Furthermore, there is a need to count items in open-world scenarios. Open-world refers to models that need to handle new objects of unknown categories or that have never been seen before, rather than just those pre-defined categories or objects processed during training. For example, when counting newly added items on a shelf, it is difficult to provide accurate counting results because the model has not seen the new item during training and therefore lacks knowledge about it. Summary of the Invention

[0005] According to one aspect of this disclosure, a method for performing object counting using a trained neural network is provided. The neural network includes a counting error prediction module and a density map estimation module. The density map estimation module includes a first branch capable of generating a density map based on an original image of a first modality, a second branch capable of generating a density map based on a depth map of a second modality, and a third branch capable of generating a density map based on a segmented image of a third modality, wherein the density map indicates the number and location of objects contained in the image of the corresponding modality. The method includes: inputting at least one of an original image visual sample and a text prompt to the neural network, and an original image containing one or more objects; having the counting error prediction module predict the following errors based on the input original image: a first error relative to the ground truth density map when the density map is generated by the first branch, a second error relative to the ground truth density map when the density map is generated by the second branch, and a third error relative to the ground truth density map when the density map is generated by the third branch; determining the minimum error among the first error, the second error, and the third error; generating a density map by one of the first branch, the second branch, and the third branch of the density map estimation module that corresponds to the minimum error; and counting the objects based on the generated density map.

[0006] According to another aspect of this disclosure, an apparatus for performing object counting is provided, comprising: a memory storing a program, and one or more processors. The processors are configured to execute the method described above for performing object counting using a trained neural network by executing the program.

[0007] According to another aspect of this disclosure, a method for training a neural network is provided. The neural network is used to perform object counting and includes a counting error prediction module and a density map estimation module. The density map estimation module includes a first branch capable of generating a density map based on an original image of a first modality, a second branch capable of generating a density map based on a depth map of a second modality, and a third branch capable of generating a density map based on a segmented image of a third modality, wherein the density map indicates the number and location of objects contained in the image of the corresponding modality. The counting error prediction module predicts the following errors based on the original image: a first error relative to the ground truth density map when the density map is generated by the first branch, a second error relative to the ground truth density map when the density map is generated by the second branch, and a third error relative to the ground truth density map when the density map is generated by the third branch. The method includes: independently training a first branch, a second branch, and a third branch of the density map estimation module, wherein a loss function for training the density map estimation module is set based on the first error, the second error, the third error, and a classification loss about the object; and alternately training the density map estimation module and the counting error prediction module, wherein, in training the counting error prediction module, the first error, the second error, and the third error of the density map generated by the trained density map estimation module relative to the ground truth density map are used as the ground truth.

[0008] According to another aspect of this disclosure, a method for performing data augmentation using a trained neural network is provided. The neural network is based on a diffusion generative model and includes a denoising module, a text encoder, and a density map encoder. The method includes: inputting a predetermined density map and a text prompt to the neural network, wherein the text prompt contains a description of an object; converting the text prompt into a representation that the neural network can process by the text encoder, and inputting it to the denoising module; converting the density map into a representation that the neural network can process by the density map encoder, and inputting it to the denoising module; and generating an image containing the object by the denoising module based on an initial random noise image, by progressively predicting noise and removing the predicted noise, wherein the number and location of the object in the image are defined by the density map.

[0009] According to another aspect of this disclosure, an apparatus for performing data augmentation is provided, comprising: a memory storing a program, and one or more processors. The processors are configured to execute the program to perform the method described above for performing data augmentation using a trained neural network.

[0010] According to another aspect of this disclosure, a method for performing open-world object counting using a trained neural network is provided. The neural network includes a first neural network performing the object counting method described above and a second neural network performing the data augmentation method described above. The method includes: generating a plurality of images by the second neural network based on text prompts and a plurality of predetermined density maps, wherein the text prompts contain descriptions of new objects not processed by the first neural network, and each of the generated plurality of images contains the new objects; training the first neural network using the plurality of images as training images; inputting another image containing the new objects into the trained first neural network; and counting the new objects in the other image by the trained first neural network.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable medium storing a program that, when executed by a computer, causes the computer to perform the various methods described above.

[0012] In summary, this disclosure proposes an improved technique for object counting using MLLM, which enhances counting accuracy. Furthermore, it presents a density map-based data augmentation scheme, allowing control over the number and location of objects in the generated images. The generated images can also be used to train an MLLM capable of performing open-world object counting. Attached Figure Description

[0013] Figure 1A and Figure 1B The prediction results of image counting based on different modalities are illustrated schematically.

[0014] Figure 2A and Figure 2B A block diagram of a system for performing a counting task according to the present disclosure is shown schematically.

[0015] Figure 3 A flowchart of a method for performing counting according to this disclosure is shown.

[0016] Figure 4 A conceptual diagram illustrating data enhancement based on this disclosure is shown schematically.

[0017] Figure 5A The training process of a data augmentation system according to this disclosure is illustrated schematically.

[0018] Figure 5B The image generation process of a data augmentation system according to this disclosure is illustrated schematically.

[0019] Figure 6 A flowchart of a data augmentation method based on this disclosure is shown.

[0020] Figure 7 A block diagram of an open-world counting system according to this disclosure is shown schematically.

[0021] Figure 8 A schematic block diagram illustrating the configuration of computer hardware implementing the present disclosure is shown. Detailed Implementation

[0022] In existing methods for counting objects in a given image using MLLM, only the original image (e.g., an RGB image) is typically used. However, the inventors of this invention have recognized that, depending on the specific content of the given image, in some cases, performing the counting task using a depth map or segmented image generated from the original image can achieve higher accuracy. For example, for images containing densely packed objects with clear outlines (such as goods on a shelf or books on a bookshelf), counting based on a segmented image is more effective. For images containing the shadows of objects on the ground or reflections in water, counting based on a depth map has higher accuracy. Figure 1A and Figure 1B The illustration schematically shows the prediction results of performing counts based on images of different modalities (e.g., RGB images, depth maps, and segmented images).

[0023] Figure 1AThis example illustrates counting chess pieces in an image. The first line shows a visual example of the objects (chess pieces) to be counted, which can be multiple local images containing chess pieces specified by the user in the image. The second line shows the RGB image, along with the segmented image and depth map generated based on the RGB image. The third line shows the count prediction results based on the RGB image, segmented image, and depth map, with each small black or white dot in the image identifying a chess piece. Furthermore, the text in the third line, "Prediction results: R 28; S 46; D24," indicates that the count prediction result based on the RGB image (R) is 28 objects, the count prediction result based on the segmented image (S) is 46 objects, and the count prediction result based on the depth map (D) is 24 objects. The fourth line shows the true value, i.e., there are 24 objects in the image. It can be seen that the count prediction result based on the depth map matches the true value, indicating that for this image containing a chessboard and chess pieces, counting chess pieces based on the depth map is the most effective method, because all chess pieces have the same depth, and the depth values ​​of the chess pieces can be distinguished from the depth values ​​of the chessboard, which serves as the background of the image. On the other hand, the complex patterns and colors of the chessboard and pieces lead to poor prediction results based on RGB images and segmented images. For example, the ellipse in the lower right corner of the image in the third row indicates the error in the prediction result; that is, the two circular patterns on the chessboard were incorrectly predicted as two pieces.

[0024] Similarly, in Figure 1B The image shows an example of counting stamps in an image. From the third and fourth rows, we can see that the prediction based on the RGB image (R) is 31 objects, the prediction based on the segmentation image (S) is 22 objects, and the prediction based on the depth map (D) is 65 objects, while the true value is 22 objects. For this image containing stamps and a stamp album, the depth map-based prediction is poor because the stamps are thin and difficult to distinguish from the stamp album, which forms the background, in terms of depth values. Furthermore, the ellipse in the RGB image-based prediction in the third row indicates the error in the prediction; that is, a portion of the stamp album was incorrectly predicted as a stamp. In contrast, the segmentation image-based prediction matches the true value because the stamp outlines are clear, the segmentation accuracy is high, and therefore, detecting individual stamps based on the segmentation image is highly accurate.

[0025] It should be noted that, although in Figure 1A and Figure 1B In the examples, predictions based on depth maps or segmented images outperformed those based on RGB images, but depth maps or segmented images cannot completely replace RGB images. Therefore, in this disclosure, depending on the specific content of the image, the most suitable of the three modalities is selected to perform the object counting task.

[0026] In counting tasks, density maps are a very common and effective method. Density maps are typically generated by placing a 2D Gaussian kernel (with a peak at the center and decaying outwards) at the center of each object in the image, and then stacking all the Gaussian kernels to form a continuous heatmap. Because the center point of each object is transformed into a continuous Gaussian distribution, the value of each pixel in the density map exists as a probability density value. In the density map, the peak of the Gaussian distribution (i.e., the center of the Gaussian kernel) indicates the location of the center point of an object.

[0027] Figure 2A A block diagram of a system for performing a counting task according to this disclosure is shown schematically. Figure 2A As shown, the system includes a density map estimation (DME) module 210 and a counting error prediction (CEP) module 220. The input to the DME module 210 includes the original image and visual samples VE specified by the user in the original image. R And user-provided text prompts (TP). Visual example (VE) R The object to be counted is one or more partial images of the original image containing the object. The text prompt (TP) is a textual description of the object to be counted, which may include, for example, the object's name.

[0028] In the following description, an RGB image will be used as an example of the original image to illustrate the techniques according to this disclosure. However, this disclosure can also be applied to situations where the original image is a different modality (e.g., a single-channel grayscale image or an infrared image).

[0029] The DME module 210 can generate a density map DM based on the input RGB image. R Density map DM is generated based on the depth map corresponding to the RGB image. D And generating density maps DM based on segmented images corresponding to RGB images. S .

[0030] The input to CEP module 220 includes the original image (such as an RGB image). Based on the RGB image, CEP module 220 can predict the following error: the error E relative to the ground truth density map when DME module 210 generates a density map based on the RGB image. R When the DME module 210 generates a density map based on a depth map, the error E relative to the ground truth density map is... D The error E relative to the ground truth density map when the DME module 210 generates a density map based on a segmented image. S .

[0031] When performing the counting task, for a given RGB image, the CEP module 220 first predicts errors corresponding to the RGB image, depth map, and segmentation image, respectively. Then, the DME module 210 generates a density map based on the image mode corresponding to the minimum error among the predicted errors. Finally, object counting is performed based on this density map.

[0032] Figure 2B It shows Figure 2A The more specific structure of the counting system shown.

[0033] like Figure 2B As shown, in DME module 210, a depth map and a segmentation image are generated based on the RGB image. Those skilled in the art can use appropriate known techniques to generate the depth map and segmentation image, and this disclosure does not limit this. As an example, the Depth Anything V2 depth estimation model can be used to generate the depth map, and the SegmentAnything Model 2 (SAM2) model can be used to generate the segmentation image. Furthermore, based on the visual sample VE... R The position in the RGB image can be determined at the corresponding position in the depth map and the segmented image, respectively. (Depth map visual example VE) D Visual examples of segmented images VE S .

[0034] The depth map, RGB image, and segmented image are input to the same image encoder 2110. The image encoder 2110 converts each image into multiple representations that the system can process to obtain a token for the depth map. D Token of RGB image R and the token for segmenting the image. S Accordingly, the image encoder 2110 can generate tokens for each visual example, which will be referred to as Tokens in the following text. VED Token VER and Token VES .

[0035] In addition, the text prompt TP is input to the text encoder 2120. The text encoder 2120 converts the text prompt TP into multiple representations that the system can process to obtain the text prompt's token. T .

[0036] In this disclosure, for ease of description, a Token represents a vector representation of a unit of input content (e.g., a word, an image patch). That is, the process of generating a Token involves segmenting the input content to obtain input units and mapping these input units to vectors (embeddings).

[0037] As an example, the image encoder and text encoder in a pre-trained Grounding DINO model can be used as the image encoder 2110 and text encoder 2120 in the DME module 210. However, this disclosure is not limited thereto, and those skilled in the art can implement the image encoder 2110 and text encoder 2120 using appropriate known techniques.

[0038] exist Figure 2B In the first branch of the system shown, the token of the depth map D Tokens for depth map visual examples VED And the token for text prompts T Input to feature enhancer FE D In the second branch, the token of the RGB image. R Tokens for RGB image visual examples VER And the token for text prompts T Input to feature enhancer FE R In the third branch, the token for segmenting the image. S Tokens for segmented image visual examples VES And the token for text prompts T Input to feature enhancer FE S .

[0039] In each branch, the feature enhancer performs the following operations:

[0040] Combine the tokens of the visual examples and the tokens of the text prompts. T Perform concatenation, and apply self-attention to the concatenated token, thereby merging the token of the visual example with the token of the text prompt. T Mutual integration;

[0041] Perform self-attention on the tokens of the image;

[0042] Perform image-text cross-attention and text-image cross-attention for the following: the token of the image, the token of the stitched visual sample, and the token of the text cue.

[0043] like Figure 2B As shown, each feature enhancer outputs two sets of features: one set is query features, which correspond to the tokens of the concatenated visual sample and text prompt, and the other set is image features, which correspond to the tokens of the image.

[0044] In the first branch, query feature F QD and image features F IDInput to cross-modal decoder CMD D In cross-modal decoder CMD D In the first step, the image features F are determined. ID Among them, the query feature F QD The image features with the highest similarity are selected, and then the query feature F is used. QD Cross-attention is performed on the image features with the highest similarity, thereby outputting image feature F. ID (Also known as the second image feature). Next, the cross-modal decoder CMD is computed. D The output image features F ID 'With query feature F QD The similarity between them is used to form a similarity matrix.

[0045] In the second branch, query feature F QR and image features F IR Input to cross-modal decoder CMD R In cross-modal decoder CMD R In the first step, the image features F are determined. IR Among them, the query feature F QR The image features with the highest similarity are selected, and then the query feature F is used. QR Cross-attention is performed on the image features with the highest similarity, thereby outputting image feature F. IR Next, the cross-modal decoder CMD is computed. R The output image features F IR 'With query feature F QR The similarity between them is used to form a similarity matrix.

[0046] In the third branch, query feature F QS and image features F IS Input to cross-modal decoder CMD S In cross-modal decoder CMD S In the first step, the image features F are determined. IS Among them, the query feature F QS The image features with the highest similarity are selected, and then the query feature F is used. QS Cross-attention is performed on the image features with the highest similarity, thereby outputting image feature F. IS Next, the cross-modal decoder CMD is computed. S The output image features F IS 'With query feature F QS The similarity between them is used to form a similarity matrix.

[0047] It should be noted that image feature F ID FIR F IS These can be collectively referred to as the first image features, image features F ID '、F IR '、F IS These can be collectively referred to as second image features.

[0048] In each branch, assuming the image features output by the cross-modal decoder have a dimension of M, the query features have a dimension of N, and the resulting similarity matrix has a dimension of M × N, we first determine the maximum value in each row of the similarity matrix, thus obtaining M first maximum values. Then, among these M first maximum values, we determine a second maximum value that is greater than a predetermined threshold. In this way, one or more second maximum values ​​greater than the predetermined threshold can be obtained in each branch.

[0049] The state of objects in the corresponding modality image can be determined based on the obtained second maximum value. Specifically, the number of second maximum values ​​can be determined as the number of objects in the corresponding modality image. Furthermore, based on the image features corresponding to the second maximum value in the similarity matrix, the position in the corresponding modality image can be determined, which can be identified as the location where the object exists. Thus, in each branch, a density map DM corresponding to the corresponding modality image can be generated based on the obtained second maximum value. D DM R DM S .

[0050] Next, the density map DM generated by DME module 210 can be calculated. D DM R DM S The error E relative to the truth density map in each of the data is... D E R E S This error indicates the deviation of the number and location of objects predicted by the DME module 210 from the true value.

[0051] In addition, such as Figure 2B As shown, the input to the Counting Error Prediction (CEP) module 220 is only an RGB image. Based on this RGB image, the CEP module 220 can predict the following errors: a first error relative to the ground truth density map when generating a density map based on the RGB image (prediction error of the first branch); a second error relative to the ground truth density map when generating a density map based on a depth map corresponding to the RGB image (prediction error of the second branch); and a third error relative to the ground truth density map when generating a density map based on a segmented image corresponding to the RGB image (prediction error of the third branch).

[0052] CEP module 220 can be implemented using an image classifier. Those skilled in the art can implement CEP module 220 using any suitable known image classification model, and this disclosure does not limit this. As an example, CEP module 220 can be implemented based on the SwinTransformer model.

[0053] exist Figure 2B The training phase of the system shown begins with training the DME module 210. The three branches of the DME module 210 can be trained independently. Loss functions can be set based on the error between the predicted density map and the ground truth density map, and the classification loss for determining the object category. As an example, the loss regarding the density map can use the Euclidean distance between the predicted and ground truth density maps, and the classification loss can use the cross-entropy loss function. However, this disclosure is not limited to these, and those skilled in the art can use other appropriate loss functions.

[0054] As an example, when implementing the image encoder 2110 and text encoder 2120 using the corresponding encoders in the pre-trained Grounding DINO model, the training can be performed solely for the purpose of optimizing the feature enhancer and the cross-modal decoder to ensure that each branch has robust performance for the corresponding modality of the image.

[0055] After training the DME module 210 for several epochs, the DME module 210 and the CEP module 220 are trained alternately. Specifically, the prediction error E of the DME module 210 can be used as the training metric. D E R E S The CEP module 220 is trained using the ground truth value so that, for the input training image, the CEP module 220 can generate a result consistent with the error E. D E R E S The error is essentially the same. As an example, a squared error loss function (i.e., the square of the difference between the error generated by CEP module 220 and the true value) can be used. However, this disclosure is not limited to this, and those skilled in the art can use other appropriate loss functions.

[0056] After training the CEP module 220, the training image set can be divided into three categories (corresponding to images of the three modalities) to continue training the three branches in the DME module 210. Then, the prediction error E of the trained DME module 210 is used again... D E R E S The CEP module 220 is trained using the ground truth values. This alternating training can be repeated, for example, two to three rounds, to achieve optimal system performance.

[0057] The following will combine Figure 3 The flowchart shown describes Figure 2A and Figure 2B The application stage of the object counting system. For example... Figure 3 As shown, in step S310, the trained counting system is input with an RGB image and a visual sample VE. R And text prompt TP. In step S320, the trained CEP module 220 predicts errors corresponding to three modal images (RGB image, depth map, and segmentation image) based on the input RGB image. In step S330, the minimum error among the predicted errors is determined. As an example, the minimum error is the error corresponding to the depth map. In step S340, a density map (e.g., DM) is generated using the branch (e.g., the first branch) in the trained DME module 210 corresponding to the determined minimum error. D Then, in step S350, object counting is performed based on the density map. Specifically, the number of objects is determined based on the number of Gaussian distribution peaks (Gaussian kernel centers) contained in the density map.

[0058] Furthermore, although the above describes the... Figure 2A and Figure 2B The system inputs an RGB image, a visual example, and a text prompt, but this disclosure is not limited to this implementation. According to this disclosure, it is possible to input either an RGB image and a visual example, or an RGB image and a text prompt. That is, one or both of a visual example and a text prompt can be input. If the user inputs either a visual example or a text prompt, the above implementation can be modified accordingly. For example, the feature enhancer in each branch will not input the tokens of the visual example and the text prompt. T Instead of concatenating tokens, this only applies to tokens for visual examples or text prompts. T Self-attention is performed. Furthermore, for example, in the absence of input RGB visual samples, the DME module 210 does not need to generate depth map visual samples and segmentation image visual samples. Those skilled in the art can easily make adaptive modifications.

[0059] Furthermore, the inventors recognized that if counting is performed based on an RGB image, using both visual examples and text prompts may improve counting accuracy. However, if counting is performed based on a segmented image or depth map, using visual examples may yield higher performance, while text prompts may negatively impact counting accuracy. Therefore, this disclosure also includes the following implementation: even if the user inputs both visual examples and text prompts, the first and third branches in the DME module 210 can be based on a token using only the visual example, without using the token for the text prompt. T .

[0060] Table 1 below shows a performance comparison between the object counting system according to this disclosure and existing models. Mean absolute error (MAE) is a metric used to evaluate prediction accuracy; a smaller value indicates better performance. As can be seen from Table 1, the counting prediction accuracy of the system according to this disclosure is superior to that of existing models.

[0061] [Table 1]

[0062]

[0063]

[0064] According to another aspect of this disclosure, a data augmentation method is also provided. Specifically, an image is generated based on textual prompts describing objects and a predetermined density map, wherein the generated image contains one or more objects, and the number and layout of the objects are defined by the predetermined density map.

[0065] Figure 4 A conceptual diagram illustrating data enhancement based on this disclosure is shown schematically. For example... Figure 4 As shown, the first row contains multiple pre-defined density maps. The second row shows multiple images generated based on the text prompt "heart-shaped bowl, strawberries" and the corresponding density maps. It can be seen that the number and position of the strawberries in each image are essentially consistent with the small white dots (centers of the Gaussian kernel) in the corresponding density map. Similarly, the third row shows multiple images generated based on the text prompt "round bowl, cherry tomatoes" and the corresponding density maps, with the layout of the cherry tomatoes in each image essentially consistent with the small white dots (centers of the Gaussian kernel) in the corresponding density map.

[0066] Compared to traditional techniques that generate images based solely on text prompts, the images generated by the method according to this disclosure meet the following requirements: they contain objects that correspond to the text prompts, and the number and location of the objects are consistent with a given density map. In other words, the method according to this disclosure allows for precise control over the number and location of objects in the generated image.

[0067] Figure 5A The training process of a data augmentation system based on this disclosure is illustrated schematically. Figure 5B The image generation process of a data augmentation system according to this disclosure is illustrated schematically.

[0068] The data augmentation system according to this disclosure is based, for example, on a diffusion generative model (DGM). DGM is a type of generative model based on progressive denoising, which mainly includes a forward process and a backward process. In the forward process, noise is progressively added to the input image, eventually making it a purely noisy image. In the backward process, starting from a random noisy image, noise is progressively predicted and removed, thereby reconstructing the image from the noise. Stable Diffusion (SD) is one of the representative diffusion generative models. The data augmentation system according to this disclosure will be described below based on the Stable Diffusion 2.1 (SD21) model.

[0069] Reference Figure 5A The image encoder 520, image decoder 530, and text encoder 540 are inherited from the SD21 model and remain unchanged. Furthermore, the image encoder 520 and image decoder 530 can form a denoising module. Compared to the SD21 model, the data augmentation system according to this disclosure adds an image encoder 510 for the density map, which can be copied from the encoder 520 of the SD21 model.

[0070] Density maps are prepared in advance before training; specifically, density maps can be generated based on known training images. This yields pairs of training images and density maps.

[0071] exist Figure 5A During the training process shown, a noisy image is input into the denoising module. This noisy image is generated by adding a specific level of random noise to the training image. Additionally, a text prompt is input to the text encoder 540, which generates a token for the text prompt. T The data is fed into the denoising module. Additionally, the density map corresponding to the training image is fed into the image encoder 510, and the token of the density map generated by the image encoder 510 is fed into the denoising module. The denoising module predicts the noise to be added to the training image. A training loss function (e.g., mean squared error, MSE) is set based on the error between the predicted noise and the actual added noise. As an example, training can be performed by optimizing only the image encoder 510 used for the density map.

[0072] exist Figure 5BIn the image generation process shown, a density map and a text prompt (e.g., "The image shows a close-up of several green watermelons stacked together") are taken as input to the system. The system can then generate an image corresponding to the text prompt and the density map. Initially, a randomly noisy image is generated. Based on this image, a denoising module iteratively predicts and removes noise, eventually obtaining a clean image. The generated image not only contains objects that match the text prompt (e.g., green watermelons), but the number and location of these objects also correspond to the input density map.

[0073] Figure 6 A flowchart illustrating a data augmentation method according to this disclosure is shown. Figure 6 As shown, in step S610, a predetermined density map and a text prompt containing a description of the object are input to the data augmentation system. In step S620, the text prompt is converted into a token by the text encoder 540. T The density map is input to the denoising module. In step S630, the image encoder 510 converts the density map into tokens, which are then input to the denoising module. In step S640, the denoising module generates a final image based on the initial random noise image by progressively predicting and removing the predicted noise. The final image contains the object described by the text prompt, and the number and location of the objects in the final image are consistent with the density map.

[0074] In addition, for example, Figure 5A and Figure 5B The data augmentation system shown can be used with Figure 2A and Figure 2B The combined counting systems shown are particularly suitable for object counting in open-world scenarios. In open-world counting tasks (e.g., counting new items on a shelf), users require the counting system to count new objects that the system has not seen before. Because the system lacks knowledge about these new objects, it is difficult to provide accurate counting predictions.

[0075] In response to this issue, this disclosure proposes the following: Figure 7 The open-world counting system shown. (Refer to...) Figure 7 The Open World Counting System 700 includes the Data Augmentation System 710 (such as...). Figure 5A and Figure 5B (as shown) and counting system 720 (as shown) Figure 2A and Figure 2B (As shown). First, using a trained data augmentation system 710, multiple images (e.g., ...) are generated based on multiple pre-prepared density maps and text cues describing a new object (e.g., strawberry). Figure 4(As shown). Each generated image contains one or more new objects in number and location consistent with the corresponding density map.

[0076] Then, the multiple images generated by the data augmentation system 710 are used as a training image set for training the counting system 720, wherein the pre-prepared density map corresponding to a certain image is used as the ground truth density map during training.

[0077] By training with these generated images, the counting system 720 will be able to learn about new objects. After training, the counting system 720 can be used to count the new object (e.g., strawberries).

[0078] In particular, as another implementation, images generated by the data augmentation system 710 (synthetic images) can be combined with training images from other sources (such as the FSC-147 dataset) in a certain proportion to form a training image set for training the counting system 720.

[0079] Table 2 below shows Figure 7 The table shows a performance comparison between the open-world counting system and the existing model (CountGD). Mean absolute error (MAE) and root mean square error (RMSE) are evaluation metrics; lower values ​​indicate better performance. Table 2 shows that the counting prediction accuracy of the system based on this disclosure is superior to that of the existing model.

[0080] [Table 2]

[0081]

[0082] The technology of this disclosure has been described above with reference to specific embodiments. This disclosure provides an improved technical solution for performing object counting using MLLM, wherein the error when performing counting based on RGB images, depth maps, and segmented images is predicted respectively, and the image of the modality with the smallest error is selected for actual counting. Therefore, compared with conventional techniques using only RGB images, the technology of this disclosure can improve the accuracy of counting. Furthermore, this disclosure provides an image generation scheme based on density maps. Compared with conventional techniques that generate images based solely on text, the technology of this disclosure can precisely control the number and position of objects in the generated image. In addition, this disclosure also provides a scheme for performing object counting in open-world scenes.

[0083] The methods described in the above embodiments can be implemented by software, hardware, or a combination of software and hardware. Programs included in the software can be stored beforehand in a storage medium located internally or externally to the device. As an example, during execution, these programs are written to random access memory (RAM) and executed by a processor (e.g., a CPU) to implement the various methods and processes described herein.

[0084] This disclosure provides computer program code and computer program products for implementing the methods described above, as well as a computer-readable storage medium on which the computer program code is recorded.

[0085] Figure 8 An example configuration block diagram of computer hardware is shown for performing the methods of this disclosure according to a program.

[0086] like Figure 8 As shown, in computer 800, central processing unit (CPU) 801, read-only memory (ROM) 802 and random access memory (RAM) 803 are connected to each other via bus 804.

[0087] The input / output interface 805 is further connected to the bus 804. The input / output interface 805 is connected to the following components: an input device 806 formed by a keyboard, mouse, microphone, etc.; an output device 807 formed by a display, speaker, etc.; a storage device 808 formed by a hard disk, non-volatile memory, etc.; a communication device 809 formed by a network interface card (such as a local area network (LAN) card, modem, etc.); and a driver 810 for driving a removable medium 811, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0088] In a computer with the above structure, the CPU 801 loads a program stored in the storage device 808 into the RAM 803 via the input / output interface 805 and the bus 804, and executes the program to perform the method described above.

[0089] The program to be executed by the computer (CPU 801) can be recorded on a removable medium 811, which is formed as a packaging medium, such as a magnetic disk (including a floppy disk), an optical disk (including a compact optical disk-read-only memory (CD-ROM)), a digital multifunction optical disk (DVD), etc.), a magneto-optical disk, or a semiconductor memory. Furthermore, the program to be executed by the computer (CPU 801) can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0090] When the removable medium 811 is installed in the drive 810, the program can be installed in the storage device 808 via the input / output interface 805. Alternatively, the program can be received by the communication device 809 via a wired or wireless transmission medium and installed in the storage device 808. Alternatively, the program can be pre-installed in the ROM 802 or the storage device 808.

[0091] A program executed by a computer may be a program that performs processing in the order described in this disclosure, or it may be a program that performs processing in parallel or when needed (such as when invoked).

[0092] The modules or devices described in this disclosure are for logical purposes only and do not strictly correspond to physical devices or entities. For example, the function of each module described in this disclosure may be implemented by multiple physical entities, or the function of multiple modules described in this disclosure may be implemented by a single physical entity. Furthermore, features, components, elements, steps, etc., described in one embodiment are not limited to that embodiment, but can also be applied to other embodiments, such as replacing specific features, components, elements, steps, etc., in other embodiments, or in combination with them.

[0093] The scope of this disclosure is not limited to the specific embodiments described herein. Those skilled in the art will understand that various modifications or variations can be made to the embodiments described herein, depending on design requirements and other factors, without departing from the principles of this disclosure. The scope of this disclosure is defined by the appended claims and their equivalents.

[0094] In addition, this disclosure also includes the following implementation methods.

[0095] Implementation 1. A method for performing object counting using a trained neural network, wherein the neural network includes a counting error prediction module and a density map estimation module, the density map estimation module including a first branch capable of generating a density map based on an original image of a first modality, a second branch capable of generating a density map based on a depth map of a second modality, and a third branch capable of generating a density map based on a segmented image of a third modality, wherein the density map indicates the number and location of objects contained in the image of the corresponding modality.

[0096] The method includes:

[0097] The neural network is input with at least one of an original image visual sample and a text prompt, as well as an original image containing one or more objects;

[0098] The counting error prediction module predicts the following errors based on the input original image: a first error relative to the ground truth density map when the density map is generated by the first branch, a second error relative to the ground truth density map when the density map is generated by the second branch, and a third error relative to the ground truth density map when the density map is generated by the third branch.

[0099] Determine the minimum error among the first error, the second error, and the third error;

[0100] The density map is generated from one of the first, second, and third branches of the density map estimation module that corresponds to the minimum error.

[0101] The objects are counted based on the generated density map.

[0102] Implementation Method 2. The method according to Implementation Method 1 further includes:

[0103] The density map estimation module generates the depth map and the segmentation image based on the input original image;

[0104] When the original image visual sample is input, the density map estimation module generates a depth map visual sample and a segmentation image visual sample based on the original image visual sample.

[0105] Implementation 3. According to the method of Implementation 2, wherein the density map estimation module further includes a text encoder and an image encoder, and wherein each of the first branch, the second branch, and the third branch includes a feature enhancer and a cross-modal decoder.

[0106] The method further includes: when both the original image visual sample and the text prompt are input,

[0107] The text encoder converts the text prompt into a representation that the neural network can process;

[0108] The image encoder converts the original image, the depth map, the segmented image, the visual sample of the original image, the visual sample of the depth map, and the visual sample of the segmented image into representations that the neural network can process;

[0109] In each branch, the feature enhancer generates query features and first image features based on the representation of the text, the representation of the visual sample of the corresponding modality, and the representation of the image of the corresponding modality;

[0110] In each branch, the cross-modal decoder generates a second image feature based on the query feature and the first image feature.

[0111] Implementation 4. The method according to Implementation 3 further includes: in each branch, the feature enhancer performs the following operations:

[0112] The text representation is concatenated with the representation of the visual example of the corresponding modality, and self-attention is performed on the concatenated representation;

[0113] Perform self-attention on the image representation for the corresponding modality;

[0114] Cross-attention is performed on the image representation of the corresponding modality and the stitched representation.

[0115] Implementation 5. The method according to Implementation 3 further includes: in each branch, the cross-modal decoder performs the following operations:

[0116] Identify the image features among the first image features that have the highest similarity to the query feature;

[0117] Cross-attention is performed on the query feature and several image features with the highest similarity to generate the second image feature.

[0118] Implementation 6. The method according to implementation 3 further includes: in each branch,

[0119] Calculate the similarity between the second image feature and the query feature to form a similarity matrix;

[0120] Determine the maximum value of each row in the similarity matrix to obtain multiple first maximum values;

[0121] Among the plurality of first maximum values, determine one or more second maximum values ​​that are greater than a predetermined threshold;

[0122] A density map corresponding to the corresponding modality is generated based on the determined second maximum value.

[0123] Implementation 7. According to the method of Implementation 3, wherein the first branch, the second branch, and the third branch of the density map estimation module are trained independently to adjust the parameters of the feature enhancer and the cross-modal decoder, and

[0124] The loss function used to train the density map estimation module is set based on the first error, the second error, the third error, and the classification loss about the object.

[0125] Implementation Method 8. According to the method of Implementation Method 7, wherein the density map estimation module and the counting error prediction module are trained alternately, and

[0126] In the training of the counting error prediction module, the first error, the second error, and the third error of the density map generated by the trained density map estimation module relative to the true density map are used as the true value.

[0127] Implementation 9. An apparatus for performing object counting, comprising:

[0128] Memory containing programs; and

[0129] One or more processors configured to execute the method of performing object counting using a trained neural network according to embodiments 1-6 by executing the program.

[0130] Implementation 10. A method for training a neural network, wherein the neural network is used to perform object counting, and includes a counting error prediction module and a density map estimation module.

[0131] The density map estimation module includes a first branch capable of generating a density map based on the original image of a first modality, a second branch capable of generating a density map based on a depth map of a second modality, and a third branch capable of generating a density map based on a segmented image of a third modality. The density map indicates the number and location of objects contained in the image of the corresponding modality.

[0132] The counting error prediction module predicts the following errors based on the original image: a first error relative to the ground truth density map when the density map is generated by the first branch, a second error relative to the ground truth density map when the density map is generated by the second branch, and a third error relative to the ground truth density map when the density map is generated by the third branch.

[0133] The method includes:

[0134] The first, second, and third branches of the density map estimation module are trained independently, wherein a loss function for training the density map estimation module is set based on the first error, the second error, the third error, and the classification loss about the object.

[0135] The density map estimation module and the counting error prediction module are trained alternately, wherein, during the training of the counting error prediction module, the first error, the second error, and the third error of the density map generated by the trained density map estimation module relative to the ground truth density map are used as the ground truth.

[0136] Implementation 11. A method for performing data augmentation using a trained neural network, wherein the neural network is based on a diffusion generative model and includes a denoising module, a text encoder, and a density map encoder.

[0137] The method includes:

[0138] The neural network is input with a predetermined density map and a text prompt, wherein the text prompt contains a description of the object;

[0139] The text encoder converts the text prompt into a representation that the neural network can process, and then inputs it into the denoising module;

[0140] The density map encoder converts the density map into a representation that the neural network can process, and then inputs it into the denoising module;

[0141] The denoising module generates an image containing the object by progressively predicting and removing noise from an initial random noise image.

[0142] The number and location of the objects in the image are defined by the density map.

[0143] Implementation Method 12. An apparatus for performing data augmentation, comprising:

[0144] Memory containing programs; and

[0145] One or more processors configured to execute the method for performing data augmentation using a trained neural network according to embodiment 11 by executing the program.

[0146] Implementation Method 13. A method for performing open-world object counting using a trained neural network, wherein the neural network includes a first neural network performing the methods described in Implementation Methods 1-8 and a second neural network performing the methods described in Implementation Method 11.

[0147] The method includes:

[0148] The second neural network generates multiple images based on text prompts and multiple predetermined density maps, wherein the text prompts contain a description of a new object that has not been processed by the first neural network, and each of the multiple generated images contains the new object;

[0149] The plurality of images are used as training images to train the first neural network;

[0150] The first trained neural network is input with an additional image containing the new object;

[0151] The trained first neural network counts the new objects in the additional image.

[0152] Implementation 14. A non-transitory computer-readable medium storing a program that, when executed by a computer, causes the computer to perform the method according to the above-described implementation.

Claims

1. A method for performing object counting using a trained neural network, wherein, The neural network includes a counting error prediction module and a density map estimation module. The density map estimation module includes a first branch capable of generating a density map based on the original image of a first modality, a second branch capable of generating a density map based on a depth map of a second modality, and a third branch capable of generating a density map based on a segmented image of a third modality. The density map indicates the number and location of objects contained in the image of the corresponding modality. The method includes: The neural network is input with at least one of an original image visual sample and a text prompt, as well as an original image containing one or more objects; The counting error prediction module predicts the following errors based on the input original image: a first error relative to the ground truth density map when the density map is generated by the first branch, a second error relative to the ground truth density map when the density map is generated by the second branch, and a third error relative to the ground truth density map when the density map is generated by the third branch. Determine the minimum error among the first error, the second error, and the third error; The density map is generated from one of the first, second, and third branches of the density map estimation module that corresponds to the minimum error. The objects are counted based on the generated density map.

2. The method according to claim 1, further comprising: The density map estimation module generates the depth map and the segmentation image based on the input original image; When the original image visual sample is input, the density map estimation module generates a depth map visual sample and a segmentation image visual sample based on the original image visual sample.

3. The method according to claim 2, wherein, The density map estimation module further includes a text encoder and an image encoder, wherein each of the first branch, the second branch, and the third branch includes a feature enhancer and a cross-modal decoder. The method further includes: when both the original image visual sample and the text prompt are input, The text encoder converts the text prompt into a representation that the neural network can process; The image encoder converts the original image, the depth map, the segmented image, the visual sample of the original image, the visual sample of the depth map, and the visual sample of the segmented image into representations that the neural network can process; In each branch, the feature enhancer generates query features and first image features based on the representation of the text, the representation of the visual sample of the corresponding modality, and the representation of the image of the corresponding modality; In each branch, the cross-modal decoder generates a second image feature based on the query feature and the first image feature.

4. The method according to claim 3, further comprising: In each branch, the feature enhancer performs the following operations: The text representation is concatenated with the representation of the visual example of the corresponding modality, and self-attention is performed on the concatenated representation; Perform self-attention on the image representation for the corresponding modality; Cross-attention is performed on the image representation of the corresponding modality and the stitched representation.

5. The method according to claim 3, further comprising: In each branch, the cross-modal decoder performs the following operations: Identify the image features among the first image features that have the highest similarity to the query feature; Cross-attention is performed on the query feature and several image features with the highest similarity to generate the second image feature.

6. The method according to claim 3, further comprising: In each branch, Calculate the similarity between the second image feature and the query feature to form a similarity matrix; Determine the maximum value of each row in the similarity matrix to obtain multiple first maximum values; Among the plurality of first maximum values, determine one or more second maximum values ​​that are greater than a predetermined threshold; A density map corresponding to the corresponding modality is generated based on the determined second maximum value.

7. The method according to claim 3, wherein, The first, second, and third branches of the density map estimation module are trained independently to adjust the parameters of the feature enhancer and the cross-modal decoder. The loss function used to train the density map estimation module is set based on the first error, the second error, the third error, and the classification loss about the object.

8. The method according to claim 7, wherein, The density map estimation module and the counting error prediction module are trained alternately, and In the training of the counting error prediction module, the first error, the second error, and the third error of the density map generated by the trained density map estimation module relative to the true density map are used as the true value.

9. A method for performing data augmentation using a trained neural network, wherein, The neural network is based on a diffusion generation model and includes a denoising module, a text encoder, and a density map encoder. The method includes: The neural network is input with a predetermined density map and a text prompt, wherein the text prompt contains a description of the object; The text encoder converts the text prompt into a representation that the neural network can process, and then inputs it into the denoising module; The density map encoder converts the density map into a representation that the neural network can process, and then inputs it into the denoising module; The denoising module generates an image containing the object by progressively predicting and removing noise from an initial random noise image. The number and location of the objects in the image are defined by the density map.

10. A method for performing open-world object counting using a trained neural network, wherein, The neural network includes a first neural network that performs the method of claim 1 and a second neural network that performs the method of claim 9. The method includes: The second neural network generates multiple images based on text prompts and multiple predetermined density maps, wherein the text prompts contain a description of a new object that has not been processed by the first neural network, and each of the multiple generated images contains the new object; The plurality of images are used as training images to train the first neural network; The first trained neural network is input with an additional image containing the new object; The trained first neural network counts the new objects in the additional image.

11. A non-transitory computer-readable medium storing a program, which, when executed by a computer, causes the computer to perform the method according to any one of claims 1-10.

Images