Target identification method and system, electronic equipment and storage medium

Abstract

The invention discloses a target recognition method and system, electronic equipment and a storage medium, and belongs to the technical field of image processing technologies. The target recognition method comprises the following steps: acquiring a sample image and a prompt text corresponding to the sample image from a training data set; coding the sample image and the prompt text by using a visual language model to obtain an image sequence and a text sequence; adjusting the image sequence by using information in the text sequence to obtain embedded features; fusing the embedded features and the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence; calculating a loss function value by utilizing the prediction density map, and adjusting parameters of the visual language model according to the loss function value; and if the target identification task is received, inputting the to-be-processed image and the prompt text of the target object into the visual language model to obtain an identification result of the target object. According to the invention, high-generalization target identification can be realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target recognition method, system, electronic device and storage medium. Background Art

[0002] In recent years, the exploration of efficient and accurate recognition and counting solutions has led to the rapid development of the performance of specific deep neural network models. After these networks are trained for a certain field, although they show certain effectiveness in recognizing and counting the expected industrial scene targets, they usually require a large number of data sets specific to the current industrial scene for training, and require a lot of time for complex data annotation.

[0003] The target recognition scheme in the related art can only recognize trained objects but cannot recognize untrained objects, resulting in poor generalization of the target recognition scheme.

[0004] Therefore, how to achieve highly generalized target recognition is a technical problem that technicians in this field currently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a target recognition method, system, electronic device and storage medium that can achieve highly generalized target recognition.

[0006] To solve the above technical problems, the present application provides a target recognition method, comprising:

[0007] Acquire a sample image and a prompt text corresponding to the sample image from a training data set; wherein the prompt text is used to describe an object in the sample image;

[0008] Encoding the sample images using a visual language model to obtain an image sequence, and encoding the prompt texts corresponding to the sample images using the visual language model to obtain a text sequence;

[0009] Using information in the text sequence to adjust the image sequence to obtain embedded features;

[0010] Fusing the embedded feature with the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence;

[0011] Calculating a loss function value using the predicted density map, and adjusting parameters of the visual language model according to the loss function value;

[0012] If a target recognition task is received, the target recognition task is parsed to obtain a prompt text of the image to be processed and the target object;

[0013] The image to be processed and the prompt text of the target object are input into the visual language model to obtain a recognition result of the target object.

[0014] Optionally, fusing the embedding feature with the text sequence to obtain a fused sequence includes:

[0015] Inputting the embedded features into the multilayer perceptron of the visual language model to obtain a first output result;

[0016] Inputting the text sequence into the multilayer perceptron of the visual language model to obtain a second output result;

[0017] Perform feature splicing on the first output result and the embedded feature to obtain a first splicing result;

[0018] Perform feature splicing on the second output result and the text sequence to obtain a second splicing result;

[0019] Performing feature integral calculation on the first splicing result and the second splicing result to obtain the fusion sequence.

[0020] Optionally, generating a predicted density map according to the fusion sequence includes:

[0021] Performing convolution kernel processing or feature reshaping on the fused sequence to obtain a first processing result;

[0022] Performing upsampling, convolution and dimension adjustment on the first processing result in sequence to obtain a second processing result;

[0023] The density distribution of the feature data in the second processing result is calculated using a kernel density estimation method to obtain the predicted density map.

[0024] Optionally, calculating a loss function value using the predicted density map includes:

[0025] Obtaining an actual density map corresponding to the sample image; wherein the predicted density map has the same size as the actual density map;

[0026] The loss function value is determined based on a difference between the actual density map and the actual density map.

[0027] Optionally, the image sequence is adjusted using information in the text sequence to obtain embedded features, including:

[0028] Substituting the text sequence and the image sequence into the adjustment formula to obtain the embedded feature;

[0029] Among them, the regulation formula is Regulator= , Regulator represents the embedded feature, represents an image sequence, Represents a text sequence, Represents matrix multiplication.

[0030] Optionally, inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a recognition result of the target object includes:

[0031] Inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a target density map;

[0032] A recognition result of the target object is generated according to the target density map.

[0033] Optionally, after obtaining the target density map, it also includes:

[0034] The target objects are counted according to the target density map to obtain the number of the target objects contained in the image to be processed.

[0035] The present application also provides a target recognition system, which includes:

[0036] A sample acquisition module, used to acquire a sample image and a prompt text corresponding to the sample image from a training data set; wherein the prompt text is used to describe an object in the sample image;

[0037] An encoding module, used to encode the sample image using a visual language model to obtain an image sequence, and to encode the prompt text corresponding to the sample image using the visual language model to obtain a text sequence;

[0038] A text adjustment module, used to adjust the image sequence using information in the text sequence to obtain embedded features;

[0039] A fusion module, used for fusing the embedded features with the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence;

[0040] A model optimization module, used to calculate a loss function value using the predicted density map, and adjust parameters of the visual language model according to the loss function value;

[0041] A task parsing module, for parsing a target recognition task to obtain a prompt text of an image to be processed and a target object if a target recognition task is received;

[0042] The task processing module is used to input the image to be processed and the prompt text of the target object into the visual language model to obtain the recognition result of the target object.

[0043] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps of the target identification method described above are implemented.

[0044] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the target recognition method when calling the computer program in the memory.

[0045] The present application provides a target recognition method, which obtains sample images and their corresponding prompt texts from a training data set, and uses the information in the text sequence to adjust the image sequence to obtain embedded features, thereby determining the part of the image that matches the text description. The present application also fuses the embedded features with the text sequence to form a fused sequence, and generates a predicted density map based on this. The parameters of the visual language model can be adjusted using the predicted density map, so that the visual language model has the ability to recognize targets. After sufficient training, the visual language model can accurately identify the target object when receiving a target recognition task. The visual language model in the present application combines information from both image and text modalities for target recognition, improves the recognition ability of untrained object types, and enhances the generalization of target detection. The present application also provides a target recognition system, a storage medium, and an electronic device, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flow chart of a target recognition method provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a method for object recognition and counting based on text prompt embedding provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a target recognition and counting principle based on text prompt embedding provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a characteristic integral calculation principle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] See below Figure 1 , Figure 1 A flowchart of a target recognition method provided in an embodiment of the present application.

[0053] Specific steps may include:

[0054] S101: Acquire a sample image and a prompt text corresponding to the sample image from a training data set;

[0055] In this embodiment, a training data set may be pre-constructed, and the training data set may include multiple sample images and prompt texts of the sample images; the prompt texts are used to describe the objects in the sample images; the prompt texts may be the names of the objects or the appearance of the objects, such as a red cuboid, a screw, a pedestrian, etc. The sample images correspond to the prompt texts one by one, and each prompt text only describes one object in the sample image.

[0056] S102: Encoding the sample image using a visual language model to obtain an image sequence, and encoding the prompt text corresponding to the sample image using the visual language model to obtain a text sequence;

[0057] The visual language model used in this embodiment may be a CLIP (Contrastive Language-Image Pre-Training, general visual-language model) model or an open source visual-language model such as Open-clip.

[0058] In this embodiment, the image encoder in the visual language model may be used to encode the sample image to obtain an image sequence, and the image encoder in the visual language model may be used to encode the prompt text corresponding to the sample image to obtain a text sequence.

[0059] An image sequence is an abstract representation of a sample image, which contains the spatial distribution of information such as objects and scenes in the image. A text sequence can contain information at the vocabulary level, as well as the relationship between words and the overall semantic structure.

[0060] S103: Using the information in the text sequence to adjust the image sequence to obtain embedded features;

[0061] Among them, this embodiment can use the information in the text sequence to adjust the image sequence so that the visual language model can introduce semantic guidance in the image features to obtain embedded features; the embedded features can make the visual language model more focused on the parts that are consistent with the text description.

[0062] S104: fusing the embedded feature and the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence;

[0063] Among them, this step fuses the embedded features with the text sequence to form a fused sequence. This process combines the dual information of image and text, enabling the visual language model to understand the image content more accurately.

[0064] The above fused sequence captures the areas and features in the image that are highly relevant to the text description. Using the rich information in the fused sequence, the visual language model can generate the corresponding predicted density map. The predicted density map reflects the number distribution of objects in the image and also retains their spatial location information, providing precise guidance for subsequent target recognition and counting tasks.

[0065] S105: Calculating a loss function value using the predicted density map, and adjusting parameters of the visual language model according to the loss function value;

[0066] Among them, this embodiment can compare the predicted density map with the actual density map to obtain a loss function value, and this embodiment can adjust the parameters of the visual language model according to the loss function value to reduce the error in the next prediction. Specifically, this embodiment can obtain the actual density map corresponding to the sample image; wherein the predicted density map has the same size as the actual density map; and determine the loss function value according to the difference between the actual density map and the actual density map.

[0067] In this embodiment, the operations of S101-105 may be executed cyclically until the loss function value meets the requirement or the cycle reaches a preset number of times.

[0068] S106: If a target recognition task is received, the target recognition task is parsed to obtain a prompt text of the image to be processed and the target object;

[0069] After receiving the target recognition task, the target recognition task can be parsed to obtain the image to be processed and the prompt text of the target object.

[0070] Specifically, the target recognition task may be a pedestrian recognition task, the image to be processed is a surveillance image of a specific area, and the target object may be a pedestrian or a vehicle. The target recognition task may be a part recognition task, the image to be processed is an image of the working area of ​​a robot arm, and the target object is a part to be sorted.

[0071] S107: Inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a recognition result of the target object.

[0072] The visual language model uses the cross-modal understanding ability learned during its training process to locate and identify the target object in the image based on the image to be processed and the prompt text of the target object, and finally generates a recognition result containing information such as the target location and category. Through the above operations, the visual language model can accurately complete the target recognition task in diverse and complex scenes.

[0073] This embodiment obtains sample images and their corresponding prompt texts from a training data set, and uses the information in the text sequence to adjust the image sequence to obtain embedded features, thereby determining the portion of the image that matches the text description. This embodiment also fuses the embedded features with the text sequence to form a fused sequence, and generates a predicted density map based on this. The predicted density map can be used to adjust the parameters of the visual language model so that the visual language model has the ability to recognize targets. After sufficient training, the visual language model can accurately recognize the target object when receiving a target recognition task. The visual language model in this embodiment combines information from both image and text modalities for target recognition, improves the ability to recognize untrained object types, and enhances the generalization of target detection.

[0074] As for Figure 1 For further introduction of the corresponding embodiment, the embedding feature can be generated by: substituting the text sequence and the image sequence into a regulation formula to obtain the embedding feature; wherein the regulation formula is Regulator= , Regulator represents the embedded features output by the feature fusion regulator, represents an image sequence, Represents a text sequence, Represents matrix multiplication.

[0075] As for Figure 1 Further introduction to the corresponding embodiment, the embedded feature and the text sequence can be fused in the following manner: input the embedded feature into the multilayer perceptron of the visual language model to obtain a first output result; input the text sequence into the multilayer perceptron of the visual language model to obtain a second output result; perform feature splicing on the first output result and the embedded feature to obtain a first splicing result; perform feature splicing on the second output result and the text sequence to obtain a second splicing result; perform feature integral calculation on the first splicing result and the second splicing result to obtain the fused sequence.

[0076] As for Figure 1 Further introduction to the corresponding embodiment can generate a predicted density map based on the fusion sequence in the following manner: perform convolution kernel processing or feature reshaping on the fusion sequence to obtain a first processing result; perform upsampling operation, convolution operation and dimensionality adjustment on the first processing result in sequence to obtain a second processing result; use a kernel density estimation method to calculate the density distribution of feature data in the second processing result to obtain the predicted density map.

[0077] As for Figure 1 According to a further introduction to the corresponding embodiment, the process of generating the recognition result of the target object includes: inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a target density map; and generating the recognition result of the target object according to the target density map.

[0078] As for Figure 1 As further described in the corresponding embodiment, after obtaining the target density map, the target objects may be counted according to the target density map to obtain the number of the target objects contained in the image to be processed.

[0079] The process described in the above embodiment is explained below through an embodiment in actual application.

[0080] Advanced target recognition and counting methods are critical in smart industrial site supervision scenarios. Researchers have been working to achieve accurate target recognition and counting in complex environments, such as construction sites with workers and mechanical parts with different characteristics. Integrating this method into smart industrial site supervision can achieve the following benefits: (1) Monitor the construction site through fixed cameras or handheld cameras, accurately identify and count targets such as operators, operating tables, and operating tools in real time, and improve the safety and efficiency of the construction site; (2) Regularly monitor and count the presence and amount of domestic waste such as plastic bottles and beverage cans on the industrial site to maintain the cleanliness and hygiene of the monitoring area; (3) Provide real-time alarm and monitoring functions. If dangerous behavior or abnormal conditions are detected, feedback can be automatically provided to the platform administrator to ensure the safety of workers and equipment on the construction site.

[0081] In the current scheme of applying machine vision to industrial site supervision, there are two traditional and relatively mature strategies for the detection and counting of targets: regression-based methods and detection-based methods. Regression-based techniques are good at predicting density maps, which represent the distribution of objects of a certain category in the entire image, and then infer the final count by summarizing the values ​​of the density map. In contrast, detection-based methods use object detectors to keenly identify and isolate individual objects, and then count the instances detected in the current image. Although these methods have their own advantages and are often more effective for tasks in a certain scenario of smart industrial site supervision with the support of sufficient training data, they are often less effective when faced with the unpredictable diversity of images in real industrial areas, and the model architecture that is effective in the current industrial scenario is often significantly limited in performance in other industrial scenarios, because it is good at calculating trained objects, but not so good at calculating and counting categories and objects that have never been seen. This inflexibility poses a great limitation in the business of industrial site supervision, often resulting in poor generalization of detection models that are effective in the current scenario to other real-world areas, especially in dynamic environments where a large number of object categories must be identified and counted simultaneously, especially in industrial environments with frequent personnel movement and diverse machinery operations.

[0082] Many specially customized target recognition and counting networks often require a lot of time investment and a large number of complex and repetitive data annotation tasks, especially in industrial environments with frequent personnel exchanges and diverse mechanical operations. It is necessary to design a highly applicable machine vision model to match the industrial environment scene in order to reduce the time cost and labor cost of large amounts of data annotation.

[0083] Although category-independent counting methods have been used in some smart industrial site supervision scenarios, they usually operate under the premise of a straightforward task of obtaining accurate bounding boxes for the identified objects of interest. This assumption naturally relies on manually annotated data, a process that is time-consuming and error-prone. The need for human intervention not only increases the complexity of the operation, but also reduces the overall user experience because it interrupts the seamless interaction expected by modern technology. Moreover, when objects belong to similar categories, the challenge is further complicated because these methods usually rely on the similarity of the visual features of the objects for judgment and comparison, resulting in machine vision models with weak transferability and often uneven performance in multiple industrial scenarios.

[0084] Although regression-based methods and detection-based methods have their own advantages, they often experience performance degradation when faced with the unpredictable diversity of industrial scene images because they are better at counting objects they have been trained on, but less good at identifying and counting objects from categories they have never seen. This reliance on visual feature cues may lead to inaccuracies because these traditional strategies may not be able to fully capture the visual feature differences between objects that appear in the field of non-training datasets, resulting in incorrect target recognition and counting. The most direct way to solve this problem is to introduce some samples of the current scene as negative samples to prevent overfitting of the model training. However, complex industrial scene sample data is often difficult to collect, and due to the limitations of sample data collected on site, excessive reliance on these data by the model will lead to a decrease in its generalization ability in unseen categories or other industrial environments, making the model less adaptable to unknown environments.

[0085] In view of the defects of the above-mentioned related technologies, the embodiment of the present application proposes a method for target recognition and counting based on text prompt embedding. By introducing a pioneering cross-modal, category-agnostic general model, it addresses the complexity of target recognition and counting so that it can more flexibly demonstrate the effects of target recognition and counting in smart industrial site supervision, and can be reused in multiple industrial scenarios without the need for manual data collection and re-labeling training. This embodiment aims to interpret natural language prompts input by humans and convert them into accurate recognition and counting of target objects in images. This embodiment establishes a meaningful connection between text clues and the visual representation of objects in images. Due to the essential difference in the semantic richness of text information and images, it requires a novel method to ensure that the model can not only understand the text, but also associate it with objects that appear repeatedly in the image. This requires a paradigm shift from merely recognizing objects to semantically anchoring them in the image context. In order to bridge the gap between text and images, this embodiment constructs a cross-modal universal text-adaptive visual language model, combining the function of zero-shot learning with the nuances of text-based counting. Different from the limitations of traditional counting methods in smart industrial on-site supervision, this method is not only closer to the supervision needs of the real production environment in the smart industrial scenario, but also has strong versatility and generalizability to adapt to the diverse and changing needs of target object recognition and counting in complex industrial environments, providing an effective target recognition and counting solution for smart industrial supervision. The proposal of this embodiment has a certain positive impact on improving the intelligence of smart industrial supervision, simplifying the supervision process, and adapting to diversified needs by enabling the cross-modal universal adaptive model to effectively combine text information and image features.

[0086] See also Figure 2 , Figure 2A flowchart of a method for object recognition and counting based on text hint embedding provided in an embodiment of the present application, sample image B i Input image encoder to get image sequence , prompt text D i "Screw" input text encoder to get text sequence . The image sequence and text sequence Input adapter, get embedded features . The text sequence and embedding features Perform feature mapping to obtain the fusion sequence , the fusion sequence The density map is input into the decoder. According to the density map, the counting result C can be obtained. pred (As shown in 8), the density loss can be calculated based on the counting results.

[0087] The overall framework constructed in this embodiment is designed to achieve wide applicability for text image counting across different modes, directly generating a density map from each input image and its category prompt word, which can be accumulated to predict the total number of objects. Specifically, the design goal of the visual language model in this embodiment is to achieve the following functions: cross-modal versatility, which can process image and text data from different fields and scenes, and realize the recognition and counting of various types of objects.

[0088] In order to better reflect the content of objects of interest in the recognition and counting task through the intrinsic product of text and image features, this embodiment finely matches the text and the image to capture the complex relationships between the text description and multiple objects in the image, and accurately reflects these relationships in the statistical significance of the recognition and counting task.

[0089] This embodiment may elaborate on the image and text feature fusion method suitable for the recognition and counting task in the partial feature projection, including feature fusion to improve the accuracy of recognition and counting by text prompts. Since understanding the recognition and counting objects of interest expressed in the text is crucial to influencing the final recognition and counting results, image and text features need to capture long-term relationships at various scales. This embodiment combines image sequences and text sequences to achieve a deeper capture of the connection between text descriptions and image content, so as to understand their interactions and associations at a more refined level.

[0090] In the process of converting multimodal feature sequences into spatial feature representations, they are carefully resized to match the dimensions of the image, thereby creating feature maps that are convenient for subsequent image-based analysis. These feature maps are the basis for applying convolution operations, which are crucial for extracting and enhancing salient features inherent in visual data.

[0091] See also Figure 3 , Figure 3 A schematic diagram of the object recognition and counting principle based on text prompt embedding provided in the embodiment of the present application, the process is as follows:

[0092] In order to train the visual language model based on text prompt embedding, this embodiment can construct a general training data set for target recognition and counting. The training data set may include a large number of images of different categories, angles, and lighting conditions and corresponding prompt texts (i.e., text prompt descriptions), where the image resolution must meet 384×384, and the image must contain at least 3 counted objects, and a single image must correspond to at least 1 text description, to ensure the adequacy and reliability of the model training data.

[0093] Input the image and the corresponding prompt text into the image editor and text editor of the visual language model to generate the corresponding text sequence and image sequence .

[0094] Prompt sequence for text Rescaling Visual Language Models for Counting Image Sequences , we can highlight the region of interest indicated by the text prompt, thereby learning to map text descriptions to embedded features , enhancing the model’s understanding of implicit count information in text descriptions.

[0095] This embodiment provides the following feature projection mechanism: the image sequence calculated by the visual language model is and text prompt sequence After inputting the Regular regulator (i.e., feature fusion regulator), the image features can be obtained (i.e. embedding features), and then the obtained image features and text sequence Fusion is performed to obtain a fusion sequence ,Achieving a deeper capture of the connection between text description and image content, thus gaining a deeper understanding of the complex interaction between text description and image content.

[0096] In this embodiment, the multimodal feature sequence Convert to spatial feature representation and generate prediction density map M d The predicted density map is used to interpret the visual content and perform object recognition and counting. This embodiment can also update the loss according to the predicted density map.

[0097] In this embodiment, the corresponding training loss can be designed, and the loss function can be calculated to quantify the difference between the predicted output and the actual target value, and the parameters of the entire model can be updated and iterated. The final training result is the object recognition and counting model based on text prompt embedding (i.e., visual language model) described in this embodiment.

[0098] This embodiment proposes a regulator network based on a visual language model text-image encoder, which can capture the complex relationship between text descriptions and multiple objects in the image through deep training on a wide range of image and text pairs, and accurately reflect these relationships in the statistical significance of the counting task. This regulator network architecture can make up for the lack of precise spatial positioning capabilities in the visual language model, and enhance the model's understanding of the implicit counting information in the text description by embedding the text into the context of the image. This embodiment uses a regulator network to fuse the text feature sequence and the image feature sequence, and maps the text description to the image features, mainly including the following steps:

[0099] Step A1: For a current image-text pair i, use the image encoder in the visual language model to encode the image B in the image-text pair i Encode and get i The image encoder sequence , and the corresponding formula is:

[0100] ;

[0101] in Represents the encoding process of the image encoder on the image, which can obtain the image sequence of the image-text pair i .

[0102] Step A2: For a current image-text pair i, use the text encoder in the visual language model to encode the text D in the image-text pair i Encode and get the text D i The text encoder sequence , and the corresponding formula is:

[0103] ;

[0104] Represents the encoding process of the text encoder on the image, which can obtain the text sequence of the image-text pair i .

[0105] The image-text pair i is passed through the image encoder and the text encoder to obtain the corresponding image sequence and text sequence After that, the image sequence and text sequence Perform text adjustment and input both into the regulator network to obtain the input image sequence Block-level embedding features , the corresponding formula is:

[0106] ;

[0107] Indicates that the image sequence and text sequence The equivalent formula for the processing after inputting into the regulator is:

[0108] ;

[0109] The role of the regulator in the visual language model is to serve as a bridge for the transition from natural language cues to the prediction of the number of target objects in the image. It can use the pre-training capabilities of the visual language model to map text descriptions to image features, and then fine-tune these image features based on text clues to enhance the recognition of target object areas.

[0110] This embodiment proposes a cross-modal feature fusion method for fusing image and text features. This method uses a multi-layer perceptron MLP to map image sequences and text prompt sequences, uses an attention mechanism to fuse the combined image-text feature sequences, and calculates a multi-modal feature fusion sequence after feature integration. This method solves the problem that calculating the cosine similarity between image sequences and text sequences is not suitable for counting tasks, and this cross-modal fusion strategy allows objects in various scenes to be accurately counted without manual annotation. Thanks to this cross-modal feature fusion method, the model can fully understand the counting objects of interest expressed in the text and make this part of knowledge ultimately act on the counting results. It has strong generalization and practicality, and specifically includes the following steps:

[0111] Step B1: Mapping the embedded features using a multi-layer perceptron (MLP) get (i.e., the first output result), using a multi-layer perceptron MLP to map the text sequence get (ie, the second output result).

[0112] in yes The text prompt sequence after multi-layer perceptron MLP mapping, yes Image sequence after multi-layer perceptron MLP mapping.

[0113] Step B2: The resulting image sequence and text prompt sequence Respectively with the original image sequence and the original text sequence Combined, that is and Perform feature stitching to obtain the first stitching result ( , ), and After feature splicing, the second splicing result is obtained ( , ), respectively, to achieve a deeper capture of the connection between text description and image content.

[0114] Step B3: Use the attention mechanism to the combined image-text features ( , ) sequence and ( , ) sequence to perform feature integral calculation, and the result after feature integral calculation is the multimodal feature fusion sequence .

[0115] See also Figure 4 , Figure 4 A schematic diagram of a characteristic integral calculation principle provided in an embodiment of the present application, ( , ) sequence and ( , ) sequence is processed by MHA (Multi-head Attention) to obtain the fused sequence , Q represents the query vector, K represents the key vector, and V represents the value vector.

[0116] This embodiment connects image and text features and inputs them into an attention mechanism, which selectively emphasizes the most important aspects of the input data, helps the model focus on relevant image areas and text details, and can be applied to processing scenarios involving multiple objects or concepts.

[0117] The multimodal feature fusion sequence obtained by using the collaboration of multi-layer perceptron and attention mechanism Finally, this paper proposes a subsequent processing method for generating intuitive density maps and loss calculation methods for design model update optimization, which specifically includes the following steps:

[0118] Step C1: By fusing the multimodal feature sequence Perform convolution kernel processing or feature reshaping to obtain a higher-level and more representative feature representation.

[0119] Step C2: Upsampling the extracted or reshaped features to expand or restore their spatial dimensions and increase the resolution so that more detailed density information can be obtained during the density estimation process.

[0120] Step C3: The feature data undergoes a convolution operation after upsampling to further process the feature information and capture spatial correlations while retaining key feature patterns.

[0121] Step C4: Adjust the shape (i.e., dimension) of the feature data to make it meet the requirements of the subsequent density estimation method and ensure the compatibility of the data structure with the density map generation process.

[0122] Step C5: Use the kernel density estimation method to calculate and display the density distribution of these feature data, that is, generate a predicted density map M d , the formula for the entire process is as follows:

[0123] ;

[0124] represents upsampling, represents the convolution operation, Represents a reshape operation.

[0125] Prediction density map M d A pixel-by-pixel distribution of predicted object instances is provided, which helps to derive an accurate count of the objects present in the image. When transitioning to the decoding stage of the density map, the present embodiment constructs a decoder that is an upsampling layer of multiple 3×3 convolutional layers, using this design to maintain the spatial integrity of the output feature map, ensuring that the output of the model is a faithful representation of the complexity of the input image. After each convolution operation, the incorporation of the rectified linear unit (RELU) activation function helps to enhance the nonlinear expression capabilities of the model, enabling it to navigate to complex image features more flexibly.

[0126] According to the specification of the loss calculation formula, this embodiment can calculate the loss function to quantify the difference between the predicted output and the actual target value, and update the model accordingly according to the loss function. The corresponding loss Densityloss calculation formula is as follows:

[0127] ;

[0128] is the density map of the model's predicted output, With density plot The size of the predicted density map is consistent with The sum of all values ​​in is the calculated object count value. Represents a density map of the same size as the image, with a point marked at the location of the target object. This embodiment reconstructs, convolves, and upsamples the multimodal feature fusion sequence, and then uses a kernel density estimation method to calculate and display the density distribution of the processed feature data, generates a predicted density map, and then designs a loss calculation formula to calculate the loss function to quantify the difference between the predicted output and the actual target value for model optimization iteration.

[0129] After training, the resulting model has effective target recognition and counting capabilities. The core mechanism of the model integrates image regulators and feature projection techniques to achieve the transition from natural language prompts to prediction of the number of target objects in images. This architecture enables the pre-trained model in target recognition tasks to accurately count objects in various scenes without the need for manual annotation, with strong generalization and practicality.

[0130] The above-mentioned visual language model is a new general cross-modal model for natural language prompt-guided target recognition and counting. The visual language model has text-image pre-training capabilities and, combined with the introduced custom optimization components, achieves effective object recognition and counting.

[0131] In order to put the performance of the above visual language model into a general practical environment, this embodiment sets up an experiment to conduct a more rigorous comparative analysis with other counting methods. The mean absolute error MAE and root mean square error RMSE can be used as evaluation indicators. The test set in the FSC-147 dataset is compared with other zero-sample counting methods Patchselection (chip selection) and VLCounter (video LAN counter). The following are the experimental results:

[0132] Table 1. Mean absolute error (MAE) comparison results

[0133]

[0134] Table 2 Root mean square error RMSE comparison results

[0135]

[0136] According to the above comparison results, the visual language model of this embodiment has a significant advantage over other general zero-shot counting models in terms of counting error. This method can achieve this result due to the two-pronged approach in the above model architecture design: deepening the model's understanding of image content through text feature description, and subsequently using density map estimation to enhance its resilience to data variability. The visual language model provided by this embodiment can not only see the target in the image, but also perceive and count the target object through rich background and details.

[0137] This embodiment proposes an innovative cross-modal, category-agnostic general model designed for smart industrial supervision, which aims to directly output the number of target objects in industrial scene images based on natural language prompts. The model not only shows excellent counting performance, but also shows significant generalization ability, achieving effective object counting without the need for a large amount of manually labeled data like traditional methods. This embodiment takes advantage of the large-scale image and text pairing pre-training of the visual language model to achieve effective fusion of text and images, thereby guiding the model to learn informative patch-level image representations for dense prediction. This method helps to distinguish similar objects in images because patch-level features can capture more detailed local information, while text guidance provides additional contextual information to assist in the distinction. This approach is well suited for complex target recognition and counting tasks used in smart industrial supervision.

[0138] This embodiment, by introducing feature projection and regulator components, can capture the complex relationships between text descriptions and multiple objects in complex industrial scene images, and accurately reflect these relationships in counting tasks. The feature projection component significantly enhances the recognition and counting capabilities of the model by combining semantic information with fine-grained features of image regions, and the regulator component further optimizes the model feature representation of the region of interest indicated by the text prompt.

[0139] Unlike other text-image matching methods for a single scene, the entire model architecture in this embodiment does not require a large number of samples for training, because it relies on a pre-trained visual language model that has been pre-trained on a large-scale dataset and has rich knowledge of image and text alignment. For objects that have never been seen before, the model of this method can guide the model to recognize and count through text feature description prompts, because the visual language model has learned how to match text descriptions with image content, which works even without a large number of samples of a specific category.

[0140] It can be seen that this embodiment proposes a target recognition and counting method that is more universal and easy to use in smart industrial site supervision than traditional target detection and counting methods. It has high research value and practical application value, and can further provide protection for safety and standardization in industrial scenes through machine vision-assisted supervision.

[0141] An object recognition system provided in an embodiment of the present application includes:

[0142] A sample acquisition module, used to acquire a sample image and a prompt text corresponding to the sample image from a training data set; wherein the prompt text is used to describe an object in the sample image;

[0143] An encoding module, used to encode the sample image using a visual language model to obtain an image sequence, and to encode the prompt text corresponding to the sample image using the visual language model to obtain a text sequence;

[0144] A text adjustment module, used to adjust the image sequence using information in the text sequence to obtain embedded features;

[0145] A fusion module, used for fusing the embedded features with the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence;

[0146] A model optimization module, used to calculate a loss function value using the predicted density map, and adjust parameters of the visual language model according to the loss function value;

[0147] A task parsing module, for parsing a target recognition task to obtain a prompt text of an image to be processed and a target object if a target recognition task is received;

[0148] The task processing module is used to input the image to be processed and the prompt text of the target object into the visual language model to obtain the recognition result of the target object.

[0149] This embodiment obtains sample images and their corresponding prompt texts from a training data set, and uses the information in the text sequence to adjust the image sequence to obtain embedded features, thereby determining the portion of the image that matches the text description. This embodiment also fuses the embedded features with the text sequence to form a fused sequence, and generates a predicted density map based on this. The predicted density map can be used to adjust the parameters of the visual language model so that the visual language model has the ability to recognize targets. After sufficient training, the visual language model can accurately recognize the target object when receiving a target recognition task. The visual language model in this embodiment combines information from both image and text modalities for target recognition, improves the ability to recognize untrained object types, and enhances the generalization of target detection.

[0150] Furthermore, the process in which the fusion module fuses the embedded features and the text sequence to obtain a fused sequence includes: inputting the embedded features into the multilayer perceptron of the visual language model to obtain a first output result; inputting the text sequence into the multilayer perceptron of the visual language model to obtain a second output result; performing feature splicing on the first output result and the embedded features to obtain a first splicing result; performing feature splicing on the second output result and the text sequence to obtain a second splicing result; and performing feature integral calculation on the first splicing result and the second splicing result to obtain the fused sequence.

[0151] Furthermore, the process of the fusion module generating a predicted density map based on the fusion sequence includes: performing convolution kernel processing or feature reshaping on the fusion sequence to obtain a first processing result; performing upsampling operation, convolution operation and dimensionality adjustment on the first processing result in sequence to obtain a second processing result; using a kernel density estimation method to calculate the density distribution of feature data in the second processing result to obtain the predicted density map.

[0152] Furthermore, the process of the model optimization module using the predicted density map to calculate the loss function value includes: obtaining the actual density map corresponding to the sample image; wherein the predicted density map has the same size as the actual density map; and determining the loss function value based on the difference between the actual density map and the actual density map.

[0153] Furthermore, the text adjustment module uses the information in the text sequence to adjust the image sequence, and the process of obtaining the embedded feature includes: substituting the text sequence and the image sequence into the adjustment formula to obtain the embedded feature; wherein the adjustment formula is Regulator= , Regulator represents the embedded feature, represents an image sequence, Represents a text sequence, Represents matrix multiplication.

[0154] Furthermore, the task processing module inputs the image to be processed and the prompt text of the target object into the visual language model, and the process of obtaining the recognition result of the target object includes: inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a target density map; and generating the recognition result of the target object according to the target density map.

[0155] Furthermore, it also includes:

[0156] The counting module is used to count the target objects according to the target density map after obtaining the target density map, so as to obtain the number of the target objects contained in the image to be processed.

[0157] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.

[0158] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0159] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.

[0160] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of this application.

[0161] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A target recognition method, characterized in that: include: Acquire a sample image and a prompt text corresponding to the sample image from a training data set; wherein the prompt text is used to describe an object in the sample image; Encoding the sample images using a visual language model to obtain an image sequence, and encoding the prompt texts corresponding to the sample images using the visual language model to obtain a text sequence; Using information in the text sequence to adjust the image sequence to obtain embedded features; Fusing the embedded feature with the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence; Calculating a loss function value using the predicted density map, and adjusting parameters of the visual language model according to the loss function value; If a target recognition task is received, the target recognition task is parsed to obtain a prompt text of the image to be processed and the target object; The image to be processed and the prompt text of the target object are input into the visual language model to obtain a recognition result of the target object.

2. The target recognition method according to claim 1, characterized in that: The embedded feature and the text sequence are fused to obtain a fused sequence, including: Inputting the embedded features into the multilayer perceptron of the visual language model to obtain a first output result; Inputting the text sequence into the multilayer perceptron of the visual language model to obtain a second output result; Perform feature splicing on the first output result and the embedded feature to obtain a first splicing result; Perform feature splicing on the second output result and the text sequence to obtain a second splicing result; Performing feature integral calculation on the first splicing result and the second splicing result to obtain the fusion sequence.

3. The target recognition method according to claim 1, characterized in that: Generating a predicted density map according to the fusion sequence includes: Performing convolution kernel processing or feature reshaping on the fused sequence to obtain a first processing result; Performing upsampling, convolution and dimension adjustment on the first processing result in sequence to obtain a second processing result; The density distribution of the feature data in the second processing result is calculated using a kernel density estimation method to obtain the predicted density map.

4. The target recognition method according to claim 1, characterized in that: Calculating the loss function value using the predicted density map includes: Obtaining an actual density map corresponding to the sample image; wherein the predicted density map has the same size as the actual density map; The loss function value is determined based on a difference between the actual density map and the actual density map.

5. The target recognition method according to claim 1, characterized in that: The image sequence is adjusted using the information in the text sequence to obtain embedded features, including: Substituting the text sequence and the image sequence into the adjustment formula to obtain the embedded feature; Among them, the regulation formula is Regulator= , Regulator represents the embedded feature, represents an image sequence, represents a sequence of text, Represents matrix multiplication.

6. The target recognition method according to claim 1, characterized in that: Inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a recognition result of the target object, including: Inputting the image to be processed and the prompt text of the target object into the visual language model to obtain a target density map; A recognition result of the target object is generated according to the target density map.

7. The target recognition method according to claim 6, characterized in that: After obtaining the target density map, it also includes: The target objects are counted according to the target density map to obtain the number of the target objects contained in the image to be processed.

8. A target recognition system, characterized in that: include: A sample acquisition module, used to acquire a sample image and a prompt text corresponding to the sample image from a training data set; wherein the prompt text is used to describe an object in the sample image; An encoding module, used to encode the sample image using a visual language model to obtain an image sequence, and to encode the prompt text corresponding to the sample image using the visual language model to obtain a text sequence; A text adjustment module, used to adjust the image sequence using information in the text sequence to obtain embedded features; A fusion module, used for fusing the embedded features with the text sequence to obtain a fused sequence, and generating a prediction density map according to the fused sequence; A model optimization module, used to calculate a loss function value using the predicted density map, and adjust parameters of the visual language model according to the loss function value; A task parsing module, for parsing a target recognition task to obtain a prompt text of an image to be processed and a target object if a target recognition task is received; The task processing module is used to input the image to be processed and the prompt text of the target object into the visual language model to obtain the recognition result of the target object.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the target recognition method according to any one of claims 1 to 7 when calling the computer program in the memory.

10. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the target recognition method according to any one of claims 1 to 7 are implemented.

Images