Beef cattle image counting method, device and equipment based on cross-modal understanding and medium
By fusing visual and textual modal information through a cross-modal understanding method, a high-quality query-encoded feature map is generated, which solves the problems of counting accuracy and computational redundancy in existing beef cattle image counting algorithms in complex scenarios, and achieves efficient and accurate beef cattle target counting.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI INSTITUTE OF PHYSICAL SCIENCE CHINESE ACADEMY OF SCIENCES
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-02
AI Technical Summary
Existing beef cattle image counting algorithms have poor generalization ability in complex scenes, making it difficult to accurately distinguish between dense occlusion and multi-scale targets. Furthermore, computational redundancy leads to frequent instances of duplicate counting or missed counting.
A cross-modal understanding approach is adopted, which integrates multimodal information through image feature encoding network and text feature encoding network, generates high-quality query encoding feature map through progressive query fusion network, performs feature alignment and decoding through cross-modal interaction network, and achieves accurate counting by combining query selector and counter.
It improves the accuracy and robustness of beef cattle target counting in complex scenarios, reduces computational redundancy, and enhances counting accuracy in dense and multi-scale occlusion environments.
Smart Images
Figure CN122134664A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and intelligent management technology for beef cattle farming, and particularly to a method, apparatus, equipment and medium for counting beef cattle images based on cross-modal understanding. Background Technology
[0002] With the rapid development of multimodal correlation technology, the application of cross-modal technology for intelligent image analysis is becoming increasingly widespread, providing strong technical support for achieving accurate counting. In the beef cattle farming sector, the efficiency and accuracy of beef cattle image counting directly affect the level of intelligent management.
[0003] However, existing cattle counting algorithms typically learn the potential target distribution from a single image modality and are highly dependent on training with massive amounts of data, resulting in poor generalization ability. Especially when facing complex real-world environments with dense occlusion and multiple scale targets, existing models often have limited understanding capabilities and struggle to accurately distinguish individuals. Furthermore, to identify densely occluded cattle targets, common algorithms tend to enlarge image sizes or stitch images together. This approach not only significantly increases computational load but also easily leads to double counting or missed counting.
[0004] Therefore, how to construct a beef cattle image counting method that can integrate multimodal information, reduce computational redundancy, and achieve high counting accuracy in complex scenarios has become an urgent technical challenge. Summary of the Invention
[0005] The main objective of this invention is to provide a method, apparatus, device, and medium for counting beef cattle images based on cross-modal understanding, aiming to construct a method for counting beef cattle images that can fuse multimodal information in complex scenarios, reduce computational redundancy, and achieve high counting accuracy.
[0006] To achieve the above objectives, this invention proposes a method for counting beef cattle images based on cross-modal understanding, which includes the following steps: Obtain the images of beef cattle to be counted, the corresponding text description information, and the query target example area in the beef cattle images; The beef cattle image is used to extract features using an image feature coding network to obtain an overall multi-scale feature map; the text description information is used to encode using a text feature coding network to obtain text feature codes. The overall multi-scale feature map, the query target example region, and the text feature encoding are input into a progressive query fusion network. Image example query features are extracted from the overall multi-scale feature map based on the query target example region. The image example query features are combined with the text feature encoding to generate joint features. The joint features are used as keys and values, and the overall multi-scale feature map is used as a query for cross-attention interaction to generate a high-quality query encoding feature map. Extract the image query code from the high-quality query code feature map based on the query target example region; The image query code, the text feature code, and the high-quality query code feature map are input into the cross-modal interaction network. Feature alignment is performed through self-attention and cross-attention mechanisms to output the cross-modal query code and the cross-modal image code. Using a query selector, K coding blocks are selected from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding. The K cross-modal target queries are input into a cross-modal decoding network for feature decoding to obtain K cross-modal candidate queries; A confidence score is calculated using a counter based on the matching degree between the K cross-modal candidate queries and the cross-modal query code, and the number of beef cattle targets is determined based on the confidence score.
[0007] Preferably, the step of inputting the overall multi-scale feature map, the query target example region, and the text feature encoding into a progressive query fusion network to generate a high-quality query encoding feature map includes the following steps: Based on the target example region, RoIAlign region of interest aligned pooling operation is applied to the feature map of the corresponding scale in the overall multi-scale feature map to extract the target example image query features at the corresponding scale. The image example query features are concatenated with the text feature encoding to form the joint features at the corresponding scale; The joint features are used as keys and values for cross-attention, and the feature maps at the corresponding scales are used as queries. The example images and text information are mapped to the sensing features at the corresponding scales through the cross-attention interaction. The sensed features are enhanced by deformable self-attention operations to obtain a query-encoded feature map at the corresponding scale; The query coding feature maps at multiple corresponding scales are fused to obtain the high-resolution, high-quality query coding feature map.
[0008] Preferably, the specific calculation formula for generating a high-quality query encoding feature map is as follows:
[0009]
[0010]
[0011] in, The first element in the overall multi-scale feature map represents the... Each network-scale layer, Query features for image examples at the corresponding scale. Encode the text features. This indicates a splicing operation. For the joint feature; This refers to the feature map at the corresponding scale. This represents a standard cross-attention layer that includes skip connection layers and layer normalization processing. The sensing feature; For deformable self-attention, This is the query encoding feature map at the corresponding scale.
[0012] Preferably, fusing multiple query coding feature maps at corresponding scales to obtain the high-resolution, high-quality query coding feature map specifically includes: The query-encoded feature maps at different scales are processed by layer-by-layer stacking using 2x upsampling, 3×3 convolution, and a multilayer perceptron. The calculation formula is as follows:
[0013]
[0014] in, This indicates a 2x upsampling. This represents a 3×3 convolution. It is a multilayer perceptron. , , and These are the query encoding feature maps at the scales of layers 2, 3, 4, and 5, respectively. This is the high-quality query encoding feature map.
[0015] Preferably, the cross-modal interaction network consists of six interconnected interaction modules; The process of inputting the image query code, the text feature code, and the high-quality query code feature map into a cross-modal interaction network, and outputting a cross-modal query code and a cross-modal image code, includes the following steps: The image query code and the text feature code are concatenated to obtain a fused code, and the fused code is subjected to self-attention processing to obtain a fused feature; The first cross-attention process is performed using the fused features and the high-quality query encoding feature map. The first cross-attention process is used to establish the connection between the image and the fused features and output the updated image information. The fusion features and the updated image information are used to perform a second cross-attention process, which is used to establish the connection between the fusion features and the image, and output the updated fusion information. Through the processing of the six interactive modules, the cross-modal image code and the cross-modal query code are output.
[0016] Preferably, the specific calculation formula for the interaction module is as follows:
[0017]
[0018]
[0019]
[0020]
[0021]
[0022] in, Encode the image query. Encode the text features. For the fusion encoding, For self-attention, The fusion feature; This is the high-quality query encoding feature map. For cross attention; It is a feedforward neural network. and The first The cross-modal image encoding and the cross-modal query encoding output by the interactive module.
[0023] Preferably, the step of using a query selector to select K coding blocks from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding specifically includes: The Einstein summation convention is used to perform a dot product calculation on the cross-modal image code and the cross-modal query code to obtain a similarity matrix; The highest relevance score for each image location is obtained by taking the maximum value in the similarity matrix. Based on the highest relevance scores, the K highest-scoring coding blocks are selected as initial query candidates, i.e., the cross-modal target query; The calculation formula is as follows:
[0024]
[0025] in, Encode the cross-modal image. Encoding the cross-modal query. To calculate the dot product using Einstein's summation convention, For the similarity matrix, For the cross-modal target query.
[0026] Preferably, the cross-modal decoding network is composed of six stacked self-attention and cross-attention modules; The step of inputting the K cross-modal target queries into a cross-modal decoding network for feature decoding includes the following steps: Self-attention processing is performed on the K cross-modal target queries to obtain the updated target queries; The updated target query is used as the first query vector, and the cross-modal image encoding is used as the first key vector and the first value vector to perform the first cross-attention processing; Using the output after the first cross-attention processing as the second query vector, and using the cross-modal query encoding as the second key vector and the second value vector, a second cross-attention processing is performed to obtain the cross-modal candidate query.
[0027] Preferably, the confidence score is calculated using a counter based on the matching degree between the K cross-modal candidate queries and the cross-modal query code, and the number of beef cattle targets is determined according to the confidence score. The specific calculation formula is as follows:
[0028] in, For the cross-modal candidate query, Encoding the cross-modal query. This represents the dot product operation. This represents the activation function. The confidence score; the determination of the number of beef cattle targets specifically involves: using a confidence threshold to... Perform filtering and count the number of target items to be retained.
[0029] Preferably, the image feature coding network adopts the Swin-B network, and the overall multi-scale feature map includes feature layers of four scales: C2, C3, C4 and C5, and each feature layer undergoes a 1×1 convolution operation to unify the number of channels to 256. The text feature encoding network uses a pre-trained BERT-base model, and the text features are encoded as a 256-dimensional label sequence.
[0030] This application also discloses a beef cattle image counting device based on cross-modal understanding, comprising: The acquisition module is used to acquire the images of beef cattle to be counted, the corresponding text description information, and the query target example area in the beef cattle images; The feature encoding module is used to extract features from the beef cattle image using an image feature encoding network to obtain an overall multi-scale feature map, and to encode the text description information using a text feature encoding network to obtain text feature codes. The progressive fusion module is used to input the overall multi-scale feature map, the query target example region, and the text feature encoding into the progressive query fusion network to generate a high-quality query encoding feature map; The extraction module is used to extract image query codes from the high-quality query code feature map based on the query target example region; The cross-modal interaction module is used to input the image query code, the text feature code, and the high-quality query code feature map into the cross-modal interaction network, perform feature alignment through self-attention mechanism and cross-attention mechanism, and output cross-modal query code and cross-modal image code; The query selection module is used to select K coding blocks from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding using a query selector. The decoding module is used to input the K cross-modal target queries into the cross-modal decoding network for feature decoding to obtain K cross-modal candidate queries; The counting module is used to calculate a confidence score based on the matching degree between the K cross-modal candidate queries and the cross-modal query code using a counter, and to determine the number of beef cattle targets based on the confidence score.
[0031] This application also discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any of the preceding claims.
[0032] This application also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described in any of the preceding claims.
[0033] The above technical solution has the following advantages: This invention integrates visual and textual modal information, making the model more flexible and robust, and enabling it to better understand image content using textual descriptions. Simultaneously, this invention introduces a progressive query fusion network, which maps image examples and textual features onto multi-scale feature maps through region-of-interest alignment, cross-attention, and deformable self-attention mechanisms, generating high-resolution, high-quality query-encoded feature maps. This mechanism significantly improves feature resolution and representational power, enabling the model to more accurately perceive targets at different scales. While reducing computational cost, it effectively improves the counting accuracy of cattle targets in dense, multi-scale, and occluded scenes. Attached Figure Description
[0034] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein: Figure 1 The flowchart shows the beef cattle image counting method based on cross-modal understanding provided in Embodiment 1 of the present invention.
[0035] Figure 2 This is a schematic diagram of the overall architecture of the beef cattle image counting model provided in Embodiment 1 of the present invention.
[0036] Figure 3 This is a schematic diagram of the progressive query fusion network provided in Embodiment 1 of the present invention.
[0037] Figure 4 This is a structural block diagram of the beef cattle image counting device based on cross-modal understanding provided in Embodiment 2 of the present invention.
[0038] Figure 5 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation
[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0040] Example 1 like Figures 1 to 3As shown, this embodiment provides a method for counting cattle images based on cross-modal understanding. This method can achieve high-precision cattle target counting in complex real-world environments, especially in scenes with dense occlusion and coexistence of multi-scale targets, by fusing visual and textual modal information and employing a progressive query fusion strategy. This method is mainly implemented by constructing and running a cattle image counting model that includes an image feature encoding network, a text feature encoding network, a progressive query fusion network, a cross-modal interaction network, a query selector, a cross-modal decoding network, and a counter.
[0041] The specific process of this method is as follows.
[0042] First, the system acquires images of the cattle to be counted, corresponding text descriptions, and the query target example regions within the cattle images. In practical applications, the cattle images can originate from surveillance cameras at the farm or be manually taken. The text descriptions are feature descriptions of the targets to be counted in the images, such as brown and white cattle. The query target example regions are typically given in the form of bounding boxes, indicating several typical cattle samples in the image so that the model understands the specific object features to be counted. The acquired cattle images undergo necessary preprocessing, such as resizing or normalizing, to adapt to the input requirements of subsequent networks.
[0043] Subsequently, an image feature encoding network is used to extract features from the beef cattle image, obtaining an overall multi-scale feature map. Specifically, in this embodiment, the image feature encoding network is constructed based on the Swin-B network. The Swin-B network can efficiently extract image features through hierarchical feature map construction and a shifted window self-attention mechanism. This network processes the input beef cattle image and outputs feature layers containing different scales, namely four feature layers of scales: C2, C3, C4, and C5. To facilitate subsequent processing, each feature layer in this embodiment is further processed... The convolution operation uniformly adjusts the number of channels to 256. Simultaneously, a text feature encoding network is used to encode the text description information, resulting in text feature encoding. The text feature encoding network employs a pre-trained BERT-based model, which has been pre-trained on phrase localization data and can effectively understand text semantics. The parameters of the text feature encoding network are frozen during training. The input text description information is processed by the BERT tokenizer to generate text tags, which are then mapped to a 256-dimensional tag sequence, i.e., the text feature encoding. .
[0044] Next, the overall multi-scale feature map, the query target example region, and the text feature encoding are input into a progressive query fusion network to generate a high-quality query encoding feature map. This step aims to combine image examples and text descriptions to extract and fuse high-resolution, semantically rich features from the multi-scale feature map, thereby enhancing the perception of dense and small-scale targets. The specific processing is as follows: First, based on the query target example region, RoIAlign region of interest alignment pooling operation is applied to the feature map of the corresponding scale in the overall multi-scale feature map to extract the target example image query features at the corresponding scale. Next, the query features of the image example are concatenated with the text feature encoding to form the joint features at the corresponding scale. The calculation formula is as follows:
[0045] in, The first element in the overall multi-scale feature map represents the... Each network-scale layer, This indicates a splicing operation.
[0046] Subsequently, the joint features are used as the keys and values for cross-attention, and the feature maps at the corresponding scales are used as queries. Through the cross-attention interaction, the example image and text information are mapped to the corresponding scale's sensing features. This cross-attention includes skip connection layers and layer normalization processing. The calculation formula is:
[0047] in, This refers to the feature map at the corresponding scale. This represents standard cross-attention. The sensing feature is described above.
[0048] Based on this, the sensed features are enhanced through deformable self-attention operations to obtain a query-encoded feature map at the corresponding scale. Deformable self-attention only needs to process a small number of neighboring queries and employs learnable displacement prediction, effectively reducing computational complexity. The calculation formula is as follows:
[0049] in, For deformable self-attention, This is the query encoding feature map at the corresponding scale.
[0050] Finally, the query coding feature maps at multiple corresponding scales are fused to obtain the high-resolution, high-quality query coding feature map. This embodiment uses a 2x upsampling... Convolutional layers and multilayer perceptrons are used to stack the query-encoded feature maps at different scales layer by layer. The specific calculation logic is as follows:
[0051]
[0052] in, This indicates a 2x upsampling. express convolution, It is a multilayer perceptron. , , and These are the query encoding feature maps at the scales of layers 2, 3, 4, and 5, respectively. This is the high-quality query encoding feature map. After obtaining the high-quality query encoding feature map, RoIAlign operation is applied again from the high-quality query encoding feature map based on the query target example region to extract the image query encoding. .
[0053] After obtaining the image query code, text feature code, and high-quality query code feature map, these three are input into the cross-modal interaction network. The cross-modal interaction network consists of six cascaded interaction modules, designed to align features through self-attention and cross-attention mechanisms, outputting cross-modal query codes and cross-modal image codes. For each interaction module, the processing steps are as follows: First, the image query code... and the text feature encoding The fused code is obtained by splicing. The fusion encoding is then subjected to self-attention processing to obtain fusion features. This helps capture the correlation information between image patches and text. The calculation formula is:
[0054]
[0055] in, For self-attention.
[0056] Then, the fused features and the high-quality query-encoded feature map are used for a first cross-attention process to establish a connection between the image and the fused features, outputting the updated image information. Next, the fused features and the updated image information are used for a second cross-attention process to establish a connection between the fused features and the image, outputting the updated fused information. The calculation formula is as follows:
[0057]
[0058] Finally, it is processed by a feedforward neural network:
[0059]
[0060] in, For cross attention, It is a feedforward neural network, which consists of two fully connected layers and a ReLU activation function. and The first The interactive modules output cross-modal image codes and cross-modal query codes. After being processed in series by six modules, the final output is the cross-modal image code used in subsequent steps. and cross-modal query coding .
[0061] Next, a query selector is used to select K coding blocks from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding. In this embodiment, K is set to 900. Specifically, the Einstein summation convention is used to perform a dot product calculation on the cross-modal image coding and the cross-modal query coding to obtain a similarity matrix. The maximum value in the similarity matrix is taken to obtain the highest relevance score for each image location. Based on the highest relevance scores, the K coding blocks with the highest scores are selected as initial query candidates, i.e., the cross-modal target queries. The calculation formula is:
[0062]
[0063] in, To calculate the dot product using Einstein's summation convention, For the similarity matrix, For the cross-modal target query.
[0064] Subsequently, the K cross-modal target queries are input into a cross-modal decoding network for feature decoding to obtain K cross-modal candidate queries. The cross-modal decoding network also consists of six stacked self-attention and cross-attention modules. Each module's processing steps include: performing self-attention processing on the K cross-modal target queries to obtain updated target queries. Then, using the updated target queries as the first query vector, the cross-modal image encoding is applied... The first key vector and the first value vector are used as the first cross-attention vector, and then the first cross-attention processing is performed. Next, the output after the first cross-attention processing is used as the second query vector, and the cross-modal query encoding is applied. As the second key vector and second value vector, a second cross-attention process is performed to obtain the cross-modal candidate query. The calculation formula for this process is as follows:
[0065]
[0066]
[0067] in, This refers to the cross-modal candidate query output.
[0068] Finally, a confidence score is calculated using a counter based on the matching degree between the K cross-modal candidate queries and the cross-modal query code, and the number of beef cattle targets is determined based on the confidence score. Specifically, a mixed similarity matrix is obtained through dot product operation, and element-wise calculation is performed using the Sigmoid function. The calculation formula is as follows:
[0069] in, This represents the dot product operation. The confidence score is used to determine the target number of beef cattle. Specifically, this is done by setting a preset confidence threshold. By filtering and counting the number of target animals retained, the accurate counting of cattle in the image can be achieved.
[0070] Example 2 like Figure 4 As shown, this embodiment provides a beef cattle image counting device based on cross-modal understanding. This device is a collection of functional modules that embodies the method described in Embodiment 1, aiming to achieve the aforementioned high-precision counting process through hardware or software logic. The device mainly includes an acquisition module 101, a feature encoding module 102, a progressive fusion module 103, an extraction module 104, a cross-modal interaction module 105, a query selection module 106, a decoding module 107, and a counting module 108. These modules are connected sequentially and work together to complete the complex image counting task.
[0071] The acquisition module 101 is mainly used for data input and preprocessing. Specifically, this module is responsible for acquiring the images of beef cattle to be counted, the corresponding text description information, and the query target example region in the beef cattle images. In actual operation, the acquisition module 101 can be a communication interface used to receive raw image data from external camera equipment or databases, and perform necessary format conversion and size normalization on it to prepare for subsequent feature extraction.
[0072] The feature encoding module 102 is connected to the acquisition module 101 and is used for deep feature mining of the input raw data. This module integrates two independent sub-networks: an image feature encoding network and a text feature encoding network. The image feature encoding network uses a Swin-B network architecture to perform multi-level convolution and attention calculations on the beef cattle images, outputting a holistic multi-scale feature map containing four scales: C2, C3, C4, and C5, with the number of channels in each feature layer adjusted to 256. Simultaneously, the text feature encoding network utilizes a pre-trained BERT-based model to transform the input textual descriptions of the beef cattle's appearance into 256-dimensional text feature codes. This dual-channel encoding design ensures that the device can simultaneously understand visual and semantic information.
[0073] The progressive fusion module 103 receives the output from the feature encoding module 102 and inputs the overall multi-scale feature map, the query target example region, and the text feature encoding into its internal progressive query fusion network. The core function of this module is to generate a high-quality query-encoded feature map. It first performs RoIAlign operations on feature maps at different scales to extract image example query features, and then concatenates these features with the text feature encoding. The concatenated features serve as keys and values, while the original scale feature map serves as the query, undergoing cross-attention interaction. Subsequently, the module enhances the features using a deformable self-attention mechanism and fuses the multi-scale feature maps layer by layer through upsampling and convolution operations, ultimately generating a high-resolution, high-quality query-encoded feature map. This process effectively addresses the problem of small targets, such as beef cattle, being easily overlooked.
[0074] The extraction module 104 extracts image query codes from the high-quality query code feature map output by the progressive fusion module 103, based on the query target example region. This step ensures that subsequent interactive networks can focus on the specific target features to be counted.
[0075] The cross-modal interaction module 105 is a key component for achieving feature alignment. It receives image query codes, text feature codes, and high-quality query-coded feature maps, and processes them through six internally concatenated interaction modules. Each interaction module includes a self-attention mechanism and a cross-attention mechanism. The self-attention mechanism first processes the fused encoding of the image and text, and then establishes bidirectional relationships from the image to the fused features and from the fused features to the image through two cross-attention processes. This module ultimately outputs cross-modal query codes and cross-modal image codes, achieving deep fusion and alignment of visual and textual features.
[0076] The query selection module 106 utilizes a query selector to filter based on the correlation between cross-modal image encoding and cross-modal query encoding. This module calculates a similarity matrix using the Einstein summation convention and selects the 900 encoding blocks with the highest similarity scores as the cross-modal target query. This filtering mechanism effectively removes background noise and retains the most likely target region for beef cattle.
[0077] The decoding module 107 inputs the selected K cross-modal target queries into the cross-modal decoding network. This network consists of 6 stacked self-attention and cross-attention modules. Through layer-by-layer decoding, the information from the cross-modal image encoding and cross-modal query encoding is further integrated into the target query to generate K cross-modal candidate queries.
[0078] Finally, the counting module 108 processes the cross-modal candidate query using a counter. This module calculates the dot product of the candidate query and the cross-modal query code, and obtains a confidence score using the Sigmoid function. Subsequently, the module filters the scores according to a preset threshold, counts the final number of retained targets, and outputs an accurate cattle count result.
[0079] Example 3 like Figure 5 As shown, this embodiment provides an electronic device designed to serve as the hardware platform for the aforementioned beef cattle image counting method. This electronic device provides powerful computing capabilities at the hardware level to support complex deep learning network operations. The electronic device mainly includes a processor 201, a memory 202, and a computer program 203 stored in the memory 202 and executable on the processor 201.
[0080] Processor 201 is the core computing unit of the electronic device, and can be one or more central processing units (CPUs), graphics processing units (GPUs), neural network processing units (NPUs), or combinations thereof. Considering that this application involves attention mechanism calculations using Swing-B, BERT, and multiple Transformer architectures, processor 201 preferably includes a high-performance GPU to accelerate matrix operations and parallel processing. Processor 201 communicates with memory 202 and other components via a bus system.
[0081] The memory 202 is used to store data and instructions. It stores not only the computer program 203, but also the pre-trained Swin-B network weights, BERT-base model weights, and parameters of various sub-networks such as the trained progressive query fusion network and cross-modal interaction network. Furthermore, the memory 202 is also used to temporarily store acquired images of beef cattle, textual descriptions, and intermediate feature maps from the computation process, such as overall multi-scale feature maps and high-quality query encoding feature maps.
[0082] When processor 201 executes computer program 203, the electronic device is able to implement the various steps of the beef cattle image counting method based on cross-modal understanding described in Embodiment 1. Specifically, this includes: controlling the input interface to acquire image and text data; calling image and text encoding networks to extract features; executing a progressive query fusion algorithm to generate high-quality features; running a cross-modal interaction and decoding network for feature alignment and decoding; and finally calculating and outputting the number of beef cattle.
[0083] Example 4 This embodiment provides a computer-readable storage medium. The storage medium can be non-volatile or volatile, and stores a computer program thereon. When executed by a processor, the computer program can implement the beef cattle image counting method based on cross-modal understanding described in Embodiment 1.
[0084] The specific forms of this storage medium include, but are not limited to, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. In practical applications, this computer program can be distributed and downloaded via the network, and installed on various computing devices, such as servers, personal computers, or embedded terminals. By running the program on this storage medium, the computing device can load a pre-trained cattle counting model and perform high-precision automatic counting of cattle images in complex input scenes, effectively solving the problem of low counting accuracy in existing technologies under dense occlusion and multi-scale target scenes.
[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for counting beef cattle images based on cross-modal understanding, characterized in that, The method includes the following steps: Obtain the images of beef cattle to be counted, the corresponding text description information, and the query target example area in the beef cattle images; The beef cattle image is used to extract features using an image feature coding network to obtain an overall multi-scale feature map; the text description information is used to encode using a text feature coding network to obtain text feature codes. The overall multi-scale feature map, the query target example region, and the text feature encoding are input into a progressive query fusion network. Image example query features are extracted from the overall multi-scale feature map based on the query target example region. The image example query features are combined with the text feature encoding to generate joint features. The joint features are used as keys and values, and the overall multi-scale feature map is used as a query for cross-attention interaction to generate a high-quality query encoding feature map. Extract the image query code from the high-quality query code feature map based on the query target example region; The image query code, the text feature code, and the high-quality query code feature map are input into the cross-modal interaction network. Feature alignment is performed through self-attention and cross-attention mechanisms to output the cross-modal query code and the cross-modal image code. Using a query selector, K coding blocks are selected from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding. The K cross-modal target queries are input into a cross-modal decoding network for feature decoding to obtain K cross-modal candidate queries; A confidence score is calculated using a counter based on the matching degree between the K cross-modal candidate queries and the cross-modal query code, and the number of beef cattle targets is determined based on the confidence score.
2. The method according to claim 1, characterized in that, The step of inputting the overall multi-scale feature map, the query target example region, and the text feature encoding into a progressive query fusion network to generate a high-quality query encoding feature map includes the following steps: Based on the target example region, RoIAlign region of interest aligned pooling operation is applied to the feature map of the corresponding scale in the overall multi-scale feature map to extract the target example image query features at the corresponding scale. The image example query features are concatenated with the text feature encoding to form the joint features at the corresponding scale; The joint features are used as keys and values for cross-attention, and the feature maps at the corresponding scales are used as queries. The example images and text information are mapped to the sensing features at the corresponding scales through the cross-attention interaction. The sensed features are enhanced by deformable self-attention operations to obtain a query-encoded feature map at the corresponding scale; The query coding feature maps at multiple corresponding scales are fused to obtain the high-resolution, high-quality query coding feature map.
3. The method according to claim 2, characterized in that, The specific calculation formula for generating a high-quality query-encoded feature map is as follows: in, The first element in the overall multi-scale feature map represents the... Each network-scale layer, Query features for image examples at the corresponding scale. Encode the text features. This indicates a splicing operation. For the joint feature; This refers to the feature map at the corresponding scale. This represents a standard cross-attention layer that includes skip connection layers and layer normalization processing. The sensing feature; For deformable self-attention, This is the query encoding feature map at the corresponding scale.
4. The method according to claim 2, characterized in that, The step of fusing multiple query coding feature maps at corresponding scales to obtain the high-resolution, high-quality query coding feature map specifically includes: The query-encoded feature maps at different scales are processed by layer-by-layer stacking using 2x upsampling, 3×3 convolution, and a multilayer perceptron. The calculation formula is as follows: in, This indicates a 2x upsampling. This represents a 3×3 convolution. It is a multilayer perceptron. , , and These are the query encoding feature maps at the scales of layers 2, 3, 4, and 5, respectively. This is the high-quality query encoding feature map.
5. The method according to claim 1, characterized in that, The cross-modal interaction network consists of six interconnected interaction modules; The process of inputting the image query code, the text feature code, and the high-quality query code feature map into a cross-modal interaction network, and outputting a cross-modal query code and a cross-modal image code, includes the following steps: The image query code and the text feature code are concatenated to obtain a fused code, and the fused code is subjected to self-attention processing to obtain a fused feature; The first cross-attention process is performed using the fused features and the high-quality query encoding feature map. The first cross-attention process is used to establish the connection between the image and the fused features and output the updated image information. The fusion features and the updated image information are used to perform a second cross-attention process, which is used to establish the connection between the fusion features and the image, and output the updated fusion information. Through the processing of the six interactive modules, the cross-modal image code and the cross-modal query code are output.
6. The method according to claim 5, characterized in that, The specific calculation formula for the interaction module is as follows: in, Encode the image query. Encode the text features. For the fusion encoding, For self-attention, The fusion feature; This is the high-quality query encoding feature map. For cross attention; It is a feedforward neural network. and The first The cross-modal image encoding and the cross-modal query encoding output by the interactive module.
7. The method according to claim 1, characterized in that, The step of using a query selector to select K coding blocks from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding specifically includes: The Einstein summation convention is used to perform a dot product calculation on the cross-modal image code and the cross-modal query code to obtain a similarity matrix; The highest relevance score for each image location is obtained by taking the maximum value in the similarity matrix. Based on the highest relevance scores, the K highest-scoring coding blocks are selected as initial query candidates, i.e., the cross-modal target query; The calculation formula is as follows: in, Encode the cross-modal image. Encoding the cross-modal query. To calculate the dot product using Einstein's summation convention, For the similarity matrix, For the cross-modal target query.
8. The method according to claim 1, characterized in that, The cross-modal decoding network consists of six stacked self-attention and cross-attention modules; The step of inputting the K cross-modal target queries into a cross-modal decoding network for feature decoding includes the following steps: Self-attention processing is performed on the K cross-modal target queries to obtain the updated target queries; The updated target query is used as the first query vector, and the cross-modal image encoding is used as the first key vector and the first value vector to perform the first cross-attention processing; Using the output after the first cross-attention processing as the second query vector, and using the cross-modal query encoding as the second key vector and the second value vector, a second cross-attention processing is performed to obtain the cross-modal candidate query.
9. The method according to claim 1, characterized in that, The confidence score is calculated using a counter based on the matching degree between the K cross-modal candidate queries and the cross-modal query code, and the number of beef cattle targets is determined based on the confidence score. The specific calculation formula is as follows: in, For the cross-modal candidate query, Encoding the cross-modal query. This represents the dot product operation. This represents the activation function. The confidence score; the determination of the number of beef cattle targets specifically involves: using a confidence threshold to... Perform filtering and count the number of target items to be retained.
10. The method according to claim 1, characterized in that, The image feature coding network adopts the Swin-B network. The overall multi-scale feature map includes feature layers of four scales: C2, C3, C4 and C5. Each feature layer undergoes a 1×1 convolution operation to unify the number of channels to 256. The text feature encoding network uses a pre-trained BERT-base model, and the text features are encoded as a 256-dimensional label sequence.
11. A beef cattle image counting device based on cross-modal understanding, characterized in that, include: The acquisition module is used to acquire the images of beef cattle to be counted, the corresponding text description information, and the query target example area in the beef cattle images; The feature encoding module is used to extract features from the beef cattle image using an image feature encoding network to obtain an overall multi-scale feature map, and to encode the text description information using a text feature encoding network to obtain text feature codes. The progressive fusion module is used to input the overall multi-scale feature map, the query target example region, and the text feature encoding into the progressive query fusion network to generate a high-quality query encoding feature map; The extraction module is used to extract image query codes from the high-quality query code feature map based on the query target example region; The cross-modal interaction module is used to input the image query code, the text feature code, and the high-quality query code feature map into the cross-modal interaction network, perform feature alignment through self-attention mechanism and cross-attention mechanism, and output cross-modal query code and cross-modal image code; The query selection module is used to select K coding blocks from the cross-modal image coding as cross-modal target queries based on the correlation between the cross-modal image coding and the cross-modal query coding using a query selector. The decoding module is used to input the K cross-modal target queries into the cross-modal decoding network for feature decoding to obtain K cross-modal candidate queries; The counting module is used to calculate a confidence score based on the matching degree between the K cross-modal candidate queries and the cross-modal query code using a counter, and to determine the number of beef cattle targets based on the confidence score.
12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 10.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 10.