Complex multi-element image popular accurate intelligent AI expression method

Abstract

This application presents a method for providing clear, accurate, and intelligent AI representations of complex, multi-element images. It employs an improved Faster R-CNN model as the image processing model, replaces VGG-16 with DenseNet for feature extraction, optimizes bounding box selection through dimensional clustering, and enhances model stability by predicting relative grid cell coordinates. A text generation architecture is constructed based on a hierarchical recursive network. The first layer, a global long short memory layer, determines the topic vector based on the fusion vector to increase the number of descriptive sentences. The second layer, a sentence BRNN layer, generates accurate and rich descriptive sentences. A fully convolutional localization layer effectively integrates and transfers block features, constructing an end-to-end architecture that collaboratively optimizes image processing and text generation. This method addresses the problems of high model complexity, inaccurate bounding box localization, and insufficient association of descriptive sentences in existing technologies, generating denser, more accurate, and natural descriptive language that conforms to language habits. It is suitable for various application scenarios such as visual intelligent question answering and video subtitle generation.

Description

Technical Field

[0001] This application relates to a simple and accurate representation method for multi-element images, and more particularly to a simple and accurate intelligent AI representation method for complex multi-element images, belonging to the field of image intelligent representation technology. Background Technology

[0002] The higher-level concept of visual analytics anticipates that after computers understand and analyze image information, they can convert it into text information that is familiar to humans and conforms to language habits, generating natural descriptive language for images, which is commonly referred to as "image description".

[0003] Generating natural language descriptions for complex, multi-element images falls under the category of image annotation in image-based storytelling. Complex, multi-element images refer to images containing relatively dense local blocks, numbering more than ten. The final result is to label each local block in the image and generate text descriptions. The development of natural language descriptions for images has progressed from initially only being able to identify simple polyhedra to now being able to identify large outdoor scenes. This research has achieved a transformation from simple to complex images, from overall objects to local attributes, and from large targets to small targets. Image-based storytelling belongs to the semantic level of scene understanding, that is, by establishing mathematical models to convert image information into text information, achieving image-to-text conversion. Based on the natural language generated from images, further research can be conducted in areas such as visual intelligent question answering and video caption generation.

[0004] Visual intelligent question answering is defined as a computer responding to a user's questions about an image based on its content. The computer first analyzes the image, including the location of objects, their descriptive attributes, and the relationships between them, then generates relevant natural text descriptions. After a user submits a question, the computer analyzes it, extracts key information, and then uses this information to locate relevant objects and their text descriptions within the image, returning the answer that best fits the question. Visual intelligent question answering also involves fine-grained discrimination, object discrimination, and behavior discrimination. The natural language used to generate the image is merely a general description of the image's semantic understanding; therefore, it forms the foundation of visual intelligent question answering. Currently, research on visual intelligent question answering is in its early stages. The algorithms and models can handle relatively simple problems, particularly those requiring significant prior knowledge. Future applications could include early childhood education, navigation for the blind, and intelligent chat services.

[0005] Video captioning is defined as the process by which a computer analyzes the content of a given video to generate a summary or detailed description of that video. Unlike understanding a single scene or image, a video may contain multiple objects and scenes, and these objects are in motion. Understanding the content of a video requires not only understanding the objects in each frame but also their motion across multiple frames and the changes in the scenes in which they are situated. The key to video captioning lies in the detailed analysis of each scene, which is essentially the task of image-to-text generation. In the process of video captioning, it is necessary to consider whether the content of the image and video is complementary, whether image-to-text and video-to-text generation can be processed simultaneously in a multi-task manner, and how to transfer image-to-text generation to video-to-text generation. Real-world applications of video-to-text generation include massive video analysis, network video surveillance, and rapid video retrieval.

[0006] Beyond the extended research described above, the natural description language task of generating images itself has broad practical application value. This includes the fact that the regulation of currently popular unmanned vehicles, drones, and unmanned supermarkets all require the perception and cognition of the external environment.

[0007] The problems existing in the prior art and the core technical difficulties of this application include:

[0008] First, traditional object detection models suffer from excessively high computational complexity and insufficient feature extraction efficiency. Existing models like Faster R-CNN rely on deep networks such as VGG-16 for feature extraction, requiring 30 billion floating-point operations for forward propagation on a single image. The sheer number of parameters leads to severe computational resource consumption. While YOLO transforms detection into a regression problem for end-to-end optimization, it frequently suffers from object localization errors. SSD employs multi-scale feature map detection but performs poorly in extracting small-scale objects. PGAN utilizes generative adversarial networks to improve the resolution of small objects, but its detection speed is slow. These models present a fundamental contradiction between accuracy and speed, making it difficult to meet the real-time processing requirements of more than ten dense blocks in complex, multi-element images. Therefore, there is an urgent need to design a lightweight image processing architecture with high capacity and low complexity.

[0009] Second, the initial bounding box setting relies on human experience and suffers from poor coordinate prediction stability. Traditional region proposal networks use prior boxes with fixed aspect ratios, failing to fully consider the true distribution of target boxes in the dataset. The statistical characteristic that the length and width of rectangular boxes are greater than their height is not effectively utilized, resulting in low matching accuracy between anchor boxes and real objects. Simultaneously, the model directly predicts the absolute coordinate parameters of the bounding boxes, which is prone to iterative oscillations in the early stages of training due to its sensitivity to prediction errors. The unstable gradient update direction in the early stages also makes model convergence difficult. How to automatically discover the optimal anchor box dimensional distribution under unsupervised conditions and design a relative position constraint mechanism to limit the predicted values ​​within an effective range is a key technical bottleneck that this application urgently needs to address.

[0010] Third, the lack of a multi-block feature fusion mechanism leads to inaccurate local semantic representation. Existing image multi-annotation methods, such as DenseCaptioning, directly generate descriptions for independent blocks. When a block contains multiple sub-targets, it is easy to misclassify "coupleofpeople" as "twoboys," resulting in distorted local information representation. Unidentified blocks cannot generate text due to the lack of feature descriptions, causing loss of semantic information. The core challenge lies in how to establish spatial association rules between blocks, and to perform max pooling fusion on the feature vectors of adjacent blocks whose center distance does not exceed one-fifth of the original image. This allows the fused D-dimensional vector to retain more accurate semantic information than the vector of a single block, achieving semantic compensation of detected sub-regions for undetected parent regions, thereby improving the linguistic accuracy and content richness of the description.

[0011] Fourth, the dense description generation architecture lacks a hierarchical recursive mechanism. Traditional methods use a single-layer RNN to generate sentences, which cannot effectively handle long-range dependencies between multiple blocks. The generated description sentences are independent and limited in number, making it difficult to construct associative text segments that conform to human cognitive habits. This application aims to overcome the challenge of collaborative training of two-layer networks: the first layer, the Global Short-Term Memory (BSOM) layer, needs to dynamically determine the number of topic vectors S_k and the fusion relationship between sentences based on K fused block vectors, and control the generation step size through the state of the 512-dimensional hidden layer; the second layer, the sentence BRNN layer, needs to convert the P-dimensional topic vectors into variable-length descriptions and design a METEOR evaluation index based on WordNet to guide the learning of the word embedding matrix, ensuring that each word is aligned only with the image block with the highest matching score. How to achieve adaptive termination of text segment generation under the triple constraints of Stop probability, N_max, and S_max is the core innovative challenge that distinguishes this application from existing technologies. Summary of the Invention

[0012] This application presents a method for providing a clear, accurate, and intelligent AI representation of complex, multi-element images. For images containing more than ten local blocks, the aim is to generate dense, accurate, and human-friendly natural language descriptions. In the image processing stage, the Faster R-CNN model is improved by replacing VGG-16 with DenseNet as the feature extraction network to reduce computational complexity. Dimensional clustering is introduced to optimize anchor box sizes, and localization stability is enhanced by predicting coordinates relative to grid cells and constraining the output range. In the language generation stage, a hierarchical recurrent network architecture is designed, including a global long short-term memory layer and a sentence-level bidirectional RNN layer. The former fuses feature vectors from multiple blocks to determine the number of topics and their association with sentences, while the latter independently generates precise descriptive sentences for each topic. Furthermore, a block spatial proximity fusion mechanism enhances the semantic expression of complex regions and objects not directly detected, achieving a comprehensive, structured, and accessible intelligent representation of multiple image elements.

[0013] To achieve the above technical effects, the technical solution adopted in this application is as follows:

[0014] This paper presents a method for representing complex, multi-element images in a clear, accurate, and intelligent way using AI. It includes an optimized and enhanced object discrimination and localization framework and a text generation architecture based on a hierarchical recursive network. First, an image set is loaded, and DenseNet replaces VGG-16 as the CNN for feature extraction. Candidate boxes generated by the block proposal network are mapped onto the DenseNet feature map. An integrated dimensional feature map is generated through RoI pooling layers. K-means clustering is used in the block proposal enhancement network to perform cluster analysis on the initial block boxes, using the width and height of the cluster centers as the anchor box dimensions. The method predicts the coordinate position relative to the grid cells, and the predicted values ​​are constrained between 0 and 1 using a logistic activation function. Collaborative training is performed on the classification probability and bounding box regression for multi-task training. Classification and regression loss functions are constructed, and a robust cost function is used to optimize the bounding box regression. The block proposal network and detection network are trained iteratively alternately. The proposed blocks are converted into M D-dimensional feature information through a fully convolutional localization layer. Candidate blocks are obtained by regressing the translation-invariant anchor boxes, and the feature vectors are mapped back to the initial image plane. For each detection... The bounding box prediction confidence score and four scalars are used to extract fixed-dimensional feature representations through bilinear interpolation. M D-dimensional feature information is loaded into a hierarchical recurrent network (RNN), which consists of a global long short-term memory (LSM) layer and a sentence BRNN layer. The LSM layer determines the number of topic vectors and the sentence fusion relationship based on multiple sets of topic sentence vectors. The LSM layer is a single-layer hidden LSM network with a dimension of 512, generating hidden states at each time step. Binary classification states are generated through mapping and a logistic classifier, and topic vectors are generated through two fully connected layers. The sentence BRNN layer converts K fused block vectors into text descriptions, uses a bidirectional recurrent neural network to generate sentences, calculates word-to-image block matching scores, and only matches each word with the block with the highest matching score. Block-to-phrase matching metrics are calculated using hidden variables and Markov random fields. A P-dimensional vector is used as input to calculate the probability distribution of each sentence state and load it into the BRNN network to generate the probability distribution of each word. The generation process is controlled by adjusting the stopping probability, the maximum number of words per sentence, and the maximum number of sentences in the text segment, ultimately generating image description sentences.

[0015] Furthermore, the optimized and enhanced object discrimination and localization framework is as follows: First, the image set is loaded, and the image set is used for feature extraction through CNN. After the image features are extracted, the candidate boxes generated by the block proposal network are mapped onto the feature map generated by the backbone network. The RoI pooling layer enables each RoI to generate a feature map with integrated dimensions. Finally, the classification probability and bounding box regression are trained in a multi-task collaborative manner using detection classification probability and detection bounding box regression.

[0016] Furthermore, the object discrimination and localization network is based on DenseNet as a convolutional generative network, consisting of dense modules and transition layers. Different dense modules are connected through transition layers. The network has L layers, each implementing a nonlinear transformation H(·), where 1 indicates the layer number, and H(·) is a fusion function. A single dense module is represented as follows:

[0017] H = BN|ReLU|Pooling|Conv ​​Equation 1

[0018] Equation 1 represents H l (·) is a fusion function consisting of three consecutive operations: batch normalization, rectified linear unit, and convolution. Batch normalization alleviates gradient vanishing, rectified linear unit uses ReLU to improve training speed while reducing gradient vanishing, and the convolution kernel size is 3×3.

[0019] The transition layer is represented as follows:

[0020] Formula 2

[0021] Introduce direct connections from any layer to all subsequent layers, where This represents the concatenation of feature maps generated in layers 0, 1, ..., l-1. The dense connection method forms DenseNet, H... l The loading of (·) is a tensor composed of the first 0 to l-1 x feature maps. The number of dense modules in the network is 3, which are composed through transition layers. The number of dense modules in the network on the ImageNet dataset is 4, which are composed of parameter structures through transition layers.

[0022] Furthermore, the block proposal enhancement network maps the generated candidate boxes onto the feature maps generated by the DenseNet backbone network, and generates integrated feature maps for each RoI through the RoI pooling layer. On the extracted feature maps, all possible candidate boxes are discriminated, and convolutional and activation layers are added so that the result generated by the convolution part is a 51×39×256-dimensional feature representation.

[0023] After determining the candidate blocks, the next step is to predict the coordinates (x, y) of the block frame. The center point of the block frame is calculated as follows:

[0024] Formula 3

[0025] After initialization, the model requires multiple iterations to stably predict sensitive object offsets. Early iterations introduce prediction errors that cause model instability. The optimization strategy involves predicting the coordinates relative to the grid blocks and using a logistic activation function to constrain the true labels between 0 and 1, thus limiting the network's predictions. The network uses a 13×13 feature map, predicting 5 bounding boxes per ce11 area, with each bounding box predicting 5 coordinate values ​​(t). x , t w , t x , t h ,t0, if the distance of this ce11 from the top left corner of the image is (c x c y The length and width of the block frame corresponding to ce11 are (p) w p h The corresponding block frame calculation formula is:

[0026] Formula 4

[0027] Establish block proposals to enhance the network.

[0028] Furthermore, collaborative training discrimination: The cost function for an image is defined as follows:

[0029] Formula 5

[0030] i represents the index number of the anchor box in a mini-batch, p i The anchor box with index i represents the predicted probability of the target. If the anchor box is positive, The value equals 1 if the anchor frame is negative. equals 0, t i It is a vector containing the four coordinate parameters of the predicted detection box, which is the coordinate vector of the ground truth detection box corresponding to the positive anchor box, and the classification loss L. cls Represents the log loss for both target and non-target categories;

[0031] Formula 6

[0032] Regression loss L reg It is expressed as follows:

[0033] Formula 7

[0034] Where R represents a robust cost function, where R is the square of x when |x| < 1, and |x| - 0.5 in other cases. The block proposal network generation consists of two parts: the score of the candidate box and the coordinate information of the candidate box. The generation part is trained for both classification and regression simultaneously. First, it is processed by a 3×3 convolutional kernel, and then by a 1×1 convolutional kernel for dimensionality reduction. When the anchor box example k=5, each pixel generates 2k=10 candidate box scores and 4k=20 coordinate information. After Reshape, Softmax, and Reshape, it is loaded into the Proposal layer, which combines the corresponding anchor box and Stride for mapping, mapping the Box to the original image. Then it is loaded into the ROI pooling layer and combined with the Conv5 feature map, that is, marking the position of the candidate on the feature map. The model learning consists of four steps:

[0035] Step 1: Independently train the RPN network: Load the pre-trained model into the RPN network as network parameters;

[0036] Step 2: Independently train the Fast R-CNN network: First, load the candidate regions generated by the RPN network into the key detection network. Specifically, the RPN network generates a candidate box, obtains the corresponding region of the original image through the candidate box, and then performs multiple convolution and pooling operations on the obtained image. Then, it generates two modules through ROI pooling layers and fully convolutional layers: one is a softmax module for object classification, and the other is a box regression module.

[0037] Step 3: After the two networks are trained independently in Step 1 and Step 2, keep the parameters of the shared part of the two networks unchanged, and only update the parameters of the rest of the RPN network;

[0038] Step 4: Keep the parameters of the shared parts of the two networks unchanged, and only update the parameters of the rest of the Fast R-CNN network.

[0039] Furthermore, a hierarchical recursive network-based text generation architecture is proposed: First, an optimized and enhanced object discrimination and localization framework is used to obtain proposed blocks and their feature information. After passing through a fully convolutional localization layer, M D-dimensional feature information is obtained. The feature information is then loaded into a hierarchical recursive network. The first layer is a global long short-term memory layer, and the second layer is a sentence BRNN layer. The global long short-term memory layer determines the number of sentences generated and the fusion between sentences based on the loaded multiple sets of topic sentence vectors, thereby increasing the number of generated sentences. The sentence BRNN layer generates descriptive sentences corresponding to each image block. The independent sentence RNN layer enables the generated descriptive sentences to be more accurate and richer.

[0040] Further, the fully convolutional localization layer: Loading and generation of the fully convolutional localization layer: In the Faster R-CNN target measurement model based on DenseNet, feature extraction is performed through DenseNet. The expression for loading the image is defined as 3×H×W, and the expression generated after feature extraction is C×H0×W0. For the convenience of model integration, W0=W / 16, H0=H / 16, and C×H0×W0 is used as the loading tensor of the convolutional localization layer. The network generates B regions of interest, each corresponding to three tensors. These tensors are the coordinates of the region, the score of the region, and the feature description of the region. The coordinates of the region represent a B×4 matrix, the score of the region represents the confidence score of the generated region of length B, and the feature description of the region represents a feature tensor of size B×C×X×Y, where X and Y represent the grid size.

[0041] Candidate Block Determination: The fully convolutional localization layer obtains candidate blocks by regressing a series of translation-invariant anchor boxes. It maps each point in the loaded feature vector (H0×W0 grid) back to the initial H×W image plane, and considers k detection boxes of different sizes centered on this mapped point. The localization layer calculates a confidence score and predicts four scalars for each detection box. The result is obtained by feeding the feature map into a convolutional layer with 256 3×3 filters, a ReLU activation layer, and a convolutional layer with 1×1 1×1 filters. The final result is a 5k×H0×W0 tensor. In the candidate box regression part, the center coordinates and width and height of the anchor box, along with the four scalars predicted by the model, are used to calculate the center and width and height of the generated region. The generation parameter calculation formula is as follows. These parameters are used to obtain the regression from the anchor box to the proposed block. The specific parameter calculation for the generated block is as follows:

[0042] Formula 8

[0043] Candidate box selection: The candidate blocks are sampled twice, and the blocks with different IoU are classified. The predicted block with the largest IoU in each initial block is also considered positive. During the testing phase, the NMS algorithm is used to select the top 500 blocks of interest based on their confidence scores. The localization layer finally obtains two tensors: B×4 including coordinates and B including the score.

[0044] Eliminating redundant boxes and finding the best object detection location: There are three detection boxes. The confidence of the first detection box is 0.7, the confidence of the second detection box is 0.6, and the confidence of the third detection box is 0.7. Non-maximum suppression selects the detection box with the highest confidence among these three boxes, which is most likely to represent the target.

[0045] Step 1: Define three windows with confidence levels of [0.7, 0.6, 0.7];

[0046] Step 2: Select the detection box with the highest confidence as the target, and then compare the intersection areas between the remaining detection boxes and the target detection box;

[0047] Step 3: If the overlapping area is larger than the set threshold, remove the detection box from the remaining detection boxes so that the confidence of the detection box is the same as the confidence of the target detection box, thus suppressing the maximum overlapping area.

[0048] Step 4: Take the second high-confidence detection box as the target, repeat steps 2 and 3, and finally get the third of the three detection boxes defined at the beginning;

[0049] Integration of candidate blocks: The sampled block suggestions vary in size and proportion. Considering the need for connection with fully connected layers and RNN language types, feature representations of the integration dimension are extracted from these regions of different sizes. Bilinear interpolation is used to establish a sampling grid G ​​of size X×Y×2. Each element in V is associated with the real-valued coordinates in U. The new feature map is obtained by interpolating the features of U. The value of the corresponding element in V is calculated based on the coordinates in U. The sampling kernel k is represented as:

[0050] Formula 9

[0051] The gradient is transmitted back to the predicted region proposal coordinates. For all sampled regions, bilinear interpolation is performed to obtain a tensor of size B×C×X×Y, which is used to generate the localization layer.

[0052] Furthermore, the hierarchical recursive network language generation network detects a block of image in the fully convolutional localization layer. This block of image includes two or more key objects and consists of sub-region 1, sub-region 2, and the remaining image. For the same image block, a feature vector D fused from multiple feature vectors retains more accurate information than a single feature vector D , which has two implications:

[0053] Meaning 1: If an image block can be detected, and the block includes two or more sub-regions that are also detected, the feature vectors of these sub-regions are fused into a single vector. This vector is more accurate than the feature vector corresponding to the image block, and the proposed textual description of this type of image block is more accurate.

[0054] Meaning 2: If an image block is not recognized and has no corresponding text description, but the image block includes two or more detected sub-regions, then the feature vectors of these sub-regions are merged into a single vector, which is used as the feature vector corresponding to the image block to generate the text description of the image block. This process uses the fusion of sub-regions to represent the undetected image block and generate a new text description for the image block.

[0055] Topic Vectors: Between the Faster R-CNN and the global layer Long Short-Term Memory (LSTM), block pooling is first performed. Specifically, K M×D block vectors are generated from B×D and fused. Vector fusion is based on the positional relationships of the original images corresponding to the B block vectors. The condition for block fusion is that the distance between the block centers does not exceed one-fifth of the original image. Then, max pooling is performed on each fused block vector to reduce the amount of information and retain the most important information. The pooling operation is as follows:

[0056] Formula 10

[0057] W pool It is a P×D matrix, where P is the dimension of the final generated vector and D is the dimension of the feature vector of each block.

[0058] (1) Global Long Short Memory Layer: This is a single-layer hidden layer with a dimension of 512, where the ce1l state is set to 0. At each time step, a series of hidden states h1→h are generated. s Each represents a sentence, and each state has two functions: one is to use h i The first step is to use a mapping and logical classifier to generate binary classification states and probability distributions. The second step is to load two fully connected layers to generate topic vectors.

[0059] (2) Statement BRNN layer: The final part of the description language generation. It obtains vectors of K fusion blocks through global layer long short-term memory. The statement BRNN layer converts these fusion block vectors into text. The description statement is represented by an h-dimensional vector, h=P. The embedding process using BRNN is as follows: Let t represent the column vector of the word at position t in the vocabulary. Suppose the word at position t is "boy," and this word appears in the vocabulary vector. If the value at the corresponding position is 1 and the position is the 6th, then... =[0, 0, 0, 0, 0, 1, 0, 0,…, 0], W W The weights of the word vector generation model are initialized to 300 dimensions, representing a word embedding matrix. The model needs to learn and train the following parameters, W. e W f W b W d and bias be b f b b b d The hidden layer size is 300 to 600 dimensions, and the activation function is f: x → max(0, x);

[0060] Formula 11

[0061] Image patch and word matching: The mapping relationship between each word and an image patch indicates a measure of how well a word fits the various bounding boxes. Each word is matched only with the image patch with the highest matching score. The number of words in each phrase within a patch is equal. The cost function is as follows:

[0062] Formula 12

[0063] Image region and phrase matching: The sentence contains N words, and the image contains M detection boxes. A hidden variable a is used. j Using Markov Random Fields (MRF), a matching metric for image blocks and phrases is calculated:

[0064] Formula 13

[0065] Description of generating the entire image: Taking the P-dimensional vector from global short-term memory as input, calculate according to the following formula:

[0066] Formula 14

[0067] Ultimately, a network is generated using a hierarchical recursive network language.

[0068] Furthermore, the hierarchical recursive network model learns as follows: each block fusion vector comprises multiple block vectors, each block vector corresponds to a statement, and one block fusion vector corresponds to multiple statements, ultimately generating a descriptive statement. The process of generating a descriptive statement from a block fusion vector is considered as generating a text segment, which actually contains only one sentence. Each block fusion is viewed as a text segment, assuming a text segment contains S statements, and each statement has N... i One word, y ij It is the j-th word of the i-th statement, in the pooled vector v p The Global Short-Term Memory (GSM) is expanded into S steps, the state probability distribution of each statement is calculated, and then loaded into the BRNN network, with each step expanded into N steps. i A probability distribution p is generated with a step size of 1. ij Regarding the j-th word of the i-th sentence, referring to the image to generate a text segment description model, the definition of the cost function corresponding to the fusion vector is as follows:

[0069] Formula 15

[0070] Three values ​​are used to control the number of products generated in each step, T. stop N represents the stopping probability. max S represents the maximum number of words generated by each statement. max This represents the maximum number of sentences included in the text segment. An image has multiple fusion vectors, and the number of fusion vectors in an image is K. The cost function for an image is defined as shown in Equation 16:

[0071] Formula 16

[0072] Where S k S represents the number of individual vectors included in the k-th fusion vector, and the step size of the long short memory model expansion is S. k ;

[0073] Based on a hierarchical recursive network text generation architecture, an improved Faster R-CNN object detection model is used to obtain proposed blocks and their feature information. After passing through a fully convolutional localization layer, these feature information is transformed into M D-dimensional feature information. This feature information is then loaded into a hierarchical recursive network. The first layer is a global long short-term memory layer, and the second layer is a sentence BRNN layer. The global long short-term memory layer determines the number of sentences generated and the fusion between sentences based on the loaded multiple sets of topic sentence vectors, thereby increasing the number of generated sentences. The sentence BRNN layer generates descriptive sentences corresponding to each image block. The independent sentence BRNN layer enables the generated descriptive sentences to be more accurate and richer.

[0074] Compared with existing technologies, the innovations and advantages of this application are as follows:

[0075] (1) Constructing a lightweight feature extraction architecture based on DenseNet fundamentally reduces the computational complexity of the model. This application innovatively replaces VGG-16 in Faster R-CNN with DenseNet as the backbone network of the CNN. Through direct inter-layer connections of dense modules and dimensionality compression of transition layers, the number of parameters and floating-point operations are significantly reduced while retaining the feature reuse capability, solving the problem of excessive computational resource consumption caused by the reliance on excessively deep networks in traditional models. Compared with the localization error defects of YOLO and the insufficient small object detection of SSD, the present solution is verified on the CIFAR and SVHN datasets. It shows that when the depth is 190 and the growth rate is 40, it can surpass the accuracy of existing models, achieving a balance between large capacity and low complexity, and significantly improving the real-time processing efficiency of dense blocks of complex multi-element images.

[0076] (2) A data-driven anchor box clustering and relative coordinate constraint mechanism is proposed to enhance the accuracy of detection box localization and training stability. This application breaks through the limitations of traditional manual setting of fixed aspect ratio prior boxes. K-means clustering is used to perform unsupervised learning on the real block boxes of ImageNet and COCO datasets, automatically discovering the statistical law that "the length of the rectangle is greater than its width". The cluster center dimension is used as the anchor box parameter to improve the IOU matching degree between the anchor box and the real target. At the same time, the bounding box coordinate prediction is innovatively changed to the position prediction relative to the grid cell. The logistic activation function is used to constrain the prediction value of the 13×13 grid between 0 and 1, avoiding model oscillation caused by sensitive offset prediction in the early iteration, making the training process more stable and convergent. The block box selection accuracy is significantly better than the traditional RPN method.

[0077] (3) Design a spatial association strategy for block fusion vectors to achieve accurate expression of local semantic information and compensation for undetected regions. This application pioneers a block vector fusion mechanism based on spatial proximity. By determining that the distance between the block centers does not exceed one-fifth of the original image, the features of the sub-blocks in the parent region containing multiple sub-targets are fused by max pooling to generate a D-dimensional fusion vector that is more accurate than the information of a single block vector. This mechanism effectively solves the problem of local information distortion caused by methods such as Dense Captioning misclassifying "couple of people" as "two boys". It can also generate compensating descriptions such as "a woman and a boy playing a game" by fusing the features of detected sub-regions to the unidentified parent region, greatly improving the accuracy and richness of semantic descriptions in complex scenes.

[0078] (4) A two-layer recursive architecture is established, which coordinates a global long short-term memory layer and a sentence BRNN layer to achieve a dual improvement in the quantity and quality of description sentences. This application breaks through the limitation of single-layer RNN generating independent sentences and innovatively designs a global LSTM with 512 hidden layers as the first layer to dynamically determine the number of topic vectors corresponding to K fusion block vectors and the fusion relationship between sentences, thereby increasing the upper limit of the number of generated sentences; the second layer, an independent sentence BRNN layer, generates variable-length descriptions for each P-dimensional topic vector, and ensures that words are accurately aligned with the image blocks with the highest matching degree through word embedding matrix and bidirectional recurrent structure. Compared with methods such as Full Image RNN and Region RNN, the validation of this architecture on the Visual Genome dataset shows that it can generate more block descriptions and has a better METEOR evaluation score, achieving an effective balance between the quantity of descriptions and the quality of generation. Attached Figure Description

[0079] Figure 1 This is a schematic diagram illustrating the calculation of the block box corresponding to the predicted coordinate values ​​of the detection box. Detailed Implementation

[0080] The technical solution of the method for expressing complex, multi-element images in a simple, accurate, and intelligent way, provided in this application, will be further described below with reference to the accompanying drawings, so that those skilled in the art can better understand this application and implement it.

[0081] Image information is one of the most beautiful and richest pieces of information that computers acquire during their perception of the outside world. Increasingly intelligent computers are not only seeing the world with their eyes, but are also beginning to understand it with their hearts. Computers' ability to understand images has evolved from simple image classification to scene analysis, from traditional image processing algorithms to the currently popular deep learning models. Their ability to learn and understand images has reached the level of a child's image cognition. Image annotation is one of the more representative research areas in image understanding, characterized by its ability to generate natural language descriptions of image content based on the image's content. Image annotation tasks can be divided into single-annotation tasks, multi-annotation tasks, and image text segment description tasks. Among these, multi-annotation tasks are the most crucial, as they can replace single-annotation tasks and serve as the basis for image text segment descriptions. However, current multi-annotation task models have high algorithmic complexity and computational cost. To improve the overall performance of the model while reducing algorithmic complexity and computational cost, and generating more descriptive language, this application proposes new optimization strategies for object detection models and text generation architectures. The research content and contributions of this application mainly include the following two aspects:

[0082] (1) Improved Faster R-CNN model as an image processing model: After loading images, DenseNet was used instead of VGG-16 as the CNN for feature extraction, reducing the complexity of the algorithm. Simultaneously, to select representative detection boxes, dimensional clustering was added to the detection box part. Clustering methods were used to cluster the block boxes in the dataset to discover the calculation patterns of the initial block boxes. Finally, to improve the model's stability, the method of predicting the correlation coefficient of the bounding boxes was replaced with predicting the coordinate position relative to the grid cells, and the predicted values ​​of the grid cells were limited to between 0 and 1.

[0083] (2) A hierarchical recurrent network-based text generation architecture is proposed: A two-layer RNN network is designed as the text generation architecture. The first layer is a global long short-term memory layer, and the second layer is a sentence BRNN layer. The global long short-term memory layer determines the final generated topic vector based on the multiple fusion vectors loaded, thereby increasing the number of generated sentences. The sentence BRNN layer generates the description sentence corresponding to each topic vector. The independent sentence BRNN layer can make the generated description sentences more accurate and richer.

[0084] The two research sections of this application are progressively structured and interconnected, together forming a complete architecture for generating natural descriptive text for complex, multi-element images. Comparison with multiple models on standard datasets demonstrates that the proposed model has lower algorithmic complexity, faster computation speed, and generates more accurate and richer descriptive language.

[0085] I. Optimized and Enhanced Object Discrimination and Localization Framework

[0086] First, the image set is loaded, and features are extracted from the image set using a CNN. After the image features are extracted, the candidate boxes generated by the block proposal network are mapped onto the feature maps generated by the backbone network. The RoI pooling layer enables each RoI to generate a feature map with integrated dimensions. Finally, the classification probability and bounding box regression are trained in a multi-task collaborative manner using detection classification probability and detection bounding box regression.

[0087] (a) Object discrimination and localization network

[0088] Based on DenseNet as a convolutional generative network, the network consists of dense modules and transition layers. Different dense modules are connected through transition layers. The network has L layers, each implementing a nonlinear transformation H(·), where 1 indicates the layer number, and H(·) is a fusion function. A single dense module is represented as follows:

[0089] H = BN|ReLU|Pooling|Conv ​​Equation 1

[0090] Equation 1 represents H l (·) is a fusion function consisting of three consecutive operations: batch normalization, rectified linear unit, and convolution. Batch normalization alleviates gradient vanishing, rectified linear unit uses ReLU to improve training speed while reducing gradient vanishing, and the convolution kernel size is 3×3.

[0091] The transition layer is represented as follows:

[0092] Formula 2

[0093] Introduce direct connections from any layer to all subsequent layers, where This represents the concatenation of feature maps generated in layers 0, 1, ..., l-1. The dense connection method forms DenseNet, H... l The loading of (·) is a tensor composed of the first 0 to l-1 x feature maps. The number of dense modules in the network is 3, which are composed of transition layers. The number of dense modules in the network on the ImageNet dataset is 4, which are composed of parameter structures through transition layers.

[0094] The network was trained using stochastic gradient descent (SGD) on CIFAR, SVHN, and ImageNet, with different network depths and growth rates. On the CIFAR dataset, the network outperformed other models when the depth was 190 and the growth rate was 40. On the SVHN dataset, the network outperformed other models when the depth was 100 and the growth rate was 24.

[0095] (ii) Block proposals enhance the network

[0096] The generated candidate boxes are mapped onto the feature maps generated by the DenseNet backbone network, and each RoI is generated into a feature map with integrated dimensions through the RoI pooling layer. On the extracted feature maps, all possible candidate boxes are discriminated, and convolutional and activation layers are added so that the result generated by the convolution part is a 51×39×256-dimensional feature representation.

[0097] K-means clustering optimization strategy: To avoid the negative effects of manual setting, the initial bounding boxes in the dataset are clustered. The computational patterns of the initial bounding boxes are discovered through a large amount of data. The number of clusters, k, is used as the anchor box, and the width and height dimensions of the k cluster center bounding boxes are used as the dimensions of the candidate boxes. The clustering results are related to IOU. The distance metric for clustering is not a simple Euclidean distance. Calculations on benchmark datasets ImageNet and COCO show that a large amount of image data indicates that the bounding box length is often greater than the width. This conclusion is used for further comparison, where a higher IOU indicates better results.

[0098] After determining the candidate blocks, the next step is to predict the coordinates (x, y) of the block frame. The center point of the block frame is calculated as follows:

[0099] Formula 3

[0100] After model initialization, multiple iterations are required to stably predict sensitive object offsets. Errors in early iterations cause model instability. The optimization strategy involves predicting the coordinates relative to the grid blocks and using a logistic activation function to constrain the true labels between 0 and 1, thus limiting the network's predictions. The network predicts 5 bounding boxes on each ce11 feature map of size 13×13. Figure 1 Each detection box predicts 5 coordinate values, namely t x , t w , t x , t h ,t0, if the distance of this ce11 from the top left corner of the image is (c x c y The length and width of the block frame corresponding to ce11 are (p) w p hThe corresponding block frame calculation formula is:

[0101] Formula 4

[0102] Establish block proposals to enhance the network.

[0103] (III) Collaborative Training Judgment

[0104] The cost function for an image is defined as follows:

[0105] Formula 5

[0106] i represents the index number of the anchor box in a mini-batch, p i The anchor box with index i represents the predicted probability of the target. If the anchor box is positive, The value equals 1 if the anchor frame is negative. equals 0, t i It is a vector containing the four coordinate parameters of the predicted detection box, which is the coordinate vector of the ground truth detection box corresponding to the positive anchor box, and the classification loss L. cls Represents the log loss for both target and non-target categories;

[0107] Formula 6

[0108] Regression loss L reg It is expressed as follows:

[0109] Formula 7

[0110] Where R represents a robust cost function, where R is the square of x when |x| < 1, and |x| - 0.5 in other cases. The block proposal network generation consists of two parts: the score of the candidate box and the coordinate information of the candidate box. The generation part is trained for both classification and regression simultaneously. First, it is processed by a 3×3 convolutional kernel, and then by a 1×1 convolutional kernel for dimensionality reduction. When the anchor box example k=5, each pixel generates 2k=10 candidate box scores and 4k=20 coordinate information. After Reshape, Softmax, and Reshape, it is loaded into the Proposal layer, which combines the corresponding anchor box and Stride for mapping, mapping the Box to the original image. Then it is loaded into the ROI pooling layer and combined with the Conv5 feature map, that is, marking the position of the candidate on the feature map. The model learning consists of four steps:

[0111] Step 1: Independently train the RPN network: Load the pre-trained model into the RPN network as network parameters;

[0112] Step 2: Independently train the Fast R-CNN network: First, load the candidate regions generated by the RPN network into the key detection network. Specifically, the RPN network generates a candidate box, obtains the corresponding region of the original image through the candidate box, and then performs multiple convolution and pooling operations on the obtained image. Then, it generates two modules through ROI pooling layers and fully convolutional layers: one is a softmax module for object classification, and the other is a box regression module.

[0113] Step 3: After the two networks are trained independently in Step 1 and Step 2, keep the parameters of the shared part of the two networks unchanged, and only update the parameters of the rest of the RPN network;

[0114] Step 4: Keep the parameters of the shared parts of the two networks unchanged, and only update the parameters of the rest of the Fast R-CNN network.

[0115] The optimized and enhanced object discrimination and localization framework uses a CNN based on DenseNets to enhance feature extraction capabilities while significantly reducing model parameters and computational cost. Two improvements are made to the block proposal network: using clustering methods to obtain the number and shape of detection boxes at anchor box centers, and constraining the range of position prediction for the coordinates of detection boxes in the grid. These two improvements reduce model parameters and make the model easier to learn and more stable.

[0116] II. Text Generation Architecture Based on Hierarchical Recursive Network

[0117] First, an optimized and enhanced object discrimination localization framework is used to obtain proposed blocks and their feature information. After passing through a fully convolutional localization layer, M D-dimensional feature information is obtained. The feature information is loaded into a hierarchical recurrent network. The first layer is a global long short memory layer, and the second layer is a statement BRNN layer. The global long short memory layer determines the number of statements generated and the fusion between statements based on the loaded multiple sets of topic statement vectors, thereby increasing the number of generated statements. The statement BRNN layer generates a description statement corresponding to each image block. The independent statement RNN layer can make the generated description statement more accurate and richer.

[0118] (a) Fully convolutional localization layer

[0119] Loading and generation of the fully convolutional localization layer: In the Faster R-CNN object measurement model based on DenseNet, feature extraction is performed through DenseNet. The expression for loading the image is defined as 3×H×W, and the expression generated after feature extraction is C×H0×W0. For the convenience of model integration, W0=W / 16, H0=H / 16, and C×H0×W0 is used as the loading tensor of the convolutional localization layer. The network generates B regions of interest, each corresponding to three tensors. These tensors are the coordinates of the region, the score of the region, and the feature description of the region. The coordinates of the region represent a B×4 matrix, the score of the region represents the confidence score of the generated region of length B, and the feature description of the region represents a feature tensor of size B×C×X×Y, where X and Y represent the grid size.

[0120] Candidate Block Determination: The fully convolutional localization layer obtains candidate blocks by regressing a series of translation-invariant anchor boxes. It maps each point in the loaded feature vector (H0×W0 grid) back to the initial H×W image plane, and considers k detection boxes of different sizes centered on this mapped point. The localization layer calculates a confidence score and predicts four scalars for each detection box. The result is obtained by feeding the feature map into a convolutional layer with 256 3×3 filters, a ReLU activation layer, and a convolutional layer with 1×1 1×1 filters. The final result is a 5k×H0×W0 tensor. In the candidate box regression part, the center coordinates and width and height of the anchor box, along with the four scalars predicted by the model, are used to calculate the center and width and height of the generated region. The generation parameter calculation formula is as follows. These parameters are used to obtain the regression from the anchor box to the proposed block. The specific parameter calculation for the generated block is as follows:

[0121] Formula 8

[0122] Candidate box selection: The candidate blocks are sampled twice, and the blocks with different IoU are classified. The predicted block with the largest IoU in each initial block is also considered positive. During the testing phase, the NMS algorithm is used to select the top 500 blocks of interest based on their confidence scores. The localization layer finally obtains two tensors: B×4 including coordinates and B including the score.

[0123] Eliminating redundant boxes and finding the best object detection location: There are three detection boxes. The confidence of the first detection box is 0.7, the confidence of the second detection box is 0.6, and the confidence of the third detection box is 0.7. Non-maximum suppression selects the detection box with the highest confidence among these three boxes, which is most likely to represent the target.

[0124] Step 1: Define three windows with confidence levels of [0.7, 0.6, 0.7];

[0125] Step 2: Select the detection box with the highest confidence as the target, and then compare the intersection areas between the remaining detection boxes and the target detection box;

[0126] Step 3: If the overlapping area is larger than the set threshold, remove the detection box from the remaining detection boxes so that the confidence of the detection box is the same as the confidence of the target detection box, thus suppressing the maximum overlapping area.

[0127] Step 4: Take the second high-confidence detection box as the target, repeat steps 2 and 3, and finally get the third of the three detection boxes defined at the beginning;

[0128] Integration of candidate blocks: The sampled block suggestions vary in size and proportion. Considering the need for connection with fully connected layers and RNN language types, feature representations of the integration dimension are extracted from these regions of different sizes. Bilinear interpolation is used to establish a sampling grid G ​​of size X×Y×2. Each element in V is associated with the real-valued coordinates in U. The new feature map is obtained by interpolating the features of U. The value of the corresponding element in V is calculated based on the coordinates in U. The sampling kernel k is represented as:

[0129] Formula 9

[0130] The gradient is transmitted back to the predicted region proposal coordinates. For all sampled regions, bilinear interpolation is performed to obtain a tensor of size B×C×X×Y, which is used to generate the localization layer.

[0131] Integration of generated feature information: A discriminant network is added after the fully convolutional localization layer. The features of each block from the localization layer are converted into a vector and passed to two fully connected layers. Both layers use ReLU units and Dropout. Each region generates a code with dimension D=4096, which includes the visual features of this region. The codes of all positive sample regions are collected into a matrix of size B×D and then passed to the RNN language model. The recognition network further refines the confidence and location of each candidate block, which are derived from the linear transformation of the D-dimensional code corresponding to each block. The final detection box is then calculated.

[0132] (II) Hierarchical Recursive Network Language Generation Network

[0133] In a fully convolutional localization layer, a block of image is detected. This block contains two or more key objects and includes sub-region 1, sub-region 2, and the remaining image. For the same image block, a feature vector D fused from multiple feature vectors retains more accurate information than a single feature vector D, which has two implications:

[0134] Meaning 1: If an image block can be detected, and the block includes two or more sub-regions that are also detected, the feature vectors of these sub-regions are fused into a single vector. This vector is more accurate than the feature vector corresponding to the image block, and the proposed textual description of this type of image block is more accurate.

[0135] Meaning 2: If an image block is not recognized and has no corresponding text description, but the image block includes two or more detected sub-regions, then the feature vectors of these sub-regions are merged into a single vector, which is used as the feature vector corresponding to the image block to generate the text description of the image block. This process uses the fusion of sub-regions to represent the undetected image block and generate a new text description for the image block.

[0136] Design process: The information B×D generated by the fully convolutional localization layer is used as the loading of the hierarchical recursive network. The hierarchical recursive network is divided into a global long short memory layer and a statement BRNN layer. The global long short memory layer is used to generate description statements corresponding to each topic vector based on the loaded multiple sets of topic statement vectors.

[0137] Topic Vectors: Between the Faster R-CNN and the global layer Long Short-Term Memory (LSTM), block pooling is first performed. Specifically, K M×D block vectors are generated from B×D and fused. Vector fusion is based on the positional relationships of the original images corresponding to the B block vectors. The condition for block fusion is that the distance between the block centers does not exceed one-fifth of the original image. Then, max pooling is performed on each fused block vector to reduce the amount of information and retain the most important information. The pooling operation is as follows:

[0138] Formula 10

[0139] W pool It is a P×D matrix, where P is the dimension of the final generated vector and D is the dimension of the feature vector of each block.

[0140] (1) Global Long Short Memory Layer: This is a single-layer hidden layer with a dimension of 512, where the ce1l state is set to 0. At each time step, a series of hidden states h1→h are generated. s Each represents a sentence, and each state has two functions: one is to use h iThe first step is to use a mapping and logical classifier to generate binary classification states and probability distributions. The second step is to load two fully connected layers to generate topic vectors.

[0141] (2) Statement BRNN layer: The final part of the description language generation. It obtains vectors of K fusion blocks through global layer long short-term memory. The statement BRNN layer converts these fusion block vectors into text. The description statement is represented by an h-dimensional vector, h=P. The embedding process using BRNN is as follows: Let t represent the column vector of the word at position t in the vocabulary. Suppose the word at position t is "boy," and this word appears in the vocabulary vector. If the value at the corresponding position is 1 and the position is the 6th, then... =[0, 0, 0, 0, 0, 1, 0, 0,…, 0], W W The weights of the word vector generation model are initialized to 300 dimensions, representing a word embedding matrix. The model needs to learn and train the following parameters, W. e W f W b W d and bias b e b f b b b d The hidden layer size is 300 to 600 dimensions, and the activation function is f: x → max(0, x);

[0142] Formula 11

[0143] Image patch and word matching: The mapping relationship between each word and an image patch indicates a measure of how well a word fits the various bounding boxes. Each word is matched only with the image patch with the highest matching score. The number of words in each phrase within a patch is equal. The cost function is as follows:

[0144] Formula 12

[0145] Image region and phrase matching: The sentence contains N words, and the image contains M detection boxes. A hidden variable a is used. j Using Markov Random Fields (MRF), a matching metric for image blocks and phrases is calculated:

[0146] Formula 13

[0147] Description of generating the entire image: Taking the P-dimensional vector from global short-term memory as input, calculate according to the following formula:

[0148] Formula 14

[0149] Ultimately, a network is generated using a hierarchical recursive network language.

[0150] (III) Learning of hierarchical recursive network models

[0151] Each block fusion vector comprises multiple block vectors, each block vector corresponding to a statement. One block fusion vector corresponds to multiple statements, ultimately generating a descriptive statement. The process of generating a descriptive statement from a block fusion vector is considered as generating a text segment, which actually contains only one sentence. Each block fusion is viewed as a text segment, assuming a text segment contains S statements, and each statement has N... i One word, y ij It is the j-th word of the i-th statement, in the pooled vector v p The Global Short-Term Memory (GSM) is expanded into S steps, the state probability distribution of each statement is calculated, and then loaded into the BRNN network, with each step expanded into N steps. i A probability distribution p is generated with a step size of 1. ij Regarding the j-th word of the i-th sentence, referring to the image to generate a text segment description model, the definition of the cost function corresponding to the fusion vector is as follows:

[0152] Formula 15

[0153] Three values ​​are used to control the number of products generated in each step, T. stop N represents the stopping probability. max S represents the maximum number of words generated by each statement. max This represents the maximum number of sentences included in the text segment. An image has multiple fusion vectors, and the number of fusion vectors in an image is K. The cost function for an image is defined as shown in Equation 16:

[0154] Formula 16

[0155] Where S k S represents the number of individual vectors included in the k-th fusion vector, and the step size of the long short memory model expansion is S. k ;

[0156] Based on a hierarchical recursive network text generation architecture, an improved Faster R-CNN object detection model is used to obtain proposed blocks and their feature information. After passing through a fully convolutional localization layer, these feature information is transformed into M D-dimensional feature information. This feature information is then loaded into a hierarchical recursive network. The first layer is a global long short-term memory layer, and the second layer is a sentence BRNN layer. The global long short-term memory layer determines the number of sentences generated and the fusion between sentences based on the loaded multiple sets of topic sentence vectors, thereby increasing the number of generated sentences. The sentence BRNN layer generates descriptive sentences corresponding to each image block. The independent sentence BRNN layer enables the generated descriptive sentences to be more accurate and richer.

[0157] III. Experimental Results and Analysis

[0158] Experiments were conducted using the Visual Genome dataset with 5000 images for comparison. Four control models were selected: FullimageRNN and RegionRNN proposed by Karpath et al., FCLN onEB proposed by Farhadie et al., and FCLN for Dense Captioning proposed by Johnsone et al. An experimental comparison figure is provided for the method in this application and the FCLN for Dense Captioning method.

[0159] The generated image patch descriptions are ten in number, and all possible image patch proposals are given. The purpose is to compare the number of patch descriptions generated by the method in this application and the comparative experiments. In the text generation architecture, this application adds a fusion vector generation process and a global long short memory layer processing process compared to the FCLN for Dense Captioning method. The final generation part uses a similar RNN model, resulting in very close generation results. Compared to the FCLN for Dense Captioning method, this application achieves better experimental results in three areas:

[0160] 1) The description generated by the block fusion vector is more accurate and richer than that generated by a single block vector: For the text description generated for block 1 in the original image, the description generated by the FCLN for Dense Captioning method is "two boys playing with a game," while the description generated by the method in this application is "a couple of people standing next to each other." Relatively speaking, the description generated by this application is more accurate and richer. The reason is that FCLN for Dense Captioning directly identifies the two people as "boys" in the proposed text description for this block, which is inconsistent with the actual situation and represents a local information error. In contrast, this application fuses the feature vectors of the blocks corresponding to the two people and generates a description after iteration, resulting in a more accurate representation of local information.

[0161] 2) The increased dimensionality of the anchor box clustering and the method of predicting coordinate positions relative to the grid cell make the selected block boxes more accurate: the three image blocks correspond to blocks 1, 7, and 8 in the experimental image. The left side shows the block boxes obtained by the FCLN for Dense Captioning method, and the right side shows the block boxes obtained by the method in this application. The block boxes obtained by the method in this application are relatively more accurate than those obtained by the FCLN for Dense Captioning method.

[0162] 3) More image block descriptions can be generated by constructing block fusion vectors and adding a global long short memory layer: The method of this application generates a description statement for block 16, "a woman and a boy playing a game offrisbee". The principle is to fuse the feature vectors of the blocks corresponding to the two characters to generate a description statement.

[0163] To further compare the model performance, the mean Average Precision metric was used to match the selected image blocks and the description language, and the METEOR method was used to generate the description language. The specific process is as follows:

[0164] 1) Language: The METEOR method is used, where m represents a pre-given set of calibrations. This calibration is based on the WordNet thesaurus and is obtained by minimizing the consecutive ordered blocks ch in the corresponding sentences. Pen represents the penalty coefficient. The METEOR method calculates the precision and recall based on the entire corpus and finally obtains the test score.

[0165] 2) Dense captioning: Mean Average Precision (mAP) is used. First, the confidence scores of N test samples are obtained using the pre-trained model. Each test sample is labeled with its ID, score, and ground truth label (0 or 1). Then, the samples are sorted by confidence score. Assuming there are M positive examples among the N test samples, M recall values ​​are calculated. For each recall value, the maximum precision value is calculated. Finally, the average of all precision values ​​is taken to obtain the mAP value.

[0166] The evaluation results for the Language and Dense captioning parts are obtained using the two methods described above. Here, mAP represents the mean Average Precision index, EB represents Edge Boxes, RPN represents Region Proposal Network, and GT represents ground truth boxes. For both parts, higher values ​​for EB, RPN, and GT are better. The method in this application is slightly superior to the FCLN for Dens eCaption method in block selection and the generated description language, and the model's computation speed is also faster than existing models.

Claims

1. A method for expressing complex, multi-element images in a simple, accurate, and intelligent AI manner, characterized by: This includes an optimized and enhanced object discrimination and localization framework and a text generation architecture based on a hierarchical recurrent network. First, an image set is loaded, and DenseNet replaces VGG-16 as the CNN for feature extraction. Candidate boxes generated by the block proposal network are mapped onto the DenseNet feature map. An integrated dimensional feature map is generated through RoI pooling layers. In the block proposal enhancement network, K-means clustering is used to cluster the initial block boxes, using the cluster center width and height as the anchor box dimensions. The predicted position relative to the grid cell coordinates is used, and the predicted values ​​are constrained between 0 and 1 using a logistic activation function. Collaborative training is performed on the discrimination probability and bounding box regression for multi-task training, constructing classification and regression loss functions. A robust cost function is used to optimize the bounding box regression, enabling iterative training of the block proposal network and the detection network. The proposed blocks are converted into M D-dimensional feature information through a fully convolutional localization layer, and the translation is then processed. Invariant anchor box regression yields candidate blocks, mapping feature vectors back to the initial image plane. For each detection box, a confidence score and four scalars are predicted, and fixed-dimensional feature representations are extracted using bilinear interpolation. M D-dimensional feature information is loaded into a hierarchical recurrent network (RNN). This RNN consists of a global long short-term memory (LSM) layer and a sentence BRNN layer, forming a two-layer RNN structure. The LSM layer determines the number of topic vectors and the fusion relationship of sentences based on multiple sets of topic sentence vectors. The LSM layer is a single-layer hidden LSM network with a dimension of 512, generating hidden states at each time step. Binary classification states are generated through mapping and a logistic classifier, and topic vectors are generated through two fully connected layers. The sentence BRNN layer transforms the K fused block vectors into text descriptions, generates sentences using a bidirectional recurrent neural network, calculates word-to-image block matching scores, and matches each word only with the block with the highest matching score. A block-to-phrase matching metric is calculated using hidden variables and a Markov random field. Using a P-dimensional vector as input, the probability distribution of each sentence state is calculated and loaded into a BRNN network to generate the probability distribution of each word. The generation process is controlled by adjusting the stopping probability, the maximum number of words per sentence, and the maximum number of sentences in the text segment, thus generating image description sentences.

2. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that: The optimized and enhanced object discrimination and localization framework is as follows: First, the image set is loaded, and the image set is used for feature extraction through CNN. After the image features are extracted, the candidate boxes generated by the block proposal network are mapped onto the feature map generated by the backbone network. The RoI pooling layer enables each RoI to generate a feature map with integrated dimensions. Finally, the classification probability and bounding box regression are trained in a multi-task collaborative manner using detection classification probability and detection bounding box regression.

3. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that: Object discrimination and localization network: Based on DenseNet as a convolutional generative network, the network consists of dense modules and transition layers. Different dense modules are connected through transition layers. The network has L layers, each implementing a nonlinear transformation H(·), where 1 indicates the layer number, and H(·) is a fusion function. A single dense module is represented as follows: H = BN|ReLU|Pooling|Conv ​​Equation 1 Equation 1 represents H l (·) is a fusion function consisting of three consecutive operations: batch normalization, rectified linear unit, and convolution. Batch normalization alleviates gradient vanishing, rectified linear unit uses ReLU to improve training speed while reducing gradient vanishing, and the convolution kernel size is 3×3. The transition layer is represented as follows: Formula 2 Introduce direct connections from any layer to all subsequent layers, where This represents the concatenation of feature maps generated in layers 0, 1, ..., l-1. The dense connection method forms DenseNet, H... l The loading of (·) is a tensor composed of the first 0 to l-1 x feature maps. The number of dense modules in the network is 3, which are composed through transition layers. The number of dense modules in the network on the ImageNet dataset is 4, which are composed of parameter structures through transition layers.

4. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that: Block Proposal Enhancement Network: The generated candidate boxes are mapped onto the feature maps generated by the DenseNet backbone network, and each RoI is generated into a feature map with integrated dimensions through the RoI pooling layer. On the extracted feature maps, all possible candidate boxes are discriminated, and convolutional and activation layers are added so that the result generated by the convolution part is a 51×39×256-dimensional feature representation. After determining the candidate blocks, the next step is to predict the coordinates (x, y) of the block frame. The center point of the block frame is calculated as follows: Formula 3 After initialization, the model requires multiple iterations to stably predict sensitive object offsets. Early iterations introduce prediction errors that cause model instability. The optimization strategy involves predicting the coordinates relative to the grid blocks and using a logistic activation function to constrain the true labels between 0 and 1, thus limiting the network's predictions. The network uses a 13×13 feature map, predicting 5 bounding boxes per ce11 area, with each bounding box predicting 5 coordinate values ​​(t). x , t w , t x , t h ,t0, if the distance of this ce11 from the top left corner of the image is (c x c y The length and width of the block frame corresponding to ce11 are (p) w p h The corresponding block frame calculation formula is: Formula 4 Establish block proposals to enhance the network.

5. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that, Collaborative training for discrimination: The cost function for an image is defined as follows: Formula 5 i represents the index number of the anchor box in a mini-batch, p i The anchor box with index i represents the predicted probability of the target. If the anchor box is positive, The value equals 1 if the anchor frame is negative. equals 0, t i It is a vector containing the four coordinate parameters of the predicted detection box, which is the coordinate vector of the ground truth detection box corresponding to the positive anchor box, and the classification loss L. cls Represents the log loss for both target and non-target categories; Formula 6 Regression loss L reg It is expressed as follows: Formula 7 Where R represents a robust cost function, where R is the square of x when |x| < 1, and |x| - 0.5 in other cases. The block proposal network generation consists of two parts: the score of the candidate box and the coordinate information of the candidate box. The generation part is trained for both classification and regression simultaneously. First, it is processed by a 3×3 convolutional kernel, and then by a 1×1 convolutional kernel for dimensionality reduction. When the anchor box example k=5, each pixel generates 2k=10 candidate box scores and 4k=20 coordinate information. After Reshape, Softmax, and Reshape, it is loaded into the Proposal layer, which combines the corresponding anchor box and Stride for mapping, mapping the Box to the original image. Then it is loaded into the ROI pooling layer and combined with the Conv5 feature map, that is, marking the position of the candidate on the feature map. The model learning consists of four steps: Step 1: Independently train the RPN network: Load the pre-trained model into the RPN network as network parameters; Step 2: Independently train the Fast R-CNN network: First, load the candidate regions generated by the RPN network into the key detection network. Specifically, the RPN network generates a candidate box, obtains the corresponding region of the original image through the candidate box, and then performs multiple convolution and pooling operations on the obtained image. Then, it generates two modules through ROI pooling layers and fully convolutional layers: one is a softmax module for object classification, and the other is a box regression module. Step 3: After the two networks are trained independently in Step 1 and Step 2, keep the parameters of the shared part of the two networks unchanged, and only update the parameters of the rest of the RPN network; Step 4: Keep the parameters of the shared parts of the two networks unchanged, and only update the parameters of the rest of the Fast R-CNN network.

6. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that, The hierarchical recursive network-based text generation architecture first utilizes an optimized and enhanced object discrimination and localization framework to obtain proposed blocks and their feature information. After passing through a fully convolutional localization layer, M D-dimensional feature information is obtained. This feature information is then loaded into a hierarchical recursive network. The first layer is a global long short-term memory layer, and the second layer is a sentence BRNN layer. The global long short-term memory layer determines the final number of generated sentences and the fusion between sentences based on the loaded multiple sets of topic sentence vectors, thereby increasing the number of generated sentences. The sentence BRNN layer generates descriptive sentences corresponding to each image block. The independent sentence RNN layer enables the generated descriptive sentences to be more accurate and richer.

7. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that, Fully Convolutional Localization Layer: Loading and Generation of the Fully Convolutional Localization Layer: In the Faster R-CNN object measurement model based on DenseNet, feature extraction is performed through DenseNet. The expression for loading the image is defined as 3×H×W, and the expression generated after feature extraction is C×H0×W0. For ease of model integration, W0=W / 16, H0=H / 16, and C×H0×W0 are used as the loading tensors of the convolutional localization layer. The network generates B regions of interest, each corresponding to three tensors. These tensors are the coordinates of the region, the score of the region, and the feature description of the region. The coordinates of the region represent a B×4 matrix, the score of the region represents the confidence score of the generated region of length B, and the feature description of the region represents a feature tensor of size B×C×X×Y, where X and Y represent the grid size. Candidate Block Determination: The fully convolutional localization layer obtains candidate blocks by regressing a series of translation-invariant anchor boxes. It maps each point in the loaded feature vector (H0×W0 grid) back to the initial H×W image plane, and considers k detection boxes of different sizes centered on this mapped point. The localization layer calculates a confidence score and predicts four scalars for each detection box. The result is obtained by feeding the feature map into a convolutional layer with 256 3×3 filters, a ReLU activation layer, and a convolutional layer with 1×1 1×1 filters. The final result is a 5k×H0×W0 tensor. In the candidate box regression part, the center coordinates and width and height of the anchor box, along with the four scalars predicted by the model, are used to calculate the center and width and height of the generated region. The generation parameter calculation formula is as follows. These parameters are used to obtain the regression from the anchor box to the proposed block. The specific parameter calculation for the generated block is as follows: Formula 8 Candidate box selection: The candidate blocks are sampled twice, and the blocks with different IoU are classified. The predicted block with the largest IoU in each initial block is also considered positive. During the testing phase, the NMS algorithm is used to select the top 500 blocks of interest based on their confidence scores. The localization layer finally obtains two tensors: B×4 including coordinates and B including the score. Eliminating redundant boxes and finding the best object detection location: There are three detection boxes. The confidence of the first detection box is 0.7, the confidence of the second detection box is 0.6, and the confidence of the third detection box is 0.

7. Non-maximum suppression selects the detection box with the highest confidence among these three boxes, which is most likely to represent the target. Step 1: Define three windows with confidence levels of [0.7, 0.6, 0.7]; Step 2: Select the detection box with the highest confidence as the target, and then compare the intersection areas between the remaining detection boxes and the target detection box; Step 3: If the overlapping area is larger than the set threshold, remove the detection box from the remaining detection boxes so that the confidence of the detection box is the same as the confidence of the target detection box, thus suppressing the maximum overlapping area. Step 4: Take the second high-confidence detection box as the target, repeat steps 2 and 3, and finally get the third of the three detection boxes defined at the beginning; Integration of candidate blocks: The sampled block suggestions vary in size and proportion. Considering the need for connection with fully connected layers and RNN language types, feature representations of the integration dimension are extracted from these regions of different sizes. Bilinear interpolation is used to establish a sampling grid G ​​of size X×Y×2. Each element in V is associated with the real-valued coordinates in U. The new feature map is obtained by interpolating the features of U. The value of the corresponding element in V is calculated based on the coordinates in U. The sampling kernel k is represented as: Formula 9 The gradient is transmitted back to the predicted region proposal coordinates. For all sampled regions, bilinear interpolation is performed to obtain a tensor of size B×C×X×Y, which is used to generate the localization layer.

8. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that, Hierarchical recursive network language generation network: It detects a block of image in the fully convolutional localization layer. This block contains two or more key objects and includes sub-region 1, sub-region 2, and the remaining image. For the same image block, a feature vector D fused from multiple feature vectors retains more accurate information than a single feature vector D, which has two implications: Meaning 1: If an image block can be detected, and the block includes two or more sub-regions that are also detected, the feature vectors of these sub-regions are fused into a single vector. This vector is more accurate than the feature vector corresponding to the image block, and the proposed textual description of this type of image block is more accurate. Meaning 2: If an image block is not recognized and has no corresponding text description, but the image block includes two or more detected sub-regions, then the feature vectors of these sub-regions are merged into a single vector, which is used as the feature vector corresponding to the image block to generate the text description of the image block. This process uses the fusion of sub-regions to represent the undetected image block and generate a new text description for the image block. Topic Vectors: Between the Faster R-CNN and the global layer Long Short-Term Memory (LSTM), block pooling is first performed. Specifically, K M×D block vectors are generated from B×D and fused. Vector fusion is based on the positional relationships of the original images corresponding to the B block vectors. The condition for block fusion is that the distance between the block centers does not exceed one-fifth of the original image. Then, max pooling is performed on each fused block vector to reduce the amount of information and retain the most important information. The pooling operation is as follows: Formula 10 W pool It is a P×D matrix, where P is the dimension of the final generated vector and D is the dimension of the feature vector of each block. (1) Global Long Short Memory Layer: This is a single-layer hidden layer with a dimension of 512, where the ce1l state is set to 0. At each time step, a series of hidden states h1→h are generated. s Each represents a sentence, and each state has two functions: one is to use h i The first step is to use a mapping and logical classifier to generate binary classification states and probability distributions. The second step is to load two fully connected layers to generate topic vectors. (2) Statement BRNN layer: The final part of the description language generation. It obtains vectors of K fusion blocks through global layer long short-term memory. The statement BRNN layer converts these fusion block vectors into text. The description statement is represented by an h-dimensional vector, h=P. The embedding process using BRNN is as follows: Let t represent the column vector of the word at position t in the vocabulary. Suppose the word at position t is "boy," and this word appears in the vocabulary vector. If the value at the corresponding position is 1 and the position is the 6th, then... =[0, 0, 0, 0, 0, 1, 0, 0,…, 0], W W The weights of the word vector generation model are initialized to 300 dimensions, representing a word embedding matrix. The model needs to learn and train the following parameters, W. e W f W b W d and bias b e b f b b b d The hidden layer size is 300 to 600 dimensions, and the activation function is f: x → max(0, x); Formula 11 Image patch and word matching: The mapping relationship between each word and an image patch indicates a measure of how well a word fits the various bounding boxes. Each word is matched only with the image patch with the highest matching score. The number of words in each phrase within a patch is equal. The cost function is as follows: Formula 12 Image region and phrase matching: The sentence contains N words, and the image contains M detection boxes. A hidden variable a is used. j Using Markov Random Fields (MRF), a matching metric for image blocks and phrases is calculated: Formula 13 Description of generating the entire image: Taking the P-dimensional vector from global short-term memory as input, calculate according to the following formula: Formula 14 Ultimately, a network is generated using a hierarchical recursive network language.

9. The method for providing a simple, accurate, and intelligent AI representation of complex, multi-element images according to claim 1, characterized in that, Hierarchical recursive network model learning: Each block fusion vector consists of multiple block vectors, each block vector corresponds to a statement, and one block fusion vector corresponds to multiple statements, ultimately generating a descriptive statement. The process of generating a descriptive statement from a block fusion vector is considered as generating a text segment, which actually contains only one sentence. Each block fusion is viewed as a text segment, assuming a text segment contains S statements, and each statement has N... i One word, y ij It is the j-th word of the i-th statement, in the pooled vector v p The Global Short-Term Memory (GSM) is expanded into S steps, the state probability distribution of each statement is calculated, and then loaded into the BRNN network, with each step expanded into N steps. i A probability distribution p is generated with a step size of 1. ij Regarding the j-th word of the i-th sentence, referring to the image to generate a text segment description model, the definition of the cost function corresponding to the fusion vector is as follows: Formula 15 Three values ​​are used to control the number of products generated in each step, T. stop N represents the stopping probability. max S represents the maximum number of words generated by each statement. max This represents the maximum number of sentences included in the text segment. An image has multiple fusion vectors, and the number of fusion vectors in an image is K. The cost function for an image is defined as shown in Equation 16: Formula 16 Where S k S represents the number of individual vectors included in the k-th fusion vector, and the step size of the long short memory model expansion is S. k ; Based on a hierarchical recursive network text generation architecture, an improved Faster R-CNN object detection model is used to obtain proposed blocks and their feature information. After passing through a fully convolutional localization layer, these feature information is transformed into M D-dimensional feature information. This feature information is then loaded into a hierarchical recursive network. The first layer is a global long short-term memory layer, and the second layer is a sentence BRNN layer. The global long short-term memory layer determines the number of sentences generated and the fusion between sentences based on the loaded multiple sets of topic sentence vectors, thereby increasing the number of generated sentences. The sentence BRNN layer generates descriptive sentences corresponding to each image block. The independent sentence BRNN layer enables the generated descriptive sentences to be more accurate and richer.

Images