Visual space description method, system and product based on depth information fusion

Abstract

The invention relates to the technical field of artificial intelligence, and particularly discloses a visual space description method, system and product based on depth information fusion, and the method comprises the steps: obtaining an image and a corresponding object text; performing vision-plane position feature extraction on the image to obtain a vision feature and a plane position feature, and performing depth feature extraction on the image to obtain a depth feature; performing text coding on the object text to obtain text features; performing feature fusion on the visual features, the plane position features, the depth features and the text features to obtain fusion features; performing spatial relationship classification on the fusion features to obtain a spatial relationship; and inputting the instruction language, the spatial relationship and the object text into a large language model to obtain a natural statement describing the spatial relationship of the object. According to the method, the spatial relationship between the objects in the image can be accurately described, the spatial perception capability of artificial intelligence is improved, and meanwhile, the increase of redundant manual annotation workload is avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a visual space description method, system and product based on deep information fusion. Background Art

[0002] With the rapid development of large language models and multimodal models in the field of artificial intelligence in recent years, natural language processing technology (NLP) and computer vision technology (CV) have received more and more attention, and the image-to-text task needs to consider the data of both modalities at the same time, so that the model can understand and describe the image scene with natural language text, such as image captioning and visual question answering. However, unlike human understanding of images, current image-to-text models generally lack the ability to understand spatial relationships in images, that is, they cannot accurately describe the spatial position relationships between objects in the image, such as "front and back" and "up and down". This requires the model to have strong prior knowledge while fully understanding the image content, which is also one of the main difficulties of this type of problem.

[0003] The existing methods construct a special image dataset, manually annotate the spatial relationship of the specified objects in the image, and then use the visual-language multimodal model to learn the spatial relationship from the two-dimensional image, and then further generate natural language description text, which is the visual spatial description task (Visual Spatial Description). However, this task currently faces many difficulties. On the one hand, it is difficult for the model to converge and achieve ideal results when learning three-dimensional spatial relationships from a limited two-dimensional image dataset; on the other hand, additional spatial relationship annotation of the image is a tedious and time-consuming task. Therefore, it is particularly important to make the model learn spatial information from the image as efficiently as possible while reducing or avoiding additional information annotation work. Summary of the invention

[0004] The present invention aims to solve the shortcomings of the above-mentioned prior art, and solve the problem of insufficient spatial understanding ability in the visual-language multimodal model while reducing or avoiding the workload of manual data annotation. To this end, the present invention provides a visual space description method, system and product based on deep information fusion, which obtains visual features, plane position features, depth features and text features according to the image and its corresponding object text. After feature fusion, the spatial relationship between objects can be classified, and then the language expression is enhanced through a large language model, which can accurately describe the spatial relationship between objects in the image.

[0005] The present invention provides a visual space description method based on depth information fusion, and the technical solution adopted is as follows: Get the image and its corresponding object text; Performing visual-plane position feature extraction on the image to obtain visual features and plane position features, and performing depth feature extraction on the image to obtain depth features; Performing text encoding on the object text to obtain text features; Fusing the visual features, plane position features, depth features and text features to obtain fused features; Classifying the spatial relationships of the fused features to obtain spatial relationships; The instruction language, the spatial relationship and the object text are input into a large language model to obtain a natural sentence describing the spatial relationship of the object.

[0006] Furthermore, the object text includes at least two text labels of objects.

[0007] Furthermore, the spatial relationship is expressed in the network model as: on, in, next, under, above, behind, in front of, left and right.

[0008] Furthermore, the process of performing visual-plane position feature extraction on the image to obtain visual features and plane position features is as follows: Use the target detection algorithm to detect objects in the image and generate multiple detection frames of the detected objects; Extracting visual features based on pixels within each of the detection frames; The area of ​​each detection frame is calculated, the detection frame coordinate information and its area are horizontally connected to form the object plane position information, and the plane position feature is obtained according to the object plane position information.

[0009] Furthermore, the process of extracting the depth feature of the image to obtain the depth feature is as follows: A monocular image depth estimation method is used to estimate depth value information of the image pixel by pixel, and a depth feature is obtained based on the depth value information in the detection frame.

[0010] Furthermore, the visual features, plane position features, depth features and text features are connected and input into a Transformer-Encoder module to obtain fused features.

[0011] The present invention also provides a visual space description system based on deep information fusion, and the technical scheme adopted is as follows: comprising: a visual-plane position feature extraction module, a deep feature extraction module, a text encoding module, a multiple feature fusion module, a spatial relationship classifier and a large language model; The visual-plane position feature extraction module is used to extract the visual-plane position feature of the image to obtain visual features and plane position features; The depth feature extraction module is used to extract the depth features of the image to obtain the depth features; The text encoding module is used to perform text encoding on the object text to obtain text features; The multiple feature fusion module is used to fuse the visual features, plane position features, depth features and text features to obtain fused features; The spatial relationship classifier is used to classify the fused features into spatial relationships to obtain spatial relationships; The large language model is used to obtain a natural sentence describing the spatial relationship of the object based on the instruction language, the spatial relationship and the object text.

[0012] Furthermore, the visual-plane position feature extraction module is ResNet50, and one of Faster-RCNN and YOLO; The depth feature extraction module is DepthAnything-V2; The multiple feature fusion module is Emedding; The spatial relationship classifier is MaxPool; The large language model is Chatgpt-4, Llama3.0 or Qwen.

[0013] Further, it is constructed through the following process: The output ends of the visual-plane position feature extraction module, the depth feature extraction module, and the text encoding module are connected to the input end of the multiple feature fusion module, and the output end of the multiple feature fusion module is connected to the input end of the spatial relationship classifier to obtain a visual spatial relationship classification neural network model; Use the visual space description dataset to train the visual space relationship classification neural network model. Input the {image, object s, object o} triples in the dataset in batches. The visual space relationship classification neural network model encodes and forward propagates, outputs the spatial position relationship p between object s and object o in the image, calculates the cross entropy loss function, and performs backpropagation to update the parameters. Repeat multiple iterations until the model converges. The output of the trained visual-spatial relationship classification neural network model is connected to the input of the large language model to obtain a visual-spatial description system based on deep information fusion.

[0014] The present invention also provides a computer program product, including a computer program, which implements the above-mentioned visual space description method based on depth information fusion when executed by a processor.

[0015] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention aims to solve the problem that the current neural network model lacks the ability to understand image space. The present invention estimates the depth information of a single image through a deep feature extraction module and encodes the deep features. Then, the deep features and visual features are innovatively integrated to classify spatial relationships. Finally, the language expression is enhanced through a large language model. The present invention can be used to solve the visual space description problem of the neural network model, improve the spatial perception ability of artificial intelligence, and avoid adding unnecessary manual annotation workload. It is suitable for application scenarios such as robotics and autonomous driving.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the method provided by the present invention.

[0019] Figure 2 It is a structural block diagram of the system provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the input and output of the deep feature extraction module provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0022] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0023] Combine the following Figures 1 to 3 The present invention is further described in detail, and a visual space description method, system and product based on depth information fusion of the present invention are described: In this embodiment, Figure 1 As shown, a visual space description method based on depth information fusion is provided, comprising the following steps: Step 1: Get the image and its corresponding object text.

[0024] Object text is a description of the target object in the image. To describe the spatial relationship between objects, the object text includes at least two object text labels. For example: Figure 2 In the image, there are two objects (object s and object o, object s is in front of object o). The object text includes the text labels of the above two objects (object s and object o).

[0025] Step 2: Extract features of images and object texts respectively.

[0026] (1) Extract visual-plane position features from the image to obtain visual features and plane position features. The specific process is as follows: Use target detection algorithms, such as YOLO and Faster-RCNN, to detect objects in the image and generate multiple detection boxes for the detected objects.

[0027] According to the pixels in each detection frame, the visual features of the object are extracted using ResNet.

[0028] For the detection frame, the area of ​​each detection frame is first calculated, and the detection frame coordinate information and its area are horizontally connected to form the object plane position information. The object plane position information passes through the fully connected layer to obtain the plane position feature.

[0029] The feature extraction process can be expressed as: in, Represents the plane position feature of an image, Represents the visual features of an image. M represents the number of objects detected in an image. is the detection box of the jth detected object, Indicates the operation of calculating the detection box area. () represents a fully connected layer, is the set of pixels corresponding to the jth detection frame, Represents the visual feature extraction method, namely the RestNet neural network. () indicates the feature tensor concatenation operation along the 0th dimension.

[0030] (2) Extracting the depth features of the image to obtain the depth features. The specific process is as follows: Use a monocular image depth estimation method, such as DepthAnything, to estimate the depth value information of the image pixel by pixel to form a depth map of the corresponding image. Use the detection box in the depth map, and the depth value information in the detection box, that is, the depth pixel set of the depth map in the detection box, is passed through 4 layers of 3×3 convolutional layers and 1 layer of fully connected layer to obtain the depth feature.

[0031] The detection frame here can reuse the detection frame in the process of extracting visual features and planar position features, or it can be calculated by the target detection algorithm.

[0032] The feature extraction process can be expressed as: in, Represents the deep features of an image, represents the set of depth pixels within the jth detection box, () represents a convolutional layer.

[0033] (3) Encode the object text to obtain text features. This process can be expressed as: in, Represents the text features obtained after encoding, represents the text label of object s, The text label representing the object o, Represents the text embedding encoding method in pytorch, Indicates the concatenation operation of feature tensors along 1 dimension.

[0034] Step 3: Fusing the visual features, plane position features, depth features and text features to obtain fused features.

[0035] In this embodiment, a 6-layer Transformer-Encoder is used to fuse features. The four features are connected and input into the 6-layer Transformer-Encoder to obtain the fused feature. : in, Represents a 6-layer Transformer-Encoder.

[0036] Step 4: Classify the spatial relationships of the fused features to obtain spatial relationships.

[0037] In this embodiment, the fused features are input into a spatial relationship classifier, which includes an average pooling and two fully connected layers, and the fused features are classified into one of nine common spatial relationships, which are expressed in the network model as: on, in, next, under, above, behind, in front of, left, and right. The types of spatial relationships can be increased or decreased according to actual needs.

[0038] in, () represents the pooling layer, represents a multi-layer perceptron, Represents the prediction result vector, with a length of 9, and each component represents the probability that two objects belong to the corresponding spatial relationship.

[0039] Step 5: Input the instruction language, the spatial relationship and the object text into the large language model to obtain a natural sentence describing the spatial relationship of the object.

[0040] Suppose the input command is: "Give you two objects: object s and object o, and the spatial relationship p between object s and object o. Use s, o, and p to generate three different sentences to describe them." If object s: power lines, object o: train, and spatial relationship p: above.

[0041] The natural sentence output by the large language model describing the spatial relationship of objects is: 1. The power lines run above the train.

[0042] 2. The train moves below the power lines.

[0043] 3. Power lines stretch over the train.

[0044] If object s: car, object o: track, relationship p: in front of.

[0045] The natural sentence output by the large language model describing the spatial relationship of objects is: 1. The car is positioned in front of the track.

[0046] 2. The track lies behind the car.

[0047] 3. The car leads the track.

[0048] This embodiment illustratively provides an instruction word, and the instruction word can be adjusted as needed.

[0049] In this embodiment, Figure 2 As shown, a visual space description system based on deep information fusion is also provided, and the technical solution adopted is as follows: including: visual-plane position feature extraction module, deep feature extraction module, text encoding module, multiple feature fusion module, spatial relationship classifier and large language model.

[0050] The visual-plane position feature extraction module is used to extract visual-plane position features from the image to obtain visual features and plane position features. Commonly used, the visual-plane position feature extraction module is ResNet50, and one of Faster-RCNN and YOLO.

[0051] The depth feature extraction module is used to extract the depth features of the image to obtain the depth features. Commonly used, the depth feature extraction module is DepthAnything-V2. The input and output diagram of the depth feature extraction module is as follows Figure 3 shown.

[0052] The text encoding module is used to perform text encoding on the object text to obtain text features.

[0053] The multiple feature fusion module is used to fuse the visual features, plane position features, depth features and text features to obtain fused features. The commonly used multiple feature fusion module is Emedding.

[0054] The spatial relationship classifier is used to classify the fusion features into spatial relationships to obtain spatial relationships. A commonly used spatial relationship classifier is MaxPool.

[0055] The large language model is used to obtain a natural sentence describing the spatial relationship of the object according to the instruction language, the spatial relationship and the object text. Commonly used large language models are Chatgpt-4, Llama3.0 or Qwen.

[0056] The visual space description system based on depth information fusion provided in this embodiment is constructed through the following process: The output ends of the visual-plane position feature extraction module, the deep feature extraction module, and the text encoding module are connected to the input end of the multiple feature fusion module, and the output end of the multiple feature fusion module is connected to the input end of the spatial relationship classifier to obtain a visual spatial relationship classification neural network model.

[0057] The visual spatial description dataset is used to train the visual spatial relationship classification neural network model. The {image, object s, object o} triples in the dataset are input in batches. The visual spatial relationship classification neural network model is encoded and forward propagated, and the spatial position relationship p between object s and object o in the image is output. The cross entropy loss function is calculated, and backward propagation is performed to update the parameters. Multiple iterations are performed until the model converges.

[0058] During model training, images are input in batches, each batch contains N images, and the sum loss of N images is calculated after forward propagation: in, represents the sign function, which takes 1 if the true category of sample i is equal to c, and takes 0 otherwise. represents the probability that sample i belongs to category c.

[0059] The output of the trained visual-spatial relationship classification neural network model is connected to the input of the large language model to obtain a visual-spatial description system based on deep information fusion.

[0060] This embodiment provides a specific implementation of a visual space description system based on depth information fusion, as shown in Table 1. Table 1 gives the network structure including the main modules, as well as the input and output.

[0061] Table 1 Module structure of a visual space description system based on depth information fusion

[0062] In Table 1, FC represents the fully connected layer, Conv represents the convolutional layer, out represents the dimension of the output feature map, k represents the size of the convolution kernel, and s represents the step size.

[0063] In this embodiment, Faster-RCNN is selected as the target detection algorithm, and 36 detection frames for each object are generated for each image. The pixels in each detection frame need to use a visual feature extraction method to extract visual features. In this embodiment, ResNet50 is used as the visual feature extraction method, and the length of the extracted visual features is 256 dimensions. For the detection frame, the area of ​​each detection frame is first calculated, and the coordinate information of the detection frame and its area are horizontally connected to form the plane position information of the object. The plane position feature is obtained through the fully connected layer. In this embodiment, the number of fully connected layers is 3, and the number of nodes in each layer is 5, 512, and 128 respectively. The activation function uses ReLU, and Dropout is used after the second layer of neurons during model training.

[0064] Use a monocular image depth estimation method to estimate the depth value information of each image pixel by pixel to form a depth map of the corresponding image. This embodiment uses DepthAnything-V2 as the depth estimation method. The detection frame in the visual-plane position feature extraction module is reused in the depth map, and the depth information of the object in the detection frame is uniformly scaled to 32×32 size, and then the depth feature is obtained through multiple neural network layers. The neural network layer of this embodiment uses 4 convolutional layers with a convolution kernel size of 3×3 and a fully connected layer. The number of output nodes of the fully connected layer is 256, and the activation function uses ReLU.

[0065] The input text {object s, object o} is encoded in the embedding layer to obtain text features. In this embodiment, nn.Embedding in the pytorch library is used for encoding, and the encoding length is 128. The multiple feature fusion module fuses the visual features, plane position features, depth features and text features. The specific method is: the visual features, plane position features, and depth features are horizontally connected, and then the text features are horizontally connected to each line, and then input into the Transformer-Encoder to obtain the fused features with a size of 36×768.

[0066] The spatial relationship is finally classified into nine spatial relationships {"on", "in", "next", "under", "above", "behind", "in front of", "left", "right"} through the spatial relationship classifier. The spatial relationship classifier includes an average pooling layer and two fully connected layers. The number of neurons in each layer is 768, 1024, and 9 respectively. The activation function uses ReLU. Dropout is used after the first layer of neurons during model training.

[0067] The visual spatial relationship classification neural network model is trained using the spatial visual description dataset. In this embodiment, the public dataset VSD-v1 is used. The three-tuple pairs of {image, object s, object o} in the training set are input in batches, and the visual spatial relationship classification neural network model is used for encoding and forward propagation. The model outputs the spatial relationship p between object s and object o in the image. The cross entropy loss function is calculated for each batch of training data, and the parameters are updated by backpropagation, and multiple iterations are performed until the model training is completed.

[0068] A large language model is then used to further generate a natural language description. This embodiment uses ChatGPT-4, inputs command language, object s, object o and the spatial relationship p between the two, and generates natural sentences in various forms that describe the spatial relationship between the objects.

[0069] Experiments were conducted on the VSDv1 and VSDv2 visual spatial understanding datasets disclosed in this embodiment. Spatial relationship natural sentence generation adopted Bleu-4 (bilingual evaluation index), SPICE (semantic proposition image caption evaluation), CIDEr (consensus-based image description evaluation index), METEOR (translation evaluation index based on weighted average and single word recall), VSRCCacc (visual spatial relationship classification accuracy), which are publicly available metrics for measuring machine translation and subtitle generation, and compared with the existing method VLBART+VSRC method. The experimental results are shown in Table 2, proving the effectiveness of the present invention.

[0070] Table 2 Comparison of experimental results of the present invention and VLBART+VSRC

[0071] This embodiment also provides a computer program product, including a computer program, which implements the above-mentioned visual space description method based on depth information fusion when executed by a processor.

[0072] The present invention focuses on designing the information fusion process, extracting the deep features of the image, and fusing the deep features with the visual features, the plane position features, and the text features to form a multimodal fusion feature, and obtains the spatial relationship between objects based on the fusion feature classification. The present invention designs each module based on the information fusion process, selects a suitable network for the module, and makes appropriate adjustments so that the structure of the network can realize the information fusion process and obtain a satisfactory natural sentence describing the spatial relationship of the object.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual space description method based on depth information fusion, characterized in that: include: Get the image and its corresponding object text; Performing visual-plane position feature extraction on the image to obtain visual features and plane position features, and performing depth feature extraction on the image to obtain depth features; Performing text encoding on the object text to obtain text features; Fusing the visual features, plane position features, depth features and text features to obtain fused features; Classifying the spatial relationships of the fused features to obtain spatial relationships; The instruction language, the spatial relationship and the object text are input into a large language model to obtain a natural sentence describing the spatial relationship of the object.

2. A visual space description method based on depth information fusion as claimed in claim 1, characterized in that: The object text includes at least two object text labels.

3. The visual space description method based on depth information fusion according to claim 1, characterized in that: The spatial relationship is expressed in the network model as: on, in, next, under, above, behind, in front of, left and right.

4. The visual space description method based on depth information fusion according to claim 1, characterized in that: The process of extracting visual-plane position features from the image to obtain visual features and plane position features is as follows: Use the target detection algorithm to detect objects in the image and generate multiple detection frames of the detected objects; Extracting visual features based on pixels within each detection frame; The area of ​​each detection frame is calculated, the detection frame coordinate information and its area are horizontally connected to form the object plane position information, and the plane position feature is obtained according to the object plane position information.

5. A visual space description method based on depth information fusion as claimed in claim 1 or 4, characterized in that: The process of extracting the deep features of the image to obtain the deep features is as follows: A monocular image depth estimation method is used to estimate depth value information of the image pixel by pixel, and a depth feature is obtained based on the depth value information in the detection frame.

6. The visual space description method based on depth information fusion according to claim 1, characterized in that: The visual features, plane position features, depth features and text features are connected and input into the Transformer-Encoder module to obtain fused features.

7. A visual space description system based on depth information fusion, characterized in that: Used to execute a visual space description method based on deep information fusion as described in any one of claims 1 to 6, comprising: a visual-plane position feature extraction module, a deep feature extraction module, a text encoding module, a multiple feature fusion module, a spatial relationship classifier and a large language model; The visual-plane position feature extraction module is used to extract the visual-plane position feature of the image to obtain visual features and plane position features; The depth feature extraction module is used to extract the depth features of the image to obtain the depth features; The text encoding module is used to perform text encoding on the object text to obtain text features; The multiple feature fusion module is used to fuse the visual features, plane position features, depth features and text features to obtain fused features; The spatial relationship classifier is used to classify the fused features into spatial relationships to obtain spatial relationships; The large language model is used to obtain a natural sentence describing the spatial relationship of the object based on the instruction language, the spatial relationship and the object text.

8. A visual space description system based on depth information fusion as claimed in claim 7, characterized in that: The visual-plane position feature extraction module is ResNet50, and one of Faster-RCNN and YOLO; The depth feature extraction module is DepthAnything-V2; The multiple feature fusion module is Emedding; The spatial relationship classifier is MaxPool; The large language model is Chatgpt-4, Llama3.0 or Qwen.

9. The visual space description system based on depth information fusion according to claim 7, characterized in that: Constructed through the following process: The output ends of the visual-plane position feature extraction module, the depth feature extraction module, and the text encoding module are connected to the input end of the multiple feature fusion module, and the output end of the multiple feature fusion module is connected to the input end of the spatial relationship classifier to obtain a visual spatial relationship classification neural network model; Use the visual space description dataset to train the visual space relationship classification neural network model. Input the {image, object s, object o} triples in the dataset in batches. The visual space relationship classification neural network model encodes and forward propagates, outputs the spatial position relationship p between object s and object o in the image, calculates the cross entropy loss function, and performs backpropagation to update the parameters. Repeat multiple iterations until the model converges. The output of the trained visual-spatial relationship classification neural network model is connected to the input of the large language model to obtain a visual-spatial description system based on deep information fusion.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements a visual space description method based on depth information fusion as described in any one of claims 1 to 6.

Images