A visual space description method, system and product based on depth information fusion

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically discloses a visual space description method, system, and product based on depth information fusion. The method comprises: obtaining an image and its corresponding object text; performing visual-plane position feature extraction on the image to obtain visual features and plane position features, performing depth feature extraction on the image to obtain depth features; performing text encoding on the object text to obtain text features; performing feature fusion on the visual features, plane position features, depth features, and text features to obtain fused features; performing spatial relationship classification on the fused features to obtain spatial relationships; and inputting instruction language, the spatial relationships, and the object text into a large language model to obtain a natural sentence describing the spatial relationships of the objects. The present invention can accurately describe the spatial relationships between objects in an image, enhance the spatial perception capability of artificial intelligence, and avoid increasing the workload of unnecessary manual annotation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a visual space description method, system and product based on depth information fusion. Background Art

[0002] With the rapid development of large language models and multimodal models in the field of artificial intelligence in recent years, natural language processing (NLP) and computer vision (CV) technologies have received increasing attention. Image-to-text tasks require simultaneous consideration of data from both modalities, enabling models to understand and describe image scenes using natural language text, such as in image captioning and visual question answering. However, unlike human understanding of images, current image-to-text models generally lack the ability to understand spatial relationships within images. Specifically, they are unable to accurately describe spatial relationships between objects in an image, such as "front-to-back" and "up-to-down." This requires models to possess strong prior knowledge in addition to a full understanding of the image content, a key challenge for this type of problem.

[0003] Existing methods construct specialized image datasets, manually annotate the spatial relationships of specified objects in the images, and then employ visual-linguistic multimodal models to learn spatial relationships from two-dimensional images, further generating natural language descriptions. This is known as the visual spatial description task. However, this task currently faces many difficulties. Firstly, it is difficult for the model to converge and achieve ideal results by learning three-dimensional spatial relationships from a limited two-dimensional image dataset. Secondly, additional spatial relationship annotation of images is tedious and time-consuming. Therefore, it is crucial to ensure that the model learns spatial information from images as efficiently as possible while reducing or avoiding the need for additional information annotation. Summary of the Invention

[0004] The present invention aims to address the shortcomings of the above-mentioned prior art, while reducing or avoiding the workload of manual data annotation and solving the problem of insufficient spatial understanding ability in visual-language multimodal models. To this end, the present invention provides a visual space description method, system and product based on deep information fusion. According to the image and its corresponding object text, visual features, planar position features, depth features and text features are obtained. After feature fusion, the spatial relationship between objects can be classified and then enhanced through a large language model. The language expression can accurately describe the spatial relationship between objects in the image.

[0005] The present invention provides a visual space description method based on depth information fusion, which adopts the following technical solutions:

[0006] Get the image and its corresponding object text;

[0007] Performing visual-plane position feature extraction on the image to obtain visual features and plane position features, and performing depth feature extraction on the image to obtain depth features;

[0008] Performing text encoding on the object text to obtain text features;

[0009] Fusing the visual features, plane position features, depth features, and text features to obtain fused features;

[0010] Performing spatial relationship classification on the fused features to obtain spatial relationships;

[0011] The instruction language, the spatial relationship and the object text are input into a large language model to obtain a natural sentence describing the spatial relationship of the object.

[0012] Furthermore, the object text includes at least two object text labels.

[0013] Furthermore, the spatial relationship is expressed in the network model as: on, in, next, under, above, behind, in front of, left and right.

[0014] Furthermore, the process of performing visual-plane position feature extraction on the image to obtain visual features and plane position features is as follows:

[0015] Use target detection algorithms to detect objects in the image and generate multiple detection frames for the detected objects;

[0016] Extracting visual features based on pixels within each detection frame;

[0017] The area of ​​each detection frame is calculated, the detection frame coordinate information and its area are horizontally connected to form the object plane position information, and the plane position feature is obtained according to the object plane position information.

[0018] Furthermore, the process of extracting depth features from the image to obtain depth features is as follows:

[0019] A monocular image depth estimation method is used to estimate the depth value information of the image pixel by pixel, and a depth feature is obtained based on the depth value information within the detection frame.

[0020] Furthermore, the visual features, planar position features, depth features and text features are connected and input into the Transformer-Encoder module to obtain fused features.

[0021] The present invention also provides a visual space description system based on depth information fusion, which adopts the following technical solutions: including: a visual-plane position feature extraction module, a depth feature extraction module, a text encoding module, a multiple feature fusion module, a spatial relationship classifier and a large language model;

[0022] The visual-plane position feature extraction module is used to extract visual-plane position features from the image to obtain visual features and plane position features;

[0023] The depth feature extraction module is used to extract depth features from the image to obtain depth features;

[0024] The text encoding module is used to perform text encoding on the object text to obtain text features;

[0025] The multiple feature fusion module is used to fuse the visual features, plane position features, depth features and text features to obtain fused features;

[0026] The spatial relationship classifier is used to perform spatial relationship classification on the fusion features to obtain spatial relationships;

[0027] The large language model is used to obtain a natural sentence describing the spatial relationship of the object based on the instruction language, the spatial relationship and the object text.

[0028] Furthermore, the visual-plane position feature extraction module is ResNet50, and one of Faster-RCNN and YOLO;

[0029] The depth feature extraction module is DepthAnything-V2;

[0030] The multiple feature fusion module is Emedding;

[0031] The spatial relationship classifier is MaxPool;

[0032] The large language model is Chatgpt-4, Llama3.0 or Qwen.

[0033] Further, it is constructed through the following process:

[0034] Connect the outputs of the visual-plane position feature extraction module, the depth feature extraction module, and the text encoding module to the input of the multiple feature fusion module, and connect the output of the multiple feature fusion module to the input of the spatial relationship classifier to obtain a visual spatial relationship classification neural network model;

[0035] Use the visual space description dataset to train a visual space relationship classification neural network model. The {image, object s, object o} triples in the dataset are input in batches. The visual space relationship classification neural network model encodes and forward propagates, outputs the spatial position relationship p between object s and object o in the image, calculates the cross entropy loss function, and performs backward propagation to update the parameters. Repeat multiple iterations until the model converges.

[0036] The output of the trained visual-spatial relationship classification neural network model is connected to the input of the large language model to obtain a visual-spatial description system based on deep information fusion.

[0037] The present invention also provides a computer program product, including a computer program, which implements the above-mentioned visual space description method based on depth information fusion when executed by a processor.

[0038] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0039] This invention addresses the problem of current neural network models lacking the ability to understand image space. By using a deep feature extraction module to estimate the depth information of a single image and encode the deep features, it then innovatively integrates these deep features with visual features to classify spatial relationships. Finally, a large language model is used to enhance language expression. This invention can be used to address the visual-spatial description issues of neural network models, improving the spatial perception capabilities of artificial intelligence while avoiding the unnecessary manual annotation workload. It is suitable for application scenarios such as robotics and autonomous driving.

[0040] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a flow chart of the method provided by the present invention.

[0043] Figure 2 It is a structural block diagram of the system provided by the present invention.

[0044] Figure 3 This is a schematic diagram of the input and output of the deep feature extraction module provided by the present invention. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0046] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0047] The following combination Figures 1 to 3 The present invention is further described in detail, which includes a visual space description method, system and product based on depth information fusion.

[0048] In this embodiment, Figure 1 As shown, a visual space description method based on depth information fusion is provided, comprising the following steps:

[0049] Step 1: Get the image and its corresponding object text.

[0050] Object text is a description of the target object in the image. To describe the spatial relationship between objects, the object text includes at least two object text labels. For example: Figure 2 In the image, there are two objects (object s and object o, object s is in front of object o). The object text includes the text labels of the above two objects (object s and object o).

[0051] Step 2: Extract features of images and object texts respectively.

[0052] (1) Perform visual-plane position feature extraction on the image to obtain visual features and plane position features. The specific process is as follows:

[0053] Use target detection algorithms, such as YOLO and Faster-RCNN, to detect objects in the image and generate multiple detection frames for the detected objects.

[0054] Based on the pixels within each detection frame, ResNet is used to extract the visual features of the object.

[0055] For the detection frame, the area of ​​each detection frame is first calculated, and the detection frame coordinate information and its area are horizontally connected to form the object plane position information. The object plane position information passes through the fully connected layer to obtain the plane position feature.

[0056] The feature extraction process can be expressed as:

[0057]

[0058] in, Represents the plane position feature of an image, Represents the visual features of an image. M represents the number of objects detected in an image. is the detection box of the jth detected object, Indicates the operation of calculating the detection box area, () represents the fully connected layer, is the set of pixels corresponding to the jth detection frame, Represents the visual feature extraction method, namely the RestNet neural network. () indicates the feature tensor concatenation operation along the 0th dimension.

[0059] (2) Extracting depth features from the image to obtain depth features. The specific process is as follows:

[0060] Using a monocular image depth estimation method, such as DepthAnything, the depth value information of the image is estimated pixel by pixel to form a depth map for the corresponding image. The depth value information within the detection box (i.e., the depth pixel set within the depth map) is passed through four layers of 3×3 convolutional layers and one fully connected layer to obtain depth features.

[0061] The detection frame here can reuse the detection frame in the process of extracting visual features and planar position features, or it can be calculated by the target detection algorithm.

[0062] The feature extraction process can be expressed as:

[0063]

[0064] in, Represents the depth feature of an image, represents the set of depth pixels within the j-th detection box, () represents a convolutional layer.

[0065] (3) Encode the object text to obtain text features. This process can be expressed as:

[0066]

[0067] in, Represents the text features obtained after encoding, represents the text label of object s, The text label representing the object o, Represents the text embedding encoding method in pytorch, Indicates the feature tensor concatenation operation along 1 dimension.

[0068] Step 3: Fusing the visual features, planar position features, depth features, and text features to obtain fused features.

[0069] In this embodiment, a 6-layer Transformer-Encoder is used to fuse features. The four features are connected and input into the 6-layer Transformer-Encoder to obtain the fused features. :

[0070]

[0071] in, Represents a 6-layer Transformer-Encoder.

[0072] Step 4: Classify the spatial relationships of the fused features to obtain spatial relationships.

[0073] In this example, the fused features are input into a spatial relationship classifier, which includes an average pooling layer and two fully connected layers. The classifier classifies the fused features into one of nine common spatial relationships, represented in the network model as: on, in, next, under, above, behind, in front of, left, and right. The types of spatial relationships can be increased or decreased based on actual needs.

[0074]

[0075] in, () represents the pooling layer, represents a multilayer perceptron, Represents the prediction result vector, with a length of 9, and each component represents the probability that two objects belong to the corresponding spatial relationship.

[0076] Step 5: Input the instruction language, the spatial relationship and the object text into the large language model to obtain a natural sentence describing the spatial relationship of the object.

[0077] Suppose the input instruction is: "Give you two objects: object s and object o, and the spatial relationship p between object s and object o. Use s, o, and p to generate three different sentences to describe them." If object s: power lines, object o: train, and spatial relationship p: above.

[0078] The natural sentence output by the large language model describing the spatial relationship of objects is:

[0079] 1. The power lines run above the train.

[0080] 2. The train moves below the power lines.

[0081] 3. Power lines stretch over the train.

[0082] If object s: car, object o: track, relationship p: in front of.

[0083] The natural sentence output by the large language model describing the spatial relationship of objects is:

[0084] 1. The car is positioned in front of the track.

[0085] 2. The track lies behind the car.

[0086] 3. The car leads the track.

[0087] This embodiment schematically provides an instruction word, and the instruction word can be adjusted as needed.

[0088] In this embodiment, Figure 2 As shown, a visual space description system based on deep information fusion is also provided, and the technical solution adopted is as follows: including: visual-plane position feature extraction module, deep feature extraction module, text encoding module, multiple feature fusion module, spatial relationship classifier and large language model.

[0089] The visual-plane position feature extraction module is used to extract visual-plane position features from the image to obtain visual features and plane position features. Commonly used visual-plane position feature extraction modules are ResNet50, Faster-RCNN, and YOLO.

[0090] The depth feature extraction module is used to extract the depth features of the image to obtain the depth features. Commonly used, the depth feature extraction module is DepthAnything-V2. The input and output diagram of the depth feature extraction module is as follows Figure 3 shown.

[0091] The text encoding module is used to perform text encoding on the object text to obtain text features.

[0092] The multi-feature fusion module is used to fuse the visual features, plane position features, depth features and text features to obtain fused features. A commonly used multi-feature fusion module is Emedding.

[0093] The spatial relationship classifier is used to classify the spatial relationship of the fused features to obtain the spatial relationship. A commonly used spatial relationship classifier is MaxPool.

[0094] The large language model is used to obtain a natural sentence describing the spatial relationship of the object based on the instruction language, the spatial relationship and the object text. Commonly used large language models are Chatgpt-4, Llama3.0 or Qwen.

[0095] This embodiment provides a visual space description system based on depth information fusion, which is constructed through the following process:

[0096] The output ends of the visual-plane position feature extraction module, the depth feature extraction module, and the text encoding module are connected to the input end of the multiple feature fusion module, and the output end of the multiple feature fusion module is connected to the input end of the spatial relationship classifier to obtain a visual spatial relationship classification neural network model.

[0097] The visual spatial description dataset is used to train the visual spatial relationship classification neural network model. The {image, object s, object o} triples in the dataset are input in batches. The visual spatial relationship classification neural network model encodes and forward propagates, outputs the spatial position relationship p between object s and object o in the image, calculates the cross entropy loss function, and performs backward propagation to update the parameters. Multiple iterations are performed until the model converges.

[0098] During model training, images are input in batches, each batch contains N images, and the sum loss of N images is calculated after forward propagation:

[0099]

[0100] in, Represents the sign function, which takes 1 if the true category of sample i is equal to c, and takes 0 otherwise. represents the probability that sample i belongs to category c.

[0101] The output of the trained visual-spatial relationship classification neural network model is connected to the input of the large language model to obtain a visual-spatial description system based on deep information fusion.

[0102] This embodiment provides a specific implementation of a visual space description system based on depth information fusion, as shown in Table 1. Table 1 provides the network structure including the main modules, as well as the input and output.

[0103] Table 1 Module structure of a visual space description system based on depth information fusion

[0104]

[0105] In Table 1, FC represents the fully connected layer, Conv represents the convolutional layer, out represents the dimension of the output feature map, k represents the size of the convolution kernel, and s represents the step size.

[0106] In this embodiment, Faster-RCNN is selected as the target detection algorithm, and 36 detection frames for detecting objects are generated for each image. The pixels in each detection frame need to use a visual feature extraction method to extract visual features. In this embodiment, ResNet50 is used as the visual feature extraction method, and the length of the extracted visual features is 256 dimensions. For the detection frame, the area of ​​each detection frame is first calculated, and the detection frame coordinate information and its area are horizontally connected to form the plane position information of the object. The plane position feature is obtained through the fully connected layer. In this embodiment, the number of fully connected layers is 3, and the number of nodes in each layer is 5, 512, and 128 respectively. The activation function uses ReLU, and Dropout is used after the second layer of neurons during model training.

[0107] A monocular image depth estimation method is used to estimate the depth value information of each image pixel by pixel to form a depth map of the corresponding image. This embodiment uses DepthAnything-V2 as the depth estimation method. The detection box in the visual-plane position feature extraction module is reused in the depth map. After the depth information of the objects in the detection box is uniformly scaled to 32×32 size, the depth features are obtained through multiple neural network layers. The neural network layer of this embodiment uses 4 convolutional layers with a 3×3 convolution kernel size and a fully connected layer. The number of output nodes of the fully connected layer is 256, and the activation function uses ReLU.

[0108] The input text {object s, object o} is encoded in the embedding layer to obtain text features. In this example, nn.Embedding from the PyTorch library is used for encoding, and the encoding length is 128. The multi-feature fusion module fuses visual features, plane position features, depth features, and text features. The specific method is as follows: the visual features, plane position features, and depth features are horizontally concatenated, and the text features are horizontally concatenated to each row. The fused features are then input into the Transformer-Encoder to obtain fused features with a size of 36×768.

[0109] Spatial relationships are finally classified into nine spatial relationships {"on", "in", "next", "under", "above", "behind", "in front of", "left", "right"} through a spatial relationship classifier. The spatial relationship classifier includes an average pooling layer and two fully connected layers. The number of neurons in each layer is 768, 1024, and 9, respectively. The activation function uses ReLU, and Dropout is used after the first layer of neurons during model training.

[0110] A visual spatial relationship classification neural network model is trained using the Spatial Visual Description Dataset (VSD-v1). In this example, the publicly available dataset, VSD-v1, is used. Triple pairs {image, object s, object o} from the training set are input in batches. The visual spatial relationship classification neural network model performs encoding and forward propagation, outputting the spatial relationship p between objects s ​​and o in the image. A cross-entropy loss function is calculated for each batch of training data, and backward propagation is performed to update the parameters. This process is repeated multiple times until model training is complete.

[0111] A large language model is then used to further generate natural language descriptions. In this embodiment, ChatGPT-4 is used to input command language, object s, object o, and the spatial relationship p between the two, and generate natural sentences in various forms describing the spatial relationship between the objects.

[0112] Experiments were conducted on the VSDv1 and VSDv2 visual spatial understanding datasets disclosed in this embodiment. Spatial relationship natural sentence generation was performed using Bleu-4 (bilingual evaluation index), SPICE (semantic proposition image caption evaluation), CIDEr (consensus-based image description evaluation index), METEOR (translation evaluation index based on weighted average and single-word recall), and VSRCCAcc (visual spatial relationship classification accuracy), which are publicly available metrics for measuring machine translation and subtitle generation. The results were compared with the existing VLBART+VSRC method. The experimental results are shown in Table 2, demonstrating the effectiveness of the present invention.

[0113] Table 2 Comparison of experimental results of the present invention and VLBART+VSRC

[0114]

[0115] This embodiment further provides a computer program product, including a computer program, which implements the above-mentioned visual space description method based on depth information fusion when executed by a processor.

[0116] This paper focuses on the design of an information fusion process, extracting deep features from images and fusing them with visual features, planar position features, and text features to form a multimodal fusion feature. Spatial relationships between objects are then classified based on these fusion features. Based on this information fusion process, the paper designs various modules, selects appropriate networks for each module, and makes appropriate adjustments to ensure that the network structure implements the information fusion process and produces a satisfactory natural statement describing the spatial relationships between objects.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A visual space description method based on depth information fusion, characterized in that: include: Get the image and its corresponding object text; The object text only includes the text labels of 2 objects; Performing visual-plane position feature extraction on the image to obtain visual features and plane position features, and performing depth feature extraction on the image to obtain depth features; Select Faster-RCNN as the target detection algorithm to generate a detection frame for the detected object. The visual features of each pixel in the detection frame are extracted using a visual feature extraction method, and ResNet50 is used as the visual feature extraction method. For the detection box, the area of ​​each detection box is first calculated, and the detection box coordinate information and its area are horizontally connected to form the object plane position information. The plane position feature is obtained through the fully connected layer, and the number of fully connected layers is 3; A monocular image depth estimation method is used to estimate the depth value information of each image pixel by pixel to form a depth map of the corresponding image. DepthAnything-V2 is used as the depth estimation method. The detection box is reused in the depth map. The depth information of the objects in the detection box is uniformly scaled to 32×32 size. The depth features are obtained through multiple neural network layers. The neural network layer uses four convolutional layers with a 3×3 convolution kernel size and a fully connected layer with 256 output nodes. The activation function is ReLU. Performing text encoding on the object text to obtain text features; Fusing the visual features, plane position features, depth features, and text features to obtain fused features; The visual features, plane position features, and depth features are horizontally connected, and the text features are horizontally connected to each row, and then input into the Transformer-Encoder module to obtain the fused features; Use a 6-layer Transformer-Encoder to fuse features, connect the four features and input them into the 6-layer Transformer-Encoder to obtain the fused features : in, represents a 6-layer Transformer-Encoder, Indicates the feature tensor concatenation operation along 1 dimension, Represents the visual features of an image, Represents the plane position feature of an image, Represents the depth feature of an image, Represents the text features obtained after encoding; Performing spatial relationship classification on the fused features to obtain spatial relationships; The instruction language, the spatial relationship and the object text are input into a large language model to obtain a natural sentence describing the spatial relationship of the object.

2. The visual space description method based on depth information fusion according to claim 1, characterized in that: The spatial relationship is expressed in the network model as: on, in, next, under, above, behind, in front of, left and right.

3. The visual space description method based on depth information fusion according to claim 1, characterized in that: The process of performing visual-plane position feature extraction on the image to obtain visual features and plane position features is as follows: Use target detection algorithms to detect objects in the image and generate multiple detection frames for the detected objects; Extracting visual features based on pixels within each detection frame; The area of ​​each detection frame is calculated, the detection frame coordinate information and its area are horizontally connected to form the object plane position information, and the plane position feature is obtained according to the object plane position information.

4. A visual space description method based on depth information fusion according to claim 1 or 3, characterized in that: The process of extracting depth features from the image to obtain depth features is as follows: A monocular image depth estimation method is used to estimate the depth value information of the image pixel by pixel, and a depth feature is obtained based on the depth value information within the detection frame.

5. A visual space description system based on depth information fusion, characterized in that: Used to execute the visual space description method based on depth information fusion according to any one of claims 1 to 4, comprising: a visual-plane position feature extraction module, a depth feature extraction module, a text encoding module, a multiple feature fusion module, a spatial relationship classifier and a large language model; The visual-plane position feature extraction module is used to extract visual-plane position features from the image to obtain visual features and plane position features; The depth feature extraction module is used to extract depth features from the image to obtain depth features; The text encoding module is used to perform text encoding on the object text to obtain text features; The multiple feature fusion module is used to fuse the visual features, plane position features, depth features and text features to obtain fused features; The spatial relationship classifier is used to perform spatial relationship classification on the fusion features to obtain spatial relationships; The large language model is used to obtain a natural sentence describing the spatial relationship of the object based on the instruction language, the spatial relationship and the object text.

6. A visual space description system based on depth information fusion according to claim 5, characterized in that: The visual-plane position feature extraction module is ResNet50, and one of Faster-RCNN and YOLO; The depth feature extraction module is DepthAnything-V2; The multiple feature fusion module is Emedding; The spatial relationship classifier is MaxPool; The large language model is Chatgpt-4, Llama3.0 or Qwen.

7. The visual space description system based on depth information fusion according to claim 5, characterized in that: Constructed through the following process: Connect the outputs of the visual-plane position feature extraction module, the depth feature extraction module, and the text encoding module to the input of the multiple feature fusion module, and connect the output of the multiple feature fusion module to the input of the spatial relationship classifier to obtain a visual spatial relationship classification neural network model; Use the visual space description dataset to train a visual space relationship classification neural network model. The {image, object s, object o} triples in the dataset are input in batches. The visual space relationship classification neural network model encodes and forward propagates, outputs the spatial position relationship p between object s and object o in the image, calculates the cross entropy loss function, and performs backward propagation to update the parameters. Repeat multiple iterations until the model converges. The output of the trained visual-spatial relationship classification neural network model is connected to the input of the large language model to obtain a visual-spatial description system based on deep information fusion.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements a visual space description method based on depth information fusion according to any one of claims 1 to 4.

Images