Monocular three-dimensional vision positioning method and device, equipment and storage medium

Abstract

The invention discloses a monocular three-dimensional visual positioning method, device and equipment and a storage medium, and relates to the technical field of computer vision and natural language processing, and the method comprises the steps: extracting text features, visual features and depth features based on a description text and a scene image of a to-be-positioned target; coupling the text feature, the visual feature and the depth feature to obtain a multi-modal coupling feature; the multi-mode coupling features are decoded, and multi-mode decoding features are obtained; and determining a three-dimensional visual positioning result of the to-be-positioned target based on the multi-mode decoding features. Through the above mode, cross-modal depth coupling among visual features, text features and depth features is realized, the expression ability of different modal features is improved, the perception ability of spatial position information and advanced semantic information is improved, a visual target most related to text description can be positioned, and the precision of three-dimensional target positioning is improved.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision and natural language processing, and in particular to a monocular three-dimensional visual positioning method, apparatus, device and storage medium. Background Art

[0002] Monocular 3D visual localization is an important research direction at the intersection of computer vision and natural language processing, with broad application potential in scenarios such as intelligent robotic grasping, autonomous driving, and robot navigation. It can introduce textual descriptions into the task of monocular 3D object detection. The core goal is to enable the detector to accurately identify the object referred to by the textual description, given an image and a natural language description. However, in practical applications, the feature differences between visual, textual, and depth modalities are large, the representation of visual and textual features is insufficient, and there is confusion between fine-grained object features.

[0003] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a monocular three-dimensional visual positioning method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that the multimodal features used in traditional monocular three-dimensional visual positioning have insufficient expression capabilities, which affects the accuracy of three-dimensional target positioning.

[0005] To achieve the above objectives, the present application provides a monocular three-dimensional visual positioning method, which includes:

[0006] Extract text features, visual features, and depth features based on the description text and scene image of the target to be located;

[0007] coupling the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature, wherein the text feature is used as an intermediate mode to connect the depth feature and the visual feature during coupling;

[0008] Decoding the multimodal coupling feature to obtain a multimodal decoding feature;

[0009] Based on the multimodal decoding features, a three-dimensional visual positioning result of the target to be positioned is determined.

[0010] In one embodiment, the step of extracting text features, visual features, and depth features based on the description text of the target to be located and the scene image includes:

[0011] Based on a preset masking rule, masking is performed on the description text to obtain a masked text, and semantic information and contextual relationships in the masked text are extracted to obtain the text features;

[0012] Extracting the visual features from the scene image, the visual features including at least texture visual features, color visual features, contour visual features, and category visual features;

[0013] Based on the visual features, a depth image corresponding to the scene image is generated, and the depth features are extracted from the depth image.

[0014] In one embodiment, the step of coupling the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature includes:

[0015] Aggregating the depth information corresponding to the depth feature to the text feature to obtain an initial text depth coupling feature;

[0016] Performing a residual connection between the text feature and the initial text deep coupling feature to obtain a text deep coupling feature;

[0017] Dispersing the text depth information corresponding to the text depth coupling feature into the visual feature to obtain an initial multimodal coupling feature;

[0018] Perform a residual connection on the text deep coupling feature and the initial multimodal coupling feature to obtain the multimodal coupling feature.

[0019] In one embodiment, the step of decoding the multimodal coupling feature to obtain a multimodal decoding feature includes:

[0020] Calculating a text-guided attention score based on the text feature and the multimodal coupling feature;

[0021] fusing the text-guided attention score with the multimodal coupling feature to obtain a multimodal enhancement feature, and fusing the text-guided attention score with the depth feature to obtain a depth enhancement feature;

[0022] The multimodal decoding feature is generated based on the multimodal enhancement feature and the depth enhancement feature.

[0023] In one embodiment, the step of calculating the first correspondence based on the text feature and the multimodal coupling feature includes:

[0024] Obtaining the first correspondence between text features, multimodal coupling features, learning parameters and text-guided attention scores;

[0025] The first corresponding relationship is determined based on the text feature, the multimodal coupling feature, the learning parameter, and the first corresponding relationship.

[0026] In one embodiment, the three-dimensional visual positioning result includes at least a three-dimensional position, a three-dimensional posture, and a category. After determining the three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features, the step further includes:

[0027] Calculating a weighted mean square error loss based on the three-dimensional position of the target to be located;

[0028] Calculating quaternion loss based on the three-dimensional posture of the target to be located;

[0029] Calculating the cross entropy loss based on the category of the target to be located;

[0030] Obtaining a second corresponding relationship among a weighted mean square error loss, a quaternion loss, a cross entropy loss, and a joint loss, and determining a joint loss based on the weighted mean square error loss, the quaternion loss, the cross entropy loss, and the second corresponding relationship;

[0031] The coupling and the decoding are optimized based on the joint loss.

[0032] In one embodiment, before the step of extracting text features, visual features, and depth features based on the description text of the target to be located and the scene image, the step further includes:

[0033] Obtain original description text and original scene image;

[0034] performing standardization processing on the original scene image to obtain the scene image;

[0035] Segmenting the original description text into corresponding vocabulary data, and converting the vocabulary data into corresponding identifiers based on a preset vocabulary table;

[0036] The description text is generated based on the identifier corresponding to the vocabulary data.

[0037] In addition, to achieve the above objectives, the present application also proposes a monocular three-dimensional visual positioning device, which includes:

[0038] The feature extraction module is used to extract text features, visual features, and depth features based on the description text and scene image of the target to be located;

[0039] a multimodal feature iterative coupling module, configured to couple the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature, wherein the text feature is used as an intermediate mode to connect the depth feature and the visual feature during the coupling;

[0040] A cross-modal cyclic enhanced feature decoding module, configured to decode the multimodal coupling feature to obtain a multimodal decoding feature;

[0041] The cross-modal cyclic enhanced feature decoding module is further used to determine the three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a monocular three-dimensional visual positioning device, which includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor, and the computer program is configured to implement the steps of the monocular three-dimensional visual positioning method as described above.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the monocular three-dimensional visual positioning method as described above are implemented.

[0044] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the monocular three-dimensional visual positioning method as described above.

[0045] The present application provides a monocular three-dimensional visual positioning method, which extracts text features, visual features and depth features based on the descriptive text and scene image of the target to be positioned; couples the text features, visual features and depth features to obtain multimodal coupling features, wherein the text features are used as the intermediate mode to connect the depth features and the visual features during coupling; decodes the multimodal coupling features to obtain multimodal decoding features; and determines the three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features. The present application uses text guidance to achieve cross-modal deep coupling between visual features, text features and depth features, fully utilizes visual, depth and text information, improves the expressive power of multimodal features, improves the perception of spatial position information and high-level semantic information, can locate the visual target most relevant to the text description, improves the positioning accuracy and robustness of the target referred to by the text in three-dimensional space, and improves the accuracy of three-dimensional target positioning, solving the technical problem that the multimodal features used in traditional monocular three-dimensional visual positioning have insufficient expressive power and affect the accuracy of three-dimensional target positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 This is a flow chart of Example 1 of the monocular 3D visual positioning method of this application;

[0049] Figure 2 This is a flow chart of Example 2 of the monocular 3D visual positioning method of this application;

[0050] Figure 3 Schematic diagram of multimodal feature iterative coupling of the monocular 3D visual positioning method provided in Example 2 of the present application;

[0051] Figure 4 This is a flow chart of Example 3 of the monocular 3D visual positioning method of this application;

[0052] Figure 5 Schematic diagram of cross-modal cyclic enhanced feature decoding of the monocular 3D visual positioning method provided in Example 3 of the present application;

[0053] Figure 6 A schematic diagram of a simplified process of the monocular 3D visual positioning method provided in Example 3 of the present application;

[0054] Figure 7 This is a schematic diagram of the module structure of the monocular three-dimensional visual positioning device according to an embodiment of the present application;

[0055] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the monocular three-dimensional visual positioning method in the embodiment of the present application.

[0056] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of the embodiment of the present application is: based on the descriptive text and scene image of the target to be located, text features, visual features and depth features are extracted; the text features, visual features and depth features are coupled to obtain multimodal coupling features, wherein the text features are used as the intermediate mode during coupling to connect the depth features and the visual features; the multimodal coupling features are decoded to obtain multimodal decoding features; based on the multimodal decoding features, the three-dimensional visual positioning result of the target to be located is determined.

[0060] At present, in practical applications, the features of traditional monocular 3D visual positioning are quite different between visual, textual, and depth modalities, and the representation of visual and textual features is insufficient.

[0061] This application provides a solution that uses text guidance to achieve cross-modal deep coupling between visual features, text features, and depth features, fully utilizes visual, depth, and text information, improves the expressiveness of multimodal features, and enhances the perception of spatial position information and high-level semantic information. It can locate the visual targets most relevant to the text description, improve the positioning accuracy and robustness of the text-referenced targets in three-dimensional space, and improve the accuracy of three-dimensional target positioning. It solves the technical problem that the expressiveness of multimodal features used in traditional monocular three-dimensional visual positioning is insufficient, which affects the accuracy of three-dimensional target positioning.

[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions, a monocular 3D visual positioning device, etc., and this embodiment does not specifically limit this. The following uses a monocular 3D visual positioning device as an example to illustrate this embodiment and the following embodiments.

[0063] The present invention provides a method for three-dimensional monocular vision positioning. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the monocular 3D visual positioning method of the present application.

[0064] In this embodiment, the monocular 3D visual positioning method includes steps S10 to S40:

[0065] Step S10: extracting text features, visual features, and depth features based on the description text of the target to be located and the scene image;

[0066] It should be noted that the target to be located refers to the object that currently requires three-dimensional visual positioning, which can be an object, and this embodiment does not specifically limit this. The description text is the text used by three-dimensional visual positioning to describe the relevant information of the target to be located, and the scene image is the image of the scene in which the target to be located is located, which is used by three-dimensional visual positioning. The scene can be determined according to actual conditions, for example: an outdoor autonomous driving scene, and there is no specific limitation on this. Generally speaking, both the description text and the scene image have been processed and can be directly used for three-dimensional visual positioning.

[0067] In addition, it should be noted that text features, visual features, and depth features are all key features extracted in this embodiment, and include text information, visual information, and depth information, respectively.

[0068] In a feasible embodiment, before step S10, it also includes: obtaining original description text and original scene image; standardizing the original scene image to obtain the scene image; dividing the original description text into corresponding vocabulary data, and converting the vocabulary data into corresponding identifiers based on a preset vocabulary table; generating the description text based on the identifier corresponding to the vocabulary data.

[0069] It should be noted that the original description text is the original text that uses natural language to describe the information related to the target to be located, and the original scene image is the original image of the scene where the target to be located is located. The original description text and the original scene image cannot be used directly for 3D visual positioning. They require certain preprocessing to obtain the corresponding description text T and scene image I.

[0070] It is understandable that the original scene images need to be standardized and resized to a uniform size for easy model processing. For example, all original scene images are resized to 480×480×3, where 480×480 represents the image resolution and 3 represents the three color channels of red, green, and blue. The scene image obtained after preprocessing is

[0071] It should be understood that the original description text usually contains information such as the characteristics and location of the target to be located. This information is segmented and divided into a series of vocabularies (words), which are lexical data. The preset vocabulary is the vocabulary of the language model used in this embodiment. The vocabulary of the language model (which has been pre-trained) is used to convert each vocabulary into a corresponding unique identifier, thereby converting the text tag into a numerical form that the model can understand. The identifiers corresponding to all vocabularies constitute a serialized description text T, that is, T = {id1, id2, ..., id m}, where id mIndicates the identifier corresponding to the mth word. In this case, the description text can be considered as an identifier sequence.

[0072] Furthermore, in a feasible implementation, step S10 includes steps S101 to S103:

[0073] Step S101, masking the description text based on a preset masking rule to obtain a masked text, and extracting semantic information and contextual relationships in the masked text to obtain the text features;

[0074] It should be noted that the preset masking rules are the rules used in the pre-set masking, which can be flexibly adjusted according to actual needs and are not specifically limited. The words in the description text are masked according to the preset masking rules to obtain the masked text, that is, the masked text T M , which is mathematically represented as T M ={id′1,id′2,…,id′ m}, where id′ m Represents the mth identifier after masking, if id′ m If it is selected for masking, it will be replaced by a special mask identifier. In this case, the mask text can be considered as a masked identifier sequence.

[0075] It is understandable that the masked text is input into the language model (which has been pre-trained) to capture the semantic information and contextual relationship of the text and generate text embedding, that is, the text feature F l , Here, m is the number of words after word segmentation, and d is the pre-set embedding dimension. Using masks can enhance language diversity and improve the model's robustness to input from different languages.

[0076] Step S102, extracting the visual features from the scene image, wherein the visual features include at least texture visual features, color visual features, contour visual features, and category visual features;

[0077] It should be noted that visual features usually include visual information such as texture, color, contour, and category. It can be seen that visual features at least include texture visual features, color visual features, contour visual features, and category visual features.

[0078] It is understandable that the scene image I is input into the deep neural network (which has been pre-trained) to mine the visual information such as texture, color, contour, category, etc. contained in the image to obtain the visual feature F v , Among them, h, w, and c represent the height, width, and number of channels of the feature map corresponding to the visual feature, respectively.

[0079] Step S103 : generating a depth image corresponding to the scene image based on the visual features, and extracting the depth features from the depth image.

[0080] It should be noted that the visual features are input into the depth prediction network (which has been pre-trained) to perform depth prediction and generate a depth image D corresponding to the scene image I. p , The whole process is based on the preset depth map D gt Supervision (obtained according to specific calculation rules) uses L1 smoothing loss, and the calculation relationship is as follows:

[0081]

[0082] Where, Loss d represents L1 smoothing loss, D p Represents the depth image corresponding to the scene image I, D gt Indicates a preset depth map obtained according to specific calculation rules.

[0083] It is understandable that the use of L1 smooth loss function can increase the stability of model training. It is smoother when the depth difference is small, and the increase rate of the loss function slows down when the depth difference is large, avoiding excessive loss values.

[0084] It should be understood that the predicted depth image D p Input into the deep encoder (pre-trained), mine the spatial position relationship, and obtain the deep feature F d , Where h′, w′, and c are the height, width, and number of channels of the depth image, respectively.

[0085] Step S20: coupling the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature, wherein the text feature is used as an intermediate mode to connect the depth feature and the visual feature during coupling;

[0086] It should be noted that multimodal coupling features are feature data obtained by cross-modal coupling of text features, visual features, and depth features.

[0087] It is understandable that textual features, visual features, and deep features are input into the text-guided multimodal feature iterative coupling module to generate multimodal coupled features. This module contains two multi-head attention mechanisms. It uses textual features as an intermediate modality to connect deep features with visual features, enhancing the semantic expressiveness of multimodal coupled features. It also adds residual connections in multiple iterations to maintain the stability of feature aggregation.

[0088] Step S30, decoding the multimodal coupling feature to obtain a multimodal decoding feature;

[0089] It should be noted that the multimodal decoding feature is feature data obtained after decoding the multimodal coupling feature.

[0090] It can be understood that text features, depth features, and multimodal coupling features are input into the cross-modal cyclic enhanced feature decoding module to obtain multimodal decoding features.

[0091] Step S40: determining a three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features.

[0092] It should be noted that the 3D visual positioning result is the final result obtained by positioning, which at least includes 3D position, 3D posture and category.

[0093] In specific implementation, the multimodal decoding features can be input into multiple multilayer perceptrons (MLPs), and multiple multilayer perceptrons are used to predict the three-dimensional position, three-dimensional posture, category and other information of the target to be located, where the category is usually reflected in the form of category probability.

[0094] Furthermore, since the three-dimensional position, three-dimensional posture and category are all predicted values and have corresponding real values, this embodiment adopts a supervised learning approach, uses three types of loss functions for supervision, and optimizes through a joint loss function.

[0095] In a feasible implementation manner, step S40 may include steps S501 to S505:

[0096] Step S501, calculating a weighted mean square error loss based on the three-dimensional position of the target to be located;

[0097] It should be noted that, assuming the three-dimensional position P s The corresponding true value (true position) is T s , weighted mean square error loss is used for supervision, and the calculation formula of weighted mean square error loss is as follows:

[0098]

[0099] Where, Loss s represents the weighted mean square error loss, represents the i-th coordinate of the predicted three-dimensional position, represents the i-th coordinate of the real position, N represents the dimension of the coordinate (X, Y, Z), w i is the weight of the i-th coordinate, which is used to adjust the error weights on different coordinate axes, λ regis the weight of the regularization term, which is used to control the strength of the regularization term. is the norm of the predicted coordinates, used for regularization.

[0100] Step S502, calculating quaternion loss based on the three-dimensional posture of the target to be located;

[0101] It is understandable that, assuming the three-dimensional posture P a The corresponding true value (true posture) is T a , quaternion loss is used for supervision, and the calculation formula of quaternion loss is as follows:

[0102]

[0103] Where, Loss a represents the quaternion loss, P a represents the predicted 3D pose, T a Showing true attitude, Represents a multiplication operation on quaternions describing the attitude.

[0104] Step S503, calculating the cross entropy loss based on the category of the target to be located;

[0105] It should be understood that, assuming that the class P c The corresponding true value (true category) is T c , cross entropy loss is used for supervision, and the calculation formula of cross entropy loss is as follows:

[0106]

[0107] Where, Loss c represents the cross entropy loss, represents the predicted probability of the i-th category, represents the true label of the i-th category, and N represents the total number of categories.

[0108] Step S504: obtaining a second corresponding relationship among the weighted mean square error loss, the quaternion loss, the cross entropy loss, and the joint loss, and determining the joint loss based on the weighted mean square error loss, the quaternion loss, the cross entropy loss, and the second corresponding relationship;

[0109] It should be noted that the joint loss is the overall loss function. The second corresponding relationship between the weighted mean square error loss, quaternion loss, cross entropy loss and the joint loss is the calculation relationship of the joint loss, as shown below:

[0110] Loss = γLoss s +βLoss a +Loss c

[0111] In the formula, Loss represents the joint loss, Loss s Represents the weighted mean square error loss, Loss a Represents quaternion loss, Loss c Represents the cross entropy loss, γ and β are hyperparameters for adjusting the weights of each loss.

[0112] Step S505 : optimizing the coupling and the decoding based on the joint loss.

[0113] It can be understood that this embodiment optimizes the coupling process and the decoding process through a joint loss function. If the intersection of the predicted result and the true value is greater than 0.25, the three-dimensional visual positioning task is considered to be successful. The success rate of the method used in this embodiment can reach 64.39%, compared with 14.38% of the VLTVG method, 30.08% of the GroundingDINO method, and 57.88% of the Mono3DVG method. The success rate of this embodiment is higher and has higher precision. If the intersection of the predicted result and the true value is greater than 0.5, the three-dimensional visual positioning task is considered to be successful. The success rate of the method used in this embodiment can reach 35.67%, compared with 7.18% of the VLTVG method, 14.34% of the GroundingDINO method, and 28.85% of the Mono3DVG method. The success rate of this embodiment is higher and has higher precision.

[0114] It should be understood that in specific implementation, the text-guided multimodal feature iterative coupling module, the cross-modal cyclic enhanced feature decoding module and the early feature extraction can be integrated into a monocular three-dimensional visual positioning model for training and application.

[0115] This embodiment provides a monocular three-dimensional visual positioning method, which extracts text features, visual features, and depth features based on the descriptive text and scene image of the target to be positioned; couples the text features, visual features, and depth features to obtain multimodal coupling features; decodes the multimodal coupling features to obtain multimodal decoding features; and determines the three-dimensional visual positioning results of the target to be positioned based on the multimodal decoding features. This embodiment uses text guidance to achieve cross-modal deep coupling between visual features, text features, and depth features, fully utilizing visual, depth, and text information to enhance the expressive power of multimodal features and the perception of spatial position information and high-level semantic information. It can locate the visual target most relevant to the text description, improve the positioning accuracy and robustness of the target referred to by the text in three-dimensional space, and improve the accuracy of three-dimensional target positioning.

[0116] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 , step S20 may include steps S201 to S202:

[0117] Step S201: Aggregate the depth information corresponding to the depth feature to the text feature to obtain an initial text depth coupling feature, and perform a residual connection between the text feature and the initial text depth coupling feature to obtain a text depth coupling feature;

[0118] It should be noted that, during the coupling process, this embodiment uses relative position encoding (RPE) to capture the relative position relationship between features, further enhancing the overall model's ability to express features. In addition, text features are used as an intermediate modality to connect deep features and visual features.

[0119] In addition, it should be noted that this embodiment uses a text-guided multimodal feature iterative coupling module to couple text features, visual features, and depth features. Figure 3 ,This module contains two multi-head attention mechanisms.

[0120] It is understandable that the first multi-head attention mechanism takes the deep feature F d As key (Key, K) and value (Value, V), text feature F l As the query (Query, Q), the attention weights of the text features and the deep features are calculated, and the depth information in the deep features is aggregated into the text features to obtain the preliminary coupling data of the text features and the deep features, that is, the initial text deep coupling features. After further processing the initial text deep coupling features, the coupling data of the text features and the deep features can be obtained, that is, the text deep coupling features. Since the text features are used as the intermediate modality for guidance in the coupling process, the text deep coupling features can be considered as text-guided deep features. Considering the stability of the training, the text features and the output features are added element by element using residual connections to obtain the text-guided deep features. That is to say, the text features are residually connected with the initial text deep coupling features to obtain the text deep coupling features F. ld , the calculation relationship is as follows:

[0121]

[0122] Where, F ld Represents the deep coupling feature of text, F l Represents text features, F d represents the depth feature, W Q 、WK 、W V are the linear transformation matrices of query, key, and value respectively, d k is the dimension of the key, and RPE stands for relative position encoding.

[0123] Step S202: Disperse the text depth information corresponding to the text depth coupling feature into the visual feature to obtain an initial multimodal coupling feature, and perform a residual connection between the text depth coupling feature and the initial multimodal coupling feature to obtain the multimodal coupling feature.

[0124] It should be noted that the text depth information is the text information and depth information contained in the text depth coupling feature. The second multi-head attention mechanism combines the text depth coupling feature F ld As the key (Key, K) and value (Value, V), the visual feature F v As the query (Query, Q), calculate the attention weights of text, depth, and visual features, disperse the text depth information into the visual features, and obtain the preliminary coupling data of text, depth, and visual features, that is, the initial multimodal coupling features. After further processing the initial multimodal coupling features, the coupling data of text, depth, and visual features can be obtained, that is, the multimodal coupling features. Similarly, the residual is used to connect the text depth coupling features and the initial multimodal coupling features to obtain the multimodal coupling features F vld , the calculation relationship is as follows:

[0125]

[0126] Where, F vld represents the multi-mode coupling characteristics, F ld Represents the deep coupling feature of text, F v represents the visual feature, W Q 、W K 、W V are the linear transformation matrices of query, key, and value respectively, d k is the dimension of the key, and RPE stands for relative position encoding.

[0127] It is understandable that using text features as an intermediate modality, iteratively coupling visual, textual, and depth features, and effectively fusing visual and depth features can supplement the vision-related text information lost in the multimodal coupling process, improve the expressiveness of features, and enhance the model's perception of spatial position information and high-level semantic information. It can effectively improve the reliability of the forward coupling between different modalities, allowing the model to pay more attention to areas related to the target referred to by the text, thereby improving the accuracy of three-dimensional target positioning.

[0128] This embodiment provides a monocular three-dimensional visual positioning method, which aggregates the depth information corresponding to the depth feature to the text feature to obtain the initial text depth coupling feature; performs residual connection on the text feature and the initial text depth coupling feature to obtain the text depth coupling feature; disperses the text depth information corresponding to the text depth coupling feature into the visual feature to obtain the initial multimodal coupling feature; performs residual connection on the text depth coupling feature and the initial multimodal coupling feature to obtain the multimodal coupling feature. This embodiment uses text guidance to achieve cross-modal deep coupling between visual features, text features, and depth features, fully utilizing visual, depth, and text information to enhance the expressive power of multimodal features, enhance the perception of spatial position information and high-level semantic information, and can locate the visual target most relevant to the text description, improve the positioning accuracy and robustness of the text-referenced target in three-dimensional space, and improve the accuracy of three-dimensional target positioning.

[0129] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be described in detail later. Figure 4 , step S30 may include steps S301 to S303:

[0130] Step S301, calculating a text-guided attention score based on the text feature and the multimodal coupling feature;

[0131] In a feasible implementation, step S301 may include: obtaining a first correspondence between text features, multimodal coupling features, learning parameters and text-guided attention scores; and determining the first correspondence based on the text features, the multimodal coupling features, learning parameters and the first correspondence.

[0132] It should be noted that the text-guided attention score is the attention score based on text guidance, and the learning parameters are the relevant parameters that the model can learn. The first correspondence between text features, multimodal coupling features, learning parameters, and the text-guided attention score is the calculation relationship of the text-guided attention score, which is as follows:

[0133]

[0134] Where S l represents the text-guided attention score, F vld represents the multi-mode coupling characteristics, F l Represents text features, F l (x) T F vld (x) represents the attention score of each point in the feature vector, and β and θ represent the learning parameters.

[0135] Step S302: fusing the text-guided attention score with the multimodal coupling feature to obtain a multimodal enhancement feature, and fusing the text-guided attention score with the depth feature to obtain a depth enhancement feature;

[0136] It should be noted that the reference Figure 5 , the text-guided attention score is fused with the multimodal coupling feature and the deep feature to obtain the enhanced feature. In this embodiment, the enhanced feature includes the multimodal enhanced feature and the deep enhanced feature.

[0137] It is understandable that the data obtained by fusing the text-guided attention score with the multimodal coupling feature is the multimodal enhanced feature, and the calculation relationship is as follows:

[0138] F′ vld =S l F vld

[0139] Where F′ vld represents the multimodal enhanced features, S l represents the text-guided attention score, F vld Represents multimodal coupling features. The data obtained by fusing the text-guided attention score with the deep features is the deep enhancement feature. The calculation relationship is as follows:

[0140] F′ d =S l F d

[0141] Where F′ d represents the depth enhancement feature, S l represents the text-guided attention score, F d Represents deep features.

[0142] Step S303: Generate the multimodal decoding feature based on the multimodal enhancement feature and the depth enhancement feature.

[0143] It should be noted that, initialize a query vector F q , used to aggregate multimodal enhanced features. Figure 5 , respectively enhance the features (multimodal enhanced features F′ vld and depth-enhanced features F′ d ), text feature F l Take turns as key (K) and value (V), query vector F q As the query (Q), the visual, depth, and text information are aggregated into the query vector through the multi-head self-attention mechanism to obtain the multimodal query vector F′ q , that is, multimodal decoding features. The intermediate result F′ of visual features and depth featuresvd , text features F l For example, the calculation relationship is as follows:

[0144]

[0145] Where F′ q represents the multimodal decoding feature, F l Represents text features, F′ vd Represents the intermediate result of visual features and depth features, W K 、W V are the linear transformation matrices of keys and values, respectively, d k is the dimension of the key, and RPE stands for relative position encoding.

[0146] It can be understood that the obtained multimodal decoding feature F′ q , are input into multiple multi-layer perceptrons respectively to predict the position, posture, and category of the target to be located in three-dimensional space described by the text. The calculation relationship is as follows:

[0147] P s ,P a ,P c =MLPs(F′ q )

[0148] Where, P s Represents the three-dimensional position of the target to be located, P a Represents the three-dimensional posture of the target to be located, P c Represents the category of the target to be located, MLPs represents multiple multi-layer perceptrons, F′ q Represents multimodal decoding features.

[0149] It should be understood that by calculating attention scores, initializing query vectors, and cyclic feature aggregation, the model can more comprehensively understand visual, depth, and text features, thereby improving the model's prediction accuracy for the target.

[0150] This embodiment provides a monocular three-dimensional visual positioning method, which calculates a text-guided attention score based on text features and multimodal coupling features; fuses the text-guided attention score with the multimodal coupling features to obtain a multimodal enhancement feature, and fuses the text-guided attention score with the depth feature to obtain a depth enhancement feature; and generates a multimodal decoding feature based on the multimodal enhancement feature and the depth enhancement feature. This embodiment calculates the text-guided attention score, applies it to enhance the multimodal coupling feature and the depth feature, iteratively fuses the enhanced feature to the query vector, and predicts through multiple multi-layer perceptrons, efficiently mining the potential associations between different modalities, improving the model's ability to represent multimodal information, improving the perception of the target's spatial position relationship and the target's fine-grained features, and improving the accuracy of three-dimensional target positioning.

[0151] For example, in order to help understand the implementation process of the monocular three-dimensional visual positioning method obtained by combining this embodiment with the above embodiment three, please refer to Figure 6 , Figure 6 A brief flowchart of a monocular 3D visual positioning method is provided. Specifically:

[0152] Step 1: Preprocess the input images and text. All images are uniformly processed into a 480*480*3 size as the network input. Natural language description text related to the target to be located is input, and the text is segmented and serialized.

[0153] Step 2: Text feature extraction: Mask random words in the input text according to specific masking rules to obtain the masked text, which is then input into a pre-trained language model to generate text embeddings.

[0154] Step 3: Visual feature extraction: Input the input image into a pre-trained deep neural network to obtain visual features.

[0155] Step 4: Depth prediction and feature extraction: Input the visual features into the pre-trained depth prediction network to obtain the corresponding depth features.

[0156] Step 5: Text-guided iterative coupling of multimodal features. Textual, visual, and deep features are input into the text-guided iterative coupling module to generate multimodal coupled features. This module includes two multi-head attention mechanisms, using textual features as an intermediate modality to connect deep and visual features, enhancing the semantic expressiveness of multimodal coupled features. Residual connections are added during multiple iterations to maintain the stability of feature aggregation.

[0157] Step 6: 3D visual localization based on cross-modal cyclic enhanced feature decoding. Text features, depth features, and multimodal coupling features are input into the cross-modal cyclic enhanced feature decoding module to obtain multimodal decoding features. Multiple multi-layer perceptrons are used to predict the position, posture, category, and other information of the 3D target, and the model is optimized using a joint loss function.

[0158] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the monocular three-dimensional visual positioning method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0159] This application also provides a monocular three-dimensional visual positioning device, please refer to Figure 7 , the monocular three-dimensional visual positioning device includes:

[0160] A feature extraction module 10 is used to extract text features, visual features, and depth features based on the description text of the target to be located and the scene image;

[0161] a multimodal feature iterative coupling module 20 for coupling the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature, wherein the text feature is used as an intermediate mode to connect the depth feature and the visual feature during coupling;

[0162] A cross-modal cyclic enhancement feature decoding module 30 is used to decode the multimodal coupling feature to obtain a multimodal decoding feature;

[0163] The cross-modal cyclic enhanced feature decoding module 30 is further configured to determine a three-dimensional visual positioning result of the target to be positioned based on the multi-modal decoding features.

[0164] In a feasible implementation manner, the feature extraction module 10 is further configured to perform masking processing on the description text based on a preset masking rule to obtain a masked text, and extract semantic information and contextual relationships in the masked text to obtain the text features;

[0165] Extracting the visual features from the scene image, the visual features including at least texture visual features, color visual features, contour visual features, and category visual features;

[0166] Based on the visual features, a depth image corresponding to the scene image is generated, and the depth features are extracted from the depth image.

[0167] In a feasible implementation manner, the multimodal feature iterative coupling module 20 is further configured to aggregate the depth information corresponding to the depth feature to the text feature to obtain an initial text depth coupling feature;

[0168] Performing a residual connection between the text feature and the initial text deep coupling feature to obtain a text deep coupling feature;

[0169] Dispersing the text depth information corresponding to the text depth coupling feature into the visual feature to obtain an initial multimodal coupling feature;

[0170] Perform a residual connection on the text deep coupling feature and the initial multimodal coupling feature to obtain the multimodal coupling feature.

[0171] In a feasible implementation manner, the cross-modal cyclic enhanced feature decoding module 30 is further configured to calculate a text-guided attention score based on the text feature and the multimodal coupling feature;

[0172] fusing the text-guided attention score with the multimodal coupling feature to obtain a multimodal enhancement feature, and fusing the text-guided attention score with the depth feature to obtain a depth enhancement feature;

[0173] The multimodal decoding feature is generated based on the multimodal enhancement feature and the depth enhancement feature.

[0174] In a feasible embodiment, the cross-modal cyclic enhanced feature decoding module 30 is further used to obtain a first correspondence between text features, multimodal coupling features, learning parameters and text-guided attention scores;

[0175] The first corresponding relationship is determined based on the text feature, the multimodal coupling feature, the learning parameter, and the first corresponding relationship.

[0176] In a feasible implementation manner, the cross-modal cyclic enhanced feature decoding module 30 is further configured to calculate a weighted mean square error loss based on the three-dimensional position of the target to be located;

[0177] Calculating quaternion loss based on the three-dimensional posture of the target to be located;

[0178] Calculating the cross entropy loss based on the category of the target to be located;

[0179] Obtaining a second corresponding relationship among a weighted mean square error loss, a quaternion loss, a cross entropy loss, and a joint loss, and determining a joint loss based on the weighted mean square error loss, the quaternion loss, the cross entropy loss, and the second corresponding relationship;

[0180] The coupling and the decoding are optimized based on the joint loss.

[0181] In a feasible implementation manner, the feature extraction module 10 is further configured to obtain original description text and original scene image;

[0182] performing standardization processing on the original scene image to obtain the scene image;

[0183] Segmenting the original description text into corresponding vocabulary data, and converting the vocabulary data into corresponding identifiers based on a preset vocabulary table;

[0184] The description text is generated based on the identifier corresponding to the vocabulary data.

[0185] The monocular 3D visual positioning device provided by this application adopts the monocular 3D visual positioning method in the above-mentioned embodiment, which can solve the technical problem that the multimodal features used in traditional monocular 3D visual positioning have insufficient expressive power, affecting the accuracy of 3D target positioning. Compared with the prior art, the beneficial effects of the monocular 3D visual positioning device provided by this application are the same as those of the monocular 3D visual positioning method provided by the above-mentioned embodiment, and the other technical features of the monocular 3D visual positioning device are the same as those disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0186] The present application provides a monocular three-dimensional visual positioning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the monocular three-dimensional visual positioning method in the above-mentioned embodiment one.

[0187] Reference below Figure 8 , which shows a schematic diagram of the structure of a monocular 3D visual positioning device suitable for implementing the embodiments of the present application. The monocular 3D visual positioning device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The monocular three-dimensional visual positioning device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Figure 8As shown, the monocular three-dimensional visual positioning device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 to the RAM (Random Access Memory) 1004. Various programs and data required for the operation of the monocular three-dimensional visual positioning device are also stored in the RAM 1004. The processing device 1001, ROM 1002 and RAM 1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the monocular three-dimensional visual positioning device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a monocular three-dimensional visual positioning device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have instead.

[0189] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0190] The monocular 3D visual positioning device provided by this application adopts the monocular 3D visual positioning method of the above-mentioned embodiment, which can solve the technical problem that the multimodal features used in traditional monocular 3D visual positioning have insufficient expressive power, affecting the accuracy of 3D target positioning. Compared with the prior art, the beneficial effects of the monocular 3D visual positioning device provided by this application are the same as the beneficial effects of the monocular 3D visual positioning method provided by the above-mentioned embodiment, and the other technical features of the monocular 3D visual positioning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0191] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0192] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0193] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the monocular three-dimensional visual positioning method in the above-mentioned embodiment.

[0194] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0195] The computer-readable storage medium may be included in the monocular three-dimensional visual positioning device; or it may exist independently without being assembled into the monocular three-dimensional visual positioning device.

[0196] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a monocular three-dimensional visual positioning device, the monocular three-dimensional visual positioning device: extracts text features, visual features and depth features based on the description text of the target to be positioned and the scene image; couples the text features, visual features and depth features to obtain multimodal coupling features, wherein the text features are used as the intermediate mode during coupling to connect the depth features and the visual features; decodes the multimodal coupling features to obtain multimodal decoding features; and determines the three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features.

[0197] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0198] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0199] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0200] The readable storage medium provided by this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned monocular three-dimensional visual positioning method, which can solve the technical problem that the multimodal features used in traditional monocular three-dimensional visual positioning have insufficient expression capabilities and affect the accuracy of three-dimensional target positioning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as the beneficial effects of the monocular three-dimensional visual positioning method provided by the above-mentioned embodiment, and will not be repeated here.

[0201] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned monocular three-dimensional visual positioning method when executed by a processor.

[0202] The computer program product provided in this application can address the technical problem that the multimodal features used in traditional monocular 3D visual positioning are insufficiently expressive, thus affecting the accuracy of 3D target positioning. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the monocular 3D visual positioning method provided in the above-mentioned embodiments, and will not be elaborated here.

[0203] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A monocular three-dimensional visual positioning method, characterized in that: The method comprises: Extract text features, visual features, and depth features based on the description text and scene image of the target to be located; coupling the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature, wherein the text feature is used as an intermediate mode to connect the depth feature and the visual feature during coupling; Decoding the multimodal coupling feature to obtain a multimodal decoding feature; Based on the multimodal decoding features, a three-dimensional visual positioning result of the target to be positioned is determined.

2. The method according to claim 1, wherein The step of extracting text features, visual features, and depth features based on the description text and scene image of the target to be located includes: Based on a preset masking rule, masking is performed on the description text to obtain a masked text, and semantic information and contextual relationships in the masked text are extracted to obtain the text features; Extracting the visual features from the scene image, the visual features including at least texture visual features, color visual features, contour visual features, and category visual features; Based on the visual features, a depth image corresponding to the scene image is generated, and the depth features are extracted from the depth image.

3. The method according to claim 1, wherein The step of coupling the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature includes: Aggregating the depth information corresponding to the depth feature to the text feature to obtain an initial text depth coupling feature; Performing a residual connection between the text feature and the initial text deep coupling feature to obtain a text deep coupling feature; Dispersing the text depth information corresponding to the text depth coupling feature into the visual feature to obtain an initial multimodal coupling feature; Perform a residual connection on the text deep coupling feature and the initial multimodal coupling feature to obtain the multimodal coupling feature.

4. The method according to claim 1, wherein The step of decoding the multimodal coupling feature to obtain a multimodal decoding feature includes: Calculating a text-guided attention score based on the text feature and the multimodal coupling feature; fusing the text-guided attention score with the multimodal coupling feature to obtain a multimodal enhancement feature, and fusing the text-guided attention score with the depth feature to obtain a depth enhancement feature; The multimodal decoding feature is generated based on the multimodal enhancement feature and the depth enhancement feature.

5. The method according to claim 1, wherein The step of calculating a first corresponding relationship based on the text feature and the multimodal coupling feature includes: Obtaining the first correspondence between text features, multimodal coupling features, learning parameters and text-guided attention scores; The first corresponding relationship is determined based on the text feature, the multimodal coupling feature, the learning parameter, and the first corresponding relationship.

6. The method according to claim 1, wherein The three-dimensional visual positioning result includes at least a three-dimensional position, a three-dimensional posture, and a category. After the step of determining the three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features, the step further includes: Calculating a weighted mean square error loss based on the three-dimensional position of the target to be located; Calculating quaternion loss based on the three-dimensional posture of the target to be located; Calculating the cross entropy loss based on the category of the target to be located; Obtaining a second corresponding relationship among a weighted mean square error loss, a quaternion loss, a cross entropy loss, and a joint loss, and determining a joint loss based on the weighted mean square error loss, the quaternion loss, the cross entropy loss, and the second corresponding relationship; The coupling and the decoding are optimized based on the joint loss.

7. The method according to any one of claims 1 to 6, characterized in that Before the step of extracting text features, visual features, and depth features based on the description text and scene image of the target to be located, the step further includes: Obtain original description text and original scene image; performing standardization processing on the original scene image to obtain the scene image; Segmenting the original description text into corresponding vocabulary data, and converting the vocabulary data into corresponding identifiers based on a preset vocabulary table; The description text is generated based on the identifier corresponding to the vocabulary data.

8. A monocular three-dimensional visual positioning device, characterized in that: The device comprises: The feature extraction module is used to extract text features, visual features, and depth features based on the description text and scene image of the target to be located; a multimodal feature iterative coupling module, configured to couple the text feature, the visual feature, and the depth feature to obtain a multimodal coupling feature, wherein the text feature is used as an intermediate mode to connect the depth feature and the visual feature during the coupling; A cross-modal cyclic enhanced feature decoding module, configured to decode the multimodal coupling feature to obtain a multimodal decoding feature; The cross-modal cyclic enhanced feature decoding module is further used to determine the three-dimensional visual positioning result of the target to be positioned based on the multimodal decoding features.

9. A monocular three-dimensional visual positioning device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the monocular three-dimensional visual positioning method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the monocular three-dimensional visual positioning method according to any one of claims 1 to 7 are implemented.

Images