Multi-view remote mixed reality image retrieval method and system
By introducing multi-view and natural language query adaptive environment perception and global feature fusion into mixed reality image retrieval, the problems of single view and insufficient cross-modal alignment in existing technologies are solved, realizing efficient and accurate multi-view remote mixed reality image retrieval and improving the retrieval accuracy and robustness in complex scenes.
Patent Information
- Application Number
- CN202511757070.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-02-27
AI Technical Summary
Existing mixed reality image retrieval technologies heavily rely on a single perspective and lack a deep understanding of the global scene and contextual semantics. This leads to decreased retrieval accuracy in complex environments, rigid traditional retrieval processes that struggle to balance efficiency and accuracy, and the failure to accurately align cross-modal features, making it difficult for the system to understand complex semantic retrieval commands from users.
By capturing the main view image stream of the target scene in real time based on MR devices, adaptively selecting the auxiliary view image that is semantically related to the current scene, and combining it with an attention mechanism network to generate a global scene multi-view image feature vector, and embedding it with the user's natural language query command into a unified semantic space, multi-view remote mixed reality image retrieval is realized.
It enhances the ability to understand the semantic meaning of scene context, balances retrieval efficiency and accuracy, optimizes resource allocation, improves the accuracy and robustness of image retrieval in complex cross-modal scenarios, and provides an intelligent and smooth mixed reality interactive experience.
Smart Images
Figure CN121579712A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image retrieval, in particular to a multi-view remote mixed reality image retrieval method and system. BACKGROUND
[0002] In the existing mixed reality image retrieval technology, the following core defects are generally present, most methods are heavily dependent on a single, local first-view image, lack of depth perception of scene global and contextual semantics, leading to a sharp decline in retrieval accuracy in complex or occluded environments; traditional retrieval processes are rigid and cannot dynamically adjust strategies according to the confidence of the query content, making it difficult to balance retrieval efficiency and accuracy, often falling into the dilemma of either redundant computation or insufficient recall; the features of images and natural language queries, two heterogeneous modalities, are often processed in independent semantic spaces, failing to achieve precise and fine-grained alignment, resulting in bias in cross-modal similarity calculation, making it difficult for the system to accurately understand and respond to users' complex semantic retrieval instructions. SUMMARY
[0003] To solve the above technical problems, a multi-view remote mixed reality image retrieval method and system are provided, which solve the above problems.
[0004] To achieve the above purposes, the technical scheme adopted by the present application is:
[0005] A multi-view remote mixed reality image retrieval method, comprising:
[0006] S1, based on the MR device, real-time capture the target scene main-view image stream, collect the user natural language query instruction, adaptively select the optimal auxiliary view image associated with the current scene semantics, realize multi-modal environment perception and context semantic enhancement;
[0007] S2, extract the target scene main-view image features, associate and pair the known local scene view feature library for preliminary retrieval, quantitatively evaluate the target scene main-view confidence, if the confidence is lower than the preset threshold, automatically trigger the accurate retrieval process;
[0008] S3, based on the automatically triggered accurate retrieval process, extract the target scene main-view image and auxiliary view image hierarchical features, use the attention mechanism network to generate the global scene multi-view image feature vector;
[0009] S4, based on the generated global scene multi-view image feature vector and the user natural language query instruction, jointly embed into a unified semantic space, calculate the cross-modal similarity with the known local scene view feature library, realize multi-view remote mixed reality image retrieval.
[0010] Preferably, step S1 specifically comprises:
[0011] Based on the remote sensor, a plurality of auxiliary perspective images of the target scene are collected, data preprocessing is performed, and an auxiliary perspective image set of the target scene is constructed;
[0012] Taking each auxiliary perspective image of the target scene as input, introducing ReLU as an activation function, designing a fully connected layer, establishing a CNN convolutional neural network model, and taking the feature vector of the auxiliary perspective image of the target scene as output through forward propagation;
[0013] Based on the feature vector of the auxiliary perspective image of the target scene, the corresponding remote sensor ID is combined to construct a local scene auxiliary perspective feature library;
[0014] A 24-hour fixed sliding window is set as a cycle, and the local scene auxiliary perspective feature library is continuously updated;
[0015] Based on the MR device, the current time main perspective image of the target scene is captured in real time, data preprocessing is performed, and a main perspective image sequence of the target scene is constructed;
[0016] Taking each main perspective image of the target scene as input, training the CNN convolutional neural network model, and executing forward propagation to take the feature vector of the main perspective image of the target scene as output.
[0017] Preferably, step S1 further comprises:
[0018] Based on the local scene auxiliary perspective feature library, the feature vector of the auxiliary perspective image of the target scene is extracted, and the dot product of the feature vector of the auxiliary perspective image of the target scene and the feature vector of the main perspective image of the target scene at the same time is calculated;
[0019] The L2 norm of the feature vector of the auxiliary perspective image of the target scene and the feature vector of the main perspective image of the target scene at the same time is calculated respectively;
[0020] The dot product of the feature vector of the auxiliary perspective image of the target scene and the feature vector of the main perspective image of the target scene at the same time is divided by the product of the L2 norm to obtain the cosine similarity of the feature vector of the auxiliary perspective image of the target scene and the feature vector of the main perspective image of the target scene at the same time, and the normalization processing is performed.
[0021] Preferably, step S1 further comprises:
[0022] The highest cosine similarity is selected as the optimal auxiliary perspective image of the current target scene semantic association, the corresponding remote sensor ID is taken as the source of the optimal auxiliary perspective, and a user natural language query instruction is generated;
[0023] Based on the user natural language query instruction, the current language query instruction video stream of the corresponding remote sensor is obtained in real time, and multi-modal environment perception and context semantic enhancement are realized.
[0024] Preferably, step S2 specifically comprises:
[0025] Based on the obtained target scene main view image features, the known local scene view feature library is associated and matched, the cosine similarity score of the target scene main view image features and the known local scene view feature library is calculated, and a cosine similarity score-scene ID pair list is obtained;
[0026] Based on the cosine similarity score-scene ID pair list, the cosine similarity score-scene ID pair list is sorted in descending order, and the top K is selected as the candidate list.
[0027] The confidence of the cosine similarity score-scene ID pair candidate list is calculated using the difference method to quantitatively evaluate the target scene main view confidence.
[0028] A preset threshold value of 0.15 is designed.
[0029] If the target scene main view confidence is less than the preset threshold value, the process is terminated without starting the accurate retrieval. If the target scene main view confidence is greater than the preset threshold value, the accurate retrieval process is automatically triggered.
[0030] Preferably, step S3 specifically comprises:
[0031] Based on the automatically triggered accurate retrieval process, the current target scene main view image is obtained.
[0032] Based on the cosine similarity score-scene ID pair candidate list, K auxiliary view images are obtained.
[0033] The current target scene main view image and the K auxiliary view images are taken as inputs to train the CNN convolutional neural network model, perform forward propagation, extract the hierarchical features of the current target scene main view image and the K auxiliary view images, and construct a hierarchical feature set of the target scene main view image and the K auxiliary view images.
[0034] Based on the hierarchical feature set of the target scene main view image and the K auxiliary view images, global average pooling is applied to obtain a same-dimension hierarchical feature vector sequence of the target scene main view image and the K auxiliary view images.
[0035] The same-dimension hierarchical feature vectors of the target scene main view image and the K auxiliary view images are spliced to obtain K target scene main-auxiliary view image comprehensive feature vectors.
[0036] The embedding vector of the user natural language query instruction is taken as the query vector, and the K target scene main-auxiliary view image comprehensive feature vectors are taken as the key vector and the value vector to design an attention mechanism network.
[0037] The similarity score of the embedding vector of the user natural language query instruction and each target scene main-auxiliary view image comprehensive feature vector is calculated, normalized by a Softmax function, and K target scene main-auxiliary view image comprehensive feature vector attention weights are obtained.
[0038] The K target scene main-auxiliary view image comprehensive feature vector attention weights are fused by using a weighted summation formula to obtain a global scene multi-view image feature vector.
[0039] The specific formula of the global scene multi-view image feature vector includes:
[0040]
[0041] In the formula, is the similarity score of the embedding vector of the user natural language query instruction and the i-th target scene main-auxiliary view image comprehensive feature vector, is the dot product of the embedding vector of the user natural language query instruction and the i-th target scene main-auxiliary view image comprehensive feature vector, is the dimension of the embedding vector of the user natural language query instruction and the i-th target scene main-auxiliary view image comprehensive feature vector; is the global scene multi-view image feature vector, is the total number of candidate auxiliary view images,
[0042] is the index of the target scene main-auxiliary view image comprehensive feature vector, is the i-th target scene main-auxiliary view image comprehensive feature vector attention weight, is a learnable projection weight matrix, is the i-th target scene main-auxiliary view image comprehensive feature vector. The specific formula of the global scene multi-view image feature vector includes:
[0043] Preferably, step S4 specifically includes:
[0044] Based on the generated global scene multi-view image feature vector, a global scene multi-view image feature is obtained.
[0045] Based on the user natural language query instruction, a user natural language query text is converted.
[0046] The user natural language query text is taken as input, a BERT pre-training language model is used, and a user natural language query text vector is taken as output to obtain a user natural language query text feature.
[0047] The global scene multi-view image features and the user natural language query text features are mapped to a unified semantic space by using a linear projection layer to generate a global scene multi-view image feature and user natural language query text feature sample set.
[0048] Preferably, the step S4 further comprises:
[0049] Based on the global scene multi-view image feature and user natural language query text feature sample set, the sample set is divided into positive samples and negative samples.
[0050] The positive sample is a semantic correlation global scene multi-view image feature and user natural language query text feature pair.
[0051] The negative sample is a user natural language query text feature and a non-matching image feature randomly selected from a known local scene view feature library.
[0052] For each user natural language query text feature, the similarity of the global scene multi-view image feature in the positive sample is calculated.
[0053] The similarity of the global scene multi-view image feature in the negative sample is calculated.
[0054] Based on the global scene multi-view image feature similarity, a minimum contrast loss function is introduced to calculate the cosine similarity of the user natural language query text feature and the known local scene view feature library, and the cosine similarity is sorted in descending order to realize multi-view remote mixed reality image retrieval.
[0055] Further, a multi-view remote mixed reality image retrieval system comprises:
[0056] A main view image stream module, a retrieval process module, a multi-view image feature vector module, and a multi-view remote mixed reality image retrieval module.
[0057] The main view image stream module is used to capture a target scene main view image stream in real time based on an MR device, collect a user natural language query instruction, and adaptively select an optimal auxiliary view image associated with the current scene semantics to realize multi-modal environment perception and context semantic enhancement.
[0058] The retrieval process module is electrically connected with the main view image stream module, extracts a target scene main view image feature, and preliminarily retrieves a known local scene view feature library in association, quantitatively evaluates a target scene main view confidence, and if the confidence is lower than a preset threshold, automatically triggers an accurate retrieval process.
[0059] The multi-view image feature vector module is electrically connected with the retrieval process module, based on automatic triggering of an accurate retrieval process, extracts hierarchical features of a target scene main-view image and an auxiliary-view image, generates a global scene multi-view image feature vector using an attention mechanism network;
[0060] The multi-view remote mixed reality image retrieval module is electrically connected with the multi-view image feature vector module and the main-view image stream module, based on the generated global scene multi-view image feature vector and the user natural language query instruction, jointly embedded in a unified semantic space, calculates the cross-modal similarity with a known local scene view feature library, and realizes multi-view remote mixed reality image retrieval.
[0061] Compared with the prior art, the beneficial effects of the present application are that:
[0062] The present application proposes a multi-view remote mixed reality image retrieval scheme, which realizes efficient and accurate multi-view remote mixed reality image retrieval through multi-modal environment perception and hierarchical feature fusion, and comprehensively enhances the context semantic understanding ability of the scene through the cooperative collection of the main-view image stream and the semantically associated optimal auxiliary view; the confidence evaluation and automatic triggering mechanism are introduced, which takes into account the retrieval efficiency and accuracy, and only starts the computationally intensive accurate retrieval when necessary, optimizing the allocation of system resources; by jointly embedding the multi-view global features and the user natural language query in a unified semantic space, using the attention mechanism and contrast learning, the image retrieval accuracy and robustness in complex cross-modal scenes are improved, providing users with a more intelligent and smooth mixed reality interactive experience. BRIEF DESCRIPTION OF DRAWINGS
[0063] Figure 1 It is a multi-view remote mixed reality image retrieval method flow chart;
[0064] Figure 2 It is a multi-view remote mixed reality image retrieval system framework diagram. DETAILED DESCRIPTION
[0065] The following description is used to disclose the present application so that those skilled in the art can implement the present application. The preferred embodiments in the following description are only as examples, and other obvious variants can be conceived by those skilled in the art.
[0066] REFERENCE Figure 1 As shown in the figure, a multi-view remote mixed reality image retrieval method comprises:
[0067] S1, based on the MR device, real-time capture of the target scene main-view image stream, collection of the user natural language query instruction, adaptive selection of the auxiliary-view image with the optimal semantic association with the current scene, realization of multi-modal environment perception and context semantic enhancement;
[0068] The step S1 specifically comprises:
[0069] Based on the remote sensor, a plurality of auxiliary perspective images of the target scene are collected, data preprocessing is performed thereon, and an auxiliary perspective image set of the target scene is constructed;
[0070] Taking each auxiliary perspective image of the target scene as input, introducing ReLU as an activation function, designing a fully connected layer, establishing a CNN convolutional neural network model, and taking the auxiliary perspective image feature vector of the target scene as output through forward propagation;
[0071] Based on the auxiliary perspective image feature vector of the target scene obtained, the corresponding remote sensor ID is combined to construct a local scene auxiliary perspective feature library;
[0072] A fixed sliding window of 24 hours is set as a period, and the local scene auxiliary perspective feature library is constantly updated;
[0073] Based on the MR device, a current time main perspective image of the target scene is captured in real time, data preprocessing is performed thereon, and a main perspective image sequence of the target scene is constructed;
[0074] Taking each main perspective image of the target scene as input, training the CNN convolutional neural network model, and executing forward propagation to take the main perspective image feature vector of the target scene as output.
[0075] The step S1 further comprises:
[0076] Based on the local scene auxiliary perspective feature library, the auxiliary perspective feature vector of the target scene is extracted, and the dot product of the auxiliary perspective feature vector of the target scene and the main perspective image feature vector of the target scene at the same time is calculated;
[0077] The L2 norm of the auxiliary perspective feature vector of the target scene and the main perspective image feature vector of the target scene at the same time is calculated respectively;
[0078] The dot product of the auxiliary perspective feature vector of the target scene and the main perspective image feature vector of the target scene at the same time is divided by the product of the L2 norm to obtain the cosine similarity of the auxiliary perspective feature vector of the target scene and the main perspective image feature vector of the target scene at the same time, and the normalization processing is performed.
[0079] The step S1 further comprises:
[0080] The highest cosine similarity is selected as the optimal auxiliary perspective image of the current target scene semantic association, the corresponding remote sensor ID is taken as the source of the optimal auxiliary perspective, and a user natural language query instruction is generated;
[0081] Based on the user natural language query instruction, the current language query instruction video stream of the corresponding remote sensor is obtained in real time, and multi-modal environment perception and context semantic enhancement are realized.
[0082] In use, in combination with the above:
[0083] In the prior art, the multi-modal environment perception system based on the MR device generally has defects such as lag in auxiliary view selection, weak semantic correlation, and insufficient context perception. Traditional methods mostly rely on fixed views or manual switching, lack of adaptive ability to dynamic scenes, high computational redundancy in feature extraction and matching process, resulting in low real-time interaction efficiency. This step introduces an adaptive auxiliary view selection mechanism, combines a CNN convolutional neural network model to extract a feature vector and calculate a cosine similarity, realizes efficient semantic matching of the main and auxiliary views, improves the accuracy and real-time performance of multi-modal perception, and enhances the timeliness and stability of the feature library through sliding window updating and normalization processing. Through optimal view selection and natural language instruction fusion, the context semantics are dynamically enhanced, effectively supporting intelligent decision-making and user experience optimization in immersive interaction scenarios.
[0084] S2, extract the target scene main view image features, associate and pair the known local scene view feature library for preliminary retrieval, quantitatively evaluate the target scene main view confidence, and if the confidence is lower than the preset threshold, automatically trigger the accurate retrieval process;
[0085] Step S2 specifically includes:
[0086] Based on the obtained target scene main view image features, the known local scene view feature library is associated and paired, the cosine similarity score of the target scene main view image features and the known local scene view feature library is calculated, and a cosine similarity score-scene ID pair list is obtained;
[0087] Based on the cosine similarity score-scene ID pair list, the list is sorted in descending order, the top K in the cosine similarity score-scene ID pair list are selected as candidate results, and a cosine similarity score-scene ID pair candidate list is constructed;
[0088] The confidence of the cosine similarity score-scene ID pair candidate list is calculated using the difference method, and the target scene main view confidence is quantitatively evaluated;
[0089] A preset threshold of 0.15 is designed;
[0090] If the target scene main view confidence the preset threshold, the process is terminated without starting the accurate retrieval, and if the target scene main view confidence the preset threshold, the accurate retrieval process is automatically triggered.
[0091] In use, in combination with the above:
[0092] In the current scene retrieval technology, there are high false matching rate, significant interference of low-quality candidate set and rough confidence evaluation mechanism due to the dependence on single similarity ranking, which makes the system prone to retrieval drift and early termination in complex lighting, view change or local occlusion scene, restricting the overall retrieval accuracy and adaptability; this step improves the robustness and discriminability of preliminary retrieval by fusing the cosine similarity correlation pairing of multiple candidate view angles, dynamic K value candidate list construction and confidence quantification mechanism based on difference method; the threshold comparison can effectively filter high certainty scenes to reduce computational overhead, and the automatic triggering of accurate retrieval can cope with boundary cases, ensuring system efficiency while enhancing adaptability to complex scenes and reliability of the overall retrieval process.
[0093] S3, based on the automatic triggering of the accurate retrieval process, extracting the hierarchical features of the target scene main view image and the auxiliary view image, using the attention mechanism network to generate the global scene multi-view image feature vector;
[0094] Step S3 includes:
[0095] Based on the automatic triggering of the accurate retrieval process, the current target scene main view image is obtained.
[0096] Based on the cosine similarity score-scene ID pair candidate list, K auxiliary view images are obtained.
[0097] The current target scene main view image and the K auxiliary view images are taken as input to train the CNN convolutional neural network model, perform forward propagation, extract the hierarchical features of the current target scene main view image and the K auxiliary view images, and construct the hierarchical feature set of the target scene main view image and the K auxiliary view images.
[0098] Based on the hierarchical feature set of the target scene main view image and the K auxiliary view images, global average pooling is applied to obtain the same dimension hierarchical feature vector sequence of the target scene main view image and the K auxiliary view images.
[0099] The same dimension hierarchical feature vectors of the target scene main view image and the K auxiliary view images are spliced to obtain K target scene main-auxiliary view image comprehensive feature vectors.
[0100] The embedding vector of the user natural language query instruction is taken as the query vector, and the K target scene main-auxiliary view image comprehensive feature vectors are taken as the key vector and the value vector to design the attention mechanism network.
[0101] The similarity score between the embedding vector of the user natural language query instruction and each target scene main-auxiliary view image comprehensive feature vector is calculated, and the Softmax function is used for normalization to obtain the attention weight of the K target scene main-auxiliary view image comprehensive feature vectors.
[0102] By using a weighted summation formula, attention weights of the comprehensive feature vectors of the main and secondary viewpoint images of the target scene are fused to obtain the global scene multi-view image feature vector.
[0103] The specific formula for the feature vector of the global scene multi-view image is as follows:
[0104]
[0105] In the formula, The embedding vector of the user's natural language query command and the first Similarity score of combined feature vectors of main and secondary viewpoint images of a target scene. The embedding vector values of the user's natural language query command and the first The dot product of the combined feature vectors of the main and secondary viewpoint images of the target scene. The embedding vector of the user's natural language query command and the first The dimension of the comprehensive feature vector of the main and secondary view images of a target scene;
[0106] This is a feature vector of a global scene from multiple perspectives. The total number of candidate auxiliary viewpoint images. Indexing of the combined feature vectors of the main and secondary viewpoint images of the target scene. For the first Attention weights for the integrated feature vectors of the main and secondary viewpoint images of the target scene. For learnable projective weight matrix, For the first A comprehensive feature vector of the main and secondary viewpoint images of the target scene.
[0107] When using it, please refer to the above:
[0108] Current cross-modal retrieval technologies for vision and language generally suffer from drawbacks such as single perspective, insufficient feature extraction granularity, and inadequate fusion of global contextual information. Most methods rely solely on single-view images or simple feature stitching, failing to effectively utilize the complementarity of multi-view images and lacking a dynamic attention weighting mechanism tailored to user queries. This results in limited understanding and retrieval accuracy in complex scenes. This step automatically triggers a precise retrieval process that fuses hierarchical features from the main view and multiple auxiliary view images. It introduces a user query-oriented attention mechanism, dynamically calculates the feature weights of each view, and then fuses them, enhancing the ability to represent the global semantics of the scene. This effectively improves the accuracy and robustness of cross-modal retrieval, especially demonstrating significant advantages in complex scenes with multiple perspectives and multi-object interactions.
[0109] S4, embedding the global scene multi-view image feature vector and the user natural language query instruction into a unified semantic space based on the generated global scene multi-view image feature vector and the user natural language query instruction, calculating the cross-modal similarity with the known local scene view feature library, and realizing multi-view remote mixed reality image retrieval;
[0110] Step S4 specifically includes:
[0111] Based on the generated global scene multi-view image feature vector, the global scene multi-view image feature is obtained.
[0112] Based on the user natural language query instruction, the user natural language query text is converted.
[0113] Taking the user natural language query text as input, using the BERT pre-training language model, taking the user natural language query text vector as output, and obtaining the user natural language query text feature.
[0114] The global scene multi-view image feature and the user natural language query text feature are mapped to a unified semantic space by using a linear projection layer, and a global scene multi-view image feature and a user natural language query text feature sample set are generated.
[0115] Step S4 further includes:
[0116] Based on the global scene multi-view image feature and the user natural language query text feature sample set, the sample set is divided into positive samples and negative samples.
[0117] The positive sample is: a semantic correlation global scene multi-view image feature and a user natural language query text feature pair.
[0118] The negative sample is: a user natural language query text feature and a randomly selected non-matching image feature from the known local scene view feature library.
[0119] For each user natural language query text feature, the similarity with the global scene multi-view image feature in the positive sample is calculated.
[0120] The similarity with the global scene multi-view image feature in the negative sample is calculated.
[0121] Based on the global scene multi-view image feature similarity, a minimum contrast loss function is introduced, the cosine similarity between the user natural language query text feature and the known local scene view feature library is calculated, and the cosine similarity is sorted in descending order, realizing multi-view remote mixed reality image retrieval.
[0122] In use, in combination with the above content:
[0123] In the prior art, the cross-modal retrieval method based on multi-view images and natural language queries often has the following defects: most methods rely on single view or local features, lack of multi-view collaborative modeling of global scene, resulting in limited retrieval accuracy; traditional cross-modal alignment methods do not fully optimize semantic space mapping, easily introducing noise, affecting similarity calculation accuracy; negative sample construction mostly adopts simple random strategy, which is difficult to effectively distinguish the semantic association in complex scenes, restricting the model discrimination ability; this step generates unified semantic space embedding of global scene multi-view image features and user query text features, introduces a contrast learning mechanism, and multi-view feature fusion can comprehensively capture scene context information, improving the robustness and coverage of retrieval; the ranking mechanism based on cosine similarity and the positive and negative sample optimization strategy enhance the cross-modal semantic alignment capability and improve the accuracy and efficiency of image retrieval under complex queries, providing reliable support for remote mixed reality applications.
[0124] Referring to Figure 2 A multi-view remote mixed reality image retrieval system is shown, comprising:
[0125] a main view image stream module, a retrieval process module, a multi-view image feature vector module, and a multi-view remote mixed reality image retrieval module;
[0126] The main view image stream module is used to capture real-time target scene main view image stream based on the MR device, collect user natural language query instructions, adaptively select the optimal auxiliary view image associated with the current scene semantics, and realize multi-modal environment perception and context semantic enhancement.
[0127] The retrieval process module and the main view image stream module are electrically connected, the target scene main view image features are extracted, the known local scene view feature library is associated and paired for preliminary retrieval, the target scene main view confidence is quantitatively evaluated, and if the confidence is lower than the preset threshold, the accurate retrieval process is automatically triggered.
[0128] The multi-view image feature vector module and the retrieval process module are electrically connected, based on the automatically triggered accurate retrieval process, the target scene main view image and auxiliary view image hierarchical features are extracted, and the attention mechanism network is used to generate global scene multi-view image feature vectors.
[0129] The multi-view remote mixed reality image retrieval module is electrically connected with the multi-view image feature vector module and the main view image stream module, based on the generated global scene multi-view image feature vectors and user natural language query instructions, which are embedded in a unified semantic space, the cross-modal similarity with the known local scene view feature library is calculated, and multi-view remote mixed reality image retrieval is realized.
[0130] In combination with the above step content, the specific implementation is as follows:
[0131] An engineer wears MR glasses to inspect a large industrial equipment, the MR device captures the main perspective image in the engineer's field of view in real time, such as the front panel of the equipment, the system calls the images taken by the two auxiliary cameras deployed on the top and side of the equipment;
[0132] After extracting features and calculating cosine similarity through the CNN convolutional neural network model, the system finds that the top camera's perspective can overlook the entire equipment structure and has the highest semantic correlation with the engineer's current main perspective, so the system automatically floats the real-time video stream of the top camera in the form of picture-in-picture in the corner of the MR field of view, and synchronously receives the natural language query spoken by the engineer: check the status of the cooling system pressure valve, realizing multi-modal environment perception and context semantic enhancement, and providing key information for subsequent retrieval.
[0133] The system compares the image features of the engineer's main perspective with the locally pre-stored equipment scene feature library, calculates the cosine similarity, and obtains a candidate list containing scene A normal operation and scene B maintenance;
[0134] Through difference method calculation, the confidence of the current main perspective is only 0.12, which is lower than the preset threshold of 0.15. This shows that relying solely on the main perspective cannot clearly match to a unique or high-confidence scene, possibly because the perspective is incomplete or has local occlusion, so the system automatically triggers the precise retrieval process, preparing to use multi-perspective information for more in-depth analysis.
[0135] Based on the precise retrieval process, the system uses the engineer's main perspective image, calls the top and side camera pictures of the top K candidate auxiliary perspective images with the highest score in the preliminary retrieval, and the CNN convolutional neural network model extracts the hierarchical features of the three perspectives, performs pooling and splicing, and forms two main-auxiliary perspective image comprehensive feature vectors;
[0136] Taking the text vector of the engineer's query to check the status of the cooling system pressure valve as a guide, it is found that the features of the top camera are much higher than those of the side camera, and the system assigns a very high attention weight to the feature vector corresponding to the top camera, and fuses to generate a global scene multi-perspective image feature vector that can accurately reflect the status of the cooling system pressure valve.
[0137] The system projects the generated global multi-perspective image feature vector and the BERT pre-training language model feature vector of the user query text to check the status of the cooling system pressure valve into a unified semantic space;
[0138] By optimizing the contrast learning loss, the system calculates the cosine similarity between the query text and all historical scene image features in the feature library, and the retrieval results are sorted in descending order of similarity. In the MR glasses, the image and maintenance log that best match the pressure valve reading abnormal history record are clearly highlighted, and the multi-view remote mixed reality image retrieval task is completed, providing precise support for the engineer's decision.
[0139] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection required by the present application is defined by the appended claims and their equivalents.
Claims
1. A multi-view remote mixed reality image retrieval method, characterized by, The application comprises the following steps: S1, based on the MR device, real-time capture target scene main view image stream, collect user natural language query instruction, adaptive select the optimal auxiliary view image associated with the current scene semantics, realize multi-modal environment perception and context semantic enhancement; S2, extract target scene main view image features, associate and match known local scene view feature library for preliminary retrieval, quantitatively evaluate target scene main view confidence, if the confidence is lower than the preset threshold, automatically trigger the accurate retrieval process; S3, based on the automatic triggering of the accurate retrieval process, extract the hierarchical features of the target scene main view image and the auxiliary view image, generate the global scene multi-view image feature vector by using the attention mechanism network; S4, based on the generated global scene multi-view image feature vector and the user natural language query instruction, embed them into a unified semantic space, calculate the cross-modal similarity with the known local scene view feature library, and realize multi-view remote mixed reality image retrieval.
2. The multi-view remote mixed reality image retrieval method of claim 1, wherein, Step S1 specifically comprises: Based on remote sensors, collect multiple auxiliary view images of the target scene, preprocess the data, and construct a target scene auxiliary view image set; Taking each target scene auxiliary view image as input, introducing ReLU as activation function, designing full connection layer, establishing CNN convolutional neural network model, through forward propagation, taking target scene auxiliary view image feature vector as output; Based on the obtained target scene auxiliary view image feature vector, combined with the corresponding remote sensor ID, construct a local scene auxiliary view feature library; Set a 24-hour fixed sliding window as a cycle, and constantly update the local scene auxiliary view feature library; Based on the MR device, real-time capture target scene current time main view image, preprocess the data, and construct target scene main view image sequence; Taking each target scene main view image as input, training CNN convolutional neural network model, executing forward propagation, taking target scene main view image feature vector as output. 3.The multi-view remote mixed reality image retrieval method of claim 2, wherein, Step S1 further comprises: Based on the local scene auxiliary view feature library, extract the target scene auxiliary view feature vector, calculate the dot product of the target scene auxiliary view feature vector and the target scene main view image feature vector at the same time; Calculate the L2 norm of the target scene auxiliary view feature vector and the target scene main view image feature vector at the same time, respectively; Divide the dot product of the target scene auxiliary view feature vector and the target scene main view image feature vector at the same time by the product of the L2 norm, to obtain the cosine similarity of the target scene auxiliary view feature vector and the target scene main view image feature vector at the same time, and perform normalization processing.
4. The multi-view remote mixed reality image retrieval method of claim 3, wherein, Step S1 further comprises: Select the highest cosine similarity as the optimal auxiliary view image associated with the current target scene semantics, take the corresponding remote sensor ID as the source of the optimal auxiliary view, and generate the user natural language query instruction; Based on the user natural language query instruction, real-time acquire the current language query instruction video stream of the corresponding remote sensor, realize multi-modal environment perception and context semantic enhancement.
5. The multi-view remote mixed reality image retrieval method of claim 4, wherein, Step S2 specifically comprises: Based on the obtained target scene main view image features, the known local scene view feature library is associated and matched, the cosine similarity scores of the target scene main view image features and the known local scene view feature library are calculated, and a cosine similarity score-scene ID pair list is obtained; Based on the cosine similarity score-scene ID pair list, the cosine similarity score-scene ID pair list is sorted in descending order, and the top K cosine similarity score-scene ID pairs in the list are selected as candidate results to construct a cosine similarity score-scene ID pair candidate list; The confidence of the cosine similarity score-scene ID pair candidate list is calculated using the difference method to quantitatively evaluate the target scene main view confidence. A preset threshold of 0.15 is designed. if the target scene main view angle confidence preset threshold, the process is terminated without starting the accurate retrieval, if the target scene main view angle confidence preset threshold, the accurate retrieval process is automatically triggered.
6. The multi-view remote mixed reality image retrieval method of claim 5, wherein, Step S3 specifically includes: Based on the automatic trigger accurate retrieval process, the current target scene main view image is obtained; Based on the cosine similarity score-scene ID pair candidate list, K auxiliary view images are obtained; The current target scene main view image and the K auxiliary view images are taken as inputs to train a CNN convolutional neural network model, perform forward propagation, extract hierarchical features of the current target scene main view image and the K auxiliary view images, and construct a hierarchical feature set of the target scene main view image and the K auxiliary view images. Based on the hierarchical feature set of the target scene main view image and the K auxiliary view images, global average pooling is applied to obtain the same dimension hierarchical feature vector sequence of the target scene main view image and the K auxiliary view images. The same dimension hierarchical feature vectors of the target scene main view image and the K auxiliary view images are spliced to obtain K target scene main-auxiliary view image comprehensive feature vectors. The embedding vector of the user natural language query instruction is taken as a query vector, and the K target scene main-auxiliary view image comprehensive feature vectors are taken as key vectors and value vectors to design an attention mechanism network. The similarity scores of the embedding vector of the user natural language query instruction and each target scene main-auxiliary view image comprehensive feature vector are calculated, normalized by a Softmax function, and K target scene main-auxiliary view image comprehensive feature vector attention weights are obtained. The K target scene main-auxiliary view image comprehensive feature vector attention weights are fused by a weighted sum formula to obtain a global scene multi-view image feature vector.
7. The multi-view remote mixed reality image retrieval method of claim 6, wherein, The specific formula of the global scene multi-view image feature vector includes: In the formula, The embedding vector of the user's natural language query command and the first Similarity score of combined feature vectors of main and secondary viewpoint images of a target scene. The embedding vector values of the user's natural language query command and the first The dot product of the combined feature vectors of the main and secondary viewpoint images of the target scene. The embedding vector of the user's natural language query command and the first The dimension of the comprehensive feature vector of the main and secondary view images of a target scene; This is a feature vector of a global scene from multiple perspectives. The total number of candidate auxiliary viewpoint images. Indexing of the combined feature vectors of the main and secondary viewpoint images of the target scene. For the first Attention weights for the integrated feature vectors of the main and secondary viewpoint images of the target scene. For learnable projective weight matrix, For the first A comprehensive feature vector of the main and secondary viewpoint images of the target scene. 8.The multi-view remote mixed reality image retrieval method of claim 7, wherein, Step S4 specifically includes: Based on the generated global scene multi-view image feature vector, a global scene multi-view image feature is obtained; Based on the user natural language query instruction, a user natural language query text is converted; The user natural language query text is taken as input, and the BERT pre-training language model is used to obtain the user natural language query text feature with the user natural language query text vector as output. The global scene multi-view image feature and the user natural language query text feature are mapped to a unified semantic space by using a linear projection layer to generate a global scene multi-view image feature and a user natural language query text feature sample set. 9.The multi-view remote mixed reality image retrieval method of claim 8, wherein, Step S4 further includes: Based on the global scene multi-view image feature and the user natural language query text feature sample set, the sample set is divided into positive samples and negative samples; The positive sample is a feature pair of semantic association global scene multi-view image feature and user natural language query text feature; The negative sample is a feature pair of user natural language query text feature and randomly selected non-matching image feature from the known local scene view feature library; For each user natural language query text feature, the similarity with the global scene multi-view image feature in the positive sample is calculated; The similarity with the global scene multi-view image feature in the negative sample is calculated; Based on the global scene multi-view image feature similarity, a minimum contrast loss function is introduced to calculate the cosine similarity between the user natural language query text feature and the known local scene view feature library, and the cosine similarity is sorted in descending order to realize multi-view remote mixed reality image retrieval.
10. A multi-view remote mixed reality image retrieval system, characterized in that, To realize any one of claims 1-9, a multi-view remote mixed reality image retrieval method comprises: A main view image stream module, a retrieval process module, a multi-view image feature vector module, and a multi-view remote mixed reality image retrieval module; The main view image stream module is used to capture the target scene main view image stream in real time based on the MR device, collect the user natural language query instruction, and adaptively select the optimal auxiliary view image associated with the current scene semantics to realize multi-modal environment perception and context semantic enhancement; The retrieval process module is electrically connected with the main view image stream module, extracts the target scene main view image feature, and correlates and pairs the known local scene view feature library for preliminary retrieval, quantitatively evaluates the target scene main view confidence, and if the confidence is lower than the preset threshold, automatically triggers the accurate retrieval process; The multi-view image feature vector module is electrically connected with the retrieval process module, extracts the hierarchical features of the target scene main view image and auxiliary view image based on the automatic triggering of the accurate retrieval process, and generates the global scene multi-view image feature vector using the attention mechanism network; The multi-view remote mixed reality image retrieval module is electrically connected with the multi-view image feature vector module and the main view image stream module, and based on the generated global scene multi-view image feature vector and the user natural language query instruction, it is embedded in a unified semantic space to calculate the cross-modal similarity with the known local scene view feature library, and realizes multi-view remote mixed reality image retrieval.