Training method of implicit problem inference model and implicit problem inference method

Abstract

The invention provides a training method of an implicit problem reasoning model and an implicit problem reasoning method, which can be applied to the field of artificial intelligence, and the training method comprises the following steps: obtaining a reasoning training set; constructing a three-dimensional model of the target area according to the scene video; inputting the scene video and the implicit question into a visual language model, and outputting semantic features; the scene video and the semantic features are input into a segmentation basic model, a plurality of mask images are output, and mask labels are formed on a plurality of initial objects corresponding to the implicit question by the mask images; processing the plurality of mask images and the semantic features by using a perceptual fusion model to obtain target fusion features; the three-dimensional model, the semantic features and the target fusion features are processed through a preset language model, a reasoning answer is output, and the initial reasoning model comprises a visual language model, a segmentation basic model, a perception fusion model and the preset language model; and adjusting model parameters of the initial reasoning model according to the reasoning answer and the answer label to obtain a target reasoning model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a training method for an implicit question reasoning model and an implicit question reasoning method. Background Art

[0002] Existing question reasoning mainly reasons about explicit instruction type questions, which require clearly pointing out the object or type to reconstruct the area related to the instruction. However, humans tend to directly express needs involving implicit types and commonalities. For example, "I'm hungry, find me something to eat" or "Find a small sanitation facility suitable for children" instead of providing clear step-by-step instructions.

[0003] In addition, in an unfamiliar environment, people don't know what objects exist, so they can only give implicit instructions. Since the existing explicit question reasoning methods have limited interaction and reasoning capabilities between the target object and the instruction, the accuracy and reasoning efficiency of implicit question reasoning are not very ideal, resulting in a poor user experience. Summary of the Invention

[0004] In view of this, the present application provides a training method for an implicit question reasoning model and an implicit question reasoning method.

[0005] One aspect of the present application provides a training method for an implicit question reasoning model, including: obtaining an inference training set, where the inference training set includes multiple training samples, each training sample includes a scene video of a target area and multiple scene instruction pairs, the scene instruction pairs include multiple implicit questions for the target area and answer labels corresponding to each implicit question, and the implicit question is a implicit expression of the target object to be found; for each training sample, constructing a three-dimensional model of the target area according to the scene video; inputting the scene video and the implicit question into a vision-language model to output semantic features; inputting the scene video and the semantic features into a segmentation base model to output multiple mask images, where mask labels are formed on multiple initial objects corresponding to the implicit question on the mask images; processing the multiple mask images and the semantic features by using a perception fusion model to obtain a target fusion feature, where the target fusion feature represents the multiple initial objects after fusion; processing the three-dimensional model, the semantic features and the target fusion feature by using a preset language model to output an inference answer, and the initial inference model includes a vision-language model, a segmentation base model, a perception fusion model and a preset language model; adjusting the model parameters of the initial inference model according to the inference answer and the answer label to obtain a target inference model.

[0006] Another aspect of the present application provides an implicit question reasoning method, including: obtaining the regional video of the area to be reasoned and the question to be reasoned; constructing a three-dimensional regional model of the area to be reasoned according to the regional video; inputting the question to be reasoned and the three-dimensional regional model into a target reasoning model, and outputting a reasoning answer corresponding to the question to be reasoned, where the reasoning answer represents the position information of the required object corresponding to the implicit question.

[0007] Another aspect of the present application provides a training device for an implicit question reasoning model, including:

[0008] A first acquisition module, configured to acquire a reasoning training set, where the reasoning training set includes a plurality of training samples, each training sample includes the scene video of the target area and a plurality of scene instruction pairs, the scene instruction pairs include a plurality of implicit questions for the target area and answer labels corresponding to each implicit question, and the implicit question is a veiled expression of the target object to be searched;

[0009] A first construction module, configured to construct a three-dimensional model of the target area according to the scene video for each training sample;

[0010] A first obtaining module, configured to input the scene video and the implicit question into a vision-language model, and output semantic features;

[0011] A segmentation module, configured to input the scene video and the semantic features into a segmentation base model, and output a plurality of mask images, where the mask images form mask labels on a plurality of initial objects corresponding to the implicit question;

[0012] A second obtaining module, configured to process the plurality of mask images and the semantic features by using a perception fusion model to obtain target fusion features, where the target fusion features represent a plurality of fused initial objects;

[0013] A first reasoning module, configured to process the three-dimensional model, the semantic features, and the target fusion features by using a preset language model, and output a reasoning answer, and the initial reasoning model includes a vision-language model, a segmentation base model, a perception fusion model, and a preset language model;

[0014] An adjustment module, configured to adjust the model parameters of the initial reasoning model according to the reasoning answer and the answer label to obtain a target reasoning model.

[0015] Another aspect of the present application provides an implicit question reasoning device, including:

[0016] A second acquisition module, configured to acquire the regional video of the area to be reasoned and the question to be reasoned;

[0017] A second construction module, configured to construct a three-dimensional regional model of the area to be reasoned according to the regional video;

[0018] A second inference module, configured to input a problem to be inferred and a three-dimensional region model into a target inference model, and output an inference answer corresponding to the problem to be inferred, where the inference answer represents the position information of a demand object corresponding to an implicit problem.

[0019] Another aspect of the present application provides a humanoid robot, including the above-mentioned implicit problem inference device, an image acquisition device, a display device, and an input device.

[0020] Another aspect of the present application provides an electronic device, including:

[0021] One or more processors;

[0022] A memory for storing one or more programs,

[0023] wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described above.

[0024] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method as described above when executed.

[0025] Another aspect of the present application provides a computer program product, which includes computer-executable instructions, and the instructions are used to implement the method as described above when executed.

[0026] According to an embodiment of the present application, for each training sample, a three-dimensional model of a target region is constructed according to a scenario video; the scenario video and an implicit problem are input into a vision-language model to output semantic features; the scenario video and the semantic features are input into a segmentation base model to output a plurality of mask images, and a perception fusion model is used to process the plurality of mask images and the semantic features to obtain target fusion features, and a preset language model is used to process the three-dimensional model, the semantic features, and the target fusion features to output an inference answer, and the model parameters of an initial inference model are adjusted according to the inference answer and an answer label to obtain a target inference model. Since different features are fused using a perception fusion model during the inference process of implicit problems, the accuracy and efficiency of implicit problem inference are improved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Through the following description of the embodiments of the present application with reference to the accompanying drawings, the above and other objects, features, and advantages of the present application will become clearer.

[0028] Figure 1 FIG. shows an exemplary system architecture to which the training method or the inference method of the implicit problem inference model according to the embodiment of the present application can be applied.

[0029] Figure 2 The flowchart of the training method of the implicit problem reasoning model according to an embodiment of the present application is shown.

[0030] Figure 3 The flowchart of the generation of the inference training set according to an embodiment of the present application is shown.

[0031] Figure 4 The flowchart of the generation of the inference answer according to an embodiment of the present application is shown.

[0032] Figure 5 The flowchart of the generation of the target fusion feature according to an embodiment of the present application is shown.

[0033] Figure 6 The flowchart of the implicit problem reasoning method according to an embodiment of the present application is shown.

[0034] Figure 7 The schematic diagram of the scenario of the implicit problem reasoning method according to an embodiment of the present application is shown.

[0035] Figure 8 The block diagram of the training device of the implicit problem reasoning model according to an embodiment of the present application is shown.

[0036] Figure 9 The block diagram of the implicit problem reasoning device according to an embodiment of the present application is shown.

[0037] Figure 10 The humanoid robot applying the implicit problem reasoning device according to an embodiment of the present application is shown.

[0038] Figure 11 The block diagram of the electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Detailed implementation manners

[0039] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In the following detailed description, for the sake of explanation, many specific details are set forth in order to provide a comprehensive understanding of the embodiments of the present application. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present application.

[0040] The terms used herein are merely for describing specific embodiments and are not intended to limit the present application. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0041] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0042] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning that those of ordinary skill in the art usually understand this expression (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0043] In the embodiments of the present application, in terms of the collection, update, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the involved data (for example, including but not limited to user personal information), they all comply with the provisions of relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and to maintain the security of user personal information and network security.

[0044] The embodiments of the present application provide a training method for an implicit question reasoning model and an implicit question reasoning method. The training method includes obtaining an inference training set, where the inference training set includes a plurality of training samples, each training sample includes a scene video of a target area and a plurality of scene instruction pairs, the scene instruction pairs include a plurality of implicit questions for the target area and answer labels corresponding to each implicit question, and the implicit question is an implicit expression of the target object to be searched; for each training sample, constructing a three-dimensional model of the target area according to the scene video; inputting the scene video and the implicit question into a vision-language model to output semantic features; inputting the scene video and the semantic features into a segmentation basic model to output a plurality of mask images, where mask labels are formed on a plurality of initial objects corresponding to the implicit questions on the mask images; using a perception fusion model to process the plurality of mask images and the semantic features to obtain a target fusion feature, where the target fusion feature represents the plurality of initial objects after fusion; using a preset language model to process the three-dimensional model, the semantic features, and the target fusion feature to output an inference answer, and the initial inference model includes a vision-language model, a segmentation basic model, a perception fusion model, and a preset language model; adjusting the model parameters of the initial inference model according to the inference answer and the answer label to obtain a target inference model.

[0045] Figure 1 An exemplary system architecture to which the training method for an implicit question reasoning model or the implicit question reasoning method according to the embodiments of the present application can be applied is shown. It should be noted that Figure 1The illustration is only an example of the system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0046] As Figure 1 shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0047] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0048] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop computers, and desktop computers, etc.

[0049] The server 105 may be a server that provides various services, such as a background management server that supports the websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server may analyze and process data such as user requests received, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0050] It should be noted that the training method of the implicit problem inference model and the implicit problem inference method provided by the embodiments of the present application can generally be executed by the server 105. Correspondingly, the training device and the implicit problem inference device of the implicit problem inference model provided by the embodiments of the present application can generally be set in the server 105. The training method of the implicit problem inference model and the implicit problem inference method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the training device and the implicit problem inference device of the implicit problem inference model provided by the embodiments of the present application can also be set in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Alternatively, the training method of the implicit problem inference model and the implicit problem inference method provided by the embodiments of the present application can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or can also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the training device and the implicit problem inference device of the implicit problem inference model provided by the embodiments of the present application can also be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or can be set in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0051] It should be understood that Figure 1 the numbers of terminal devices, networks, and servers in

[0052] Figure 2 shows a flowchart of a training method of an implicit problem inference model according to an embodiment of the present application.

[0053] As Figure 2 shown, the training method of the implicit problem inference model includes operation S210 to operation S270.

[0054] In operation S210, an inference training set is obtained, where the inference training set includes a plurality of training samples, each training sample includes a scene video of a target area and a plurality of scene instruction pairs, the scene instruction pair includes a plurality of implicit problems for the target area and an answer label corresponding to each implicit problem, and the implicit problem is an implicit expression of the target object to be searched.

[0055] In operation S220, for each training sample, a three-dimensional model of the target area is constructed according to the scene video.

[0056] In operation S230, the scene video and the implicit question are input into the vision-language model, and semantic features are output.

[0057] In operation S240, the scene video and the semantic features are input into the segmentation-based model, and multiple masked images are output, where masked labels are formed on multiple initial objects corresponding to the implicit question on the masked images.

[0058] In operation S250, the multiple masked images and the semantic features are processed by the perception fusion model to obtain target fusion features, where the target fusion features represent the multiple initial objects after fusion.

[0059] In operation S260, the 3D model, the semantic features, and the target fusion features are processed by the preset language model to output an inference answer. The initial inference model includes the vision-language model, the segmentation-based model, the perception fusion model, and the preset language model.

[0060] In operation S270, the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain the target inference model.

[0061] According to the embodiments of the present application, an explicit question is a direct expression of a user's certain need, and the target object required by the user can be directly understood from the explicit question. An implicit question is a veiled expression of a user's certain need, and it is relatively difficult to understand the target object required by the user from the implicit question. For example, the explicit question is "I need a water cup", and it can be seen from this explicit question that the target object required by the user is a water cup. The implicit question is "I'm thirsty and want to drink water". At this time, the actual target object the user wants is a water cup, but the expression is more veiled. The implicit question inference model of this embodiment can be applied to a comprehensive shopping mall to infer answers when the user's implicit questions are "I'm hungry", "I'm thirsty", "My son is a bit cold", etc.

[0062] According to the embodiments of the present application, a vision-language model (VLM) is a multi-modal generative artificial intelligence system that integrates vision and language processing capabilities, can understand image, video, and text data, and generate natural language responses. The vision-language model of this embodiment can be replaced by other multi-modal large models.

[0063] According to an embodiment of the present application, the segmentation base model is mainly used to distinguish different objects or regions in an image. For example, it can be models such as U-Net, Segment Anything Model (SAM), Structure from Motion (SFM), and Diffusion Model (DM). The preset language model can refer to a large language model (LLM).

[0064] According to an embodiment of the present application, during the training process of the implicit question reasoning model, first obtain the inference training set used for training. Based on each training sample in the inference training set, construct a three-dimensional model of the target region according to the scene video in the training sample. For example, if the scene video is a video of a certain shopping mall being filmed, a three-dimensional model of the shopping mall can be constructed based on this video.

[0065] According to an embodiment of the present application, input the scene video and the implicit question into the vision-language model to output semantic features. The semantic features can be attribute features of the target object, such as color, type, size, etc. Input the scene video and the semantic features into the segmentation base model to output multiple mask images. On these mask images, mask labels are marked on multiple initial objects of the target object to be searched for according to the implicit question. Among the multiple initial objects, several initial objects may essentially be the same object, but may be divided into different initial objects due to the perspective. At this time, the perceptual fusion model can process the multiple mask images and the semantic features to obtain the target fusion features.

[0066] According to an embodiment of the present application, input the above three-dimensional model, semantic features, and target fusion features into the preset language model for reasoning to output a reasoning answer. For example, the reasoning answer can inform the user of the specific location of the target object they need by marking on an image, or it can inform the user in text form.

[0067] According to an embodiment of the present application, input the reasoning answer and the answer label into the loss function to obtain a loss result. According to the loss result, adjust the model parameters of the initial reasoning model. After several parameter adjustments using multiple training samples, the target reasoning model, that is, the implicit question reasoning model, can be obtained.

[0068] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and the implicit question are input into the vision-language model to output semantic features; the scene video and the semantic features are input into the segmentation basic model to output multiple mask images, and the perceptual fusion model is used to process the multiple mask images and the semantic features to obtain the target fusion features. The preset language model is used to process the three-dimensional model, the semantic features and the target fusion features to output the inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain the target inference model. Since different features are fused by using the perceptual fusion model during the inference process of the implicit question, the accuracy and efficiency of the inference of the implicit question are improved, and thus the user experience is improved.

[0069] Figure 3 FIG. shows a flowchart of generating an inference training set according to an embodiment of the present application.

[0070] According to an embodiment of the present application, the inference training set is generated in the following manner: obtaining an information set of the target area; projecting the three-dimensional point cloud onto the two-dimensional image space to obtain perspective images from different perspectives; inputting the multiple perspective images into the segmentation basic model to obtain segmentation mask labels corresponding to different objects; inputting the multiple perspective images into the vision-language model to output object attribute labels corresponding to different objects; inputting the multiple perspective images, the multiple segmentation mask labels and the multiple object attribute labels into the preset language model to output multiple scene instruction pairs; and generating an inference training set according to the scene video and the multiple scene instruction pairs.

[0071] According to an embodiment of the present application, referring to Figure 3 , when constructing the inference training set, first, the three-dimensional point cloud and the scene video of the target area can be captured by a shooting device, or can be obtained from the existing ScanNetV2 dataset. For the three-dimensional point cloud, it can be projected onto the two-dimensional image space, and after the projection, the point cloud that cannot be seen from this perspective can be removed. For example, in the front view of a cup, since the back of the cup cannot be seen, the point cloud of the back can be deleted at this time, so as to obtain perspective images from k different perspectives (viewpoints).

[0072] According to an embodiment of the present application, the perspective images from multiple perspectives are input into the segmentation basic model to mark different objects (such as bathtubs, toilets, etc.), so as to obtain segmentation mask labels corresponding to different objects. At the same time, the multiple perspective images can also be input into the vision-language model to obtain object attribute labels of different objects, such as attribute labels of color, size, category and position. If the object attribute labels of different perspective images are different, the final object attribute label can be determined through a voting mechanism.

[0073] According to an embodiment of the present application, multiple perspective images, multiple segmentation mask labels, and multiple object attribute labels are input into a preset language model, and multiple scene instruction pairs are output. The implicit questions in the scene instruction pairs are such as "I'm feeling a bit uncomfortable in my stomach and want to go to the toilet" or "I want to take a bath", and the answer labels are "The toilet is at position XX" or "The bathtub is at position XX".

[0074] According to an embodiment of the present application, in order to further improve the effect of model training, the generated scene instruction pairs can be checked and polished manually or by a multimodal large model (such as the Chat Generative Pre-trained Transformer-4o model), so that the scene instruction pairs are closer to the actual scene and the human language expression. For the perspective images obtained by projection, filtering processing can also be performed on them to improve the accuracy of the perspective images.

[0075] According to an embodiment of the present application, before the segmentation base model processes the perspective image, since there are multiple feature points marked on the perspective image, such as a very large number of feature points marked on the bathtub (such as Figure 3 the red feature points in the lower left image in Figure 3 ), at this time, the K-means clustering algorithm can be used to cluster and classify the feature points on the perspective image, so that more feature points are clustered and fused to obtain fewer feature points (see

[0076] the red feature points in the image input to the segmentation base model in

[0077] ), and at the same time, the cluster center is given, and the cluster center is output to the segmentation base model to give a hint, thereby improving the accuracy of the segmentation mask label.

[0078] According to an embodiment of the present application, the scene video can be an RGB-D image collected by a video acquisition device (such as an RGB-D (Red-Green-Blue-Depth) sensor), which includes initial channel images of three color channels of red (R), green (G), and blue (B) and an initial depth image D. The Truncated Signed Distance Function (TSDF) is a voxel grid representation method widely used in the fields of 3D reconstruction and computer vision. TSDF characterizes the surface and spatial information of the scene by storing the signed distance value (SDF) of each voxel (usually a cubic unit) in 3D space.

[0079] According to an embodiment of the present application, the preset selection rule refers to using an image frame with a relative translation or relative rotation angle greater than a certain threshold (such as a translation distance exceeding 10 cm and a rotation angle exceeding 5°) as a key frame.

[0080] According to an embodiment of the present application, based on the above preset selection rule, multiple target channel images (i.e., key frames) can be filtered out from multiple initial channel images. Thus, based on the multiple target channel images, multiple target depth image frames can be filtered out from multiple initial depth images. Processing the multiple target depth image frames using the truncated signed distance function to obtain a 3D model of the target area.

[0081] Figure 4 The flowchart for generating the inference answer according to an embodiment of the present application is shown.

[0082] According to an embodiment of the present application, inputting the scene video and the implicit question into the vision-language model to output semantic features, including: inputting the scene video into the vision encoder to output a first image encoding vector; inputting the first image encoding vector and the implicit question into the multi-modal language model to output semantic features, where the semantic features characterize the object attributes of different initial objects.

[0083] According to an embodiment of the present application, the multi-modal language model can refer to a large language model, such as the Qwen2.5-VL model or the Chat Generative Pre-trained Transformer-4o model, etc.

[0084] According to an embodiment of the present application, refer to Figure 4 , input the scene video I into the vision encoder of the vision-language model for image encoding processing, thereby obtaining a first image encoding vector f vis , the first image encoding vector f vis and the implicit question are input into the multi-modal language model to output semantic features r txt, such as the color, size and other features described above.

[0085] According to an embodiment of the present application, inputting the scene video and semantic features into a segmentation base model to output a plurality of mask images, including: processing the scene video by an image encoder to obtain a second image encoding vector; processing the second image encoding vector and semantic features by an image decoder to obtain a plurality of mask images.

[0086] According to an embodiment of the present application, referring to Figure 4 , the segmentation base model includes an image encoder and an image decoder (i.e., the mask decoder in Figure 4 ). The image encoder performs image encoding processing on the scene video to obtain a second image encoding vector, and at the same time inputs the second image encoding vector and semantic features into the image decoder to obtain a plurality of mask images. The mask image can be a binary mask of a plurality of candidate objects that may become the target object (such as mask marking through shadows, wireframes, etc.), and the number of mask images can be 0, 1 or more. Figure 4 In

[0087] Figure 5 shows a flowchart of generating the target fusion feature according to an embodiment of the present application.

[0088] According to an embodiment of the present application, processing a plurality of mask images and semantic features by a perception fusion model to obtain a target fusion feature, including: for each mask image, back-projecting the mask image into a three-dimensional space to obtain a three-dimensional space model, where the three-dimensional space model includes a plurality of three-dimensional voxel features and three-dimensional boundary information corresponding to the target object; inputting a data set of a plurality of mask images into a feature extractor to output a target extraction feature, where the data set includes a plurality of three-dimensional voxel features, three-dimensional boundary information and semantic features; respectively processing the target extraction feature by a transformation network and a first multi-layer perceptron to obtain transformation features and a confidence matrix corresponding to different initial objects, where the confidence matrix includes confidences corresponding to different initial objects; for any confidence, in the case where the confidence is less than a confidence threshold, deleting matrix elements of the initial object corresponding to the confidence in the transformation feature and the confidence matrix to obtain a new transformation feature and a new confidence matrix; calculating a similarity matrix according to the new transformation feature and the new confidence matrix, where each element in the similarity matrix represents the similarity between any two initial objects; for any similarity, in the case where the similarity is greater than a similarity threshold, fusing the initial features of the two initial objects corresponding to the similarity to obtain an initial fusion feature; generating a target fusion feature according to at least one target fusion feature and a plurality of initial features that are not fused.

[0089] According to an embodiment of the present application, referring to Figure 5, when performing feature fusion, first backproject the mask image into the three-dimensional space to obtain a three-dimensional space model, and extract each three-dimensional voxel feature and the three-dimensional boundary information of the target object from this three-dimensional space model. The three-dimensional voxel feature can be represented by F voxel which is the average value of the backprojected voxel features from multiple perspectives. The three-dimensional boundary information is the 3D bounding box {x, y, z, w, h, l}, where {x, y, z} and {w, h, l} represent the center point position coordinates in the global coordinate system and the width, height, and depth of the 3D bounding box, respectively.

[0090] According to an embodiment of the present application, input the three-dimensional voxel features, three-dimensional boundary information, and semantic features of multiple mask images into a feature extractor to obtain target extraction features (for the specific processing process of the feature extractor, see below). Then, use a transformation network (such as a Transformer network) to process the target extraction features, which can obtain transformation features corresponding to different initial objects. At the same time, use a first multi-layer perceptron to process the target extraction features to obtain a confidence matrix corresponding to different initial objects.

[0091] According to an embodiment of the present application, for any confidence in the confidence matrix, when the confidence is less than the confidence threshold, delete the matrix elements of the transformation features and the initial objects corresponding to the confidence in the confidence matrix to obtain new transformation features and a new confidence matrix. The confidence threshold can be specifically set according to actual needs, for example, it is 0.6.

[0092] According to an embodiment of the present application, calculate a similarity matrix based on the new transformation features and the new confidence matrix. If a certain similarity in the similarity matrix is greater than the similarity threshold, it means that these two initial objects belong to the same object. At this time, the initial features of the two initial objects corresponding to this similarity can be fused to obtain initial fusion features. Since not every two initial objects' initial features are fused, there are multiple initial features that are not fused. At this time, according to at least one target fusion feature and the multiple initial features that are not fused, generate target fusion features. The similarity threshold can be specifically set according to actual needs, for example, it is 0.8.

[0093] According to an embodiment of the present application, the feature extractor includes a multi-layer perceptron and L mask cross-attention modules.

[0094] According to an embodiment of the present application, input the data set of multiple mask images into the feature extractor, and output target extraction features, including: using a multi-layer perceptron to process the three-dimensional boundary information to obtain boundary-aware features; using L mask cross-attention modules to process the boundary-aware features, three-dimensional voxel features, and semantic features to obtain fused target extraction features.

[0095] According to an embodiment of the present application, the multi-layer perceptron has the same structure as the first multi-layer perceptron described above.

[0096] According to an embodiment of the present application, referring to Figure 5 , for three-dimensional boundary information, the multi-layer perceptron MLP can be used to process the three-dimensional boundary information to obtain the boundary-aware feature Q box , and L masked cross-attention modules are used to process the boundary-aware feature, three-dimensional voxel feature, and semantic feature to obtain the fused target extraction feature.

[0097] According to an embodiment of the present application, using L masked cross-attention modules to process the boundary-aware feature, three-dimensional voxel feature, and semantic feature to obtain the fused target extraction feature includes: for the m-th masked cross-attention module, using the first masked cross-attention mechanism layer to process the input feature and the three-dimensional voxel feature to obtain the first attention feature, where when m = 1, the input feature is the boundary-aware feature, and when m ≠ 1, the input feature is the output feature output by the previous masked cross-attention module; using the first residual normalization layer to process the input feature and the first attention feature to generate the first normalized feature; using the second masked cross-attention mechanism layer to process the first normalized feature and the semantic feature to obtain the second attention feature; using the second residual normalization layer to process the first normalized feature and the second attention feature to generate the second normalized feature; using a feed-forward neural network to process the second normalized feature to obtain a non-linearly transformed feature; using the third residual normalization layer to process the second normalized feature and the non-linearly transformed feature to generate the output feature, where when m = L, the output feature represents the target extraction feature.

[0098] According to an embodiment of the present application, referring to Figure 5 , in the process of obtaining the fused target extraction feature, the input data of the first masked cross-attention module is the boundary-aware feature Q box and the three-dimensional voxel feature F voxel , that is, using the first masked cross-attention mechanism layer to process the boundary-aware feature Q box and the three-dimensional voxel feature F voxel to obtain the first attention feature.

[0099] According to an embodiment of the present application, using the first residual normalization layer (Add&Norm 1 ) to process the input feature and the first attention feature to generate the first normalized feature, using the second masked cross-attention mechanism layer to process the first normalized feature and the semantic feature F text , to obtain the second attention feature, using the second residual normalization layer (Add&Norm 2(0) Process the first normalized feature and the second attention feature to generate a second normalized feature, and process the second normalized feature using a Feedforward Neural Network (FNN) to obtain a non-linearly transformed feature Q. ffn , and finally use the third residual normalization layer (Add&Norm 3 ) to process the second normalized feature and the non-linearly transformed feature to generate an output feature. Any residual normalization layer is used to perform residual connection and normalization processing on the input feature.

[0100] According to an embodiment of the present application, when 1 < m ≤ L, the data processing flow of the m-th masked cross-attention module is the same as that of the first masked cross-attention module, except that the input data of the m-th masked cross-attention module is the output feature, the three-dimensional voxel feature, and the semantic feature output by the previous masked cross-attention module. When m = L, the output feature of the m-th masked cross-attention module is the target extraction feature. Among them, the specific value of L can be set according to actual needs, but the size of L affects the inference efficiency, and here it can be set to 3 to balance the inference accuracy and the inference efficiency.

[0101] According to an embodiment of the present application, a feature extractor is composed of a multi-layer perceptron and L masked cross-attention modules, and the target extraction feature is extracted through the feature extractor, thereby improving the accuracy of feature fusion and thus improving the inference accuracy.

[0102] According to an embodiment of the present application, a preset language model is used to process the three-dimensional model, the semantic feature, and the target fusion feature, and an inference answer is output, including: generating a three-dimensional fusion image according to the target fusion feature and the three-dimensional model; inputting the three-dimensional fusion image and the semantic feature into the preset language model, and outputting an inference answer corresponding to the implicit question, where the inference answer represents the position information of the target object corresponding to the implicit question in the three-dimensional fusion image.

[0103] According to an embodiment of the present application, refer to Figure 5 , fuse the target fusion feature into the three-dimensional model, thereby generating a three-dimensional fusion image, and input the three-dimensional fusion image and the semantic feature into the preset language model, then the inference answer corresponding to the implicit question can be obtained. The inference answer can not only display the specific position information of the target object, but also mark the target object in the three-dimensional fusion image and display it to the user.

[0104] Figure 6 shows a flowchart of an implicit question inference method according to an embodiment of the present application. Figure 7 shows a schematic diagram of a scenario of an implicit question inference method according to an embodiment of the present application.

[0105] As shown Figure 6 in the figure, the implicit problem reasoning method includes operation S601 to operation S603.

[0106] In operation S601, obtain the area video of the area to be inferred and the problem to be inferred.

[0107] In operation S602, construct a three-dimensional area model of the area to be inferred according to the area video.

[0108] In operation S603, input the problem to be inferred and the three-dimensional area model into the target inference model, and output the inference answer corresponding to the problem to be inferred, where the inference answer represents the position information of the required object corresponding to the implicit problem.

[0109] According to the embodiments of the present application, the area to be inferred can be areas such as shopping malls, commercial streets, or the interior of houses.

[0110] In a specific embodiment, obtain the area video of a shopping mall and the problem to be inferred "I'm thirsty", construct a three-dimensional area model of the shopping mall based on the area video, input the problem to be inferred and the three-dimensional area model into the target inference model, and output the inference answer corresponding to the problem to be inferred, such as the water dispenser is next to the xx restaurant on the second floor. Among them, the number of inference answers can be more than one.

[0111] It should be noted that the target inference model can be applied to the display device in the waiting inference area of the shopping mall. The construction of the three-dimensional area model only needs to be carried out during the first inference and can be directly used in subsequent inferences.

[0112] In another specific embodiment, refer to Figure 7 , use the geometric reconstruction module based on the truncated sign function to process the depth image frames in the area video, thereby obtaining the three-dimensional area model of the area to be inferred. Use the inference segmentation model (including the visual language model segmentation basic model) in the target inference model to process the channel images of different color channels in the area video and the problem to be inferred, generate semantic features and multiple mask images, map the three-dimensional area model, semantic features and multiple mask images, and a three-dimensional image containing multiple candidate instances (objects) can be obtained. Fuse its features and combine with the three-dimensional area model to obtain a fused instance image containing multiple target fusion features. The fused instance image has multiple object attribute labels output by the previous visual language model. Input the fused instance image and the problem to be inferred into the preset language model again, and the inference answer corresponding to the problem to be inferred can be obtained.

[0113] Among them, Figure 7 the F i 、F j and F kRepresent the images from perspectives i, j, and k respectively, including the channel image and the depth image.

[0114] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and the implicit question are input into the vision-language model to output semantic features; the scene video and the semantic features are input into the segmentation basic model to output multiple mask images, and the perception fusion model is used to process the multiple mask images and the semantic features to obtain the target fusion features. The preset language model is used to process the three-dimensional model, the semantic features, and the target fusion features to output the inference answer. The model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain the target inference model. Since different features are fused using the perception fusion model during the inference process of the implicit question, the accuracy and efficiency of the inference of the implicit question are improved, thereby improving the user experience.

[0115] Figure 8 Shows a block diagram of a training device for an implicit question inference model according to an embodiment of the present application.

[0116] As Figure 8 shown, the training device 800 for the implicit question inference model includes a first acquisition module 810, a first construction module 820, a first obtaining module 830, a segmentation module 840, a second obtaining module 850, a first inference module 860, and an adjustment module 870.

[0117] The first acquisition module 810 is used to acquire an inference training set, where the inference training set includes multiple training samples, and each training sample includes a scene video of the target area and multiple scene instruction pairs. The scene instruction pair includes multiple implicit questions for the target area and the answer label corresponding to each implicit question. The implicit question is a veiled expression of the target object to be searched for.

[0118] The first construction module 820 is used to construct a three-dimensional model of the target area based on the scene video for each training sample.

[0119] The first obtaining module 830 is used to input the scene video and the implicit question into the vision-language model to output semantic features.

[0120] The segmentation module 840 is used to input the scene video and the semantic features into the segmentation basic model to output multiple mask images, where the mask image has a mask label formed on multiple initial objects corresponding to the implicit question.

[0121] The second obtaining module 850 is used to process the multiple mask images and the semantic features using the perception fusion model to obtain the target fusion features, where the target fusion features represent multiple fused initial objects.

[0122] The first inference module 860 is used to process the 3D model, semantic features, and target fusion features using a preset language model, and output an inference answer. The initial inference model includes a vision-language model, a segmentation base model, a perception fusion model, and a preset language model.

[0123] The adjustment module 870 is used to adjust the model parameters of the initial inference model according to the inference answer and the answer label to obtain the target inference model.

[0124] According to an embodiment of the present application, for each training sample, a 3D model of the target area is constructed based on the scene video; the scene video and the implicit question are input into the vision-language model to output semantic features; the scene video and the semantic features are input into the segmentation base model to output multiple mask images, and the perception fusion model is used to process the multiple mask images and the semantic features to obtain target fusion features. The preset language model is used to process the 3D model, semantic features, and target fusion features to output an inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain the target inference model. Since different features are fused using the perception fusion model during the inference process of the implicit question, the accuracy and efficiency of the inference of the implicit question are improved, and thus the user experience is improved.

[0125] Figure 9 The block diagram of the implicit question inference device according to an embodiment of the present application is shown.

[0126] As Figure 9 shown, the implicit question inference device 900 includes a second acquisition module 910, a second construction module 920, and a second inference module 930.

[0127] The second acquisition module 910 is used to acquire the area video of the area to be inferred and the question to be inferred.

[0128] The second construction module 920 is used to construct a 3D area model of the area to be inferred based on the area video.

[0129] The second inference module 930 is used to input the question to be inferred and the 3D area model into the target inference model and output an inference answer corresponding to the question to be inferred, where the inference answer represents the position information of the required object corresponding to the implicit question.

[0130] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and the implicit question are input into the vision-language model to output semantic features; the scene video and the semantic features are input into the segmentation-based model to output multiple mask images, and the perceptual fusion model is used to process the multiple mask images and the semantic features to obtain the target fusion features. The preset language model is used to process the three-dimensional model, the semantic features, and the target fusion features to output the inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain the target inference model. Since different features are fused using the perceptual fusion model during the inference process of the implicit question, the accuracy and efficiency of the inference of the implicit question are improved, thereby improving the user experience.

[0131] Any combination of multiple or at least some functions of the modules, sub-modules, units, and sub-units according to the embodiments of the present application can be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present application can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present application can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits in hardware or firmware, or can be implemented in any one of the three implementation manners of software, hardware, and firmware, or in any appropriate combination of several of them. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present application can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions can be executed.

[0132] It should be noted that the training device and the implicit question inference device of the implicit question inference model in the embodiments of the present application correspond to the training method and the implicit question inference method of the implicit question inference model in the embodiments of the present application. For the description of the training device and the implicit question inference device of the implicit question inference model, please refer to the training method and the implicit question inference method of the implicit question inference model, which will not be elaborated here.

[0133] Figure 10 A humanoid robot showing the implicit question inference device applying the embodiments of the present application is shown.

[0134] As Figure 10 shown, the humanoid robot 1000 includes the above-mentioned implicit question inference device 900, an image acquisition device 1010, a display device 1020, and an input device 1030.

[0135] The image acquisition device 1010 can be a device such as an RGB-D sensor that can simultaneously acquire a channel image and a depth image. The display device 1020 can include a display or a voice broadcast device (such as a speaker), and the input device 1030 can include a keyboard and / or a voice acquisition device (such as a microphone).

[0136] Figure 11 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 11 The electronic device shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0137] As Figure 11 shown, the electronic device 1100 according to an embodiment of the present application includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in the read-only memory 1102 or a program loaded from the storage section 1108 into the random access memory 1103. The processor 1101 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), and so on. The processor 1101 can also include on-board memory for caching purposes. The processor 1101 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0138] In the random access memory 1103, various programs and data required for the operation of the electronic device 1100 are stored. The processor 1101, the read-only memory 1102, and the random access memory 1103 are connected to each other through a bus 1104. The processor 1101 executes various operations of the method flow according to an embodiment of the present application by executing the programs in the read-only memory 1102 and / or the random access memory 1103. It should be noted that the program can also be stored in one or more memories other than the read-only memory 1102 and the random access memory 1103. The processor 1101 can also execute various operations of the method flow according to an embodiment of the present application by executing the programs stored in the one or more memories.

[0139] According to an embodiment of the present application, the electronic device 1100 may further include an input / output (I / O) interface 1105, and the input / output (I / O) interface 1105 is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the input / output (I / O) interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. The drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read from it is installed into the storage portion 1108 as needed.

[0140] According to an embodiment of the present application, the method flow according to the embodiment of the present application may be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program codes for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication portion 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the above-described system, device, apparatus, module, unit, etc. may be implemented by computer program modules.

[0141] The present application also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiment; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0142] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0143] An embodiment of the present application also includes a computer program product, which includes a computer program. The computer program contains program code for executing the method provided by the embodiment of the present application. When the computer program product runs on an electronic device, the program code is used to cause the electronic device to implement the method provided by the embodiment of the present application.

[0144] When the computer program is executed by the processor 1101, the above functions defined in the system / apparatus of the embodiment of the present application are executed. According to an embodiment of the present application, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0145] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and is downloaded and installed through the communication part 1109, and / or installed from the removable medium 1111. The program code included in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0146] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0147] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present application, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A training method for an implicit problem reasoning model, characterized in that Including: Obtain an inference training set, where the inference training set includes a plurality of training samples, each training sample includes a scene video of a target area and a plurality of scene instruction pairs, the scene instruction pair includes a plurality of implicit questions for the target area and an answer label corresponding to each implicit question, and the implicit question is an implicit expression of the target object to be searched; For each training sample, construct a three-dimensional model of the target area according to the scene video; Input the scene video and the implicit question into a vision-language model to output semantic features; Input the scene video and the semantic features into a segmentation base model to output a plurality of mask images, where the mask images are formed with mask labels on a plurality of initial objects corresponding to the implicit questions; Process the plurality of mask images and the semantic features by using a perception fusion model to obtain target fusion features, where the target fusion features represent a plurality of initial objects after fusion; Process the three-dimensional model, the semantic features, and the target fusion features by using a preset language model to output an inference answer, and the initial inference model includes the vision-language model, the segmentation base model, the perception fusion model, and the preset language model; Adjust the model parameters of the initial inference model according to the inference answer and the answer label to obtain a target inference model.

2. The method according to claim 1, wherein The inference training set is generated by the following method: Obtain an information set of the target area, where the information set includes three-dimensional point clouds and the scene video; Project the three-dimensional point clouds into a two-dimensional image space to obtain perspective images from different perspectives; Input the plurality of perspective images into the segmentation base model to obtain segmentation mask labels corresponding to different objects; Input the plurality of perspective images into the vision-language model to output object attribute labels corresponding to different objects; Input the plurality of perspective images, the plurality of segmentation mask labels, and the plurality of object attribute labels into the preset language model to output the plurality of scene instruction pairs; Generate the inference training set according to the scene video and the plurality of scene instruction pairs.

3. The method according to claim 1, wherein The scene video includes a plurality of initial channel images of different color channels and a plurality of initial depth images corresponding to each color channel; Among them, constructing the three-dimensional model of the target area according to the scene video includes: Based on a preset selection rule, screen out a plurality of target channel images from the plurality of initial channel images; Based on the target channel images, screen out a plurality of target depth image frames from the plurality of initial depth images; Process the plurality of target depth image frames by using a truncated sigmoid function to obtain the three-dimensional model of the target area.

4. The method according to claim 1, characterized in that Inputting the scene video and the implicit question into a vision-language model to output semantic features includes: Input the scene video into a vision encoder to output a first image encoding vector; Input the first image encoding vector and the implicit question into a multi-modal language model to output the semantic features, where the semantic features represent the object attributes of different initial objects.

5. The method according to claim 1, characterized in that Input the scene video and the semantic features into a segmentation base model to output multiple mask images, including: Process the scene video using an image encoder to obtain a second image encoding vector; Process the second image encoding vector and the semantic features using an image decoder to obtain multiple mask images.

6. The method according to claim 1, characterized in that, Process multiple mask images and the semantic features using a perception fusion model to obtain a target fusion feature, including: For each mask image, back-project the mask image into a three-dimensional space to obtain a three-dimensional space model, where the three-dimensional space model includes multiple three-dimensional voxel features and three-dimensional boundary information corresponding to the target object; Input a data set of multiple mask images into a feature extractor to output a target extraction feature, where the data set includes multiple three-dimensional voxel features, the three-dimensional boundary information, and the semantic features; Process the target extraction feature using a transformation network and a first multi-layer perceptron respectively to obtain transformation features and a confidence matrix corresponding to different initial objects, where the confidence matrix includes confidences corresponding to different initial objects; For any one of the confidences, when the confidence is less than a confidence threshold, delete matrix elements of the initial object corresponding to the confidence in the transformation feature and the confidence matrix to obtain a new transformation feature and a new confidence matrix; Calculate a similarity matrix according to the new transformation feature and the new confidence matrix, where each element in the similarity matrix represents the similarity between any two initial objects; For any one of the similarities, when the similarity is greater than a similarity threshold, fuse the initial features of the two initial objects corresponding to the similarity to obtain an initial fusion feature; Generate the target fusion feature according to at least one of the target fusion features and multiple initial features that are not fused.

7. The method according to claim 6, characterized in that, The feature extractor includes a multi-layer perceptron and L mask cross-attention modules; Among them, inputting a data set of multiple mask images into the feature extractor to output a target extraction feature includes: Process the three-dimensional boundary information using the multi-layer perceptron to obtain a boundary perception feature; Process the boundary perception feature, the three-dimensional voxel features, and the semantic features using L mask cross-attention modules to obtain a fused target extraction feature.

8. The method according to claim 7, characterized in that Process the boundary perception feature, the three-dimensional voxel features, and the semantic features using L mask cross-attention modules to obtain a fused target extraction feature, including: For the m-th mask cross-attention module, process the input feature and the three-dimensional voxel features using a first mask cross-attention mechanism layer to obtain a first attention feature, where when m = 1, the input feature is the boundary perception feature, and when m ≠ 1, the input feature is the output feature output by the previous mask cross-attention module; Process the input feature and the first attention feature using a first residual normalization layer to generate a first normalized feature; Process the first normalized feature and the semantic feature using a second masked cross-attention mechanism layer to obtain a second attention feature; Process the first normalized feature and the second attention feature using a second residual normalization layer to generate a second normalized feature; Process the second normalized feature using a feed-forward neural network to obtain a non-linearly transformed feature; Process the second normalized feature and the non-linearly transformed feature using a third residual normalization layer to generate the output feature, where when m = L, the output feature represents the target extraction feature.

9. The method according to claim 1, wherein Process the three-dimensional model, the semantic feature, and the target fusion feature using a preset language model to output an inference answer, including: Generate a three-dimensional fusion image according to the target fusion feature and the three-dimensional model; Input the three-dimensional fusion image and the semantic feature into the preset language model to output the inference answer corresponding to the implicit question, where the inference answer represents the position information of the target object corresponding to the implicit question in the three-dimensional fusion image.

10. An implicit problem reasoning method, characterized in that, Including: Obtain the regional video of the area to be inferred and the question to be inferred; Construct a three-dimensional regional model of the area to be inferred according to the regional video; Input the question to be inferred and the three-dimensional regional model into the target inference model to output an inference answer corresponding to the question to be inferred, where the inference answer represents the position information of the required object corresponding to the implicit question; Wherein, the target inference model is trained by the method according to any one of claims 1 to 9.

Images