Training method and implicit question reasoning method of implicit question reasoning model

Abstract

The application provides a training method and an implicit problem reasoning method of an implicit problem reasoning model, which can be applied to the field of artificial intelligence. The training method comprises the following steps: obtaining a reasoning training set; constructing a three-dimensional model of a target area according to a scene video; inputting the scene video and an implicit question into a visual language model to output semantic features; inputting the scene video and the semantic features into a segmentation base model to output a plurality of mask images, the mask images being formed with mask labels on a plurality of initial objects corresponding to the implicit question; processing the plurality of mask images and the semantic features by using a perception fusion model to obtain target fusion features; processing the three-dimensional model, the semantic features and the target fusion features by using a preset language model to output a reasoning answer, the initial reasoning model comprising the visual language model, the segmentation base model, the perception fusion model and the preset language model; and adjusting model parameters of the initial reasoning model according to the reasoning answer and an answer label to obtain a target reasoning model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a training method for an implicit question reasoning model and an implicit question reasoning method. Background Art

[0002] Existing problem reasoning primarily focuses on problems with explicit instructions, requiring explicit specification of the object or type to reconstruct the area related to the instruction. However, humans tend to directly express needs involving implicit and common needs, such as "I'm hungry, help me find something to eat" or "find a small toilet facility suitable for children," rather than providing explicit step-by-step instructions.

[0003] Furthermore, in unfamiliar environments, people don't know what objects are present, so they can only give implicit instructions. Existing explicit question reasoning methods have limited interaction and reasoning capabilities between target objects and instructions, resulting in suboptimal reasoning accuracy and efficiency for implicit questions, leading to a poor user experience. Summary of the Invention

[0004] In view of this, the present application provides a training method for an implicit question reasoning model and an implicit question reasoning method.

[0005] One aspect of the present application provides a training method for an implicit question reasoning model, comprising: obtaining a reasoning training set, wherein the reasoning training set includes multiple training samples, each training sample includes a scene video of a target area and multiple scene instruction pairs, the scene instruction pairs include multiple implicit questions for the target area and answer labels corresponding to each implicit question, and the implicit question is an implicit expression of the target object to be found; for each training sample, constructing a three-dimensional model of the target area based on the scene video; inputting the scene video and the implicit question into a visual language model to output semantic features; inputting the scene video and the semantic features into a segmentation base model to output multiple mask images, wherein the mask images form mask labels on multiple initial objects corresponding to the implicit questions; using a perceptual fusion model to process the multiple mask images and semantic features to obtain target fusion features, wherein the target fusion features represent multiple fused initial objects; using a preset language model to process the three-dimensional model, semantic features and target fusion features, and outputting an inference answer, wherein the initial inference model includes a visual language model, a segmentation base model, a perceptual fusion model and a preset language model; adjusting the model parameters of the initial inference model according to the inference answer and the answer label to obtain a target inference model.

[0006] Another aspect of the present application provides an implicit question reasoning method, including: obtaining a regional video of the area to be reasoned and the question to be reasoned; constructing a three-dimensional regional model of the area to be reasoned based on the regional video; inputting the question to be reasoned and the three-dimensional regional model into a target reasoning model, and outputting a reasoning answer corresponding to the question to be reasoned, wherein the reasoning answer represents the location information of the required object corresponding to the implicit question.

[0007] Another aspect of the present application provides a training device for an implicit question reasoning model, comprising:

[0008] a first acquisition module, configured to acquire an inference training set, wherein the inference training set includes a plurality of training samples, each training sample including a scene video of a target area and a plurality of scene instruction pairs, the scene instruction pairs including a plurality of implicit questions regarding the target area and an answer label corresponding to each implicit question, the implicit question being an implicit expression of the target object to be sought;

[0009] The first construction module is used to construct a three-dimensional model of the target area according to the scene video for each training sample;

[0010] The first module is used to input the scene video and implicit question into the visual language model and output semantic features;

[0011] a segmentation module configured to input the scene video and semantic features into a segmentation base model and output a plurality of mask images, wherein the mask images are provided with mask labels on a plurality of initial objects corresponding to the implicit question;

[0012] The second obtaining module is used to process the multiple mask images and semantic features using the perceptual fusion model to obtain a target fusion feature, wherein the target fusion feature represents the multiple fused initial objects;

[0013] The first reasoning module is used to process the 3D model, semantic features, and target fusion features using a preset language model and output an inference answer. The initial reasoning model includes a visual language model, a segmentation base model, a perception fusion model, and a preset language model.

[0014] The adjustment module is used to adjust the model parameters of the initial reasoning model according to the reasoning answer and the answer label to obtain the target reasoning model.

[0015] Another aspect of the present application provides an implicit question reasoning device, comprising:

[0016] The second acquisition module is used to obtain the regional video of the area to be inferred and the problem to be inferred;

[0017] The second construction module is used to construct a three-dimensional regional model of the area to be inferred based on the regional video;

[0018] The second reasoning module is used to input the problem to be reasoned and the three-dimensional area model into the target reasoning model, and output the reasoning answer corresponding to the problem to be reasoned, wherein the reasoning answer represents the position information of the required object corresponding to the implicit question.

[0019] Another aspect of the present application provides a humanoid robot comprising the above-mentioned implicit question reasoning device, an image acquisition device, a display device and an input device.

[0020] Another aspect of the present application provides an electronic device, comprising:

[0021] one or more processors;

[0022] a memory for storing one or more programs,

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0024] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0025] Another aspect of the present application provides a computer program product, which includes computer-executable instructions. When the instructions are executed, they are used to implement the method described above.

[0026] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and implicit question are input into the visual language model to output semantic features; the scene video and semantic features are input into the segmentation base model to output multiple mask images, the multiple mask images and semantic features are processed using the perceptual fusion model to obtain target fusion features, the three-dimensional model, semantic features and target fusion features are processed using a preset language model to output an inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain a target inference model. Since the perceptual fusion model is used to fuse different features during the inference process of the implicit question, the accuracy and efficiency of the implicit question inference are improved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings.

[0028] Figure 1 An exemplary system architecture is shown to which an implicit question reasoning model training method or an implicit question reasoning method can be applied according to an embodiment of the present application.

[0029] Figure 2 A flowchart of a method for training an implicit question reasoning model according to an embodiment of the present application is shown.

[0030] Figure 3 A flow chart for generating an inference training set according to an embodiment of the present application is shown.

[0031] Figure 4 A flow chart of generating an inference answer according to an embodiment of the present application is shown.

[0032] Figure 5 A flow chart of generating target fusion features according to an embodiment of the present application is shown.

[0033] Figure 6 A flowchart of an implicit question reasoning method according to an embodiment of the present application is shown.

[0034] Figure 7 A schematic diagram of a scenario of an implicit question reasoning method according to an embodiment of the present application is shown.

[0035] Figure 8 A block diagram of a training device for an implicit question reasoning model according to an embodiment of the present application is shown.

[0036] Figure 9 A block diagram of an implicit question reasoning device according to an embodiment of the present application is shown.

[0037] Figure 10 A humanoid robot to which the implicit question reasoning device according to an embodiment of the present application is applied is shown.

[0038] Figure 11 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0040] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0041] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the use of any terms herein should be interpreted to have the meaning commonly understood by one of ordinary skill in the art, consistent with the context of the specification, and should not be interpreted in an idealized or overly formal sense.

[0042] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted that the expression includes at least one of A, B, and C, etc. (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0043] In the embodiments of the present application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) comply with the relevant legal regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and to maintain user personal information security and network security.

[0044] The embodiments of the present application provide a training method of an implicit question reasoning model and an implicit question reasoning method. The training method comprises obtaining a reasoning training set, wherein the reasoning training set comprises a plurality of training samples, each training sample comprises a scene video of a target area and a plurality of scene instruction pairs, the scene instruction pair comprises a plurality of implicit questions for the target area and an answer label corresponding to each implicit question, and the implicit question is an implicit expression of a sought target object; for each training sample, a three-dimensional model of the target area is constructed according to the scene video; the scene video and the implicit question are input into a visual language model to output semantic features; the scene video and the semantic features are input into a segmentation base model to output a plurality of mask images, wherein the mask image forms a mask label on a plurality of initial objects corresponding to the implicit question; the plurality of mask images and the semantic features are processed by using a perception fusion model to obtain target fusion features, wherein the target fusion features represent a plurality of fused initial objects; the three-dimensional model, the semantic features, and the target fusion features are processed by using a preset language model to output a reasoning answer, the initial reasoning model comprises the visual language model, the segmentation base model, the perception fusion model, and the preset language model; and model parameters of the initial reasoning model are adjusted according to the reasoning answer and the answer label to obtain a target reasoning model.

[0045] Figure 1 An exemplary system architecture to which the training method of an implicit question reasoning model or the implicit question reasoning method according to the embodiments of the present application can be applied is shown. It should be noted that, Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0046] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0047] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0048] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0049] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0050] It should be noted that the training method of the implicit question reasoning model and the implicit question reasoning method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the training device of the implicit question reasoning model and the implicit question reasoning device provided in the embodiment of the present application can generally be set in the server 105. The training method of the implicit question reasoning model and the implicit question reasoning method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the training device of the implicit question reasoning model and the implicit question reasoning device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the implicit question reasoning model training method and implicit question reasoning method provided in the embodiments of the present application may also be performed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the implicit question reasoning model training device and implicit question reasoning device provided in the embodiments of the present application may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be provided in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0052] Figure 2 A flowchart of a method for training an implicit question reasoning model according to an embodiment of the present application is shown.

[0053] like Figure 2 As shown, the training method of the implicit question reasoning model includes operations S210 to S270.

[0054] In operation S210, an inference training set is obtained, wherein the inference training set includes multiple training samples, each training sample includes a scene video of a target area and multiple scene instruction pairs, the scene instruction pairs include multiple implicit questions for the target area and answer labels corresponding to each implicit question, and the implicit question is an implicit expression of the target object being sought.

[0055] In operation S220 , for each training sample, a three-dimensional model of the target area is constructed according to the scene video.

[0056] In operation S230 , the scene video and the implicit question are input into a visual language model, and a semantic feature is output.

[0057] In operation S240 , the scene video and the semantic features are input to a segmentation base model, and a plurality of mask images are output, wherein the mask images are formed with mask labels on a plurality of initial objects corresponding to the implicit question.

[0058] In operation S250 , the plurality of mask images and semantic features are processed using a perceptual fusion model to obtain a target fusion feature, wherein the target fusion feature represents the plurality of fused initial objects.

[0059] In operation S260, a preset language model is used to process the three-dimensional model, semantic features, and target fusion features, and an inference answer is output. The initial inference model includes a visual language model, a segmentation base model, a perception fusion model, and a preset language model.

[0060] In operation S270 , model parameters of the initial reasoning model are adjusted according to the reasoning answer and the answer label to obtain a target reasoning model.

[0061] According to the embodiments of the present application, explicit questions are direct expressions of a user's needs, and the target object the user needs can be directly understood from the explicit questions, while implicit questions are implicit expressions of a user's needs, and it is more difficult to understand the target object the user needs from the implicit questions. For example, the explicit question is "I need a water cup", and it can be seen from the explicit question that the target object the user needs is the water cup. The implicit question is "I am thirsty and want to drink water", and the target object the user actually wants is the water cup, but the expression is more implicit. The implicit question reasoning model of this embodiment can be applied to comprehensive shopping malls to infer answers to users' implicit questions such as "I am hungry", "I am thirsty", "My son is a little cold", etc.

[0062] According to an embodiment of the present application, the Vision Language Model (VLM) is a multimodal generative artificial intelligence system that integrates vision and language processing capabilities. It can understand image, video, and text data and generate natural language responses. The Vision Language Model in this embodiment can be replaced with other large multimodal models.

[0063] According to embodiments of the present application, the segmentation base model is primarily used to distinguish different objects or regions in an image. Examples include U-Net, Segment Anything Model (SAM), Structure from Motion (SFM), and Diffusion Model (DM). The preset language model may be a Large Language Model (LLM).

[0064] According to an embodiment of the present application, during the training process of the implicit problem reasoning model, the reasoning training set used for training is first obtained. Based on each training sample in the reasoning training set, a three-dimensional model of the target area is constructed according to the scene video in the training sample. For example, the scene video is a video shot of a shopping mall, and a three-dimensional model of the shopping mall can be constructed based on the video.

[0065] According to an embodiment of the present application, a scene video and an implicit question are input into a visual language model, and semantic features are output. The semantic features may be attribute features of the target object, such as color, type, size, etc. The scene video and semantic features are input into a segmentation base model, and multiple mask images are output. The mask images are marked with mask labels on multiple initial objects of the target object to be found in the implicit question. Some of the multiple initial objects may be essentially the same object, but may be divided into different initial objects due to the perspective. At this time, the multiple mask images and semantic features can be processed by the perceptual fusion model to obtain the target fusion feature.

[0066] According to an embodiment of the present application, the above-mentioned three-dimensional model, semantic features and target fusion features are input into a preset language model for reasoning, and an inference answer is output. For example, the inference answer can be informing the user of the specific location of the target object required by marking on an image, or it can be informing the user in the form of text.

[0067] According to an embodiment of the present application, the inference answer and the answer label are input into the loss function to obtain the loss result. The model parameters of the initial inference model are adjusted according to the loss result. After several parameter adjustments using multiple training samples, the target inference model, that is, the implicit question inference model, can be obtained.

[0068] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and implicit question are input into the visual language model to output semantic features; the scene video and semantic features are input into the segmentation base model to output multiple mask images, the multiple mask images and semantic features are processed using the perceptual fusion model to obtain target fusion features, the three-dimensional model, semantic features and target fusion features are processed using a preset language model to output an inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain a target inference model. Since the perceptual fusion model is used to fuse different features during the inference process of the implicit question, the accuracy and efficiency of the implicit question inference are improved, thereby improving the user experience.

[0069] Figure 3 A flow chart for generating an inference training set according to an embodiment of the present application is shown.

[0070] According to an embodiment of the present application, an inference training set is generated in the following manner: obtaining an information set of the target area; projecting a three-dimensional point cloud into a two-dimensional image space to obtain perspective images under different perspectives; inputting multiple perspective images into a segmentation basic model to obtain segmentation mask labels corresponding to different objects; inputting multiple perspective images into a visual language model to output object attribute labels corresponding to different objects; inputting multiple perspective images, multiple segmentation mask labels and multiple object attribute labels into a preset language model to output multiple scene instruction pairs; generating an inference training set based on the scene video and the multiple scene instruction pairs.

[0071] According to the embodiments of this application, referring to Figure 3 When constructing an inference training set, you can first use a camera to capture a 3D point cloud and scene video of the target area, or you can obtain it from the existing ScanNetV2 dataset. The 3D point cloud can be projected into a 2D image space. After projection, point clouds that are not visible from that perspective can be removed. For example, in a front view of a cup, since the back of the cup is not visible, the point cloud on the back can be removed, thereby obtaining k perspective images from different perspectives (viewpoints).

[0072] According to an embodiment of the present application, images from multiple viewpoints are fed into a segmentation base model to label different objects (e.g., a bathtub, toilet, etc.), thereby obtaining segmentation mask labels corresponding to the different objects. Simultaneously, these images from multiple viewpoints can be fed into a visual language model to obtain object attribute labels, such as color, size, category, and position. If the object attribute labels differ between images from different viewpoints, a voting mechanism can be used to determine the final object attribute label.

[0073] According to an embodiment of the present application, multiple perspective images, multiple segmentation mask labels and multiple object attribute labels are input into a preset language model, and multiple scene instruction pairs are output. The implicit questions in the scene instruction pairs are such as "I feel a little uncomfortable in my stomach and want to go to the toilet" or "I want to take a bath", and the answer labels are "The toilet is at XX position" or "The bathtub is at XX position".

[0074] According to embodiments of the present application, to further improve model training, the generated scene-command pairs can be manually verified and refined, either by humans or using a large multimodal model (such as the Chat Generative Pre-trained Transformer-40 model), thereby ensuring they more closely align with real-world scenarios and human language. The projected perspective images can also be filtered to improve their accuracy.

[0075] According to an embodiment of the present application, before the segmentation base model processes the perspective image, since a plurality of feature points are marked on the perspective image, such as a bathtub, a large number of feature points are marked (such as Figure 3 The red feature points in the lower left corner of the image) can be clustered and classified using the K-means clustering algorithm, thereby clustering and merging more feature points to obtain fewer feature points (see Figure 3 The red feature points in the image of the segmentation base model are input into the segmentation base model), and the cluster center is given at the same time, and the cluster center is output to the segmentation base model to give a hint, thereby improving the accuracy of the segmentation mask label.

[0076] According to an embodiment of the present application, a scene video includes a plurality of initial channel images of different color channels and a plurality of initial depth images corresponding to each color channel.

[0077] According to an embodiment of the present application, a three-dimensional model of a target area is constructed based on a scene video, including: based on a preset selection rule, screening out a plurality of target channel images from a plurality of initial channel images; based on the target channel images, screening out a plurality of target depth image frames from a plurality of initial depth images; and using a truncated sign function to process the plurality of target depth image frames to obtain a three-dimensional model of the target area.

[0078] According to an embodiment of the present application, a scene video may be an RGB-D image captured by a video capture device (such as an RGB-D (Red-Green-Blue-Depth) sensor), which includes initial channel images of the three color channels (red (R), green (G), and blue (B)) and an initial depth image D. The Truncated Signed Distance Function (TSDF) is a voxel grid representation method widely used in the fields of 3D reconstruction and computer vision. The TSDF represents the surface and spatial information of a scene by storing a signed distance value (SDF) for each voxel (typically a cubic unit) in 3D space.

[0079] According to an embodiment of the present application, the preset selection rule is to use image frames whose relative translation or relative rotation angle is greater than a certain threshold (such as a translation distance greater than 10 cm, a rotation angle greater than 5°) as key frames.

[0080] According to an embodiment of the present application, multiple target channel images (i.e., key frames) can be screened from multiple initial channel images based on the aforementioned preset selection rules. Multiple target depth image frames can then be screened from multiple initial depth images based on the multiple target channel images. The multiple target depth image frames are processed using a truncated sign function to obtain a three-dimensional model of the target area.

[0081] Figure 4 A flow chart of generating an inference answer according to an embodiment of the present application is shown.

[0082] According to an embodiment of the present application, a scene video and an implicit question are input into a visual language model to output semantic features, including: inputting a scene video into a visual encoder to output a first image coding vector; inputting the first image coding vector and the implicit question into a multimodal language model to output semantic features, wherein the semantic features represent object attributes of different initial objects.

[0083] According to an embodiment of the present application, the multimodal language model may refer to a large language model, such as a Qwen2.5-VL model or a Chat Generative Pre-trained Transformer-4o model.

[0084] According to the embodiments of this application, see Figure 4 , the scene video I is input into the visual encoder of the visual language model for image coding processing, thereby obtaining the first image coding vector f vis , the first image encoding vector f vis and implicit questions are input into the multimodal language model and the semantic feature r is output txt, such as the color, size and other characteristics recorded above.

[0085] According to an embodiment of the present application, a scene video and semantic features are input into a segmentation base model, and multiple mask images are output, including: using an image encoder to process the scene video to obtain a second image coding vector; using an image decoder to process the second image coding vector and semantic features to obtain multiple mask images.

[0086] According to the embodiments of this application, see Figure 4 , the segmentation base model includes an image encoder and an image decoder (i.e. Figure 4 The image encoder encodes the scene video to obtain a second image code vector. This second image code vector and semantic features are then input into the image decoder to obtain multiple mask images. These mask images can be binary masks of multiple candidate objects that may be the target object (e.g., masked by shading, wireframes, etc.). The number of mask images can be zero, one, or multiple. Figure 4 [SEG] represents a mask.

[0087] Figure 5 A flow chart of generating target fusion features according to an embodiment of the present application is shown.

[0088] According to an embodiment of the present application, a perceptual fusion model is used to process multiple mask images and semantic features to obtain target fusion features, including: for each mask image, the mask image is back-projected into a three-dimensional space to obtain a three-dimensional space model, wherein the three-dimensional space model includes multiple three-dimensional voxel features and three-dimensional boundary information corresponding to the target object; a data set of multiple mask images is input into a feature extractor, and a target extracted feature is output, wherein the data set includes multiple three-dimensional voxel features, three-dimensional boundary information and semantic features; the target extracted features are processed using a conversion network and a first multi-layer perceptron respectively to obtain conversion features and confidence matrices corresponding to different initial objects, wherein the confidence matrix includes corresponding different the confidence of the same initial object; for any confidence, when the confidence is less than the confidence threshold, the matrix elements of the initial object corresponding to the confidence in the conversion feature and the confidence matrix are deleted to obtain new conversion features and a new confidence matrix; based on the new conversion features and the new confidence matrix, a similarity matrix is ​​calculated, wherein each element in the similarity matrix represents the similarity between any two initial objects; for any similarity, when the similarity is greater than the similarity threshold, the initial features of the two initial objects corresponding to the similarity are fused to obtain an initial fused feature; a target fused feature is generated based on at least one target fused feature and multiple initial features that have not been fused.

[0089] According to the embodiments of this application, referring to Figure 5When performing feature fusion, the mask image is first back-projected into the three-dimensional space to obtain a three-dimensional space model. The three-dimensional voxel features and the three-dimensional boundary information of the target object are extracted from the three-dimensional space model. The three-dimensional voxel features can be used to voxel It represents the average value of the back-projected voxel features under multiple viewpoints, and the three-dimensional boundary information is the 3D bounding box {x, y, z, w, h, l}, where {x, y, z} and {w, h, l} represent the center point position coordinates in the global coordinate system and the width, height, and depth of the 3D bounding box, respectively.

[0090] According to an embodiment of the present application, 3D voxel features, 3D boundary information, and semantic features of multiple mask images are input into a feature extractor to obtain target extracted features (the specific feature extractor processing process is described below). These target extracted features are then processed using a transformation network (e.g., a Transformer network) to obtain transformed features corresponding to different initial objects. Simultaneously, these target extracted features are processed using a first multilayer perceptron to obtain confidence matrices corresponding to the different initial objects.

[0091] According to an embodiment of the present application, for any confidence in the confidence matrix, when the confidence is less than the confidence threshold, the matrix elements of the conversion feature and the initial object corresponding to the confidence in the confidence matrix are deleted to obtain a new conversion feature and a new confidence matrix, wherein the confidence threshold can be set according to actual needs, for example, 0.6.

[0092] According to an embodiment of the present application, a similarity matrix is ​​calculated based on the new conversion features and the new confidence matrix. If a similarity in the similarity matrix is ​​greater than a similarity threshold, it indicates that the two initial objects belong to the same object. In this case, the initial features of the two initial objects corresponding to the similarity can be fused to obtain an initial fused feature. Since the initial features of not every two initial objects are fused, there are multiple initial features that are not fused. In this case, a target fused feature can be generated based on at least one target fused feature and multiple initial features that are not fused. The similarity threshold can be set according to actual needs, for example, 0.8.

[0093] According to an embodiment of the present application, the feature extractor includes a multi-layer perceptron and L masked cross-attention modules.

[0094] According to an embodiment of the present application, a data set of multiple mask images is input into a feature extractor, and a target extraction feature is output, including: using a multi-layer perceptron to process three-dimensional boundary information to obtain boundary perception features; using L mask cross-attention modules to process boundary perception features, three-dimensional voxel features and semantic features to obtain fused target extraction features.

[0095] According to an embodiment of the present application, the multilayer perceptron has the same structure as the first multilayer perceptron described above.

[0096] According to the embodiments of this application, referring to Figure 5 For the three-dimensional boundary information, the multi-layer perceptron MLP can be used to process the three-dimensional boundary information to obtain the boundary perception feature Q box , L masked cross-attention modules are used to process boundary perception features, 3D voxel features and semantic features to obtain fused target extraction features.

[0097] According to an embodiment of the present application, L masked cross-attention modules are used to process boundary perception features, three-dimensional voxel features and semantic features to obtain fused target extraction features, including: for the mth masked cross-attention module, the first masked cross-attention mechanism layer is used to process input features and three-dimensional voxel features to obtain first attention features, wherein when m=1, the input features are boundary perception features, and when m≠1, the input features are output features output by the previous masked cross-attention module; the first residual normalization layer is used to process the input features and the first attention features to generate first normalized features; the second masked cross-attention mechanism layer is used to process the first normalized features and the semantic features to obtain second attention features; the second residual normalization layer is used to process the first normalized features and the second attention features to generate second normalized features; the feedforward neural network is used to process the second normalized features to obtain nonlinear transformation features; the third residual normalization layer is used to process the second normalized features and the nonlinear transformation features to generate output features, wherein, when m=L, the output features represent target extraction features.

[0098] According to the embodiments of this application, referring to Figure 5 In the process of obtaining the fused target extraction feature, the input data of the first mask cross attention module is the boundary perception feature Q box and 3D voxel features F voxel , that is, using the first mask cross attention mechanism layer to process the boundary perception feature Q box and 3D voxel features F voxel , and get the first attention feature.

[0099] According to an embodiment of the present application, the first residual normalization layer (Add & Norm 1 ) processes the input features and the first attention features, generates the first normalized features, and uses the second mask cross attention mechanism layer to process the first normalized features and semantic features F text , get the second attention feature, and use the second residual normalization layer (Add&Norm 2) processes the first normalized feature and the second attention feature to generate the second normalized feature, and uses the feedforward neural network (FNN) to process the second normalized feature to obtain the nonlinear transformation feature Q ffn Finally, the third residual normalization layer (Add&Norm 3 ) processes the second normalized features and nonlinear transformation features to generate output features. Any residual normalization layer is used to perform residual connection and normalization on the input features.

[0100] According to the embodiment of the present application, in 1 <m≤L时,第m个掩码交叉注意力模块的数据处理流程与第一个掩码交叉注意力模块的数据处理流程相同,区别仅在于第m个掩码交叉注意力模块的输入数据为前一个掩码交叉注意力模块输出的输出特征、三维体素特征和语义特征。在m=L时,该第m个掩码交叉注意力模块的输出特征即目标提取特征。其中,L的具体数值可以根据实际需求具体设置,但是L的大小影响推理的效率,在此可以设置为3以在推理准确性和推理效率之间进行平衡。

[0101] According to an embodiment of the present application, a feature extractor is formed by a multi-layer perceptron and L mask cross-attention modules, and the target extraction features are extracted by the feature extractor, thereby improving the accuracy of feature fusion and thus improving the accuracy of reasoning.

[0102] According to an embodiment of the present application, a preset language model is used to process a three-dimensional model, semantic features, and target fusion features, and output an inference answer, including: generating a three-dimensional fusion image based on the target fusion features and the three-dimensional model; inputting the three-dimensional fusion image and semantic features into the preset language model, and outputting an inference answer corresponding to the implicit question, wherein the inference answer represents the position information of the target object corresponding to the implicit question in the three-dimensional fusion image.

[0103] According to the embodiments of this application, see Figure 5 , the target fusion features are fused into the 3D model to generate a 3D fused image. The 3D fused image and semantic features are input into the preset language model to obtain the inference answer corresponding to the implicit question. The inference answer can not only display the specific location information of the target object, but also mark the target object in the 3D fused image and display it to the user.

[0104] Figure 6 A flowchart of an implicit question reasoning method according to an embodiment of the present application is shown. Figure 7 A schematic diagram of a scenario of an implicit question reasoning method according to an embodiment of the present application is shown.

[0105] like Figure 6 As shown, the implicit question reasoning method includes operations S601 to S603.

[0106] In operation S601 , a regional video of an area to be inferred and a question to be inferred are obtained.

[0107] In operation S602 , a three-dimensional region model of the region to be inferred is constructed based on the region video.

[0108] In operation S603 , the question to be inferred and the three-dimensional region model are input into the target reasoning model, and a reasoning answer to the question to be inferred is output, wherein the reasoning answer represents the position information of the required object corresponding to the implicit question.

[0109] According to an embodiment of the present application, the area to be inferred may be a shopping mall, a commercial street, or the interior of a house.

[0110] In a specific embodiment, a video of a shopping mall and a question to be inferred, "I am thirsty," are obtained. A 3D model of the shopping mall is constructed based on the video. The question to be inferred and the 3D model are input into a target inference model, which then outputs an inference answer to the question to be inferred, such as "The water dispenser is on the second floor, next to the xx restaurant." The number of inference answers can be more than one.

[0111] It should be noted that the target reasoning model can be applied to the display equipment in the waiting reasoning area of ​​the shopping mall. The construction of the three-dimensional area model only needs to be constructed during the first reasoning and can be directly used in subsequent reasoning.

[0112] In another specific embodiment, referring to Figure 7 , using a geometric reconstruction module built based on a truncated sign function to process the depth image frames in the regional video, thereby obtaining a three-dimensional regional model of the region to be inferred. The inference segmentation model in the target inference model (including the visual language model segmentation base model) is used to process the channel images of different color channels in the regional video and the problem to be inferred, generating semantic features and multiple mask images. The three-dimensional regional model, semantic features, and multiple mask images are mapped to obtain a three-dimensional image containing multiple candidate instances (objects). By fusing these features and combining them with the three-dimensional regional model, a fused instance image containing multiple target fused features can be obtained. This fused instance image has multiple object attribute labels output by the previous visual language model. The fused instance image and the problem to be inferred are then input into the preset language model to obtain the inference answer corresponding to the problem to be inferred.

[0113] in, Figure 7 F in i 、F j and F kRepresent the images under perspectives i, j, and k respectively, including channel images and depth images.

[0114] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and implicit question are input into the visual language model to output semantic features; the scene video and semantic features are input into the segmentation base model to output multiple mask images, the multiple mask images and semantic features are processed using the perceptual fusion model to obtain target fusion features, the three-dimensional model, semantic features and target fusion features are processed using a preset language model to output an inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain a target inference model. Since the perceptual fusion model is used to fuse different features during the inference process of the implicit question, the accuracy and efficiency of the implicit question inference are improved, thereby improving the user experience.

[0115] Figure 8 A block diagram of a training device for an implicit question reasoning model according to an embodiment of the present application is shown.

[0116] like Figure 8 As shown, the training device 800 of the implicit question reasoning model includes a first acquisition module 810, a first construction module 820, a first obtaining module 830, a segmentation module 840, a second obtaining module 850, a first reasoning module 860 and an adjustment module 870.

[0117] The first acquisition module 810 is used to obtain an inference training set, wherein the inference training set includes multiple training samples, each training sample includes a scene video of the target area and multiple scene instruction pairs, the scene instruction pairs include multiple implicit questions for the target area and answer labels corresponding to each implicit question, and the implicit question is an implicit expression of the target object being found.

[0118] The first construction module 820 is used to construct a three-dimensional model of the target area according to the scene video for each training sample.

[0119] The first obtaining module 830 is used to input the scene video and implicit question into the visual language model and output semantic features.

[0120] The segmentation module 840 is used to input the scene video and semantic features into the segmentation basic model, and output a plurality of mask images, wherein the mask images are formed with mask labels on a plurality of initial objects corresponding to the implicit question.

[0121] The second obtaining module 850 is used to process the multiple mask images and semantic features using the perceptual fusion model to obtain a target fusion feature, wherein the target fusion feature represents the multiple fused initial objects.

[0122] The first reasoning module 860 is used to process the three-dimensional model, semantic features and target fusion features using a preset language model and output an inference answer. The initial reasoning model includes a visual language model, a segmentation base model, a perception fusion model and a preset language model.

[0123] The adjustment module 870 is used to adjust the model parameters of the initial reasoning model according to the reasoning answer and the answer label to obtain the target reasoning model.

[0124] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and implicit question are input into the visual language model to output semantic features; the scene video and semantic features are input into the segmentation base model to output multiple mask images, the multiple mask images and semantic features are processed using the perceptual fusion model to obtain target fusion features, the three-dimensional model, semantic features and target fusion features are processed using a preset language model to output an inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain a target inference model. Since the perceptual fusion model is used to fuse different features during the inference process of the implicit question, the accuracy and efficiency of the implicit question inference are improved, thereby improving the user experience.

[0125] Figure 9 A block diagram of an implicit question reasoning device according to an embodiment of the present application is shown.

[0126] like Figure 9 As shown, the implicit question reasoning device 900 includes a second acquisition module 910 , a second construction module 920 , and a second reasoning module 930 .

[0127] The second acquisition module 910 is used to obtain a regional video of the area to be inferred and a question to be inferred.

[0128] The second construction module 920 is used to construct a three-dimensional regional model of the area to be inferred based on the regional video.

[0129] The second reasoning module 930 is used to input the question to be reasoned and the three-dimensional area model into the target reasoning model, and output the reasoning answer to the question to be reasoned, wherein the reasoning answer represents the position information of the required object corresponding to the implicit question.

[0130] According to an embodiment of the present application, for each training sample, a three-dimensional model of the target area is constructed based on the scene video; the scene video and implicit question are input into the visual language model to output semantic features; the scene video and semantic features are input into the segmentation base model to output multiple mask images, the multiple mask images and semantic features are processed using the perceptual fusion model to obtain target fusion features, the three-dimensional model, semantic features and target fusion features are processed using a preset language model to output an inference answer, and the model parameters of the initial inference model are adjusted according to the inference answer and the answer label to obtain a target inference model. Since the perceptual fusion model is used to fuse different features during the inference process of the implicit question, the accuracy and efficiency of the implicit question inference are improved, thereby improving the user experience.

[0131] According to the embodiments of the present application, any number of modules, submodules, units, and subunits, or at least part of the functions of any number of them, can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, which can perform the corresponding functions when the computer program module is executed.

[0132] It should be noted that the training device and implicit problem reasoning device of the implicit problem reasoning model in the embodiments of the present application correspond to the training method and implicit problem reasoning method of the implicit problem reasoning model in the embodiments of the present application. The description of the training device and implicit problem reasoning device of the implicit problem reasoning model specifically refers to the training method and implicit problem reasoning method of the implicit problem reasoning model, and will not be repeated here.

[0133] Figure 10 A humanoid robot to which the implicit question reasoning device according to an embodiment of the present application is applied is shown.

[0134] like Figure 10 As shown, the humanoid robot 1000 includes the above-mentioned implicit question reasoning device 900, an image acquisition device 1010, a display device 1020 and an input device 1030.

[0135] The image acquisition device 1010 can be a device such as an RGB-D sensor that can simultaneously acquire channel images and depth images. The display device 1020 can include a display or a voice broadcast device (such as a speaker). The input device 1030 can include a keyboard and / or a voice acquisition device (such as a microphone).

[0136] Figure 11 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 11 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0137] like Figure 11 As shown, the electronic device 1100 according to an embodiment of the present application includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory 1102 or a program loaded from a storage portion 1108 into a random access memory 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0138] Various programs and data required for the operation of the electronic device 1100 are stored in the random access memory 1103. The processor 1101, the read-only memory 1102, and the random access memory 1103 are connected to each other via a bus 1104. The processor 1101 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the read-only memory 1102 and / or the random access memory 1103. It should be noted that the program can also be stored in one or more memories other than the read-only memory 1102 and the random access memory 1103. The processor 1101 can also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0139] According to an embodiment of the present application, electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to bus 1104. Electronic device 1100 may also include one or more of the following components connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or modem. Communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1110 as needed, so that computer programs read from the removable media can be installed into storage section 1108 as needed.

[0140] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0141] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0142] According to embodiments of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0143] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0144] When the computer program is executed by the processor 1101, the above functions defined in the system / device of the embodiment of the present application are performed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0145] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1109, and / or installed from removable media 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0146] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0147] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A training method for an implicit question reasoning model, characterized in that: include: Obtaining an inference training set, wherein the inference training set includes a plurality of training samples, each of the training samples includes a scene video of a target area and a plurality of scene instruction pairs, the scene instruction pairs include a plurality of implicit questions for the target area and an answer label corresponding to each implicit question, the implicit question being an implicit expression of the target object to be found; For each of the training samples, constructing a three-dimensional model of the target area according to the scene video; Inputting the scene video and the implicit question into a visual language model and outputting semantic features; Inputting the scene video and the semantic features into a segmentation base model, and outputting a plurality of mask images, wherein the mask images form mask labels on a plurality of initial objects corresponding to the implicit question; Processing the plurality of mask images and the semantic features using a perceptual fusion model to obtain a target fusion feature, wherein the target fusion feature represents the plurality of fused initial objects; Processing the three-dimensional model, the semantic features, and the target fusion features using a preset language model to output an inference answer, wherein the initial inference model includes the visual language model, the segmentation base model, the perceptual fusion model, and the preset language model; Adjusting the model parameters of the initial reasoning model according to the reasoning answer and the answer label to obtain a target reasoning model; The inference training set is generated in the following way: Acquire an information set of the target area, wherein the information set includes a three-dimensional point cloud and the scene video; Projecting the three-dimensional point cloud into a two-dimensional image space to obtain perspective images at different perspectives; Inputting the plurality of perspective images into the segmentation basic model to obtain segmentation mask labels corresponding to different objects; Inputting the plurality of perspective images into the visual language model, and outputting object attribute labels corresponding to different objects; Inputting the plurality of perspective images, the plurality of segmentation mask labels, and the plurality of object attribute labels into the preset language model, and outputting a plurality of the scene instruction pairs; The reasoning training set is generated according to the scene video and a plurality of the scene instruction pairs.

2. The method according to claim 1, characterized in that The scene video includes a plurality of initial channel images of different color channels and a plurality of initial depth images corresponding to each color channel; Wherein, constructing the three-dimensional model of the target area according to the scene video includes: Based on a preset selection rule, screening out a plurality of target channel images from the plurality of initial channel images; Based on the target channel image, screening out a plurality of target depth image frames from the plurality of initial depth images; A truncated sign function is used to process a plurality of target depth image frames to obtain a three-dimensional model of the target area.

3. The method according to claim 1, characterized in that The scene video and the implicit question are input into a visual language model, and semantic features are output, including: Inputting the scene video into a visual encoder and outputting a first image coding vector; The first image encoding vector and the implicit question are input into a multimodal language model, and the semantic features are output, wherein the semantic features represent object attributes of different initial objects.

4. The method according to claim 1, wherein The scene video and the semantic features are input into a segmentation base model, and a plurality of mask images are output, including: Processing the scene video using an image encoder to obtain a second image coding vector; The second image coding vector and the semantic feature are processed by an image decoder to obtain a plurality of mask images.

5. The method according to claim 1, wherein The perceptual fusion model is used to process the plurality of mask images and the semantic features to obtain a target fusion feature, including: For each mask image, back-projecting the mask image into a three-dimensional space to obtain a three-dimensional space model, wherein the three-dimensional space model includes a plurality of three-dimensional voxel features and three-dimensional boundary information corresponding to the target object; Inputting a data set of the plurality of mask images into a feature extractor and outputting a target extracted feature, wherein the data set includes the plurality of the three-dimensional voxel features, the three-dimensional boundary information, and the semantic features; Processing the target extracted features using a conversion network and a first multi-layer perceptron respectively to obtain conversion features and confidence matrices corresponding to different initial objects, wherein the confidence matrices include confidences corresponding to different initial objects; For any of the confidence levels, if the confidence level is less than a confidence threshold, deleting the matrix elements of the initial object corresponding to the confidence level in the conversion feature and the confidence matrix to obtain a new conversion feature and a new confidence matrix; Calculating a similarity matrix based on the new conversion features and the new confidence matrix, wherein each element in the similarity matrix represents the similarity between any two initial objects; For any of the similarities, when the similarity is greater than a similarity threshold, fusing initial features of two initial objects corresponding to the similarity to obtain an initial fused feature; The target fused feature is generated according to at least one of the target fused features and a plurality of initial features that have not been fused.

6. The method according to claim 5, characterized in that The feature extractor includes a multi-layer perceptron and L masked cross-attention modules; The data set of multiple mask images is input into the feature extractor, and the target extracted features are output, including: Processing the three-dimensional boundary information using the multi-layer perceptron to obtain boundary perception features; L masked cross-attention modules are used to process the boundary perception features, the three-dimensional voxel features and the semantic features to obtain fused target extraction features.

7. The method according to claim 6, characterized in that Using L masked cross-attention modules to process the boundary perception feature, the three-dimensional voxel feature, and the semantic feature to obtain a fused target extraction feature, including: For the mth masked crisscross attention module, use the first masked crisscross attention mechanism layer to process the input feature and the three-dimensional voxel feature to obtain a first attention feature, wherein when m=1, the input feature is the boundary-aware feature, and when m≠1, the input feature is the output feature output by the previous masked crisscross attention module; Processing the input feature and the first attention feature using a first residual normalization layer to generate a first normalized feature; Processing the first normalized feature and the semantic feature using a second masked cross-attention mechanism layer to obtain a second attention feature; Processing the first normalized feature and the second attention feature using a second residual normalization layer to generate a second normalized feature; Processing the second normalized features using a feedforward neural network to obtain nonlinear transformation features; The second normalized features and the nonlinear transformation features are processed by a third residual normalization layer to generate the output features, wherein when m=L, the output features represent the target extraction features.

8. The method according to claim 1, characterized in that Processing the three-dimensional model, the semantic features, and the target fusion features using a preset language model to output an inference answer includes: generating a three-dimensional fused image according to the target fusion feature and the three-dimensional model; The three-dimensional fused image and the semantic features are input into the preset language model, and the inference answer corresponding to the implicit question is output, wherein the inference answer represents the position information of the target object corresponding to the implicit question in the three-dimensional fused image.

9. An implicit problem reasoning method, characterized in that include: Obtain the regional video and the question to be reasoned of the area to be reasoned; Constructing a three-dimensional regional model of the area to be inferred based on the regional video; Inputting the question to be inferred and the three-dimensional region model into a target reasoning model, and outputting a reasoning answer corresponding to the question to be inferred, wherein the reasoning answer represents the position information of the required object corresponding to the implicit question; Wherein, the target reasoning model is obtained by training using the method described in any one of claims 1 to 8.

Images