Multimodal interactive question answering method, device, equipment, product and storage medium

Abstract

The application relates to the technical field of interactive question and answer, and provides a multi-modal interactive question and answer method, device, equipment, product and storage medium. The method comprises the following steps: inputting multi-modal interactive information into a trained multi-modal interactive question and answer model to obtain answer information for question text information output by the trained multi-modal interactive question and answer model; the trained multi-modal interactive question and answer model is used for fusing image information and question text information into image text features, fusing image information and interactive region selection information into image interaction features, and generating answer information based on the fusion of the image text features and the image interaction features. The application introduces interactive region selection information of a user as instant feedback of the user, and can realize effective fusion of instant feedback of the user and traditional multi-modal information such as image information and text information, so that the efficiency and accuracy of model output are significantly improved.

Description

Technical Field

[0001] This application relates to the field of interactive question-answering technology, specifically to a multimodal interactive question-answering method, apparatus, device, product, and storage medium. Background Technology

[0002] In the field of multimodal large-scale models, deep learning models that integrate multiple data types (such as text, images, and audio) have been widely researched and applied. These models, by combining different types of input information, can provide more comprehensive and accurate results.

[0003] However, existing applications of large multimodal models often focus on processing static data inputs, and some do not consider real-time user feedback. In complex and nuanced tasks such as intelligent network expert question-and-answer scenarios, real-time user feedback is crucial. It enables the model to fully understand the user's intent and output the required answer more efficiently and accurately. Although some applications consider real-time user feedback, they fail to effectively integrate it with other traditional multimodal information, thus failing to fully leverage the guiding role of real-time user feedback in the model's answer, resulting in very limited improvements in the model's efficiency and accuracy.

[0004] In summary, existing multimodal large models applied to intelligent online expert question-answering scenarios either fail to consider real-time user feedback or, while taking real-time user feedback into account, cannot effectively integrate real-time user feedback with traditional multimodal information. Consequently, there is still significant room for improvement in the efficiency and accuracy of model output. Summary of the Invention

[0005] This application provides a multimodal interactive question-answering method, apparatus, device, product, and storage medium to address the technical problem that existing multimodal large model applications in network intelligent expert question-answering scenarios cannot effectively integrate real-time user feedback with traditional multimodal information, and the efficiency and accuracy of model output still have considerable room for improvement.

[0006] In a first aspect, embodiments of this application provide a multimodal interactive question-answering method, including: Acquire multimodal interactive information input by the user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; The multimodal interactive information is input into the trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

[0007] In one embodiment, fusing the image information and the question text information into image text features includes: The image information is encoded to obtain image encoding features; The interactive area selection information is used to generate prompts, resulting in interactive image prompts. Based on the image interactive prompts, the image encoding features are decoded to obtain the region image features corresponding to the interactive region selection information; Extract the problem text features from the problem text information; The image features of the region and the text features of the question are fused to obtain image-text features.

[0008] In one embodiment, the interactive region selection information includes multimodal region selection information; the trained multimodal interactive question answering model includes an interactive information aggregation layer, a visual interaction fusion layer, and a deep interaction processing layer. The step of fusing the image information and the interactive region selection information into image interaction features includes: The interactive information aggregation layer is used to aggregate the multimodal region selection information to obtain guidance enhancement features; The visual interaction fusion layer is used to fuse the guidance enhancement features and the image features in the image information to obtain multi-scale fusion features; Using the deep interaction processing layer, the multi-scale fusion features are semantically enhanced to obtain image interaction features.

[0009] In one embodiment, the aggregation of the multimodal region selection information to obtain guided enhancement features includes: The multimodal region selection information is interactively encoded to obtain region selection encoding features; The region selection coding features and the image features in the image information are hierarchically fused to obtain the guided enhancement features.

[0010] In one embodiment, fusing the guidance enhancement features and the image features in the image information to obtain multi-scale fused features includes: For the image features in the image information, calculate the cross attention between each layer of image features and the guided enhancement features to obtain the cross-layer interaction features of each layer of image features; The cross-layer interaction features of the multi-layer image features are adjusted at different scales to obtain multi-scale features; The multi-scale features are fused to obtain multi-scale fused features.

[0011] In one embodiment, the semantic enhancement of the multi-scale fused features to obtain image interaction features includes: The nonlinear semantics of the multi-scale fusion features are enhanced to obtain the first semantically enhanced features; The global contextual semantics of the multi-scale fusion features are enhanced to obtain the second semantically enhanced features; The first semantic enhancement feature and the second semantic enhancement feature are fused together to obtain the image interaction feature.

[0012] In one embodiment, generating response information based on the fusion of the image text features and the image interaction features includes: The image text features and the image interaction features are fused to obtain visual perception fusion features; Based on the aforementioned visual perception fusion features, an encoded answer is generated for the textual information of the question; The encoded response is decoded to obtain the response information in natural language form.

[0013] In one embodiment, the trained multimodal interactive question-answering model is obtained based on the following method: Acquire historical multimodal interactive information; the historical multimodal interactive information includes historical image information, historical interactive region selection information for the historical image information, historical question-answer pair example text information for the historical interactive region selection information, and historical description information for the historical interactive region selection information; The multimodal interactive question-answering model is trained based on the historical multimodal interactive information to obtain the trained multimodal interactive question-answering model.

[0014] Secondly, embodiments of this application provide a multimodal interactive question-answering device, comprising: A multimodal interactive information acquisition module is used to: acquire multimodal interactive information input by a user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; A multimodal interactive question answering module is used to: input the multimodal interactive information into a trained multimodal interactive question answering model, and obtain the answer information for the question text information output by the trained multimodal interactive question answering model; The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the multimodal interactive question-and-answer method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal interactive question-and-answer method described in the first aspect.

[0017] Fifthly, embodiments of this application provide a non-transitory computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the multimodal interactive question-and-answer method described in the first aspect.

[0018] The multimodal interactive question answering method, apparatus, device, product, and storage medium provided in this application acquire multimodal interactive information input by the user. The multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information. The multimodal interactive information is input into a trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question answering model is used to fuse image information and question text information into image text features, and to fuse image information and interactive region selection information into image interaction features. Based on the fusion of image text features and image interaction features, the answer information is generated. The multimodal interactive question-answering model trained in this application is applied to intelligent expert question-answering scenarios in the network. On the one hand, it introduces the user's interactive region selection information as the user's immediate feedback. On the other hand, it fuses this interactive region selection information with image information, that is, it embeds the user's immediate feedback into the image information to obtain image interaction features. It then fuses the image information with the question text information to obtain image text features. Finally, it generates answer information based on the fusion of image interaction features and image text features. This achieves effective fusion of user's immediate feedback with traditional multimodal information such as image and text information. This process not only enhances the interaction between the user and the model but also makes the model output more aligned with the user's needs and intentions, thereby significantly improving the efficiency and accuracy of the model's output answer information. In summary, this application introduces user's interactive region selection information as the user's immediate feedback and can effectively fuse user's immediate feedback with traditional multimodal information such as image and text information, thereby significantly improving the efficiency and accuracy of the model's output. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 2 This is the second flowchart of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 3 This is the third flowchart of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 4 This is the fourth flowchart of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 5 This is the fifth flowchart illustrating the multimodal interactive question-answering method provided in the embodiments of this application; Figure 6 This is the sixth flowchart of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 7 This is the seventh flowchart of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 8 This is the eighth flowchart of the multimodal interactive question-answering method provided in the embodiments of this application; Figure 9 This is a schematic diagram of historical multimodal interactive information provided in an embodiment of this application; Figure 10 This is a schematic diagram of the rack image information input and the area information selected by the closed rectangle for the rack image information in the application example provided in this application embodiment; Figure 11 This is a schematic diagram of the structure of the multimodal interactive question-and-answer device provided in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Figure 1 This is one of the flowcharts illustrating the multimodal interactive question-answering method provided in this application embodiment. (Refer to...) Figure 1 This application provides a multimodal interactive question-answering method, which may include: Step 101: Obtain multimodal interactive information input by the user; Multimodal interactive information includes image information, interactive region selection information for image information, and question text information for interactive region selection information; Step 102: Input the multimodal interactive information into the trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model.

[0023] The trained multimodal interactive question answering model is used to fuse image information and question text information into image text features, and to fuse image information and interactive region selection information into image interaction features. Based on the image text features and image interaction features, the model generates answer information.

[0024] In step 101, the interactive region selection information may include region information selected by the user in the image using various interactive methods, such as point selection, drawing selection, and mask selection. Point selection means that the user selects an image region by clicking, drawing selection means that the user selects an image region by drawing lines, frames, or other closed or open shapes, and mask selection means that the user selects an image region by covering a closed shape such as a mask. Therefore, the interactive region selection information may include region information selected by a set of point coordinates, region information selected by the border coordinates of closed or open shapes, etc.

[0025] Furthermore, all of the above interaction methods can be implemented through gestures, touch, etc., and the relevant information can be input into the trained multimodal interactive question answering model to interact with the model.

[0026] It should be noted that different regions of the same image can be selected using only one of the point selection, drawing selection, or mask selection methods, or two or more of these methods can be used to select different regions of the same image. Similarly, the same region of the same image can be selected using only one of the point selection, drawing selection, or mask selection methods, or two or more of these methods can be used to select the same region of the same image. No limitation is imposed here.

[0027] The question text information refers to the specific question text information that the user raises regarding the selected area information, such as, "Please describe this area." In step 102, after inputting multimodal interactive information into the trained multimodal interactive question-answering model, the model fuses traditional multimodal information, namely image information and text information, into image-text features, and embeds interactive region selection information into image information, thereby fusing the two into image interactive features. Then, the image-text features are fused with the image interactive features to achieve effective fusion of interactive region selection information and traditional multimodal information, that is, effective fusion of user instant feedback and traditional multimodal information. As a result, the model can output more efficient and accurate answers based on the user's instant feedback, enhance the user experience, and significantly improve its flexibility and adaptability. It can better serve various complex network intelligent expert question-answering tasks and provide answers that are more in line with user needs.

[0028] The multimodal interactive question answering method provided in this embodiment acquires multimodal interactive information input by the user. The multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information. The multimodal interactive information is input into a trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question answering model is used to fuse image information and question text information into image text features, and to fuse image information and interactive region selection information into image interaction features. Based on the fusion of image text features and image interaction features, the answer information is generated. This embodiment applies the trained multimodal interactive question-answering model to a network-based intelligent expert question-answering scenario. On one hand, it incorporates the user's interactive region selection information as immediate feedback. On the other hand, it fuses this interactive region selection information with image information, embedding the user's immediate feedback into the image information to obtain image interaction features. It then fuses the image information with the question text information to obtain image text features. Finally, it generates answer information based on the fusion of image interaction features and image text features. This achieves effective fusion of user's immediate feedback with traditional multimodal information such as image and text information. This process not only enhances the interaction between the user and the model but also makes the model output more aligned with the user's needs and intentions, thereby significantly improving the efficiency and accuracy of the model's output answer information. In summary, this embodiment introduces user's interactive region selection information as immediate feedback and effectively fuses it with traditional multimodal information such as image and text information, thus significantly improving the efficiency and accuracy of the model's output.

[0029] Furthermore, this embodiment provides users with multiple interaction methods such as point selection, drawing selection, and mask selection, as well as multiple implementation methods such as gestures and touch control. This not only enriches the means of inputting user interaction information, but also allows users to adjust the area selection information in real time according to their specific needs, thereby improving the efficiency and accuracy of information exchange.

[0030] Figure 2 This is the second flowchart illustrating the multimodal interactive question-answering method provided in this application embodiment. (Refer to...) Figure 2 In one embodiment, step 102 may include: Step 201: Encode the image information to obtain image coding features; Step 202: Generate prompts for the interactive area selection information to obtain interactive prompts for the image; Step 203: Based on the interactive prompts in the image, decode the image encoding features to obtain the region image features corresponding to the interactive region selection information; Step 204: Extract the problem text features from the problem text information; Step 205: Fuse the regional image features and the question text features to obtain image text features.

[0031] In step 201, an image encoder can be used to encode the image information, thereby converting the image information into image encoded features containing rich feature information; the image encoder can adopt architectures such as ResNet (Residual Network), Swin (Shifted windows), and ViT (Vision Transformer).

[0032] In step 202, for each interaction method, corresponding image interactive prompts are generated for the selected region information. These prompts can be used to guide subsequent image processing and information fusion.

[0033] In step 203, an image decoder can be used to decode the image encoding features based on the guidance of the image interactive prompts, and extract the image features of the region information selected for each interaction method, i.e., the region image features.

[0034] In step 204, any method can be used to extract the problem text features from the problem text information, such as bag-of-words model, word embedding, term frequency-inverse document frequency, etc., and no limitation is made here.

[0035] In step 205, the regional image features and the question text features are fused together. That is, the regional image features of the user interactive selection area are fused with the question text features of the user interactive selection area to obtain an intermediate representation of the user interactive selection area that combines image features and text features, namely image-text features.

[0036] In this embodiment, interactive region selection information is also introduced to guide the fusion of image information and question text information, so that the fused image information and question text information are focused on the user's interactive selection area. The regional image features and question text features of this area are fused to obtain image and text features that accurately represent the user's interactive selection area.

[0037] Figure 3 This is the third flowchart illustrating the multimodal interactive question-answering method provided in this application. (Refer to...) Figure 3 In one embodiment, the interactive region selection information includes multimodal region selection information, namely, region information selected by various interaction methods such as point selection, drawing selection, and mask selection; the trained multimodal interactive question answering model includes an interactive information aggregation layer, a visual interaction fusion layer, and a deep interaction processing layer; step 102 may include: Step 301: Using the interactive information aggregation layer, aggregate the multimodal region selection information to obtain guided enhancement features; Step 302: Using the visual interaction fusion layer, the guided enhancement features and image features in the image information are fused to obtain multi-scale fusion features; Step 303: Using a deep interaction processing layer, semantic enhancement is performed on the multi-scale fusion features to obtain image interaction features.

[0038] In step 301, as mentioned above, the region information selected by different interaction methods is represented in different forms, such as a set of point coordinates, or the coordinates of the border of a closed or open shape. If multiple interaction methods are used to select the same region in the image at the same time, then the region corresponds to multiple modalities of region selection information. By using the interaction information aggregation layer to aggregate these modalities of region selection information, all forms of region information can be integrated, significantly enhancing the ability to understand and represent user intentions and needs, and obtaining more guiding enhancement features.

[0039] In step 302, the image features in the image information can be the aforementioned regional image features. Since the guidance enhancement features have stronger guidance capabilities, the visual interaction fusion layer can be used to extract and fuse the refined features in the regional image features that match the user's needs and intentions based on the guidance enhancement features. These refined features are multi-scale, thus obtaining multi-scale fusion features.

[0040] In step 303, the deep interaction processing layer is used to semantically enhance the multi-scale fusion features, so that the obtained image interaction features not only retain the original spatial detail information, but also significantly enhance the semantic representation ability.

[0041] This embodiment utilizes an interactive information aggregation layer to aggregate region selection information from multiple modalities corresponding to the same area, resulting in more guiding enhancement features. These enhanced features are then financially fused with regional image features to extract refined features from the regional image features that align with user needs and intentions. A visual interaction fusion layer is then used to fuse these refined features into multi-scale fusion features. Finally, a deep interaction processing layer is used to semantically enhance the multi-scale fusion features, thereby obtaining image interaction features that accurately represent the user's interactive selection region and possess both refined spatial detail information and enhanced semantic representation capabilities.

[0042] Figure 4 This is the fourth flowchart illustrating the multimodal interactive question-answering method provided in this application. (Refer to...) Figure 4 In one embodiment, step 301 may include: Step 401: Perform interactive encoding on the multimodal region selection information to obtain region selection encoded features; Step 402: Hierarchically fuse the region selection coding features and image features in the image information to obtain guided enhancement features.

[0043] In step 401, the region selection coding features can be obtained based on the following formula. : ; in, For the point set of the area selected by the user using a point-and-click method, for The coordinates of any point in the middle, This refers to the bounding box set of the same area selected by the user using either drawing selection or masking selection methods. for The coordinates of any border in the middle, The region weight selected for the point selection method. The region weights selected for drawing selection and masking selection methods. Let G be the coordinates of any point in the image. Let be the standard deviation of the Gaussian distribution, then Indicates The region selected by the point selection method is encoded using the center of the Gaussian kernel function. Indicates generation The corresponding binary mask; where, and All of them can be adaptively adjusted according to specific application scenarios to dynamically adjust the contribution of different interaction methods and optimize the quality of feature representation.

[0044] In step 402, the image features in the image information can be the aforementioned regional image features, and the guided enhancement features can be obtained based on the following formula. : ; in, express convolution, express convolution, , For regional image features, then Indicates will and After splicing, the splicing features are analyzed. Convolution operation, Indicates will and After splicing, the splicing features are analyzed. Convolution operations, specifically convolution operations with two kernels of different scales, can... and Features are respectively merged into features at different levels of abstraction, and then features at different levels of abstraction are merged into... ,but exist Further integration on the basis It possesses all the key features and has a stronger guiding ability. To adjust parameters, used to balance the relative importance of features at different levels of abstraction. It can be flexibly configured according to specific application scenarios.

[0045] This embodiment first interactively encodes the multimodal region selection information of the same area to better capture user needs and intentions, obtaining region selection encoding features. Then, it uses convolution with convolution kernels of different scales to hierarchically fuse the region selection encoding features and the region image features, so as to integrate all the key features of the region image features into the region selection encoding features, resulting in more guiding enhancement features. These guiding enhancement features can not only accurately reflect the user's interaction needs and intentions, but also enhance the model's resistance to noise and variant samples due to the adoption of a multi-level feature fusion strategy. They can adapt to various user interaction modes more flexibly, greatly improving the model's stability and generalization ability in complex scenarios.

[0046] Figure 5 This is the fifth flowchart illustrating the multimodal interactive question-answering method provided in this application. (Refer to...) Figure 5 In one embodiment, step 302 may include: Step 501: For the image features in the image information, calculate the cross attention between each layer of image features and the guided enhancement features to obtain the cross-layer interaction features of each layer of image features. Step 502: Adjust the cross-layer interaction features of the multi-layer image features at different scales to obtain multi-scale features; Step 503: Fuse the multi-scale features to obtain multi-scale fused features.

[0047] In step 501, the image features in the image information can be the aforementioned regional image features. The image features include multiple layers of image features at different levels of abstraction, and the first layer can be obtained based on the following formula. Layer image features Cross-layer interaction features : ; in, These are adaptive weights used to adjust the contribution of features from each layer. The set of attention parameters learned. Indicates in Under the influence of the calculation and Deep cross attention, The various parameters work together to achieve and More accurate feature fusion.

[0048] because Fusion Key features at all levels, thus enabling guidance in deep cross-attention computation. In-depth attention middle The multi-level key features, and thus obtain The cross-layer perceptual features are then fused with the image features of the current layer to obtain the final cross-layer interactive features. .

[0049] In steps 502 to 503, the image features of each layer are first upsampled at different scales to obtain multi-scale features. Then, the image features of different scales from each layer are fused based on gating coefficients to obtain multi-scale fused features. Specifically, the multi-scale fused features can be obtained based on the following formula. : ; in, for The number of image feature layers, The gating coefficients obtained after training, and the weights This relates to controlling the contribution ratio of image features in each layer. For those with adjustable parameters The upsampling operation allows the model to flexibly adjust the scale of the feature map as needed.

[0050] The design of adaptive weights and adjustable parameters enables the visual interaction fusion layer to flexibly cope with various application scenarios, reduces unnecessary computation, enhances the model's versatility, and improves the model's processing efficiency; the design of the gating coefficients effectively prevents overfitting and improves the model's stability and generalization ability.

[0051] This embodiment first calculates the cross-attention between each layer of image features and the guided enhancement features to more effectively capture the deep correlation between features at different levels, thus obtaining the cross-layer interaction features of each layer of image features. Then, the scale of the cross-layer image features of each layer of image features is adjusted to adapt to the expression requirements of cross-layer image features of different levels of image features, thus obtaining multi-scale features. Finally, the multi-scale features are fused based on the gating coefficient to improve the fusion effect. Due to the adoption of a more refined feature fusion strategy, the final multi-scale fused features have higher resolution and stronger expressiveness, which can significantly improve the performance of subsequent visual tasks.

[0052] Figure 6 This is the sixth flowchart illustrating the multimodal interactive question-answering method provided in this application. (Refer to...) Figure 6 In one embodiment, step 303 may include: Step 601: Enhance the nonlinear semantics of the multi-scale fusion features to obtain the first semantically enhanced features; Step 602: Enhance the global contextual semantics of the multi-scale fusion features to obtain the second semantically enhanced features; Step 603: Fuse the first semantic enhancement feature and the second semantic enhancement feature to obtain the image interaction feature.

[0053] In step 601, any enhancement method can be used to enhance the nonlinear semantics of the multi-scale fusion features; no limitation is made here. In this embodiment, a multilayer perceptron can be used to enhance the nonlinear semantics of the multi-scale fusion features.

[0054] In step 602, any enhancement method can be used to enhance the global contextual semantics of the multi-scale fused features; no limitation is made here. In this embodiment, a Transformer block can be used to enhance the global contextual semantics of the multi-scale fused features.

[0055] In step 603, the image interaction features can be obtained based on the following formula. : ; in: ; in, It is a multilayer perceptron. For Transformer blocks, A dynamic weight between 0 and 1 This is global average pooling. If it is an activation function, then For the first semantic enhancement feature, This is a second semantic enhancement feature. Used according to The global average pooling result is used to adaptively adjust the contribution ratio of the multilayer perceptron and Transformer blocks.

[0056] Among them, the multilayer perceptron can adopt a deeper and wider network structure to enhance the nonlinear semantics of multi-scale fusion features, while the Transformer block introduces position encoding and attention mechanisms to capture the long-distance dependence of multi-scale fusion features and enhance their global contextual semantics.

[0057] This embodiment combines the advantages of multilayer perceptron and Transformer blocks to enhance the nonlinear semantics and global contextual semantics of multi-scale fusion features, and adopts a dynamic weight adjustment mechanism to fuse the enhanced first and second semantic features. This allows the model to flexibly adjust the expression requirements of different semantic enhancement features according to different image content, so that the image interaction features not only have enhanced semantic features adapted to the image content, but also inject multi-scale features adapted to the user's interaction guidance intention, thereby significantly improving the model's performance on complex visual tasks.

[0058] Figure 7 This is the seventh flowchart illustrating the multimodal interactive question-answering method provided in this application. (Refer to...) Figure 7 In one embodiment, step 102 may include: Step 701: Fuse the image text features and image interaction features to obtain visual perception fusion features; Step 702: Based on visual perception fusion features, generate an encoded answer for the question text information; Step 703: Decode the encoded response to obtain the response information in natural language form.

[0059] In step 701, the image text features and image interaction features are fused together to obtain visual perception fusion features that effectively integrate real-time user feedback with traditional multimodal information such as image information and text information.

[0060] In step 702, a large language model can be invoked, and the visual perception fusion features can be input into the large language model. The large language model can understand and reason based on the visual perception fusion features to generate an encoded answer for the question text information.

[0061] In step 703, a text decoder can be used to decode the encoded answer and convert it into natural language text that is easy for users to understand, i.e., answer information. The answer information could be "Two people can be seen working in this area. They are inspecting and surveying the well." Reference Figure 8 In one embodiment, the entire process of the multimodal interactive question-answering method of this application is briefly described as follows: Acquire multimodal interactive information from user input; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; the question text information may be "Please describe this region."; The multimodal interactive information is input into the trained multimodal interactive question-answering model. The internal process of the model includes: Image information is encoded using an image encoder to obtain image encoding features. Interactive region selection information is then generated to obtain interactive image prompts. Based on these interactive image prompts, the image encoding features are decoded using an image decoder to obtain the region image features corresponding to the interactive region selection information. Image-text fusion mapping is performed, which involves extracting the problem text features from the problem text information, fusing the region image features and the problem text features to obtain image-text features; Image interaction fusion mapping is performed by using an interaction information aggregation layer to aggregate multimodal region selection information to obtain guidance enhancement features, using a visual interaction fusion layer to fuse guidance enhancement features and region image features to obtain multi-scale fusion features, and using a deep interaction processing layer to semantically enhance the multi-scale fusion features to obtain image interaction features. Visual perception fusion mapping is performed, which involves fusing image text features and image interaction features to obtain visual perception fusion features; By calling the large language model and inputting the visual perception fusion features into the large language model, an encoded answer is generated for the text information of the question. The encoded response is decoded using a text decoder to obtain the response information in natural language form; this response information could be, "Two people can be seen working in this area; they are inspecting and surveying the well."

[0062] In one embodiment, the trained multimodal interactive question-answering model can be obtained in the following way: Acquire historical multimodal interactive information; historical multimodal interactive information includes historical image information, historical interactive region selection information for historical image information, historical question-answer pair example text information for historical interactive region selection information, and historical description information for historical interactive region selection information; The multimodal interactive question answering model is trained based on historical multimodal interactive information to obtain the trained multimodal interactive question answering model.

[0063] In the field of electronic equipment assembly, such as the assembly of equipment in various cabinets in a computer room, it is possible to acquire image information of each cabinet, point selection area information, drawing selection area information, masking selection area information, question-and-answer pair example text information for these areas, and descriptive information for these areas, and use this information as historical multimodal interactive information.

[0064] The question-and-answer example text and description information can be derived from industry-certified usage methods and standards for equipment and facilities within data centers, as well as usage methods and standards for construction external line equipment and facilities. Furthermore, colored text labels can be added to historical multimodal interactive information to differentiate between rack structures and equipment power-on status. For example, using interactive area selection information as the area selected by a closed rectangle, historical multimodal interactive information can be displayed as follows: Figure 9 As shown.

[0065] Furthermore, an application example is provided for the trained multimodal interactive question-answering model: The input rack image information and the area information selected by the closed rectangle for that rack image information are as follows: Figure 10 As shown, if the input question text regarding the information in this area is "What device is this in the picture, and what maintenance guidelines are available?", then the answer information output by the trained multimodal interactive question answering model is as follows: ## I. Image Equipment Description: XXX Series OTN Judging from the image: - The central slot area is Service boards (such as OCH, OTU, ODU, XCS, etc.) - The upper part is Optical cross-connect / uplink fiber access - The lower part is Power input, fan module, ground connection, etc. - Dense fiber optic network, standardized modules, basically in compliance with regulations. XXX Series equipment layout This equipment is typically deployed in: - City Core / Gathering Layer - Wavelength division multiplexing, OTN scheduling, and transparent forwarding at the transport layer - Interoperable with SDH / PTN / Ethernet, enabling converged L0 / L1 / L2 transport. --- ## II. XXX Operation and Maintenance Guide ### 1. Routine Inspection Items (Daily automated inspection recommended): | Inspection Items | Description | |---------------|----------------------------------------------| | Board Status | Check if all service boards, power boards, and cross-connect boards are online and in normal status | | Interface Status | Check if each OCH / OTU interface is up and if the optical power is abnormal | | Alarm Check | Check for local alarms and remote alarms (LOS, LOF, AIS, SF, etc.) | | Environmental Monitoring | Are the temperature, humidity, fan speed, and power input normal? | | Power Supply Check | Check if the dual power supply is stable, battery status, and grounding connection. | | Optical Module Check | Check if the transmit and receive power are within the allowable range (±5dBm) | --- ### 2. Operation and maintenance commands (via XXX command line or network management XXX platform): Typically accessed via serial port / SSH / network management platform. ####

Basic Status Check

Optical Power Measurement

Port Configuration and Performance

[0066] --- ## III. Recommended Operation and Maintenance Tools - XXX Network Management System - Supports graphical alarms, end-to-end paths, optical power trend graphs, batch upgrade configurations, etc. - XXX (command line) - Serial port / SSH access, suitable for scripted operation and maintenance; - XXX Assistant - A mobile app, in conjunction with a QR code, can quickly identify circuit boards and locate faults.

[0067] --- ## Summarize The XXX series OTN device in your picture is an important component of the core transmission system, mainly used for: - Metropolitan Area Backbone Wavelength Division Dispatch - DWDM / OTN circuit transparent transmission - L1 / L2 / L3 hierarchical support key convergence platform Recommended use Network management system + command-line scripted inspection The dual-mechanism system, along with the standard operating procedure (SOP), ensures stable operation 24 / 7.

[0068] The English abbreviations in this embodiment are explained as follows: - OLT (Optical Line Terminal); - SPN (Slicing Packet Network); - OTN (Optical Transport Network); - OCH (Optical Channel, or Optical Channel Layer); - OUT (Optical channel Transport Unit); - ODU (Optical Channel Data Unit); - XCS (Cross-Connect and Synchronization). - SDH (Synchronous Digital Hierarchy). - PTN (Packet Transport Network); - AIS (Alarm Indication Signal). - SSH (Secure Shell, or Secure Remote Login Protocol). - Telnet (Teletype Network or Telecommunications Network, remote login protocol); - DWDM (Dense Wavelength Division Multiplexing).

[0069] This embodiment extends the multimodal interactive question-and-answer model to the field of electronic device assembly. Users can interact directly with the model through gestures or touch to obtain the required answers more accurately and efficiently, thus solving the applicability and efficiency problems of existing technologies in this field.

[0070] The multimodal interactive question-answering device provided in the embodiments of this application is described below. The multimodal interactive question-answering device described below can be referred to in correspondence with the multimodal interactive question-answering method described above.

[0071] Figure 11 This is a schematic diagram of the structure of the multimodal interactive question-and-answer device provided in an embodiment of this application. (Refer to...) Figure 11 This application provides a multimodal interactive question-answering device, which may include: The multimodal interactive information acquisition module 1101 is used to: acquire multimodal interactive information input by the user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; The multimodal interactive question answering module 1102 is used to: input the multimodal interactive information into the trained multimodal interactive question answering model, and obtain the answer information for the question text information output by the trained multimodal interactive question answering model; The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

[0072] The multimodal interactive question-answering device provided in this embodiment acquires multimodal interactive information input by the user. The multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information. The multimodal interactive information is input into a trained multimodal interactive question-answering model to obtain the answer information for the question text information output by the trained multimodal interactive question-answering model. The trained multimodal interactive question-answering model is used to fuse image information and question text information into image text features, and to fuse image information and interactive region selection information into image interaction features. Based on the fusion of image text features and image interaction features, the answer information is generated. This embodiment applies the trained multimodal interactive question-answering model to a network-based intelligent expert question-answering scenario. On one hand, it incorporates the user's interactive region selection information as immediate feedback. On the other hand, it fuses this interactive region selection information with image information, embedding the user's immediate feedback into the image information to obtain image interaction features. It then fuses the image information with the question text information to obtain image text features. Finally, it generates answer information based on the fusion of image interaction features and image text features. This achieves effective fusion of user's immediate feedback with traditional multimodal information such as image and text information. This process not only enhances the interaction between the user and the model but also makes the model output more aligned with the user's needs and intentions, thereby significantly improving the efficiency and accuracy of the model's output answer information. In summary, this embodiment introduces user's interactive region selection information as immediate feedback and effectively fuses it with traditional multimodal information such as image and text information, thus significantly improving the efficiency and accuracy of the model's output.

[0073] Furthermore, this embodiment provides users with multiple interaction methods such as point selection, drawing selection, and mask selection, as well as multiple implementation methods such as gestures and touch control. This not only enriches the means of inputting user interaction information, but also allows users to adjust the area selection information in real time according to their specific needs, thereby improving the efficiency and accuracy of information exchange.

[0074] In one embodiment, the multimodal interactive question-answering module 1102 is specifically used for: The image information is encoded to obtain image encoding features; The interactive area selection information is used to generate prompts, resulting in interactive image prompts. Based on the image interactive prompts, the image encoding features are decoded to obtain the region image features corresponding to the interactive region selection information; Extract the problem text features from the problem text information; The image features of the region and the text features of the question are fused to obtain image-text features.

[0075] In one embodiment, the multimodal interactive question-answering module 1102 is specifically used for: The interactive region selection information includes multimodal region selection information; the trained multimodal interactive question answering model includes an interactive information aggregation layer, a visual interaction fusion layer, and a deep interaction processing layer. The interactive information aggregation layer is used to aggregate the multimodal region selection information to obtain guidance enhancement features; The visual interaction fusion layer is used to fuse the guidance enhancement features and the image features in the image information to obtain multi-scale fusion features; Using the deep interaction processing layer, the multi-scale fusion features are semantically enhanced to obtain image interaction features.

[0076] In one embodiment, the multimodal interactive question-answering module 1102 is specifically used for: The multimodal region selection information is interactively encoded to obtain region selection encoding features; The region selection coding features and the image features in the image information are hierarchically fused to obtain the guided enhancement features.

[0077] In one embodiment, the multimodal interactive question-answering module 1102 is specifically used for: For the image features in the image information, calculate the cross attention between each layer of image features and the guided enhancement features to obtain the cross-layer interaction features of each layer of image features; The cross-layer interaction features of the multi-layer image features are adjusted at different scales to obtain multi-scale features; The multi-scale features are fused to obtain multi-scale fused features.

[0078] In one embodiment, the multimodal interactive question-answering module 1102 is specifically used for: The nonlinear semantics of the multi-scale fusion features are enhanced to obtain the first semantically enhanced features; The global contextual semantics of the multi-scale fusion features are enhanced to obtain the second semantically enhanced features; The first semantic enhancement feature and the second semantic enhancement feature are fused together to obtain the image interaction feature.

[0079] In one embodiment, the multimodal interactive question-answering module 1102 is specifically used for: The image text features and the image interaction features are fused to obtain visual perception fusion features; Based on the aforementioned visual perception fusion features, an encoded answer is generated for the textual information of the question; The encoded response is decoded to obtain the response information in natural language form.

[0080] In one embodiment, a model building module (not shown in the figure) is also included for: Acquire historical multimodal interactive information; the historical multimodal interactive information includes historical image information, historical interactive region selection information for the historical image information, historical question-answer pair example text information for the historical interactive region selection information, and historical description information for the historical interactive region selection information; The multimodal interactive question-answering model is trained based on the historical multimodal interactive information to obtain the trained multimodal interactive question-answering model.

[0081] Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 can call a computer program in the memory 1230 to execute the steps of a multimodal interactive question-and-answer method, such as including: Acquire multimodal interactive information input by the user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; The multimodal interactive information is input into the trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

[0082] Furthermore, the logical instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the multimodal interactive question-answering method provided in the above embodiments, such as including: Acquire multimodal interactive information input by the user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; The multimodal interactive information is input into the trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

[0084] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, the computer program being used to cause a processor to execute the steps of the multimodal interactive question-answering method provided in the above embodiments, for example including: Acquire multimodal interactive information input by the user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; The multimodal interactive information is input into the trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

[0085] The non-transitory computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal interactive question-answering method, characterized in that, include: Acquire multimodal interactive information input by the user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; The multimodal interactive information is input into the trained multimodal interactive question answering model to obtain the answer information for the question text information output by the trained multimodal interactive question answering model. The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

2. The multimodal interactive question-answering method according to claim 1, characterized in that, The process of fusing the image information and the question text information into image text features includes: The image information is encoded to obtain image encoding features; The interactive area selection information is used to generate prompts, resulting in interactive image prompts. Based on the image interactive prompts, the image encoding features are decoded to obtain the region image features corresponding to the interactive region selection information; Extract the problem text features from the problem text information; The image features of the region and the text features of the question are fused to obtain image-text features.

3. The multimodal interactive question-answering method according to claim 1, characterized in that, The interactive region selection information includes multimodal region selection information; the trained multimodal interactive question answering model includes an interactive information aggregation layer, a visual interaction fusion layer, and a deep interaction processing layer. The step of fusing the image information and the interactive region selection information into image interaction features includes: The interactive information aggregation layer is used to aggregate the multimodal region selection information to obtain guidance enhancement features; The visual interaction fusion layer is used to fuse the guidance enhancement features and the image features in the image information to obtain multi-scale fusion features; Using the deep interaction processing layer, the multi-scale fusion features are semantically enhanced to obtain image interaction features.

4. The multimodal interactive question-answering method according to claim 3, characterized in that, The aggregation of the multimodal region selection information to obtain guided enhancement features includes: The multimodal region selection information is interactively encoded to obtain region selection encoding features; The region selection coding features and the image features in the image information are hierarchically fused to obtain the guided enhancement features.

5. The multimodal interactive question-answering method according to claim 3, characterized in that, The process of fusing the guided enhancement features and the image features in the image information to obtain multi-scale fusion features includes: For the image features in the image information, calculate the cross attention between each layer of image features and the guided enhancement features to obtain the cross-layer interaction features of each layer of image features; The cross-layer interaction features of the multi-layer image features are adjusted at different scales to obtain multi-scale features; The multi-scale features are fused to obtain multi-scale fused features.

6. The multimodal interactive question-answering method according to claim 3, characterized in that, The semantic enhancement of the multi-scale fused features to obtain image interaction features includes: The nonlinear semantics of the multi-scale fusion features are enhanced to obtain the first semantically enhanced features; The global contextual semantics of the multi-scale fusion features are enhanced to obtain the second semantically enhanced features; The first semantic enhancement feature and the second semantic enhancement feature are fused together to obtain the image interaction feature.

7. The multimodal interactive question-answering method according to claim 1, characterized in that, The process of generating response information based on the fusion of the image text features and the image interaction features includes: The image text features and the image interaction features are fused to obtain visual perception fusion features; Based on the aforementioned visual perception fusion features, an encoded answer is generated for the textual information of the question; The encoded response is decoded to obtain the response information in natural language form.

8. The multimodal interactive question-answering method according to claim 1, characterized in that, The trained multimodal interactive question-answering model was obtained based on the following method: Acquire historical multimodal interactive information; the historical multimodal interactive information includes historical image information, historical interactive region selection information for the historical image information, historical question-answer pair example text information for the historical interactive region selection information, and historical description information for the historical interactive region selection information; The multimodal interactive question-answering model is trained based on the historical multimodal interactive information to obtain the trained multimodal interactive question-answering model.

9. A multimodal interactive question-and-answer device, characterized in that, include: A multimodal interactive information acquisition module is used to: acquire multimodal interactive information input by a user; the multimodal interactive information includes image information, interactive region selection information for the image information, and question text information for the interactive region selection information; A multimodal interactive question answering module is used to: input the multimodal interactive information into a trained multimodal interactive question answering model, and obtain the answer information for the question text information output by the trained multimodal interactive question answering model; The trained multimodal interactive question-answering model is used to fuse the image information and the question text information into image text features, and to fuse the image information and the interactive region selection information into image interaction features. Based on the fusion of the image text features and the image interaction features, the model generates answer information.

10. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal interactive question-answering method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal interactive question-answering method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal interactive question-answering method according to any one of claims 1 to 8.

Images