Semantic tendency detection method and device based on multi-modal feedback, and storage medium

Abstract

The application discloses a kind of based on multi-modal feedback semantic tendency detection method, device and storage medium, it is related to artificial intelligence technical field.Therein, the method comprises: in the case where obtaining user authorization, obtain the text feedback information submitted by user, and at least one aspect word is extracted from text feedback information;According to text feedback information, construct syntax structure;Based on syntax structure and target image associated with text feedback information submitted by user, generate multi-modal semantic feature for aspect word;Text feedback information and multi-modal semantic feature are combined as structured prompt sequence, and structured prompt sequence is input to target model, and the target semantic tendency information corresponding to aspect word is generated by target model to structured prompt sequence carries out semantic reasoning.The application solves the technical problem that the accuracy of semantic tendency analysis of multi-modal feedback is low due to the lack of fine-grained semantic alignment of aspect word in the prior art.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a semantic tendency detection method, apparatus, and storage medium based on multimodal feedback. Background Technology

[0002] In financial services scenarios, users often express their feedback on financial products or services through text comments accompanied by images. However, existing methods, when processing such multimodal feedback, typically fuse text and images globally, without establishing fine-grained semantic alignment mechanisms for specific aspect terms explicitly mentioned by the user. This lack of fine-grained aspect-level analysis capabilities results in low accuracy in semantic tendency analysis of multimodal feedback. Furthermore, due to the lack of precise matching between the syntactic relevance of aspect terms in the text's syntactic structure and their semantic orientation with image content, the model is prone to misincorporating irrelevant visual information or unrelated textual components into the tendency judgment, leading to a significant deviation between the output and the user's true intent.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a semantic tendency detection method, apparatus, and storage medium based on multimodal feedback, to at least solve the technical problem in the prior art where the accuracy of semantic tendency analysis based on multimodal feedback is low due to the lack of fine-grained semantic alignment of counterpart words.

[0005] According to one aspect of the embodiments of this application, a semantic tendency detection method based on multimodal feedback is provided, comprising: obtaining text feedback information submitted by the user with user authorization, and extracting at least one aspect word from the text feedback information, wherein the aspect word is a noun phrase mentioned by the user in the text feedback information to express an objective evaluation of a product, service, or process; constructing a syntactic structure based on the text feedback information, wherein the syntactic structure is a tree structure with words as nodes and grammatical dependencies as directed edges, used to represent the grammatical dominance relationships between words; and based on the syntactic structure and the user-submitted text feedback information related to the text feedback information... The target image is linked to generate multimodal semantic features for aspect words. These multimodal semantic features include syntactic features representing aspect words in text feedback information extracted from syntactic structure, and visual semantic descriptions related to aspect words inferred from the target image. The text feedback information and multimodal semantic features are combined into a structured prompt sequence, which is then input into the target model. The target model performs semantic reasoning on the structured prompt sequence to generate target semantic tendency information corresponding to aspect words. The structured prompt sequence is a text string generated by concatenating aspect words, syntactic features, and visual semantic descriptions according to a preset template.

[0006] Optionally, at least one aspect term is extracted from the text feedback information, including: based on a preset financial domain dictionary, detecting keywords describing the user's perceived object from the text feedback information as aspect terms; wherein, the user's perceived object is the entity or attribute that the user expresses an objective evaluation in the text feedback information, the entity includes products, services, systems or institutions, and the attribute includes performance, function, experience or process.

[0007] Optionally, constructing a syntactic structure based on text feedback information includes: performing dependency parsing on the text feedback information to generate a tree structure with words as nodes and grammatical dependency relations as directed edges, as the syntactic structure, wherein dependency parsing is used to automatically detect the grammatical dependency relations between each word in the text feedback information.

[0008] Optionally, based on the syntactic structure and the target image associated with the text feedback information submitted by the user, multimodal semantic features are generated for the aspect words, including: generating syntactic features of the aspect words based on the syntactic structure, wherein the syntactic features are a sequence of the shortest dependency distances between the aspect words and each word in the syntactic structure; generating visual semantic descriptions of the aspect words based on the target image, wherein the visual semantic descriptions are semantic expression information related to the aspect words obtained after question-answering reasoning on the target image through a visual question-answering model, the visual question-answering model is a multimodal pre-trained model based on a visual encoder and a text decoder, the visual encoder is used to extract regional features of the target image, and the text decoder is used to generate the corresponding answer according to the input question; and structurally fusing the syntactic features and the visual semantic descriptions to form multimodal semantic features.

[0009] Optionally, based on the syntactic structure, the syntactic features of the aspect words are generated, including: obtaining the shortest dependency path between the aspect words and each word in the syntactic structure, and using the shortest dependency path as the dependency distance between the aspect words and the word; arranging all dependency distances according to the original order of each word in the text feedback information to form a structured sequence, and using the structured sequence as the syntactic feature.

[0010] Optionally, based on the target image, a visual semantic description of the aspect word is generated, including: for the aspect word, constructing at least one visual question-and-answer question based on a preset question-and-answer template, wherein the visual question-and-answer question includes an existence question or an attribute question, the existence question is used to filter irrelevant image content, and the attribute question is used to extract semantically biased descriptive information related to the aspect word; inputting the target image and the visual question-and-answer question into a visual question-and-answer model to obtain the corresponding natural language answer; performing semantic filtering on the natural language answer to remove background information or non-biased descriptive information irrelevant to the aspect word to obtain the target answer; and converting the target answer into a declarative sentence form as the visual semantic description of the aspect word.

[0011] Optionally, the target model receives a structured prompt sequence and performs semantic reasoning analysis based on the aspect words, syntactic features, and visual semantic descriptions included in the structured prompt sequence. After generating candidate semantic tendency information corresponding to the aspect words and confidence scores corresponding to the candidate semantic tendency information, the model selects the candidate semantic tendency information with the highest confidence score as the target semantic tendency information of the aspect words.

[0012] Optionally, the semantic tendency detection method based on multimodal feedback further includes: generating an analysis report based on the target semantic tendency information of aspect words, wherein the analysis report includes the distribution data of the semantic tendency information of aspect words and the position of aspect words in the text feedback information; and transmitting the analysis report to the target system in a structured data format through an interface to trigger a service response.

[0013] According to another aspect of the embodiments of this application, a semantic tendency detection device based on multimodal feedback is also provided, comprising: an acquisition unit, configured to acquire text feedback information submitted by a user and extract at least one aspect word from the text feedback information, wherein the aspect word is a noun phrase mentioned by the user in the text feedback information to express an objective evaluation of a product, service, or process; a construction unit, configured to construct a syntactic structure based on the text feedback information, wherein the syntactic structure is a tree structure with words as nodes and grammatical dependencies as directed edges, used to represent the grammatical dominance relationships between words; and a generation unit, configured to generate based on the syntactic structure and the text feedback information submitted by the user. The associated target image generates multimodal semantic features for aspect words. These multimodal semantic features include syntactic features extracted from the syntactic structure representing the aspect words in the text feedback information, and visual semantic descriptions inferred from the target image related to the aspect words. The inference unit combines the text feedback information and multimodal semantic features into a structured prompt sequence, and inputs the structured prompt sequence into the target model. The target model performs semantic inference on the structured prompt sequence to generate target semantic tendency information corresponding to the aspect words. The structured prompt sequence is a text string generated by concatenating aspect words, syntactic features, and visual semantic descriptions according to a preset template.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described semantic tendency detection method based on multimodal feedback.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described semantic tendency detection method based on multimodal feedback.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described semantic tendency detection method based on multimodal feedback.

[0017] In this embodiment, a structured prompt sequence is formed by combining textual feedback information, syntactic structural features, and visual semantic descriptions. The target model then performs semantic reasoning based on this structured prompt sequence. This achieves the goal of unifying the grammatical roles of aspect words in text and their semantic expressions in images by using aspect words as semantic anchors. This results in the technical effect of directional alignment of fine-grained semantic associations in multimodal feedback, thereby solving the technical problem in the prior art where the lack of fine-grained semantic alignment of aspect words leads to low accuracy in semantic tendency analysis of multimodal feedback. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart of an optional semantic tendency detection method based on multimodal feedback according to an embodiment of this application;

[0020] Figure 2 This is an optional network architecture diagram according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional semantic tendency detection device based on multimodal feedback according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] It should also be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding operation entry points for them to choose to agree to or refuse automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0025] According to an embodiment of this application, a method embodiment for semantic tendency detection based on multimodal feedback is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] According to the embodiments of this application, a semantic tendency detection system based on multimodal feedback (hereinafter referred to as the system) can be used as the execution subject of the semantic tendency detection method based on multimodal feedback in the embodiments of this application. The semantic tendency detection system based on multimodal feedback can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiments of this application can also be other forms of execution subject, such as devices or equipment. Those skilled in the art should know that this application does not particularly limit the specific form of the execution subject.

[0027] Figure 1 This is a flowchart of an optional semantic tendency detection method based on multimodal feedback according to an embodiment of this application, such as... Figure 1As shown, the method includes the following steps:

[0028] Step S101: With the user's authorization, obtain the text feedback information submitted by the user, and extract at least one aspect word from the text feedback information, wherein the aspect word is a noun phrase mentioned by the user in the text feedback information to express an objective evaluation of the product, service or process.

[0029] Optionally, text feedback refers to evaluative content submitted by users in written form regarding bank products, services, or business processes, such as text records left by customers in the bank's application software review section, online questionnaires, or customer service conversations. Additionally, sensitive information contained in text feedback needs to be anonymized.

[0030] Optionally, aspect terms are specific entities or attributes that users explicitly mention in their text feedback as the objects of evaluation. They are usually noun phrases, such as transfer speed, customer service response, and financial product returns, and are the core of the user's intent.

[0031] Optionally, after receiving the original text submitted by the user, the system performs word matching and phrase recognition based on a pre-defined financial domain dictionary. The dictionary includes frequently occurring evaluation objects in banking scenarios, covering product names, service types, and operational procedures. The system segments and tags the text, selecting continuous word strings that conform to a noun structure and have a mapping in the dictionary as candidate aspect words. After structured filtering, phrases that are semantically complete and can independently represent the evaluation object are retained, forming the final extraction result.

[0032] Step S102: Construct a syntactic structure based on the text feedback information. The syntactic structure is a tree structure with words as nodes and grammatical dependencies as directed edges, used to represent the grammatical dominance relationships between words.

[0033] Optionally, syntactic structure is a graph of the grammatical organization within a language, where nodes represent each word in a sentence, and edges represent grammatical dependencies between words, such as modification, subject-verb, and verb-object relationships. Syntactic structure reveals the logical paths of control between words, rather than simple co-occurrence. For example, in "slow transfer speed," "speed" is the head word, and "slow" modifies "speed," forming a directed relationship of "slow-speed."

[0034] Optionally, the system invokes natural language processing tools to perform dependency parsing on the text feedback information, automatically identifying the syntactic role of each word and its relationship with the headword, forming a complete syntactic tree. This tree is rooted at the predicate verb of the sentence, with other words expanding layer by layer according to grammatical dependency relationships, forming a hierarchical grammatical network. Each edge is labeled with its dependency type, such as noun modification, adverb modification, object, etc., constituting a structured grammatical representation.

[0035] Step S103: Based on the syntactic structure and the target image associated with the text feedback information submitted by the user, generate multimodal semantic features for aspect words. The multimodal semantic features include syntactic features extracted from the syntactic structure that represent aspect words in the text feedback information, and visual semantic descriptions related to aspect words inferred from the target image.

[0036] Optionally, syntactic features refer to the grammatical association patterns between aspect words and other words in the syntactic structure. Specifically, they are the shortest dependency path length sequences between aspect words and other words in the sentence, reflecting the local influence of aspect words in the semantic network.

[0037] Optionally, visual semantic description refers to extracting non-textual semantic information directly related to aspect terms by analyzing image content, such as service scenarios, personnel status, and equipment status presented in the image, to corroborate or supplement the evaluation content in the text.

[0038] Optionally, the system first calculates the shortest dependency distance between the aspect term and each word in the sentence based on the syntactic tree, forming a numerical sequence arranged in word order, which serves as the syntactic feature of the aspect term. Simultaneously, the system pairs the target image with the aspect term, calls the visual question-answering model, inputs a preset question (such as "Does this image reflect a state related to ××?"), and extracts the natural language description output by the model. After semantic filtering, statements directly related to the aspect term are retained, such as "There are many people queuing at the window" and "The teller is smiling," as visual semantic descriptions.

[0039] Optionally, syntactic features quantify the semantic structure within the text, while visual semantic descriptions capture the objective state of the external scene. Together, they constitute multimodal semantic features, thereby avoiding coarse-grained processing of the entire image or text and allowing the analysis to focus on the evaluation objects that users truly care about, thus improving the pertinence of semantic tendency judgment.

[0040] Step S104: Combine the text feedback information with multimodal semantic features into a structured prompt sequence, and input the structured prompt sequence into the target model. The target model performs semantic reasoning on the structured prompt sequence to generate target semantic tendency information corresponding to aspect words. The structured prompt sequence is a text string generated by concatenating aspect words, syntactic features and visual semantic descriptions according to a preset template.

[0041] Optionally, the structured prompt sequence is a text instruction formed by splicing aspect words, their syntactic features, and visual semantic descriptions according to a fixed word order and format, which serves as an input form to guide the reasoning of the language model.

[0042] Optionally, the target model refers to a large-scale language model that has been pre-trained and has the ability to understand and reason about language, and performs zero-shot or few-shot reasoning through prompts.

[0043] Optionally, the target semantic tendency information is the model's judgment result on the objective tendency expressed by the aspect words, such as "negative", "neutral" and "positive", without involving emotional intensity or subjective judgment.

[0044] Optionally, the system concatenates the extracted aspect words, syntactic feature sequences, and visual semantic descriptions according to a unified template to form a structured prompt sequence, such as: "Aspect word: transfer speed; Syntactic association: speed and slowness have a dependency distance of 1; Image description: there are many people queuing at the ATM." The structured prompt sequence is used as the complete prompt input target model. Based on built-in language knowledge, the model understands the semantic relationships conveyed by the input string and outputs a standardized tendency category label and its confidence level.

[0045] Optionally, the system transforms the originally scattered multimodal information into natural language instructions that the model can understand, achieving non-parametric and interpretable semantic reasoning. The model can generalize to different contexts without retraining, thereby reducing its dependence on labeled data. The final output of sentiment information directly corresponds to the user's evaluation object, providing an actionable decision-making basis for the bank's automatic classification, risk warning, and service optimization.

[0046] In one optional embodiment, at least one aspect word is extracted from the text feedback information, including: detecting keywords describing the user-perceived object from the text feedback information based on a preset financial domain dictionary, as aspect words; wherein, the user-perceived object is the entity or attribute that the user expresses an objective evaluation in the text feedback information, the entity includes products, services, systems or institutions, and the attribute includes performance, function, experience or process.

[0047] Optionally, the financial dictionary is manually verified and structured from bank historical work orders, customer service dialogues, and customer survey texts. It includes thousands of high-frequency entities and attribute phrases, categorized and coded according to semantic type, such as "product category: fixed-term wealth management, credit card installment", "process category: account opening review, repayment plan", and "experience category: ease of operation, clarity of voice navigation".

[0048] Optionally, the user-perceived object refers to the entity or attribute that the user explicitly points to in the text feedback information and can be objectively evaluated. The entity covers the specific products, services, information systems or operating institutions provided by the bank, such as wealth management products, mobile banking, counter services, and loan approval processes; the attribute refers to the measurable characteristics embodied by the entity, such as transfer speed, interface response, customer service attitude, and interest rate transparency.

[0049] Optionally, after receiving text feedback, the system first performs word segmentation and part-of-speech tagging on the text to form a sequence of basic language units. Subsequently, it calls a financial dictionary and uses a sliding window matching and word frequency weighting algorithm to locate continuous word strings in the text that completely or partially match dictionary entries. It then filters out non-evaluative nouns based on syntactic position and finally retains semantically complete phrases with clear evaluative implications as aspect terms for output.

[0050] Optionally, the system uses pre-defined domain knowledge to semantically focus on the original text, avoiding misjudging generalized words or non-evaluative nouns as analysis objects. This transforms vague user feedback into structured, quantifiable evaluation units, providing stable and reusable semantic anchors for subsequent syntactic analysis and multimodal alignment. The system can then identify the evaluation dimensions that users truly care about, thereby improving the overall relevance and accuracy of the analysis.

[0051] In one optional embodiment, constructing a syntactic structure based on text feedback information includes: performing dependency parsing on the text feedback information to generate a tree structure with words as nodes and grammatical dependency relations as directed edges, as the syntactic structure, wherein the dependency parsing is used to automatically detect the grammatical dependency relations between each word in the text feedback information.

[0052] Optionally, dependency parsing is used to automatically identify the grammatical dominance relationships between each word in a sentence and other words, such as subject-verb, verb-object, and modifier. The final output of dependency parsing is a hierarchical syntax tree, which reflects the logical organization of the language.

[0053] Optionally, after receiving text feedback, the system invokes a general natural language processing framework to execute a dependency parser based on statistics or neural networks. This parser performs part-of-speech tagging and syntactic structure prediction on the input text, assigning each word a role in the sentence and determining the parent node and dependency type directly governed by that word. For example, in the phrase "transfer speed is too slow," "slow" is the predicate, "speed" is the subject, and "too" is an adverb modifying "slow." The system connects these relationships with directed edges, forming a tree structure rooted at the predicate and expanding outwards layer by layer. This structure fully preserves the grammatical governance paths between words.

[0054] Optionally, the system can construct syntactic structures to transform unstructured natural language into a computable semantic relation graph. This enables the system to identify the grammatical position of keywords in user reviews within sentences, determine which words directly modify those words, and identify which words are irrelevant components. This provides an accurate structural basis for subsequent extraction of syntactic features and reduces semantic misjudgments caused by disordered word order, elliptical sentences, or colloquial expressions.

[0055] In one optional embodiment, multimodal semantic features are generated for aspect words based on syntactic structure and a target image associated with text feedback information submitted by the user. This includes: generating syntactic features of aspect words based on syntactic structure, wherein the syntactic features are sequences of shortest dependency distances between aspect words and each word in the syntactic structure; generating visual semantic descriptions of aspect words based on the target image, wherein the visual semantic descriptions are semantic expression information related to aspect words obtained after question-answering reasoning on the target image through a visual question-answering model, the visual question-answering model being a multimodal pre-trained model based on a visual encoder and a text decoder, the visual encoder being used to extract regional features of the target image, and the text decoder being used to generate corresponding answers based on the input question; and structurally fusing the syntactic features and the visual semantic descriptions to form multimodal semantic features.

[0056] Optionally, the syntactic features are specifically represented as an ordered sequence of the shortest dependency path lengths between aspect words and each word in the syntactic structure. This sequence is ordered by the original position of the words in the sentence, recording the minimum number of steps between each word and the aspect word in the dependency tree, reflecting the strength of semantic relevance. For example, if the aspect word is "transfer speed," and "slow" is directly connected to "transfer speed," the distance is 1; if "customer service" and "transfer speed" are separated by two nodes, the distance is 3. After generating the syntactic structure, the system traverses all words in the tree, calculates the shortest path between all words and aspect words, and arranges them in word order to form a fixed-length numerical vector, which serves as the syntactic feature of that aspect word.

[0057] Optionally, visual semantic description refers to a natural language description directly related to aspect terms, obtained through semantic question-answering reasoning on the target image. This process is completed by a visual question-answering model, which consists of a visual encoder and a text decoder. The former extracts semantic features from multiple regions in the image through a convolutional neural network, while the latter generates corresponding text answers based on pre-trained language capabilities and the input question. The system constructs a semantically relevant question for each aspect term, such as "Does the image reflect a state related to ××?" or "Is there any anomaly in the environment associated with ××?", and inputs the question and image into the model. The original answer output by the model undergoes semantic filtering, retaining objective statements that are semantically relevant to the aspect term, which serve as the visual semantic description of that aspect term. Sensitive information involved in the target image needs to be desensitized.

[0058] Optionally, the system then concatenates the syntactic features and visual semantic descriptions in a fixed order to form a structured text string. The syntactic features are presented as a sequence of numbers, and the visual semantic descriptions are presented as short natural language sentences; the two are combined using a preset template.

[0059] Optionally, through the above steps, the system achieves a collaborative expression of fine-grained text association based on syntactic structure and image semantic alignment based on semantic question answering. Syntactic features reveal the logical position of aspect words in the language, while visual semantic descriptions supplement the observable objective state in the image. The two corroborate each other, reducing the one-sidedness of single-modal information. This allows the system to no longer rely on overall image features or global text semantics, but instead focus on the evaluation objects that users truly care about, ensuring that semantic tendency judgments are based on authentic, interpretable, and verifiable semantic evidence.

[0060] Optionally, in the multimodal information fusion architecture of this application, to achieve accurate guidance of image content by text semantics, a bidirectional interaction mechanism is introduced between the image understanding layer and the syntactic analysis layer. This mechanism, through a multi-level attention structure, enables visual features in the image to be dynamically focused based on the grammatical relationships and semantic focus of aspect words in the text.

[0061] Specifically, the system uses the text features output by the syntactic analysis layer as the query signal and the regional features extracted by the image understanding layer as the key and value signals, respectively, as shown in Formula (1). The query signal reflects the grammatical position and semantic association strength of aspect words in the sentence, while the key and value signals characterize the spatial semantic content of each region in the image. During the interaction, the system calculates the matching degree between the query and the key, generating a set of weight distributions for weighted aggregation of image value vectors. This process is repeated multiple times, each time using different parameter sets for independent calculation, to capture multi-dimensional association patterns between text and image. This allows image features to no longer be processed as a whole, but rather, based on the semantic needs of aspect words in the text, automatically strengthens the visual regions related to them and suppresses irrelevant background or interfering information. For example, when the text mentions "service attitude," the system can guide the image processing to focus on facial expressions or the service environment; when the text describes "interface lag," the system prioritizes the screen display status or operation feedback traces. Through this semantically driven visual focus, image information is no longer passively supplemented, but actively responds to the text's intent, improving the accuracy of information alignment between modalities.

[0062] (1)

[0063] Where A represents the attention weight matrix, Q is the text query, and K and V are the image keys. The feature dimension of the key vector is represented by t, where t is the transpose symbol.

[0064] In one optional embodiment, generating syntactic features of aspect words based on syntactic structure includes: obtaining the shortest dependency path between the aspect word and each word in the syntactic structure, and using the shortest dependency path as the dependency distance between the aspect word and the word; arranging all dependency distances according to the original order of each word in the text feedback information to form a structured sequence, and using the structured sequence as the syntactic feature.

[0065] Alternatively, the shortest dependency path refers to the minimum number of steps along directed edges from an aspect word node to any other lexical node in a tree diagram of a syntactic structure. This path reflects the direct or indirect grammatical dominance relationship between two words; the shorter the path, the stronger the semantic association.

[0066] Optionally, the dependency distance is the length of the shortest dependency path, which is a non-negative integer used to represent syntactic semantic proximity.

[0067] Optionally, the structured sequence is an ordered list of numbers consisting of all words in the sentence arranged in the order of their appearance in the original text and corresponding to their dependency distances to aspect words. The length of the sequence is the same as the number of words in the sentence, and the value at each position represents the grammatical distance between the word at that position and the aspect word.

[0068] Optionally, after completing dependency parsing and generating the syntactic structure, the system uses a breadth-first search algorithm to traverse the entire tree, starting with aspect words, and calculates the shortest dependency path length between each aspect word and every lexical node in the sentence, obtaining a set of dependency distance values. Subsequently, the system reorders all words in the sentence according to their order in the original text feedback information, and extracts the dependency distance corresponding to each word sequentially, forming a numerical sequence that is completely consistent with the word order of the original sentence. In this numerical sequence, if a word is itself an aspect word, its distance is 0; if it has no syntactic connection with an aspect word, it is set to the maximum value by default, which helps to ensure the integrity and stability of the numerical sequence.

[0069] Optionally, the system transforms the originally abstract grammatical structure into a computable and comparable numerical sequence, enabling the system to objectively identify which words are directly related to aspect words and which words are semantic interference items. For example, adjectives or adverbs modifying aspect words are usually 1 or 2 distances, while irrelevant subjects, objects, or background information are farther away. Through structured representation, the system allows subsequent models to prioritize text components closely related to the grammar of aspect words when fusing visual information, thereby suppressing noise interference and improving the accuracy and consistency of semantic reasoning.

[0070] In one optional embodiment, generating a visual semantic description of an aspect term based on a target image includes: constructing at least one visual question-and-answer question for the aspect term based on a preset question-and-answer template, wherein the visual question-and-answer question includes an existence question or an attribute question, the existence question being used to filter irrelevant image content, and the attribute question being used to extract semantically biased descriptive information related to the aspect term; inputting the target image and the visual question-and-answer question into a visual question-and-answer model to obtain a corresponding natural language answer; performing semantic filtering on the natural language answer to remove background information or non-biased descriptive information irrelevant to the aspect term to obtain a target answer; and converting the target answer into a declarative sentence form as the visual semantic description of the aspect term.

[0071] Optionally, visual question answering questions refer to natural language questions designed to guide the model to focus on specific aspect words. They are divided into two categories: existence questions are used to determine whether an entity or scene associated with an aspect word appears in an image to eliminate irrelevant background interference; attribute questions are used to inquire about the state, emotion, or performance characteristics associated with the aspect word in order to extract descriptive content that can support semantic tendency judgment.

[0072] Optionally, the visual question answering model is a multimodal system consisting of a visual encoder and a text decoder. First, the visual encoder extracts regional features from the image, and then the text decoder generates a natural language answer based on the input question.

[0073] Optionally, the system generates at least one visual question-and-answer question for each aspect term according to a preset template. Existence questions use the structure "Does the image contain entities or scenes related to ××?" to determine whether the image carries the physical carrier of that aspect, such as "Does the image show the status of equipment related to transfer speed?" Attribute questions use the structure "What characteristics does the environment or performance related to ×× have?" to uncover biased clues, such as "Are the facial expressions of personnel related to customer service positive?" After the questions are generated, the system inputs the target image and the questions into the visual question-and-answer model. The model performs cross-modal reasoning based on image region features and question semantics, outputting the original natural language answer. Subsequently, the system performs semantic filtering on the answers, removing background descriptions unrelated to the aspect term (such as "There is a window in the picture," "The background is blue") and neutral statements (such as "The image is clear," "Someone is present"), retaining only biased statements pointing to the state of the aspect term, such as "There are many people queuing at the counter," "The staff are smiling." Finally, all retained answers are uniformly converted into standardized declarative sentences as the visual semantic description of that aspect term.

[0074] Optionally, the system filters out a large amount of visual interference unrelated to aspect terms by using existence questions, avoiding misjudging irrelevant scenes as evidence for judging semantic tendencies; by using attribute questions, it focuses on semantic content that can support the judgment of tendencies, transforming ambiguous images into objective semantic expressions, thereby helping the output information not to contain subjective assumptions or generalized descriptions, but only retaining verifiable and explainable factual statements, achieving accurate alignment between image information and text evaluation objects.

[0075] In one optional embodiment, the target model receives a structured prompt sequence and performs semantic reasoning analysis based on the aspect words, syntactic features and visual semantic descriptions included in the structured prompt sequence. After generating candidate semantic tendency information corresponding to the aspect words and confidence scores corresponding to the candidate semantic tendency information, the model selects the candidate semantic tendency information with the highest confidence score as the target semantic tendency information of the aspect words.

[0076] Optionally, the system takes the structured prompt sequence as input and passes it completely to the target model. Based on built-in language patterns and common-sense knowledge, the model comprehensively judges three types of information in the prompts: aspect words locate the analyzed object, syntactic features provide semantic clues about the internal semantic relationships of the text, and visual semantic descriptions supplement the objective state of the external scene. During the inference process, the model simultaneously generates multiple preference judgment options, each accompanied by a probability-based confidence score reflecting the model's confidence in each option. Subsequently, the system extracts all candidate results and their corresponding confidence scores, compares the magnitudes of the confidence scores, selects the highest one as the final output, and discards the remaining options. This helps to ensure that the output has the strongest semantic support, improving the reliability of the judgment.

[0077] Optionally, in this embodiment, to enhance the semantic collaboration between text and image information, a dedicated interaction layer is added to the target model. This interaction layer employs a multi-layer structured attention mechanism, alternately receiving text feature representations and image feature representations. Through bidirectional information alignment and dynamic weighting, it gradually extracts the association patterns and semantic conflicts between the two. The processing of this interaction layer does not rely on the independent encoding of a single modality. Instead, it alternately inputs the feature vectors of text and image at time steps, allowing each modality to be guided by the other modality in each round of computation. After each round of interaction, the system concatenates the output features of the two modalities to form a joint representation. Through an adaptive weighting mechanism, it learns which content shows consistency and which content shows contradiction. For example, when the text mentions "abnormal transactions" while the image presents an abnormal operating interface or a high-risk scene, the system can identify the semantic linkage between the two; conversely, if the text expresses dissatisfaction while the image shows a calm environment, the system can determine that there is information deviation between the modalities. This interaction mechanism can enhance the target model's ability to perceive cross-modal semantic conflicts and consistency, enabling the system to accurately establish logical connections between text keywords and visual evidence in financial risk identification scenarios.

[0078] Optionally, to reduce the computational overhead and storage burden during the target model adaptation process, this application employs a parameter-efficient optimization strategy. This strategy, while retaining all original weights of the base model, introduces only a set of low-dimensional parameter matrices for training. The dimension of this matrix is ​​much smaller than the original parameter size of the model, and the training scale is only on the order of one percent of the full parameter update, thereby reducing resource consumption and training cycle while achieving model domain adaptation.

[0079] In an optional embodiment, the semantic tendency detection method based on multimodal feedback further includes: generating an analysis report based on the target semantic tendency information of aspect words, wherein the analysis report includes distribution data of semantic tendency information of aspect words and the position of aspect words appearing in text feedback information; transmitting the analysis report to the target system in a structured data format through an interface to trigger a service response.

[0080] Optionally, target semantic tendency information refers to the final classification result output by the system for each aspect word, such as "negative", "neutral", and "positive". It originates from the semantic reasoning of the target model and represents the user's true evaluation tendency for that aspect.

[0081] Optionally, the analysis report is a structured summary of the semantic tendency distribution of multiple aspects of words and their occurrence positions in the original text. It includes fields such as tendency category statistics, frequency distribution, and text source index, which are used to intuitively reflect the overall structure and key issues of the feedback content.

[0082] Optionally, structured data format refers to the use of a parsable, standardized data structure, which helps data to be automatically read, verified, and processed across different systems.

[0083] Optionally, the target system refers to the bank's internal business support system used for customer service, risk monitoring, or operational optimization, such as an intelligent customer service ticketing system, a public opinion monitoring platform, or a customer satisfaction analysis platform.

[0084] Optionally, after outputting the target semantic tendency information for all aspect terms, the system aggregates the data according to a preset template: first, it counts the frequency of occurrence of each tendency category (negative, neutral, positive) to form a distribution ratio; second, it records the start and end positions of each aspect term in the original text to locate its context in the text feedback information. This information is organized into structured data objects with clearly defined fields, including a list of aspect terms, tendency distribution, and text position index. The generated report, after lightweight validation, is sent to the target system through a pre-defined interface. The interface protocol follows the bank's unified internal data exchange standards and supports authentication, encryption, and logging. Upon receiving the data, the target system parses the content and automatically triggers response actions according to preset rules, such as automatically converting aspect terms with high negative tendencies into high-priority work orders, pushing them to account managers, or generating risk warning alerts.

[0085] Optionally, the system can automatically generate structured analysis reports and trigger the target system's response through an interface, thereby achieving seamless, automated, and auditable implementation of semantic tendency analysis results into business systems.

[0086] Figure 2 This is an optional network architecture diagram according to an embodiment of this application. For example... Figure 2 As shown, the input is a target image and text feedback information. Syntactic analysis is performed on the text feedback information to obtain multiple words, including aspect words. Then, the distances between words are calculated; for example, the distance from word 1 to word 2 is 1, from word 1 to word 3 is 2, and from word 1 to word 3 is 3. Additionally, image understanding of the target image is required, including constructing visual question-answering questions (existence questions and attribute questions) and generating the target answer. Constructing the visual question-answering question involves passing the target image through a self-attention layer and a feedforward neural network layer, fusing the visual question-answering question through a bidirectional self-attention layer, a cross-attention layer, and a feedforward neural network layer, and then fusing the natural language answer through a causal self-attention layer, a cross-attention layer, and a feedforward neural network layer to obtain the target answer. Finally, the feature vectors obtained from syntactic analysis and image understanding are processed through prompt word engineering and the target model to output target language tendency information.

[0087] Figure 3This is a schematic diagram of an optional semantic tendency detection device based on multimodal feedback according to an embodiment of this application. According to another aspect of an embodiment of this application, a semantic tendency detection device based on multimodal feedback is also provided, including: an acquisition unit 301, a construction unit 302, a generation unit 303, and a reasoning unit 304.

[0088] The acquisition unit 301, with user authorization, acquires text feedback information submitted by the user and extracts at least one aspect term from the text feedback information. The aspect term is a noun phrase mentioned by the user in the text feedback information, used to express an objective evaluation of a product, service, or process. The construction unit 302 applies a syntactic structure based on the text feedback information. The syntactic structure is a tree structure with words as nodes and grammatical dependencies as directed edges, used to represent the grammatical dominance relationships between words. The generation unit 303, based on the syntactic structure and the target image submitted by the user and associated with the text feedback information, generates... Aspect words generate multimodal semantic features, which include syntactic features representing aspect words in text feedback information extracted from syntactic structure, and visual semantic descriptions related to aspect words inferred from target images; inference unit 304 is used to combine text feedback information and multimodal semantic features into a structured prompt sequence, and input the structured prompt sequence into the target model, and perform semantic inference on the structured prompt sequence through the target model to generate target semantic tendency information corresponding to aspect words, wherein the structured prompt sequence is a text string generated by concatenating aspect words, syntactic features and visual semantic descriptions according to a preset template.

[0089] Optionally, the acquisition unit 301 includes: a detection subunit, used to detect keywords describing user-perceived objects from text feedback information based on a preset financial domain dictionary, as aspect words; wherein, the user-perceived object is the entity or attribute that the user expresses an objective evaluation in the text feedback information, the entity includes products, services, systems or institutions, and the attribute includes performance, function, experience or process.

[0090] Optionally, the construction unit 302 includes: an analysis subunit, used to perform dependency parsing on the text feedback information to generate a tree structure with words as nodes and grammatical dependency relations as directed edges, as a syntactic structure, wherein the dependency parsing is used to automatically detect the grammatical dependency relations between each word in the text feedback information.

[0091] Optionally, the generation unit 303 includes: a first generation subunit, used to generate syntactic features of aspect words based on syntactic structure, wherein the syntactic features are a sequence of shortest dependency distances between aspect words and each word in the syntactic structure; a second generation subunit, used to generate visual semantic descriptions of aspect words based on the target image, wherein the visual semantic descriptions are semantic expression information related to aspect words obtained after question-answering reasoning on the target image through a visual question-answering model, the visual question-answering model is a multimodal pre-trained model based on a visual encoder and a text decoder, the visual encoder is used to extract regional features of the target image, and the text decoder is used to generate corresponding answers based on the input question; and a third generation subunit, used to structurally fuse the syntactic features and the visual semantic descriptions to form multimodal semantic features.

[0092] Optionally, the first generation subunit includes: an acquisition module, used to acquire the shortest dependency path between the aspect word and each word in the syntactic structure, and use the shortest dependency path as the dependency distance between the aspect word and the word; and a sorting module, used to arrange all dependency distances according to the original order of each word in the text feedback information to form a structured sequence, and use the structured sequence as a syntactic feature.

[0093] Optionally, the second generation subunit includes: a question construction module, used to construct at least one visual question-and-answer question based on a preset question-and-answer template for aspect words, wherein the visual question-and-answer question includes an existence question or an attribute question, the existence question is used to filter irrelevant image content, and the attribute question is used to extract semantically biased descriptive information related to aspect words; a question answering module, used to input the target image and the visual question-and-answer question into a visual question-and-answer model to obtain the corresponding natural language answer; a filtering module, used to perform semantic filtering on the natural language answer, removing background information or non-biased descriptive information irrelevant to aspect words to obtain the target answer; and a transformation module, used to transform the target answer into a declarative sentence form as a visual semantic description of aspect words.

[0094] Optionally, the target model receives a structured prompt sequence and performs semantic reasoning analysis based on the aspect words, syntactic features, and visual semantic descriptions included in the structured prompt sequence. After generating candidate semantic tendency information corresponding to the aspect words and confidence scores corresponding to the candidate semantic tendency information, the model selects the candidate semantic tendency information with the highest confidence score as the target semantic tendency information of the aspect words.

[0095] Optionally, the semantic tendency detection device based on multimodal feedback further includes: a report generation unit, used to generate an analysis report based on the target semantic tendency information of aspect words, wherein the analysis report includes the distribution data of the semantic tendency information of aspect words and the position of aspect words in the text feedback information; and a transmission unit, used to transmit the analysis report in a structured data format to the target system through an interface to trigger a service response.

[0096] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described semantic tendency detection method based on multimodal feedback.

[0097] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described semantic tendency detection method based on multimodal feedback.

[0098] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described semantic tendency detection method based on multimodal feedback.

[0099] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0105] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A semantic tendency detection method based on multimodal feedback, characterized in that, include: With user authorization, obtain text feedback information submitted by the user, and extract at least one aspect word from the text feedback information, wherein the aspect word is a noun phrase mentioned by the user in the text feedback information to express an objective evaluation of the product, service or process; A syntactic structure is constructed based on the text feedback information. The syntactic structure is a tree structure with words as nodes and grammatical dependencies as directed edges, used to represent the grammatical dominance relationships between words. Based on the syntactic structure and the target image submitted by the user and associated with the text feedback information, multimodal semantic features are generated for the aspect words, wherein the multimodal semantic features include syntactic features extracted from the syntactic structure that characterize the aspect words in the text feedback information, and visual semantic descriptions inferred from the target image that are related to the aspect words; The text feedback information and the multimodal semantic features are combined into a structured prompt sequence, and the structured prompt sequence is input into the target model. The target model performs semantic reasoning on the structured prompt sequence to generate target semantic tendency information corresponding to the aspect words. The structured prompt sequence is a text string generated by concatenating the aspect words, the syntactic features and the visual semantic description according to a preset template.

2. The method according to claim 1, characterized in that, Extract at least one aspect word from the text feedback information, including: Based on a pre-set financial domain dictionary, keywords describing the user's perceived object are detected from the text feedback information and used as the aspect words; The user-perceived object is the entity or attribute in which the user expresses an objective evaluation in the text feedback information. The entity includes products, services, systems, or organizations, and the attribute includes performance, function, experience, or process.

3. The method according to claim 1, characterized in that, Constructing a syntactic structure based on the text feedback information includes: The text feedback information is subjected to dependency parsing to generate a tree structure with words as nodes and grammatical dependency relations as directed edges, which serves as the syntactic structure. The dependency parsing is used to automatically detect the grammatical dependency relations between each word in the text feedback information.

4. The method according to claim 1, characterized in that, Based on the syntactic structure and the target image submitted by the user and associated with the text feedback information, multimodal semantic features are generated for the aspect words, including: Based on the syntactic structure, syntactic features of the aspect words are generated, wherein the syntactic features are a sequence of the shortest dependency distances between the aspect words and each word in the syntactic structure; Based on the target image, a visual semantic description of the aspect word is generated, wherein the visual semantic description is semantic expression information related to the aspect word obtained after question-answering reasoning on the target image through a visual question-answering model, the visual question-answering model is a multimodal pre-trained model based on a visual encoder and a text decoder, the visual encoder is used to extract regional features of the target image, and the text decoder is used to generate the corresponding answer according to the input question; The syntactic features are structurally fused with the visual semantic description to form the multimodal semantic features.

5. The method according to claim 4, characterized in that, Based on the syntactic structure, the syntactic features of the aspect words are generated, including: Obtain the shortest dependency path between the aspect term and each word in the syntactic structure, and use the shortest dependency path as the dependency distance between the aspect term and the word. All the dependency distances are arranged in the original order of each word in the text feedback information to form a structured sequence, and this structured sequence is used as the syntactic feature.

6. The method according to claim 4, characterized in that, Based on the target image, a visual semantic description of the aspect word is generated, including: For the aforementioned aspect words, at least one visual question-and-answer question is constructed based on a preset question-and-answer template. The visual question-and-answer question includes an existence question or an attribute question. The existence question is used to filter irrelevant image content, and the attribute question is used to extract semantic tendency description information related to the aspect words. The target image and the visual question are input into the visual question answering model to obtain the corresponding natural language answer. The natural language response is semantically filtered to remove background information or non-biased descriptive information that is irrelevant to the aspect words, thereby obtaining the target response; The target answer is transformed into a declarative sentence form, which serves as a visual semantic description of the aspect words.

7. The method according to claim 1, characterized in that, The target model receives the structured prompt sequence and performs semantic reasoning analysis based on the aspect words, syntactic features, and visual semantic descriptions included in the structured prompt sequence. After generating candidate semantic tendency information corresponding to the aspect words and confidence scores corresponding to the candidate semantic tendency information, the model selects the candidate semantic tendency information with the highest confidence score as the target semantic tendency information of the aspect words.

8. The method according to claim 1, characterized in that, The method further includes: Based on the target semantic tendency information of the aspect words, an analysis report is generated, wherein the analysis report includes the distribution data of the semantic tendency information of the aspect words and the position of the aspect words in the text feedback information; The analysis report is transmitted to the target system in a structured data format via an interface to trigger a service response.

9. A semantic tendency detection device based on multimodal feedback, characterized in that, include: The acquisition unit is configured to acquire text feedback information submitted by the user with the user's authorization, and extract at least one aspect word from the text feedback information, wherein the aspect word is a noun phrase mentioned by the user in the text feedback information to express an objective evaluation of the product, service or process. The construction unit applies the construction of a syntactic structure based on the text feedback information. The syntactic structure is a tree structure with words as nodes and grammatical dependencies as directed edges, used to represent the grammatical dominance relationships between words. The generation unit is configured to generate multimodal semantic features for the aspect words based on the syntactic structure and the target image submitted by the user and associated with the text feedback information. The multimodal semantic features include syntactic features extracted from the syntactic structure that characterize the aspect words in the text feedback information, and visual semantic descriptions inferred from the target image that are related to the aspect words. The reasoning unit is used to combine the text feedback information with the multimodal semantic features into a structured prompt sequence, input the structured prompt sequence into the target model, and perform semantic reasoning on the structured prompt sequence through the target model to generate target semantic tendency information corresponding to the aspect words. The structured prompt sequence is a text string generated by concatenating the aspect words, the syntactic features and the visual semantic description according to a preset template.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device containing the computer-readable storage medium performs the semantic tendency detection method based on multimodal feedback as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the semantic tendency detection method based on multimodal feedback as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the semantic tendency detection method based on multimodal feedback as described in any one of claims 1 to 8.

Images